Test Plan Types
At a high level test plans come in two flavors:
- Zero Configuration (zero conf)
- Data Driven
Zero Configuration vs. Data Driven
| Zero Configuration | Data Driven | |
|---|---|---|
| Time to Value | Instant | Requires One Time Configuration |
| Test Category Coverage | Reduced | Comprehensive |
| Efficacy of Test Results | Moderate | Superior |
| Horizontal Authorization Abuse Coverage (BOLA / IDOR) | No | Yes |
| Vertical Authorization Abuse Coverage (Privilege Escalation) | Yes | Yes |
| Coverage for API Methods (GET, POST, etc.) | All Methods |
|
| Primary Use Case | Rapid Assessments | Deep / Comprehensive Assessments |
Zero Configuration
Zero Configuration test plans are instantly runnable, and only require specifying user authentication information.
These test plans use type information specified in the API schema to autogenerate data for various parameters required by the API.
For example is a string field in a POST body is of format type email, the test plan will autogenerate syntactically valid email addresses.
However there is no guarantee that the generated values are truly valid for the API endpoint being tested (the values may not exist in the backend database, etc.). Since parameter data is autogenerated, these test plans provide less security test coverage, and lower efficacy that Data Driven test plans.
For example, APIs often return 404 Not Found in lieu of 401 Unauthorized or 403 Forbidden during authentication/authorization failures. This is prevent hackers from enumerating resources.
Zero Configuration test plans cannot truly distinguish between an authentication/authorization failure and the absence of the requested resource in the API's backend database.
This results in lower efficacy and reduced test coverage (results might have false negatives).
Data Driven
As the name suggests, parameter data for APIs is end user supplied via fixtures.
By configuring parameter data, end users have complete control of what data is send when making API calls.
This dramatically increases efficacy of test results and test coverage, as the test plans can clearly distinguish between, authentication/authorization failures and the absence of the requested resource.