Blackbox API Testing with Postman/Newman and Docker

When developing an API, automated testing is a must. For every programming language and framework several testing frameworks exist. All of them are usually wired to the way the API is implemented and to the programming language, that is used.
Another approach is to perform a blackbox API testing: The testing routines are not wired to any internal details of the API implementation.
Such a test can be performed with standard software and even included in a CI/CD-pipeline, like Jenkins.

Requirements

Postman for Windows to create the API tests.
NodeJS to run „newman“.
newman to run the Postman tests in the CLI.
wait-on to make sure the API server is ready before running newman.
Docker to simplify API server start/stop.

Creating Postman tests

Create the Postman tests as a collection. If needed you can define different environments.

After the collection and environments are defined export both as JSON and store them in your project’s repository.

TIP: The JSONs can also be imported into Postman to edit them further.

Running tests with newman

Use the run command to run all tests of the previously exported Postman collection:

newman run examples/sample-collection.json

A environment file can be added with the --environment parameter:

newman run examples/sample-collection.json --environment examples/sample-environment-1.json

Example of newman run summary:

┌─────────────────────────┬───────────────────┬──────────────────┐
│                         │          executed │           failed │
├─────────────────────────┼───────────────────┼──────────────────┤
│              iterations │                 1 │                0 │
├─────────────────────────┼───────────────────┼──────────────────┤
│                requests │               211 │                0 │
├─────────────────────────┼───────────────────┼──────────────────┤
│            test-scripts │               202 │                0 │
├─────────────────────────┼───────────────────┼──────────────────┤
│      prerequest-scripts │                 7 │                0 │
├─────────────────────────┼───────────────────┼──────────────────┤
│              assertions │               204 │                0 │
├─────────────────────────┴───────────────────┴──────────────────┤
│ total run duration: 9.5s                                       │
├────────────────────────────────────────────────────────────────┤
│ total data received: 341.85KB (approx)                         │
├────────────────────────────────────────────────────────────────┤
│ average response time: 22ms [min: 2ms, max: 659ms, s.d.: 65ms] │
└────────────────────────────────────────────────────────────────┘

Generate a test report with newman

To generate a JUnit test report that can be interpreted by Jenkins the --reporters parameter can be used:

newman run examples/sample-collection.json --reporters junit

By default the report is stored in the newman folder.

Automated newman testing against a dockerized API server

If the API server is dockerized it is easy to automate the process of running the Postman tests against it. The following example assumes your API server is built as a Docker image with the name „apiServer“.

First a API server container with the name apiServerContainer is started. The name is important to clean up afterwards. The wait-on utility is used to make sure the API server container is started and ready for requests before the newman tests start. In the following example the API server listens on TCP port 8080.
Next the newman run command is used to run the tests and generate the JUint report.
After newman is finished the API server container is stopped and removed to clean up the system.

docker run --name apiServerContainer

wait-on tcp:8080
newman run examples/sample-collection.json --reporters junit

docker stop apiServerContainer
docker rm apiServerContainer

Jenkins job integration of newman

First configure the job to run the previously defined newman command with the --reporters option set.

To interpret the results in a Jenkins job the post-build action „Publish JUnit test result report“ must be added to the job configuration. As „Test report XMLs“ we add the newman/*.xml regular expression.

Now Jenkins will interpret any test report generated by newman and include the tests in the job status: