Since we now have a fully runnable assembly (jar), we can add Product Acceptance Tests (PATs) to our automated build. PATs are a type of test which tests the product as a black-box - meaning that we work directly with the contracts defined on the exposed API without internal knowledge of the system. This allows us to test the system with more real-world style tests whereas unit tests usually go for a lot more edge-case tests for all possible input scenarios.
In our use-case, we will run our PATs against the defined REST endpoints and ensure the proper response codes and content are returned for each call. We will also simulate scenarios for our guestbook application.
Design
To help build these PATs, we are going to utilize Behavior Driven Development (BDD) Testing - specifically the Cucumber library:
Cucumber and its language Gherkin are a framework which have multiple implementations. For our use-case, we will be using the python implementation called "behave":
The reason for choosing python is just to use something different than the implementation language of our REST service. Also, it helps show that these PATs are completely separate from the actual service.
Test Setup
To run our tests, we will start our REST service as an in-memory process. This is facilitated via Cucumber with before-all and after-all setup stages:
from behave import *
import requests
import subprocess
import time
def before_all(context):
print('Starting server')
process = subprocess.Popen(['java', '-jar', 'cicd-series-assembly.jar'])
time.sleep(2)
print('Saving process to context')
context.proc = process
def after_all(context):
print('Terminating server')
context.proc.terminate()
print('Server terminated')
Running our service as an in-memory process ensures that we are running our tests against the fully built artifact from our "assemble" CI stage. Thus, these tests are running against the jar that we would push to a DEV or PROD environment instead of an one-off build.
Test Implementation
To build our tests, we write the actual test in the Gherkin language. This is a more natural language than most programming languages and can be understood without having much knowledge of its structure. Also, it uses the "Given/When/Then" style which is familiar to BDD Testing:
- Given = setting up the service to be in a given state
- When = the action to perform against the service
- Then = the assertions to perform after the action
For the actual tests, we are building more real-world style use-cases - some of which are very similar to the unit tests we built previously. For example, we can build a test to ensure we cannot add a duplicate guest to our guestbook:
Scenario: conflict if a guest is added twice
Given a guestbook with one guest
When we add a guest
Then the response should be 409
And a single guest should be found with /guests
And with Cucumber, each "Given/When/Then" line maps to actual code. For the above test, our python code looks like:
from behave import *
import requests
@given('a guestbook with one guest')
def step_impl(context):
url = 'http://localhost:8080/guests'
guest = {'name': 'Dan', 'age': 31}
post_response = requests.post(url, json=guest)
assert post_response.status_code == 201
list_response = requests.get(url)
assert list_response.status_code == 200
assert 'guests' in list_response.json()
guests = list_response.json()['guests']
assert len(guests) == 1
assert guests[0] == guest
@when('we add a guest')
def step_impl(context):
url = 'http://localhost:8080/guests'
json = {'name': 'Dan', 'age': 31}
post_response = requests.post(url, json=json)
context.response = post_response
@then('the response should be 409')
def step_impl(context):
assert context.response.status_code == 409
@then('a single guest should be found with /guests')
def step_impl(context):
url = 'http://localhost:8080/guests'
guest = {'name': 'Dan', 'age': 31}
list_response = requests.get(url)
assert list_response.status_code == 200
assert 'guests' in list_response.json()
guests = list_response.json()['guests']
assert len(guests) == 1
assert guests[0] == guest
CI/CD
Now that we have our tests setup, we can plug them into our automated builds. Again, this will ensure that our system adheres to its contracts on every build and should something fail we will get automated build failure notifications.
The first step in plugging these tests into our build is to pass the built artifact from our "assemble" stage to our "pats" stage. We want to pass the built artifact between stages since we already have a dedicated stage to ensuring the build works as expected, hence, there is no reason to do the build twice. This artifact passing can be done using GitHub Actions:
Next, we want to define our PAT stage within our build. Since we are running both a Java application and Python tests, we need to choose a docker image which has all of our prerequisites installed by default (or build a custom image). Luckily, there is a docker image available which has Java 8 and Python 3 installed:
After that, we just need to install the required python dependencies and run our tests with "behave":
pats:
runs-on: ubuntu-latest
container: openkbs/jre-mvn-py3:v1.0.6
needs: assemble
steps:
- name: Checkout Repo
uses: actions/checkout@v2
# Download artifact
- name: Download Artifact
uses: actions/download-artifact@v2
with:
name: cicd-series-assembly.jar
# Verify artifact
- name: List Files
run: ls -al
# This is needed because the artifact is downloaded with the original file name (includes version)
- name: Rename Artifact
run: mv cicd-series-assembly-*.jar cicd-series-assembly.jar
# This is needed because download artifacts are not runnable
- name: Change Permissions
run: chmod a+rx cicd-series-assembly.jar
# Verify artifact
- name: List Files
run: ls -al
# Install python dependencies
- name: Install Dependencies
run: pip install -r requirements.txt
# Run behave tests
- name: Run PATs
run: behave
Conclusion
We now have added automated PATs to run on every Pull Request to the master branch of our repo. They were built using python and the Cucumber library to perform BDD Tests. Also, should anything fail, we will get automated build failure notifications.
All of the code above and more can be found on this pull request.
