Let's Play with Pact

Abstract

This is an attempt to create the simplest demo to describe a Consumer-Driven Contract Testing workflow with Pact.

Prerequisites

This scenario has been developed on Windows.

This scenario is using node, curl and jq.

This scenario is also using pact-stub-server and pact_verifier_cli.

Node.js

node --version

v16.13.2

cURL

curl --version

curl 7.79.1 (Windows) libcurl/7.79.1 Schannel
Release-Date: 2021-09-22
Protocols: dict file ftp ftps http https imap imaps pop3 pop3s smtp smtps telnet tftp
Features: AsynchDNS HSTS IPv6 Kerberos Largefile NTLM SPNEGO SSL SSPI UnixSockets

jq

jq --version

jq-1.6

Standalone Pact Stub Server

pact-stub-server --version

pact-stub-server v0.4.4
pact stub server version  : v0.4.4
pact specification version: v3.0.0

Pact Verifier CLI

pact_verifier_cli --version

pact_verifier_cli 0.9.7
pact verifier version   : v0.9.7
pact specification      : v4.0
models version          : v0.2.7

Workflow

1. The consumer gets (typically via a developer portal) the OpenAPI (f.k.a. Swagger) document of an API. For this demo, we will use the "Thingies API", thingies-api.oas2.yaml.

2. The consumer wants to use the "Thingies API" with her/his own test data. For instance when she/he sends the request:

GET localhost:8000/thingies/123

she/he expects to receive the following response:

{
  "id": "123",
  "name": "stuff"
}

3. Therefore, the consumer creates a Pact file that follows her/his scenario/interaction, consumer.pact.json.

Warning. There is, of course, a risk that the consumer does not respect the actual behaviour of the API when defining her/his scenario/interaction, that's why there is a verification step afterwards.

4. And then, the consumer can start a Pact Stub Server using the pact-stub-server command with the Pact file:

pact-stub-server --file consumer.pact.json --port 8000

in order to run her/his test script, e.g.consumer.test.cmd:

consumer.test.cmd

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    27  100    27    0     0     27      0  0:00:01 --:--:--  0:00:01   114
  "name": "stuff"
Passed.

Remark. At this stage, this test will obviously work as it is the same consumer that decides, within the test script, what the response should be, but also decides, within the Pact file run by the Pact Stub Server, what the response will be. This test is like a self-fulfilling prophecy! Again, that's why there is a verification step afterwards, during which the provider will verify that the Pact file created by the consumer makes sense or not.

5. Now, the consumer passes her/his Pact file, consumer.pact.json, to the provider so she/he can verify it.

6. The provider runs her/his implementation of the "Thingies API", e.g. provider.app.v1.js:

node provider.app.v1

Provider service is running at localhost:3000...

7. The provider can then verify the consumer Pact file, consumer.pact.json, with the Pact Verifier CLI using the pact_verifier_cli command. This tool will read the Pact file the other way around: it will send the request of the interaction to the actual implementation of the "Thingies API" and check if the actual response corresponds to the response defined by the consumer in the Pact file.

pact_verifier_cli --file consumer.pact.json --hostname localhost --port 3000

  Given has one thingy with '123' as an thingyId
    WARNING: State Change ignored as there is no state change URL provided
09:15:55 [WARN]

Please note:
We are tracking events anonymously to gather important usage statistics like Pact version and operating system. To disable tracking, set the 'PACT_DO_NOT_TRACK' environment variable to 'true'.



Verifying a pact between Thingies Consumer Example and Thingies Provider Example

  get one thingy
    returns a response which
      has status code 200 (FAILED)
      includes headers
        "Content-Type" with value "application/json; charset=utf-8" (FAILED)
      has a matching body (FAILED)


Failures:

1) Verifying a pact between Thingies Consumer Example and Thingies Provider Example Given has one thingy with '123' as an thingyId - get one thingy
    1.1) has a matching body
           / -> Expected body Present(27 bytes) but was empty
    1.2) has status code 200
           expected 200 but was 404
    1.3) includes header 'Content-Type' with value '"application/json; charset=utf-8"'
           Expected header 'Content-Type' to have value '"application/json; charset=utf-8"' but was ''

There were 1 pact failures

The verification obviously fails as the test data invented by the consumer does not match the test/real data used by the provider.

This is where the Provider States come to the rescue by "allowing you to set up data on the provider by injecting it straight into the data source before the interaction is run, so that it can make a response that matches what the consumer expects." But, how does this "injection" work? No magic, just before sending the request of an interaction the pact_verifier_cli sends the Provider State, i.e. the free text representing the Provider State, to the provider using a POST / endpoint with the following JSON structure:

{
  "action": "setup",
  "params": {},
  "state": "has one thingy with '123' as an thingyId"
}

So, it means that, as a provider you have to manually map (by writing some new specific code) this long string representing the Provider State invented by the consumer with the injection of some specific test data. Something like this, implemented in provider.app.v2.js:

app.post('/', (req, res) => {
  const providerState = req.body.state
  switch(providerState) {
    case "has one thingy with '123' as an thingyId":
      thingies.push({
        id: "123",
        name: "stuff"
      })
      break
    default:
      res.status(404).end()
      return
  }
  res.status(201).end()
})

Warning. This is not production-ready code ;-)

So, you can see that this technique can be error-prone and maybe difficult to scale when a provider has a lot of customers, but this effort needed must be put in perspective with the work needed to maintain and prepare "connected test environment(s)" in order to perform valuable end-to-end testing. As a reminder, the goal of Consumer-Driven Contract Testing with Pact is to remove the need of end-to-end testing!

Running the updated version of implementation of the "Thingies API", provider.app.v2.js:

node provider.app.v2

We can re-run the pact_verifier_cli command specifying the URL of the POST endpoing using the --state-change-url option:

pact_verifier_cli --file consumer.pact.json --hostname localhost --port 3000 --state-change-url http://localhost:3000

  Given has one thingy with '123' as an thingyId
09:18:15 [WARN]

Please note:
We are tracking events anonymously to gather important usage statistics like Pact version and operating system. To disable tracking, set the 'PACT_DO_NOT_TRACK' environment variable to 'true'.



Verifying a pact between Thingies Consumer Example and Thingies Provider Example

  get one thingy
    returns a response which
      has status code 200 (OK)
      includes headers
        "Content-Type" with value "application/json; charset=utf-8" (OK)
      has a matching body (OK)

Yeah!