Introduction

Link to original blog: this blog here. This will be the basis of this blog so it's required reading as I won't be treading over old ground. It's also an amazing read by a great test engineer.

Co-authored with the help of Nathan Hardy, who came up with the idea for this solution. If blogs aren't for you, you look at my code example on this on my GitHub here.

One of the complexities of building distributed and advanced software is testing the integrations between the different technologies. At Kablamo, when we are building client-facing applications we tend to use a known tech stack. React/TS for our front end, Golang for our backend and a RESTful API; following an OpenApi 3.0 spec. This means that we have a handy YAML file to compare our front end to. I've wanted for years to be able to leverage this file for automated testing as this is often our source of truth for both the Testing team and downstream systems.

This is a producer-driven contract test. As opposed to a consumer-driven contract test (which you should also have), and this will allow a producer of a microservice to be able to check the validity of the objects and types they are producing without having to rely on a mocking service.

What you need to get started

1. An API to test against, with an OpenAPI 3.0 spec that it conforms to.
2. The Playwright framework installed in a project.
3. An OpenAPI spec file that includes the request and response schemas that you want to test with.

1. Configuring Orval

Orval is the brain of this module. It is a restful client generator that takes in a swagger schema file and generates type-safe code. It has a bunch of uses from react-query to axios-functions.

Using Orval in this way allows us to take the supplied OpenAPI spec and do two things:

1. Generate API request functions using Axios to streamline our testing, giving type safety and lowering our script maintenance
2. Generate Zod schemas to compare expected responses to actual responses.

We chose to use Axios as our requests library over the playwright requests library because of how easily we can generate the request functions using orval, allowing us the step of defining our endpoints, which we would have to do manually if we used the base playwright library.

To install Orval use the command below:

bash
npm install --save-dev orval

Here is the example of the orval config that we are using:

typescript
import { defineConfig } from "orval";
import "dotenv/config";
import path from "path";

const target = path.join(__dirname, "pkg", "openapi.yml"); // Path to OpenAPI spec file
export default defineConfig({
  Zod: {
    input: {
      target: target,
      validation: false,
    },
    output: {
      mode: "tags",
      target: ".zod/index.ts",
      schemas: ".zod/types",
      client: "zod",
    },
  },
  axios: {
    input: {
      target: target,
      validation: false,
    },
    output: {
      mode: "tags",
      target: ".axios/index.ts",
      schemas: ".axios/types",
      client: "axios-functions",
    },
  },
});

One side note is Orval's integration with Zod is quite new as of 2024, so expect some rough edges if your swagger specs some advanced or seldom-used features. One thing we encountered were issues around anyOf and allOf at the top-level of a request or response body which affected our ability to generate responses. We had to flatten the schemas to solve this issue, so there might be post-processing of your OpenAPI spec file required.

2. What is Zod

Zod is a Typescript-based validation library. Zod uses the concept of a schema to compare the shape of one object to another. This is more useful than the scope of an API and can be used for comparing any two JSON objects.

The way we are using Zod is to compare the response that we get from Axios (or playwright requests) to a pre-generated z.object. This gives us quick type-safe testing for APIs and allows our front end team to have confidence in the schema they are hooking into.

It is worth saying that we are autogenerating this schema but if your code had odd requirements or had external dependencies you needed to test, you can manually define a zod schema to extend this testing beyond Orval.

You can install Zod easily using:

bash
npm install zod

3. Bringing it all together

Once all the prerequisites are installed you can then just generate a handy package.json script or run it in your shell.

json
"codegen": "orval --config orval.config.ts"

You should see generated files called .axios and .zod. I would recommend adding these to your .gitignore as they should be generated at the point of testing to avoid getting stale.

Also, expect some schemas like this Axios:

typescript
export const getApiV1Authors = <TData = AxiosResponse<Author[]>>(
  options?: AxiosRequestConfig,
): Promise<TData> => {
  return axios.get(`/api/v1/Authors`, options);
};
export const postApiV1Authors = <TData = AxiosResponse<Author>>(
  author: Author,
  options?: AxiosRequestConfig,
): Promise<TData> => {
  return axios.post(`/api/v1/Authors`, author, options);
};

and from Zod:

typescript
export const getApiV1AuthorsResponseItem = zod.object({
  id: zod.number().optional(),
  idBook: zod.number().optional(),
  firstName: zod.string().nullish(),
  lastName: zod.string().nullish(),
});
export const getApiV1AuthorsResponse = zod.array(getApiV1AuthorsResponseItem);

export const postApiV1AuthorsBody = zod.object({
  id: zod.number().optional(),
  idBook: zod.number().optional(),
  firstName: zod.string().nullish(),
  lastName: zod.string().nullish(),
});

export const postApiV1AuthorsResponse = zod.object({
  id: zod.number().optional(),
  idBook: zod.number().optional(),
  firstName: zod.string().nullish(),
  lastName: zod.string().nullish(),
});

Then we can combine all of this in a playwright script:

typescript
import { test, expect } from "@playwright/test";
import { getApiV1Authors } from "../.axios/authors";
import { getApiV1AuthorsResponse } from "../.zod/authors";

test("Get Authors", async () => {
  const response = await getApiV1Authors();

  expect(response.status).toBe(200);
  expect(response.data).toMatchSchema(getApiV1AuthorsResponse);
});

And just like that you have schema matching in Playwright! Enjoy!