26.9.2019 - By Rob Staples

Testing in Containers

I put all my integration tests in containers. Here's why you should too

Developing reliable integration tests requires you to control every part of your execution environment, from the operating system up to the source code itself. Running your tests inside containers takes extra work up front, so why do it?

Your environment is defined

Could you tell me what packages are on your local machine? Could you tell me what your version of chrome is at this moment? Me neither. Could I tell you what I’m running in my containers?

In a heartbeat.

There's no conflicting node packages, no chance that the browser updates, no chance you installed a conflicting node package, you know exactly what is there.

You limit extraneous problems in your testing.

A problem I used to get constantly was (Webdriver Exception: Cannot find a matching set of capabilities).

This is a useless error message.

If you don’t know what the problem is (this error was trying to tell me my firefox, geckodriver and/or selenium versions were incompatible), it gives you no guidance on solving the problem.

You can match your environment to your pipeline

When you run your tests in a pipeline you spin up a container. That container is defined.

How can your tests truly align to what’s in the pipeline if you can't even know your environment?

You can create a container that matches your pipeline.

The hurdles

As always when trying new things there are always hurdles. Here are some problems that I faced when I started this journey.

You need to learn how to containerize your environment

You need to get used to the concept of thinking about your environment. You can't take it for granted.

What version of Firefox do you want to use?

What version of python or node?

You should already know the answer to these, but until you do you can't easily containerize your tests.

You need to know the environment of the base image

The first image I used was fedora. I picked this image because I thought it was light weight and it would be similar to Ubuntu.

Unfortunately I didn’t know how to do things different in Fedora. It caused a few problems and slowed me down.

So if you are creating an image make sure you understand the environment you are using.

How to containerize your first test

First you need to install docker.

Once you have installed Docker you should get to building your Dockerfile. I chose ubuntu for my base image but if you wanted to go somthing a little more lightweight you could go alpine or mint.

Dockerfile
FROM ubuntu:16.04

Install your dependencies. Robot Framework requires python, so we need to install that first.

Dockerfile
RUN apt-get update && apt-get install --quiet --assume-yes \
    python3 python3-pip unzip

Next install robot via pip3.

Dockerfile
RUN pip3 install --upgrade robotframework

Last step is to copy your tests into the container. In this case, copy everything in the same directory as the Dockerfile.

Dockerfile
COPY . /robot

Your Dockerfile should look like this. Name it Dockerfile and place in the base directory.

Dockerfile
FROM ubuntu:16.04

RUN apt-get update && apt-get install --quiet --assume-yes \
    python3 python3-pip unzip firefox wget

RUN pip3 install --upgrade robotframework

COPY . /robot

This blog isn't about robot, so I'm going to just supply a basic test that builds and passes right away.

Create a file called my-first-test.robot and place it next to the Dockerfile you just made.

robot
my-first-test.robot
*** Variables ***
${test}  Pass

*** Test Cases ***
Test
   Pass Execution  ${test}

Finally with all this done you can build and run the container.

bash
# Build the image and tag it as myfirstdockertest
$ docker build -t myfirstdockertest .

# Start the container with the image we just built
# We're also mounting our current directory
$ docker run --name myfirstdockertest -td -v ${PWD} myfirstdockertest

# Jump into the container and run the robot test
$ docker exec -t myfirstdockertest robot robot/my-first-test.robot

# Stop the container
$ docker stop myfirstdockertest

# Remove the container
$ docker container rm myfirstdockertest

After it runs, you should see this in the console.

bash
===========================================================================
My-First-Test
===========================================================================
Test                                                               | PASS |
Pass
---------------------------------------------------------------------------
My-First-Test                                                      | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
===========================================================================