Home > English, java, work > Running your java integration tests with Gradle and docker-compose with only one command (optionally with Bitbucket Pipelines)

Running your java integration tests with Gradle and docker-compose with only one command (optionally with Bitbucket Pipelines)

This article will show you how to run your java integration tests with Docker containers running via docker-compose by only running the Gradle build-step and without needing to fiddle with Docker or docker-compose.
As an addition I’ll also show how to do it on Bitbucket Pipelines, but of course you can skip this step.

Note that this is one of my first projects I set up using Gradle, most project I worked on were with Maven, so please help me improve this article when I don’t do things the ‘Gradle-way’.

Separation of unit and integration test

I’m a huge proponent of not using mocks when integration testing so you’re testing with a realistic environment. When using docker-compose it’s so easy to not mock things that you’ll won’t even think about trying it. When you really want to be strict you can even make a physical separation between integration and unit-test (see this article on how to make this happen).

The big advantage using a docker-compose is that you can keep your containers running, even after your tests finish. This will increase you development speed since it is now much faster to run your integration tests. Be careful of course to clean up after you finish, otherwise you might get ‘dirty’ data.

It even is possible to use the running containers when running your application locally.

Example code on Github

I created a Github repo with example code so you can run everything yourself (assuming you have Java 11 and Docker installed).

For this project I started a fresh Spring Gradle project via start.spring.io with the following dependencies :

  • PostgreSQL Driver
  • Spring Data JDBC

Setup of docker-compose.yaml with a postgresql database

In order to use docker-compose you have to put a docker-compose.yaml in the root directory of you project. I used a simple compose file that will start a postgres image (yes this could’ve been done with Docker only, but this is to illustrate how to do it with docker-compose).

I took the inspiration for the docker-compose.yaml from this article.

Adding the gradle-docker-compose-plugin to your build.gradle

After some searching for plugins I found a nice plugin that does the trick. It doesn’t have a lot of stars yet, but you can change that if you like the plugin.

Add the following line to the plugins task (in your build.gradle) :

id 'com.avast.gradle.docker-compose' version '0.13.2'

And also add this task to the build.gradle:

dockerCompose {
	isRequiredBy(test)

	// enable following line when debugging container issues
	// captureContainersOutput(true)
}

Handling random port numbers

When running tests chances are you already have a postgres instance running on your machine. To make sure there are no conflicts with the ports you can omit the host port and Docker will pick a random port.
To access this port in your application add the following to you build.gradle:

test.doFirst {
	// the postgres port is mapped to a random port number that is located in a map
	def dbPort = dockerCompose.servicesInfos.database.postgres_db.tcpPorts[5432]
	environment.put("spring.datasource.url", "jdbc:postgresql://localhost:$dbPort/postgres")
}

This sets the environment variable that will be picked up by Spring. More information on the inner workings can be found in the ContainerInfo.groovy class or the project readme

Now run ./gradlew composeUp to spin up the containers. Spinning up the containers will happen in the test step, it just nice to see it work.

With ./gradlew composeDown you can shut down the containers.

If you want to use the port number (for example to run your application locally or access the database with a separate client) a nice table is displayed with the port mappings during startup :

+-------------+----------------+-----------------+
| Name        | Container Port | Mapping         |
+-------------+----------------+-----------------+
| postgres_db | 5432           | localhost:32772 |
+-------------+----------------+-----------------+

Running the integration test

For creating the integration test I’ve been a bit lazy and just used @SpringBootTest with a simple jdbc query (see JdbcTest.java.

Spring isn’t really needed to use this mechanism it’s just nice for demo purposes, but you can of course also use this in your vert.x application.

Keep your containers running when your tests finish

Of course it takes time to spin up containers and in many cases you can re-use the containers. To keep the containers running after a test copy this script to ~/.gradle/init.gradle

Note that now the ./gradlew composeDown doesn’t actually stop the containers anymore. Due to some constraints it was not possible. In the past the application didn’t show a warning. So I filed an issue that was fixed and released within a few days! So kudo’s for that, now there is a nice warning.

> Task :composeDown
You're trying to stop the containers, but stopContainers is set to false. Please use composeDownForced task instead.

To stop the containers anyway you can use ./gradlew composeDownForced.

Adding Docker to Bitbucket Pipelines

When running on Bitbucket there are few tweaks you need in order to run docker-compose with Bitbucket Pipelines.
First you need to enable Docker by adding it to the steps where you need Docker. For example :

pipelines:
  default:
    - step:
        name: Build and test
        script:
          - ./gradlew build
        services:
          - docker

There is also an option to add it to all steps, but I think it is nicer to only add it where you need it.
For more information see Run Docker commands in Bitbucket Pipelines.

Note that when you’re using the Gradle release plugin by Researchgate you should be aware that the release step also does a full test run and needs the same Docker configuration as your build step!

Adding docker-compose to Bitbucket Pipelines

When running docker-compose you need an image that includes docker-compose. A quick, temporary fix is to install docker-compose by adding a script step before the ./gradlew build (for example - ci/dependencies.sh

#!/usr/bin/env sh

set -eu

# prevent 'ImportError: No module named ssl_match_hostname'
apt-get remove python-configparser --yes

pip install docker-compose
docker-compose -v

Note that this script is for an Ubuntu image and highly depends on which image you’re using. Please use it as inspiration, just like I did from Docker-compose and pipelines and Using docker-compose in Bitbucket Pipelines.

Fixing working directory and $BITBUCKET_CLONE_DIR issues

Bitbucket is a bit picky on where you can put data. I soon ran into the following error:

authorization denied by plugin pipelines: ... only supports $BITBUCKET_CLONE_DIR and its subdirectories

I solved it by using the PWD environment variable that’s available on Bitbucket and your command line.

For example a mysql config will look like (inside the docker-compose.yaml) :

  mysql-myapp-main:
    image: mysql:5.7
    restart: always
    environment:
      MYSQL_DATABASE: 'myapp-main'
      MYSQL_PASSWORD: 'myapp123'
      MYSQL_USER: 'myapp'
      MYSQL_RANDOM_ROOT_PASSWORD: 'true'
    ports:
      - 3306:3306
    volumes:
      - '${PWD}/build/docker-volumes/myapp-mysql-main_mysql:/var/lib/mysql'
      - '${PWD}/build/docker-volumes/myapp-mysql-main_files/db:/var/files'

When using Intellij (or maybe other IDE’s) the PWD variable might not be available. The error you’ll see might be a bit cryptic since Gradle basically omits the variable :

ERROR: for b233df47a809387947f80c7fecdff6b7_myapp__mysql-myapp-main_1  Cannot start service mysql-myapp-main: Mounts denied: 
The paths /build/docker-volumes/myapp-mysql-main_mysql and /build/docker-volumes/myapp-mysql-main_files/db
are not shared from OS X and are not known to Docker.
You can configure shared paths from Docker -> Preferences... -> File Sharing.
See https://docs.docker.com/docker-for-mac/osxfs/#namespaces for more info.

I solved this by adding

environment.put("PWD", getenv("PWD") ?: projectDir)

Conclusion

I hope you enjoyed the article, if you have any improvements/comments let me know!

Categories: English, java, work Tags: , , ,

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: