Running your JUnit 5 integration test with an embedded elasticsearch on a random port (and optionally Spring Boot)

With recent versions of elasticsearch (5+) the learning curve for an integration test became a bit steeper but will result in a cleaner solution in the end. In this article I will describe how to set up your test with JUnit 5 to run your elasticsearch integration tests. I will also discuss how to make it work with Spring-Boot Test.

So why did they make it harder you’re probably wondering? It is described in this article. In short : security and class loading problems. It is also good to know the TransportClient will be deprecated soon so the high level rest client is also an upgrade that should be on your radar.

As of version 5 you have to spin up a standalone server and run your integration test against that server. David Pilato (from elastic) suggests a few tools and I found embedded-elasticsearch after some research. Elastic has given a maven plugin low priority so they suggest the tools David mentioned.

Requirements

When evaluating the tools it was important for us to be able to run the test from within the IDE and use Nexus (preferably without hard coding a url in the application) to download the elasticsearch distributions (since there is no direct internet connection on our Jenkins build server).

Random ports

Before we start with embedded-elasticsearch I’ll explain how to run any server on a random port. Even on your local machine it is a bad idea to use static ports in your tests. On a Jenkins server you immediately run into flaky tests when multiple builds use the same port at the same time. On your local machine ports might clash with already running applications. Se be a good citizen and set it up properly from the start.

With the call new ServerSocket(0) java will allocate a random port for you. With getLocalPort this port number is revealed.

In code it will look like this :

private static Integer findRandomPort() throws IOException {
  try (ServerSocket socket = new ServerSocket(0)) {
    return socket.getLocalPort();
  }
}

Setup embedd-elasticsearch

I will explain how to setup your server followed by instructions on how to setup Nexus (this step is optional).

First add embedded-elasticsearch as a dependency to your pom.xml (you might also have to add commons-io when it is not already in your project).

    <dependency>
      <groupId>pl.allegro.tech</groupId>
      <artifactId>embedded-elasticsearch</artifactId>
      <version>2.8.0</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>commons-io</groupId>
      <artifactId>commons-io</artifactId>
      <version>2.6</version>
      <scope>test</scope>
    </dependency>

I assume you already use JUnit 5 (and otherwise this might be a good time to start using it).
The first setup looks like :

public class EmbeddedElasticTest {

  private static final String ELASTIC_VERSION = "6.5.4";
  private static EmbeddedElastic embeddedElastic;
  private static Integer port;

  @BeforeAll
  public static void beforeClass() throws Exception {

    port = findRandomPort();

    final URL esUrl = new URL(String.format("https://secret-internal-host.local/nexus/repository/elasticsearch/elasticsearch-%s.zip", ELASTIC_VERSION));

    embeddedElastic = EmbeddedElastic.builder()
        .withElasticVersion(ELASTIC_VERSION)
        .withSetting(PopularProperties.TRANSPORT_TCP_PORT, findRandomPort())
        .withSetting(PopularProperties.HTTP_PORT, port)
        .withSetting(PopularProperties.CLUSTER_NAME, UUID.randomUUID())
        .withDownloadUrl(esUrl)
        .build()
        .start();
  }
}

I decided to start the elasticsearch server once, it is of course possible to start/stop a server every test.

When you don’t use Nexus it suffices to use https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-%s.zip as the url. Also don’t forget to pass the url as a configuration property.

Note that, even though you don’t use it, you have to setup the tcp transport port since multiple processes claiming the same port can lead to problems.

Don’t forget to clean up after you’re done :

  @AfterAll
  public static void afterClass() {
    embeddedElastic.stop();
  }

Right now you have a running server. It is time to add your tests, create indexes and add mappings. Since this is very specific for your needs and a basic setup is provided in the readme of embedded-elasticsearch I decided to skip this.

The important thing to know is that there is a elasticsearch REST server available on http://localhost:{port}

Not hardcoding the elasticsearch version in your test

I’m not happy with hardcoding the version so I submitted a pull request that will scan your classpath for elasticsearch client jars and use the highest version found. It’s a bit cumbersome, but it will save you the hassle of keeping two versions in sync.
When you want to try it out have a look at commit 04bfba7473. When you include AutoDetectElasticVersion in your project you can
set the version by replacing ELASTIC_VERSION with AutoDetectElasticVersion.detect() (works with elasticsearch 5 and higher).

Setup Nexus

To setup Nexus correctly click on the server admin cog in Nexus. Click on repositories, create repository and add a ‘raw (proxy)’ with the following settings :
* name : elasticsearch
* remote storage : https://artifacts.elastic.co/downloads/elasticsearch/
* Blob store : not sure what you should enter here, our config has a default option

When reviewing the setup it should look like this :

The only downside of this configuration is that we have to hardcode the url in our application. The elasticsearch-maven-plugin uses the Maven distribution model (but you can’t run it from your IDE from within the integration test).

Setup random port with SpringBootTest

Since many people use Spring Boot and it is not trivial to set the port I’ll explain it here.
Our project uses the configuration property elasticsearch.port to define the port number. Since it is random now there is no way to set it in application-test.yml.

Luckily Phill Webb has a great suggestion.

Add an inner class in your test :

static class Initializer
    implements ApplicationContextInitializer<ConfigurableApplicationContext> {

  @Override
  public void initialize(
      ConfigurableApplicationContext configurableApplicationContext) {
    TestPropertyValues.of("elasticsearch.port=" + port)
        .applyTo(configurableApplicationContext.getEnvironment());
  }

}

And add the following annotations to your test :

@ExtendWith(SpringExtension.class)
@SpringBootTest(classes = SomeApplication.class)
@ContextConfiguration(initializers = EmbeddedElasticTest.Initializer.class)
public class EmbeddedElasticTest

This will override the elasticsearch.port property when starting your tests.

Improvements/aftertoughts

I hope this article will help you. Improvements/suggestions are more than welcome in the comments.

Advertisement