Home > English, java, Uncategorized, work > Running your JUnit 5 integration test with an embedded elasticsearch on a random port (and optionally Spring Boot)

Running your JUnit 5 integration test with an embedded elasticsearch on a random port (and optionally Spring Boot)

With recent versions of elasticsearch (5+) the learning curve for an integration test became a bit steeper but will result in a cleaner solution in the end. In this article I will describe how to set up your test with JUnit 5 to run your elasticsearch integration tests. I will also discuss how to make it work with Spring-Boot Test.

So why did they make it harder you’re probably wondering? It is described in this article. In short : security and class loading problems. It is also good to know the TransportClient will be deprecated soon so the high level rest client is also an upgrade that should be on your radar.

As of version 5 you have to spin up a standalone server and run your integration test against that server. David Pilato (from elastic) suggests a few tools and I found embedded-elasticsearch after some research. Elastic has given a maven plugin low priority so they suggest the tools David mentioned.

Requirements

When evaluating the tools it was important for us to be able to run the test from within the IDE and use Nexus (preferably without hard coding a url in the application) to download the elasticsearch distributions (since there is no direct internet connection on our Jenkins build server).

Random ports

Before we start with embedded-elasticsearch I’ll explain how to run any server on a random port. Even on your local machine it is a bad idea to use static ports in your tests. On a Jenkins server you immediately run into flaky tests when multiple builds use the same port at the same time. On your local machine ports might clash with already running applications. Se be a good citizen and set it up properly from the start.

With the call new ServerSocket(0) java will allocate a random port for you. With getLocalPort this port number is revealed.

In code it will look like this :

private static Integer findRandomPort() throws IOException {
  try (ServerSocket socket = new ServerSocket(0)) {
    return socket.getLocalPort();
  }
}

Setup embedd-elasticsearch

I will explain how to setup your server followed by instructions on how to setup Nexus (this step is optional).

First add embedded-elasticsearch as a dependency to your pom.xml (you might also have to add commons-io when it is not already in your project).

    <dependency>
      <groupId>pl.allegro.tech</groupId>
      <artifactId>embedded-elasticsearch</artifactId>
      <version>2.8.0</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>commons-io</groupId>
      <artifactId>commons-io</artifactId>
      <version>2.6</version>
      <scope>test</scope>
    </dependency>

I assume you already use JUnit 5 (and otherwise this might be a good time to start using it).
The first setup looks like :

public class EmbeddedElasticTest {

  private static final String ELASTIC_VERSION = "6.5.4";
  private static EmbeddedElastic embeddedElastic;
  private static Integer port;

  @BeforeAll
  public static void beforeClass() throws Exception {

    port = findRandomPort();

    final URL esUrl = new URL(String.format("https://secret-internal-host.local/nexus/repository/elasticsearch/elasticsearch-%s.zip", ELASTIC_VERSION));

    embeddedElastic = EmbeddedElastic.builder()
        .withElasticVersion(ELASTIC_VERSION)
        .withSetting(PopularProperties.TRANSPORT_TCP_PORT, findRandomPort())
        .withSetting(PopularProperties.HTTP_PORT, port)
        .withSetting(PopularProperties.CLUSTER_NAME, UUID.randomUUID())
        .withDownloadUrl(esUrl)
        .build()
        .start();
  }
}

I decided to start the elasticsearch server once, it is of course possible to start/stop a server every test.

When you don’t use Nexus it suffices to use https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-%s.zip as the url. Also don’t forget to pass the url as a configuration property.

Note that, even though you don’t use it, you have to setup the tcp transport port since multiple processes claiming the same port can lead to problems.

Don’t forget to clean up after you’re done :

  @AfterAll
  public static void afterClass() {
    embeddedElastic.stop();
  }

Right now you have a running server. It is time to add your tests, create indexes and add mappings. Since this is very specific for your needs and a basic setup is provided in the readme of embedded-elasticsearch I decided to skip this.

The important thing to know is that there is a elasticsearch REST server available on http://localhost:{port}

Not hardcoding the elasticsearch version in your test

I’m not happy with hardcoding the version so I submitted a pull request that will scan your classpath for elasticsearch client jars and use the highest version found. It’s a bit cumbersome, but it will save you the hassle of keeping two versions in sync.
When you want to try it out have a look at commit 04bfba7473. When you include AutoDetectElasticVersion in your project you can
set the version by replacing ELASTIC_VERSION with AutoDetectElasticVersion.detect() (works with elasticsearch 5 and higher).

Setup Nexus

To setup Nexus correctly click on the server admin cog in Nexus. Click on repositories, create repository and add a ‘raw (proxy)’ with the following settings :
* name : elasticsearch
* remote storage : https://artifacts.elastic.co/downloads/elasticsearch/
* Blob store : not sure what you should enter here, our config has a default option

When reviewing the setup it should look like this :

The only downside of this configuration is that we have to hardcode the url in our application. The elasticsearch-maven-plugin uses the Maven distribution model (but you can’t run it from your IDE from within the integration test).

Setup random port with SpringBootTest

Since many people use Spring Boot and it is not trivial to set the port I’ll explain it here.
Our project uses the configuration property elasticsearch.port to define the port number. Since it is random now there is no way to set it in application-test.yml.

Luckily Phill Webb has a great suggestion.

Add an inner class in your test :

static class Initializer
    implements ApplicationContextInitializer<ConfigurableApplicationContext> {

  @Override
  public void initialize(
      ConfigurableApplicationContext configurableApplicationContext) {
    TestPropertyValues.of("elasticsearch.port=" + port)
        .applyTo(configurableApplicationContext.getEnvironment());
  }

}

And add the following annotations to your test :

@ExtendWith(SpringExtension.class)
@SpringBootTest(classes = SomeApplication.class)
@ContextConfiguration(initializers = EmbeddedElasticTest.Initializer.class)
public class EmbeddedElasticTest

This will override the elasticsearch.port property when starting your tests.

Improvements/aftertoughts

I hope this article will help you. Improvements/suggestions are more than welcome in the comments.

Advertisements
  1. 6 February 2019 at 17:24

    Hi, very informative post. I’ve given a shot to this approach, however since I run behind a proxy, it gets a bit messy… I can’t find a clean way around defining the proxy.

    I have a different approach to run an embedded Elasticsearch that relies on Maven for downloading the distribution zip, and populating the Elasticsearch configuration. I have to specify the target Elasticsearch version though, which is not big hassle, since I only use the REST API to query ES. However setting random ports is a great idea I’ll certainly replicate šŸ˜‰

    • Jeroen van Wilgenburg
      6 February 2019 at 17:38

      I’m glad my post helped you further.
      I don’t think the library has support for proxies yet. Maybe you can create a pull request (or just create an issue, it shouldn’t be too hard to implement).

    • 1 September 2019 at 22:15

      Hi,

      Yes i like the idea to use maven to download the ES artifact as then you can benefit from maven repository caching in the build system. Can you detail how you acheived this?

      I am thinking, upload or proxy the ES download location as detailed here. Add a dependency on ES. Need a pom to represent the artifact. Then point the downloadUrl to your local maven repository

      • Jeroen van Wilgenburg
        2 September 2019 at 05:56

        We don’t use the Maven mechanism for downloading the zip. We don’t have a direct connection to internet and use Nexus. Since Nexus does the downloading/caching we’re basically getting it for free.

        We tried your idea when we didn’t know about the internet connection yet. It works with two minor caveats : the first time (or first version update) you run your test from your IDE the artifact probably isn’t cached/downloaded yet and you have to figure out a way to get it working on everybody’s machine (and Jenkins), we solved it with an environment variable M2_REPO.

  2. 18 April 2019 at 09:59

    This post helped us a lot, thanks.
    We did however have to improve it somewhat. As Spring re-uses ApplicationContexts across multiple test classes if their configuration is the same, the following problem arises with your suggested implementation:

    For example:
    When running the first test class, MyTest1.java:
    – Before running the tests, an ElasticSearch instance is started on a random port, say 54321.
    – Spring spins up an ApplicationContext for the test, and in the Initializer the port 54321 is passed as a property to that ApplicationContext to contact ElasticSearch.
    – The tests in MyTest1.java finish and the ElasticSearch instance running on port 54321 is stopped.

    Now when running MyTest2.java (configured with the same Spring configuration):
    – A new ElasticSearch instance is started on a NEW port: for instance: 12345
    – Spring does NOT start a new ApplicationContext, but re-uses the one from MyTest1.java that tries to contact ElasticSearch on port 54321.
    – Tests fail because there is no ElasticSearch running on port 54321 (only on port 12345).

    To fix this issue we moved the code that starts ElasticSearch to the Initializer (where we also set the property with the ElasticSearch port for Spring). So not in an @BeforeAll method.
    This way each ApplicationContext has its own ElasticSearch instance.
    We don’t stop the ElasticSearch instance in an @AfterAll. It gets stopped after running all tests anyway.
    Now when an ApplicationContext is reused it can still contact the same ElasticSearch instance. It will still be there.

    Hope that helps others running into the same issue.

    • Jeroen van Wilgenburg
      18 April 2019 at 10:02

      Cool, thanks for the improvement. We probably can use that in our projects too!

  3. Abhidemon
    22 April 2019 at 20:13

    I am getting NoNodeAvailableException.

  4. Igor
    16 September 2019 at 23:05

    How data cleanup is being done? I suspect there are data folders being created after each execution of the test, are those being cleaned up? What’s the path to the server?

    • Jeroen van Wilgenburg
      17 September 2019 at 06:23

      I’m not sure how it’s done, but I guess a Java temp folder or cleanup is taken care of by the library itself. You can dig in the source code on how it’s done exactly. Search for ‘installation’ on https://github.com/allegro/embedded-elasticsearch to get you started.

  1. 28 January 2019 at 07:31
  2. 1 February 2019 at 17:05
  3. 8 July 2019 at 06:23

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: