Which JDK to Choose on Kubernetes

Which JDK to Choose on Kubernetes

In this article, we will make a performance comparison between several most popular JDK implementations for the app running on Kubernetes. This post also answers some questions and concerns about my Twitter publication you see below. I compared Oracle JDK with Eclipse Temurin. The result was quite surprising for me, so I decided to tweet to get some opinions and feedback.

jdk-kubernetes-tweet

Unfortunately, those results were wrong. Or maybe I should say, were not averaged well enough. After this publication, I also received interesting materials presented on London Java Community. It compares the performance of the Payara application server running on various JDKs. Here’s the link to that presentation (~1h). The results showed there seem to confirm my results. Or at least they confirm the general rule – there are some performance differences between Open JDK implementations. Let’s check it out.

This time I’ll do a very accurate comparison with several repeats to get reproducible results. I’ll test the following JVM implementations:

  • Adoptium Eclipse Temurin
  • Alibaba Dragonwell
  • Amazon Corretto
  • Azul Zulu
  • BellSoft Liberica
  • IBM Semeru OpenJ9
  • Oracle JDK
  • Microsoft OpenJDK

For all the tests I’ll use Paketo Java buildpack. We can easily switch between several JVM implementations with Paketo. I’ll test a simple Spring Boot 3 app that uses Spring Data to interact with the Mongo database. Let’s proceed to the details!

If you have already built images with Dockerfile it is possible that you were using the official OpenJDK base image from the Docker Hub. However, currently, the announcement on the image site says that it is officially deprecated and all users should find suitable replacements. In this article, we will compare all the most popular replacements, so I hope it may help you to make a good choice 🙂

Testing Environment

Before we run tests it is important to have a provisioned environment. I’ll run all the tests locally. In order to build images, I’m going to use Paketo Buildpacks. Here are some details of my environment:

  1. Machine: MacBook Pro 32G RAM Intel 
  2. OS: macOS Ventura 13.1
  3. Kubernetes (v1.25.2) on Docker Desktop: 14G RAM + 4vCPU

We will use Java 17 for app compilation. In order to run load tests, I’m going to leverage the k6 tool. Our app is written in Spring Boot. It connects to the Mongo database running on the same instance of Kubernetes. Each time I’m testing a new JVM provider I’m removing the previous version of the app and database. Then I’m deploying the new, full configuration once again. We will measure the following parameters:

  1. App startup time (the best  
  2. result and average) – we will read it directly from the Spring Boot logs 
  3. Throughput – with k6 we will simulate 5 and 10 virtual users. It will measure the number of processing requests 
  4. The size of the image
  5. The RAM memory consumed by the pod during the load tests. Basically, we will execute the kubectl top pod command

We will also set the memory limit for the container to 1G. In our load tests, the app will insert data into the Mongo database. It is exposing the REST endpoint invoked during the tests. To measure startup time as accurately as possible I’ll restart the app several times.

Let’s take a look at the Deployment YAML manifest. It injects credentials to the Mongo database and set the memory limit to 1G (as I already mentioned):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-spring-boot-on-kubernetes-deployment
spec:
  selector:
    matchLabels:
      app: sample-spring-boot-on-kubernetes
  template:
    metadata:
      labels:
        app: sample-spring-boot-on-kubernetes
    spec:
      containers:
      - name: sample-spring-boot-on-kubernetes
        image: piomin/sample-spring-boot-on-kubernetes
        ports:
        - containerPort: 8080
        env:
          - name: MONGO_DATABASE
            valueFrom:
              configMapKeyRef:
                name: mongodb
                key: database-name
          - name: MONGO_USERNAME
            valueFrom:
              secretKeyRef:
                name: mongodb
                key: database-user
          - name: MONGO_PASSWORD
            valueFrom:
              secretKeyRef:
                name: mongodb
                key: database-password
          - name: MONGO_URL
            value: mongodb
        readinessProbe:
          httpGet:
            port: 8080
            path: /readiness
            scheme: HTTP
          timeoutSeconds: 1
          periodSeconds: 10
          successThreshold: 1
          failureThreshold: 3
        resources:
          limits:
            memory: 1024Mi

Source Code and Images

If you would like to try it by yourself, you may always take a look at my source code. In order to do that you need to clone my GitHub repository. You will also find all the images in my Docker Hub repository piomin/sample-spring-boot-on-kubernetes. Every single image is tagged with the vendor’s name.

Our Spring Boot app exposes several endpoints, but I’ll test the POST /persons endpoint for inserting data into Mongo. In the integration with Mongo, I’m using the Spring Data MongoDB project and its CRUD repository pattern.

// controller

@RestController
@RequestMapping("/persons")
public class PersonController {

   private PersonRepository repository;

   PersonController(PersonRepository repository) {
      this.repository = repository;
   }

   @PostMapping
   public Person add(@RequestBody Person person) {
      return repository.save(person);
   }

   // other endpoints implementation
}


// repository

public interface PersonRepository extends CrudRepository<Person, String> {

   Set<Person> findByFirstNameAndLastName(String firstName, 
                                          String lastName);
   Set<Person> findByAge(int age);
   Set<Person> findByAgeGreaterThan(int age);

}

The Size of the Image

The size of the image is the simplest option to measure. If you would like to check what is exactly inside the image you can use the dive tool. The difference in the size between vendors results from the number of java tools and binaries included inside. From my perspective, the smaller the size the better. I’d rather not use anything that is inside the image. Of course, except all the staff required to run my app successfully. But you may have a different case. Anyway, here’s the content of the app for the Oracle JDK after executing the dive piomin/sample-spring-boot-on-kubernetes:oracle command. As you see, JDK takes up most of the space.

jdk-kubernetes-dive

On the other hand, we can analyze the smallest image. I think it explains the differences in image size since Zulu contains JRE, not the whole JDK.

Here are the result ordered from the smallest image to the biggest.

  • Azul Zulu: 271MB
  • IBM Semeru OpenJ9: 275MB
  • Eclipse Temurin: 286MB
  • BellSoft Liberica: 286MB
  • Oracle OpenJDK: 446MB
  • Alibaba Dragonwell: 459MB
  • Microsoft OpenJDK: 461MB
  • Amazon Corretto: 463MB

Let’s visualize our first results. I think it excellent shows which image contains JDK and which JRE.

jdk-kubernetes-memory

Startup Time

Honestly, it is not very easy to measure a startup time, since the difference between the vendors is not large. Also, the subsequent results for the same provider may differ a lot. For example, on the first try the app starts in 5.8s and after the pod restart 8.4s. My methodology was pretty simple. I restarted the app several times for each JDK provider to measure the average startup time and the fastest startup in the series. Then I repeated the same exercise again to verify if the results are repeatable. The proportions between the first and second series of startup time between corresponding vendors were similar. In fact, the difference between the fastest and the slowest average startup time is not large. I get the best result for Eclipse Temurin (7.2s) and the worst for IBM Semeru OpenJ9 (9.05s).

Let’s see the full list of results. It shows the average startup time of the application from the fastest one.

  • Eclipse Temurin: 7.20s
  • Oracle OpenJDK: 7.22s
  • Amazon Corretto: 7.27s
  • BellSoft Liberica: 7.44s
  • Oracle OpenJDK: 7.77s
  • Alibaba Dragonwell: 8.03s
  • Microsoft OpenJDK: 8.18s
  • IBM Semeru OpenJ9: 9.05s

Once again, here’s the graphical representation of our results. The differences between vendors are sometimes rather cosmetic. Maybe, if the same exercise once again from the beginning the results would be quite different.

jdk-kubernetes-startup

As I mentioned before, I also measured the fastest attempt. This time the best top 3 are Eclipse Temurin, Amazon Corretto, and BellSoft Liberica.

  • Eclipse Temurin: 5.6s
  • Amazon Corretto: 5.95s
  • BellSoft Liberica: 6.05s
  • Oracle OpenJDK: 6.1s
  • Azul Zulu: 6.2s
  • Alibaba Dragonwell: 6.45s
  • Microsoft OpenJDK: 6.9s
  • IBM Semero OpenJ9: 7.85s

Memory

I’m measuring the memory usage of the app under the heavy load with a test simulating 10 users continuously sending requests. It gives me a really large throughput at the level of the app – around 500 requests per second. The results are in line with the expectations. Almost all the vendors have very similar memory usage except IBM Semeru, which uses OpenJ9 JVM. In theory, OpenJ9 should also give us a better startup time. However, in my case, the significant difference is just in the memory footprint. For IBM Semeru the memory usage is around 135MB, while for other vendors it varies in the range of 210-230MB.

  • IBM Semero OpenJ9: 135M
  • Oracle OpenJDK: 211M
  • Azul Zulu: 215M
  • Alibaba DragonwellOracle OpenJDK: 216M
  • BellSoft Liberica: 219M
  • Microsoft OpenJDK: 219M
  • Amazon Corretto: 220M
  • Eclipse Temurin: 230M

Here’s the graphical visualization of our results:

Throughput

In order to generate high incoming traffic to the app I used the k6 tool. It allows us to create tests in JavaScript. Here’s the implementation of our test. It is calling the HTTP POST /persons endpoint with input data in JSON. Then it verifies if the request has been successfully processed on the server side.

import http from 'k6/http';
import { check } from 'k6';

export default function () {

  const payload = JSON.stringify({
      firstName: 'aaa',
      lastName: 'bbb',
      age: 50,
      gender: 'MALE'
  });

  const params = {
    headers: {
      'Content-Type': 'application/json',
    },
  };

  const res = http.post(`http://localhost:8080/persons`, payload, params);

  check(res, {
    'is status 200': (res) => res.status === 200,
    'body size is > 0': (r) => r.body.length > 0,
  });
}

Here’s the k6 command for running our test. It is possible to define the duration and number of simultaneous virtual users. In the first step, I’m simulating 5 virtual users:

$ k6 run -d 90s -u 5 load-tests.js

Then, I’m running the tests for 10 virtual users twice per vendor.

$ k6 run -d 90s -u 10 load-tests.js

Here are the sample results printed after executing the k6 test:

I repeated the exercise per the JDK vendor. Here are the throughput results for 5 virtual users:

  • BellSoft Liberica: 451req/s
  • Amazon Corretto: 433req/s
  • IBM Semeru OpenJ9: 432req/s
  • Oracle OpenJDK: 420req/s
  • Microsoft OpenJDK: 418req/s
  • Azul Zulu: 414req/s
  • Eclipse Temurin: 407req/s
  • Alibaba Dragonwell: 405req/s

Here are the throughput results for 10 virtual users:

  • Eclipse Temurin: 580req/s
  • Azul Zulu: 567req/s
  • Microsoft OpenJDK: 561req/s
  • Oracle OpenJDK: 561req/s
  • IBM Semeru OpenJ9: 552req/s
  • Amazon Corretto: 552req/s
  • Alibaba Dragonwell: 551req/s
  • BellSoft Liberica: 540req/s

Final Thoughts

After repeating the load tests several times I need to admit that there are no significant differences in performance between all JDK vendors. We were using the same JVM settings for testing (set by the Paketo Buildpack). Probably, the more tests I will run, the results between different vendors would be even more similar. So, in summary, the results from my tweet have not been confirmed. Ok, so let’s back to the question – which JDK to choose on Kubernetes?

Probably it somehow depends on where you are running your cluster. If for example, it’s Kubernetes EKS on AWS it’s worth using Amazon Corretto. However, if you are looking for the smallest image size you should choose between Azul Zulu, IBM Semeru, BellSoft Liberica, and Adoptium Eclipse Temurin. Additionally, IBM Semeru will consume significantly less memory than other distributions, since it is built on top of OpenJ9.

Don’t forget about best practices when deploying Java apps on Kubernetes. Here’s my article about it.

18 COMMENTS

comments user
Elytscha

How you can do auch a comparison without qarkus / graalvm? Which is the only JVM which is suitable for knative?

    comments user
    piotr.minkowski

    You can use all the JVMs for Knative. GraalVM makes a difference in startup time and memory usage. But the throughput is lowest (regarding the theory)

comments user
jonathan

How about distroless from google ?

    comments user
    piotr.minkowski

    I didn’t try them. I was testing only what is available via Paketo Buildpacks

comments user
Tilo

Maybe add Red Hat JDK runtime

    comments user
    piotr.minkowski

    I would do it, but there is no Red Hat OpenJDK in Paketo buildpacks.

comments user
Fra

For IBM semeru did you use all the flags that optimize startup time?

    comments user
    piotr.minkowski

    In the article, I didn’t test it. However, in my tests after the article, I set `-Xquickstart -Xshareclasses:cacheDir=/cache` with mounted volume and startup time was significantly improved from ~9s to ~4-5s. I also tried to enable CRIU support, but on Kubernetes it may be quite, so I avoided it

comments user
Raymond

Thanks for the extensive tests! I tested this also on my own Spring Boot local app and can confirm that IBM Semeru has the slowest startup time, but also the lowest memory footprint. Temurin starts fast (only GraalVM started faster in my case).

Some ideas for a follow-up article:

1. How does JDK 20 performs when it comes out next month?
2. How does GraalVM performs in JIT?
3. How does GraalVM performs in AOT?
4. How does Google distroless performs (can also be local test with an tar.gz version)?
5. How does Spring Boot 3.0 performs against Spring Boot 2.7.x and Spring Boot 3.1?

    comments user
    piotr.minkowski

    Thanks for that suggestions. Maybe I’ll be back once again to that topic – to add your suggestions and several other

comments user
vaano94

Apart from memory footprint and throughput it might be effective to look at current (and fixed) vulnerabilities, support for different jvm flags and existing diagnostic tools inside a container

    comments user
    piotr.minkowski

    Yes, for example, Azul or OpenJ9 provides some features for decreasing startup time. Maybe I create a similar article in the future with a deep dive. For that article, my idea was to compare all the distros with the same JVM flags

comments user
peterreji

how about using an alpine image and install jre on top of that and use that image as the base image? from ECR scanning, i noticed that the vulnerabilities were zero on the alpine+jre combo whereas there were more than 60 vulnerabilities in the openjdk image. please let me know your thoughts

    comments user
    piotr.minkowski

    From my perspective, if you use alpine with jre it’s fine. if there are less vulnerabilities than in standard openjdk image it is even better

comments user
Gary DeVal

For a throughput comparison to be meaningful, CPU usage needs to be accounted for somehow. For example, if system A handles 500 req/s at 30% cpu-util, and system B handles 500 req/s at 70% cpu-util, reporting A and B as equal in throughput is misleading; system A is probably capable of higher throughput than system B.

Measured throughput can be normalized for CPU consumption by driving enough load to saturate the CPU (cpu-util ~100%). If that is not possible, at least the CPU consumed by each system during the load test should be reported, and throughput can be normalized by dividing req/s by the cpu used; in the preceding example, system A’s normalized throughput would be (500 / 0.3 =) 1667 req/s, while system B’s would be (500 / 0.7 =) 714 req/s.

~~~~~
Java systems also depend on Just-In-Time (JIT) compilation of heavily used methods to improve throughput. The system needs to be run under load for some time to allow the JVM’s internal profiler to identify hot paths and the JIT compiler to optimize the busy methods.

So for a throughput measurement of a Java system, load should be applied twice: first for a “warmup” period to let the JIT compiler do its job, followed by a “measurement” period, the results of which are reported as the throughput of the system. A warmup period of 5 minutes (under heavy load) is typically sufficient for JIT to do most of its optimization work; after the warmup load terminates, apply measurement load for 2-3 minutes to get a realistic view of the JDK’s performance.

    comments user
    piotr.minkowski

    Ok. Thanks for the suggestions. Yes, I did some warmups before starting tests (usually 1-2 minutes with avg traffic). Then 3x90s tests

comments user
Songkiet Pombuppa

Are you using the ‘alpine’ version of each JDK?
Currently I am using Openj9-alpine, which I think it is the best in term of less memory, startup.

    comments user
    piotr.minkowski

    No, I’m not using alpine. Yes, but you can as well use alpine for all other JDKs.

Leave a Reply