Tweakable JVM setting per helm release
Circa 2019 I had the chance to run many Java server applications, I wrote down out our best practices in order to iterate faster to find the right values for the JVM, but instead how to leverage Helm to try JVM settings.
As a pre-requisite, the reader must know about
-
Docker : It is an open-source project that helps to produce self-sufficient containers that can run on any platform supporting containers. In particular, it is useful for deployment automation. Moreover, it is practical tool to use for development.
-
Kubernetes : It is an open-source container orchestration system. In particular, it can be configured to automate certain task such as deployment, scaling, and management. It was originally created by Google, but is now part of the CNCF. It’s usually abbreviated as
k8s. -
Helm : Helm is an open-source project destined to manage applications running on Kubernetes. In particular, Helm serves a way to version, configure and deploy applications. It’s main document is known as a Helm Chart.
Choosing a better value for the Java heap
This is usually the first concrete flag we have to do to configure a JVM.
If you’ve followed the improvements of the JDK you know since JDK 11 it is
_kinda_supporting containers via the addition of the -XX:*RAMPercentage
flags. Or not.
Circa 2019 I basically did try to set a magical fixed value for the heap size
through -XX:MaxRAMPercentage=xx set in the Dockerfile.
CMD [ "/usr/bin/java", \
"-Dfile.encoding=UTF-8", \
"-Duser.timezone=UTC", \
"-Djava.security.egd=file:/dev/./urandom", \
"-XX:InitialRAMPercentage=85.0", \ (1)
"-XX:MaxRAMPercentage=85.0", \ (1)
"-XX:NativeMemoryTracking=summary", \
"-Xlog:os,safepoint*,gc*,gc+ref=debug,gc+ergo*=debug,gc+age*=debug,gc+phases*:file=/gclogs/%t-gc.log:time,uptime,tags:filecount=5,filesize=10M", \
"-javaagent:/agent-1.jar", \
"-javaagent:/agent-2.jar", \
"-jar", \
"/java-app-boot.jar", \
"--spring.config.additional-location=/etc/java-app/config.yaml", \
"--server.port=8080" ]| 1 | Sets the heap size to 85% of the total RAM (equivalent to setting
Xms and Xmx at the same value). |
Of course this didn’t work really well for two reasons :
-
The value didn’t adapt well to actual trafic conditions. It is related to how
-XX:*RAMPercentageis designed to work, in my opinion these flags should be avoided.I believe now this flag family is impractical at best if not an anti-pattern when used in production containers. -
Since the values was set in a Dockerfile as they were not supposed to be adjusted that much, it was long to actually publish and try new settings. Each change had to go through the whole build pipeline (even if some shortcuts were possible, thanks to Gradle in this regard).
-
The container had to be deployed in different environments with different conditions, hence different resource requirement.
In the aftermath I still feel that I was dumb to even thought it could work.
Since Helm was used it was time to leverage it.
Make the Docker image memory settings tweakable per environment
As seen at the above, RAM settings are part of the command declaration. First these arguments turned out to be incorrect, but they are more difficult to change or tweak. In addition, the deployment requirements / limits are likely to differ depending on the cluster / environment ; this can happen when you need to reduce the money spent on the cloud provider for non-production clusters, like staging, pre-production, etc.
The first tip is to use the JDK_JAVA_OPTIONS
environment variable for more flexibility and remove the RAM percentage in the
CMD directive.
ARG REGISTRY
FROM $REGISTRY/corretto-java:11.0.6.10.1
+ ENV JDK_JAVA_OPTIONS="" (1)
RUN mkdir -p /gclogs /etc/java-app
COPY ./build/libs/java-app-boot.jar \
./build/java-agents/agent-1.jar \
./build/java-agents/agent-2.jar \
./src/serviceability/*.sh \
/
CMD [ "/usr/bin/java", \
"-Dfile.encoding=UTF-8", \
"-Duser.timezone=UTC", \
"-Djava.security.egd=file:/dev/./urandom", \
- "-XX:InitialRAMPercentage=85.0", \ (2)
- "-XX:MaxRAMPercentage=85.0", \
"-XX:NativeMemoryTracking=summary", \
"-Xlog:os,safepoint*,gc*,gc+ref=debug,gc+ergo*=debug,gc+age*=debug,gc+phases*:file=/gclogs/%t-gc.log:time,uptime,tags:filecount=5,filesize=10M", \
"-javaagent:/agent-1.jar", \
"-javaagent:/agent-2.jar", \
"-jar", \
"/java-app-boot.jar", \
"--spring.config.additional-location=/etc/java-app/config.yaml", \
"--server.port=8080" ]| 1 | Defines a default empty JDK_JAVA_OPTIONS |
| 2 | Removes the RAM percentage settings to get default values. |
Now let’s test this locally.
$ DOCKER_BUILDKIT=1 docker build \
--tag test-java-app \ (1)
--build-arg REGISTRY=eu.gcr.io/cd-registry \
--file _infra/Dockerfile \
.
[+] Building 1.4s (9/9) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 1.34kB 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 35B 0.0s
=> [internal] load metadata for eu.gcr.io/cd-registry/corretto-java:11.0.6.10.1 0.0s
=> CACHED [1/4] FROM eu.gcr.io/cd-registry/corretto-java:11.0.6.10.1 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 1.32kB 0.0s
=> [2/4] RUN mkdir -p /gclogs /etc/java-app 0.3s
=> [3/4] COPY ./build/async-profiler/linux-x64 /async-profiler 0.0s
=> [4/4] COPY ./build/libs/java-app-boot.jar ./build/java-agents/agent-1.jar ./build/java-agents/agent-2.jar ./src/serviceability/*.sh / 0.6s
=> exporting to image 0.4s
=> => exporting layers 0.4s
=> => writing image sha256:5ceef8f5a4e23cb3bea7ca7cb7c90c0e338386b7f37992c92861cb119c312cb9 0.0s
=> => naming to docker.io/library/test-java-app| 1 | Custom tag to avoid collision with regular images in my cache |
Run the container locally with the Java app
In this local test series, I’m using 3 GiB as a memory limit, and I chose 70%
for the heap percentage.
JDK_JAVA_OPTIONS$ docker run --rm --memory="3gb" --name j-mem test-java-app
Picked up JDK_JAVA_OPTIONS:
10:14:53.566 [main] INFO org.springframework.core.KotlinDetector - Kotlin reflection implementation not found at runtime, related features won't be available.
2020-03-20 10:14:55.616 [] WARN --- [kground-preinit] o.s.h.c.j.Jackson2ObjectMapperBuilder : For Jackson Kotlin classes support please add "com.fasterxml.jackson.module:jackson-module-kotlin" to the classpath
...JDK_JAVA_OPTIONS$ docker run --rm --memory="3gb" --env JDK_JAVA_OPTIONS="-XX:InitialRAMPercentage=70.0 -XX:MaxRAMPercentage=70.0" --name j-mem test-java-app
Picked up JDK_JAVA_OPTIONS: -XX:InitialRAMPercentage=70.0 -XX:MaxRAMPercentage=70.0
10:14:53.566 [main] INFO org.springframework.core.KotlinDetector - Kotlin reflection implementation not found at runtime, related features won't be available.
2020-03-20 10:14:55.616 [] WARN --- [kground-preinit] o.s.h.c.j.Jackson2ObjectMapperBuilder : For Jackson Kotlin classes support please add "com.fasterxml.jackson.module:jackson-module-kotlin" to the classpath
...Then we can make sure we have the correct flags.
JDK_JAVA_OPTIONS$ docker exec -it j-mem bash -c "jcmd \$(pgrep java) VM.flags | tr ' ' '\n'"
6:
...
-XX:MaxHeapSize=805306368 (1)
-XX:MaxNewSize=482344960
-XX:MinHeapDeltaBytes=1048576
...| 1 | Max heap is about 768 MiB |
JDK_JAVA_OPTIONS❯ docker exec -it j-mem bash -c "jcmd \$(pgrep java) VM.flags | tr ' ' '\n'"
6:
...
-XX:InitialHeapSize=2256535552
-XX:InitialRAMPercentage=70.000000
-XX:MarkStackSize=4194304
-XX:MaxHeapSize=2256535552 (1)
-XX:MaxNewSize=1353711616
-XX:MaxRAMPercentage=70.000000
...| 1 | Max heap is about 2.1 GiB |
Notice when there’s no RAM settings the JVM computed the max heap size at 25%
of 3 GiB memory limit, and at 70% the jvm uses 2.1 GiB. Also, the heap values
are the only one affected.