Category Archives: Java

Running a JVM in a Container Without Getting Killed II

Posted on by Carlos Sanchez
4

A follow up to Running a JVM in a Container Without Getting Killed

In Java 10 there is improved container integration.
No need to add extra flags, the JVM will use 1/4 of the container memory for heap.

$ docker run -m 1GB openjdk:10 java -XshowSettings:vm \
    -version
VM settings:
    Max. Heap Size (Estimated): 247.50M
    Using VM: OpenJDK 64-Bit Server VM

openjdk version "10.0.1" 2018-04-17
OpenJDK Runtime Environment (build 10.0.1+10-Debian-4)
OpenJDK 64-Bit Server VM (build 10.0.1+10-Debian-4, mixed mode)

Java 10 obsoletes the -XX:MaxRAM parameter, as the JVM will correctly detect the value.

You can still use the -XX:MaxRAMFraction=1 option to squeeze all the memory from the container.

$ docker run -m 1GB openjdk:10 java -XshowSettings:vm \
    -XX:MaxRAMFraction=1 -version
OpenJDK 64-Bit Server VM warning: Option MaxRAMFraction was deprecated in version 10.0 and will likely be removed in a future release.
VM settings:
    Max. Heap Size (Estimated): 989.88M
    Using VM: OpenJDK 64-Bit Server VM

openjdk version "10.0.1" 2018-04-17
OpenJDK Runtime Environment (build 10.0.1+10-Debian-4)
OpenJDK 64-Bit Server VM (build 10.0.1+10-Debian-4, mixed mode)

But it can be risky if your container uses off heap memory, as almost all the container memory is allocated to heap. You would have to either set -XX:MaxRAMFraction=2 and use only 50% of the container memory for heap, or resort to Xmx.

Running a JVM in a Container Without Getting Killed

Posted on by Carlos Sanchez
24

No pun intended

The JDK 8u131 has backported a nice feature in JDK 9, which is the ability of the JVM to detect how much memory is available when running inside a Docker container.

I have talked multiple times about the problems running a JVM inside a container, how it will default in most cases to a max heap of 1/4 of the host memory, not the container.

For example in my machine

$ docker run -m 100MB openjdk:8u121 java -XshowSettings:vm -version
VM settings:
    Max. Heap Size (Estimated): 444.50M
    Ergonomics Machine Class: server
    Using VM: OpenJDK 64-Bit Server VM

Wait, WAT? I set a container memory of 100MB and my JVM sets a max heap of 444M ? It is very likely that it is going to cause the Kernel to kill my JVM at some point.

Let’s try the JDK 8u131 with the experimental option -XX:+UseCGroupMemoryLimitForHeap

$ docker run -m 100MB openjdk:8u131 java \
  -XX:+UnlockExperimentalVMOptions \
  -XX:+UseCGroupMemoryLimitForHeap \
  -XshowSettings:vm -version
VM settings:
    Max. Heap Size (Estimated): 44.50M
    Ergonomics Machine Class: server
    Using VM: OpenJDK 64-Bit Server VM

Ok this makes more sense, the JVM was able to detect the container has only 100MB and set the max heap to 44M.

Let’s try in a bigger container

$ docker run -m 1GB openjdk:8u131 java \
  -XX:+UnlockExperimentalVMOptions \
  -XX:+UseCGroupMemoryLimitForHeap \
  -XshowSettings:vm -version
VM settings:
    Max. Heap Size (Estimated): 228.00M
    Ergonomics Machine Class: server
    Using VM: OpenJDK 64-Bit Server VM

Mmm, now the container has 1GB but JVM is only using 228M as max heap. Can we optimize this even more, given that nothing else other than the JVM is running in the container? Yes we can!

$ docker run -m 1GB openjdk:8u131 java \
  -XX:+UnlockExperimentalVMOptions \
  -XX:+UseCGroupMemoryLimitForHeap \
  -XX:MaxRAMFraction=1 -XshowSettings:vm -version
VM settings:
    Max. Heap Size (Estimated): 910.50M
    Ergonomics Machine Class: server
    Using VM: OpenJDK 64-Bit Server VM

Using -XX:MaxRAMFraction we are telling the JVM to use available memory/MaxRAMFraction as max heap. Using -XX:MaxRAMFraction=1 we are using almost all the available memory as max heap.

UPDATE: follow up for Java 10+ at Running a JVM in a Container Without Getting Killed II

Running Docker containers as non root

Posted on by Carlos Sanchez
3

Running containers as root is a bad practice, but many Docker images available in the Docker Hub have the user set to root by default, so what can we do about it?

TL;DR Use -u 65534 -w /tmp -e _JAVA_OPTIONS=-Duser.home=/tmp for a typical Java image, plus any tool specific environment variable needed

Option 1 (build time):

Create a new derived image that creates a new user and changes the default to that one


FROM openjdk:8-jdk
RUN useradd --create-home -s /bin/bash user
WORKDIR /home/user
USER user

This is simple, but forces us to republish all these derived images, creating a maintenance nightmare.

Option 2 (runtime):

Use docker run -u option to choose what user to run the container as

docker run -ti --rm -u 1000 openjdk:8-jdk

This may work, but we can hit some issues, let’s see

$docker run -ti --rm -u 1000 openjdk:8-jdk git clone https://github.com/jenkinsci/docker
fatal: could not create work tree dir 'docker'.: Permission denied

Well, we obviously don’t have permissions to write to the default workdir, let’s fix it using -w and a dir that is writable, for instance /tmp

$ docker run -ti --rm -u 1000 -w /tmp openjdk:8-jdk git clone https://github.com/jenkinsci/docker
Cloning into 'docker'...
remote: Counting objects: 1498, done.
remote: Total 1498 (delta 0), reused 0 (delta 0), pack-reused 1498
Receiving objects: 100% (1498/1498), 287.46 KiB | 0 bytes/s, done.
Resolving deltas: 100% (772/772), done.
Checking connectivity... done.
fatal: unable to look up current user in the passwd file: no such user

 

Git does not like being run as an user that does not exist, so we need to pick one of the existing users

UPDATE: git 2.6.5+ removes this requirement, so we can run as any user even if it is not in the passwd file. For previous versions setting the GIT_COMMITTER_NAME and GIT_COMMITTER_EMAIL environment variables also works.

$ docker run -ti --rm -u 1000 -w /tmp openjdk:8-jdk cat /etc/passwd
root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
bin:x:2:2:bin:/bin:/usr/sbin/nologin
sys:x:3:3:sys:/dev:/usr/sbin/nologin
sync:x:4:65534:sync:/bin:/bin/sync
games:x:5:60:games:/usr/games:/usr/sbin/nologin
man:x:6:12:man:/var/cache/man:/usr/sbin/nologin
lp:x:7:7:lp:/var/spool/lpd:/usr/sbin/nologin
mail:x:8:8:mail:/var/mail:/usr/sbin/nologin
news:x:9:9:news:/var/spool/news:/usr/sbin/nologin
uucp:x:10:10:uucp:/var/spool/uucp:/usr/sbin/nologin
proxy:x:13:13:proxy:/bin:/usr/sbin/nologin
www-data:x:33:33:www-data:/var/www:/usr/sbin/nologin
backup:x:34:34:backup:/var/backups:/usr/sbin/nologin
list:x:38:38:Mailing List Manager:/var/list:/usr/sbin/nologin
irc:x:39:39:ircd:/var/run/ircd:/usr/sbin/nologin
gnats:x:41:41:Gnats Bug-Reporting System (admin):/var/lib/gnats:/usr/sbin/nologin
nobody:x:65534:65534:nobody:/nonexistent:/usr/sbin/nologin
systemd-timesync:x:100:103:systemd Time Synchronization,,,:/run/systemd:/bin/false
systemd-network:x:101:104:systemd Network Management,,,:/run/systemd/netif:/bin/false
systemd-resolve:x:102:105:systemd Resolver,,,:/run/systemd/resolve:/bin/false
systemd-bus-proxy:x:103:106:systemd Bus Proxy,,,:/run/systemd:/bin/false
messagebus:x:104:108::/var/run/dbus:/bin/false

The user nobody:65534 could be a good candidate, and as it is present in the default debian and alpine Docker images it will be present also in most images in the hub.

$ docker run -ti --rm -u 65534 -w /tmp openjdk:8-jdk \
git clone https://github.com/jenkinsci/docker
Cloning into 'docker'...
remote: Counting objects: 1498, done.
remote: Total 1498 (delta 0), reused 0 (delta 0), pack-reused 1498
Receiving objects: 100% (1498/1498), 287.46 KiB | 0 bytes/s, done.
Resolving deltas: 100% (772/772), done.
Checking connectivity... done.

Ok, that worked! Now let’s try to run something else, like a maven build


$ docker run -ti --rm -u 65534 -w /tmp maven:3 \
bash -c "git clone https://github.com/jenkinsci/kubernetes-plugin.git && \
cd kubernetes-plugin && mvn package"
touch: cannot touch ‘/root/.m2/copy_reference_file.log’: Permission denied
Can not write to /root/.m2/copy_reference_file.log. Wrong volume permissions?

This means entering in the domain of each tool and checking how to configure it. The maven docker image instructs us to use MAVEN_CONFIG and pass -Duser.home otherwise we would get an error [ERROR] Could not create local repository at /nonexistent/.m2/repository -> [Help 1]

Here is the working solution


$ docker run -ti --rm -u 65534 -w /tmp -e MAVEN_CONFIG=/tmp maven:3 \
bash -c "git clone https://github.com/jenkinsci/kubernetes-plugin.git && \
cd kubernetes-plugin && mvn -Duser.home=/tmp package"

Can we generalize this a bit for other Java apps? yes! By using the env var _JAVA_OPTIONS we can pass the user.home property to any Java app.

$ docker run -ti --rm -u 65534 -w /tmp -e _JAVA_OPTIONS=-Duser.home=/tmp \
-e MAVEN_CONFIG=/tmp maven:3 \
bash -c "git clone https://github.com/jenkinsci/kubernetes-plugin.git && \
cd kubernetes-plugin && mvn package"

 

JavaOne: From Monolith to Docker Distributed Applications

Posted on by Carlos Sanchez
Reply

I’ll be speaking again this year at JavaOne: From Monolith to Docker Distributed Applications, sharing our experience running the Jenkins platform on Docker containers using Apache Mesos.

You can also find me in the CloudBees booth in the exhibitors area.

Docker is revolutionizing the way people think about applications and deployments. It provides a simple way to run and distribute Linux containers for a variety of use cases, from lightweight virtual machines to complex distributed microservice architectures.

But migrating an existing Java application to a distributed microservice architecture is no easy task, requiring a shift in the software development, networking, and storage to accommodate the new architecture.

This presentation provides insights into the experience of the speaker and his colleagues in creating a Jenkins platform based on distributed Docker containers running on Apache Mesos and Marathon and applicable to all types of applications, especially Java- and JVM-based ones.

JavaOne slides: Enterprise Build and Test in the Cloud

Posted on by Carlos Sanchez
2

I have uploaded the slides from my talk Enterprise Build and Test in the Cloud at JavaOne in San Francisco.

You can check also the code, and an introduction in previous posts

Enterprise build and Test in the Cloud with Selenium I
and
Enterprise build and Test in the Cloud with Selenium II.

Follow me on twitter

JavaOne talk: Enterprise Build and Test in the Cloud

Posted on by Carlos Sanchez
Reply

I’ll be presenting Enterprise Build and Test in the Cloud at JavaOne in San Francisco, Wednesday June 3rd 11:05am Esplanade 301 and will be around the whole week.

You can check the slides from the previous talk at ApacheCON, the code, and an introduction in previous posts

Enterprise build and Test in the Cloud with Selenium I
and
Enterprise build and Test in the Cloud with Selenium II.

Follow me on twitter

Conference season: JavaOne

Posted on by Carlos Sanchez
Reply

If last week I mentioned the two conferences I got talks accepted, ApacheCON and EclipseCON, now I just got the confirmation that my talk Enterprise build and test in the cloud was accepted for JavaOne, June 2-5 in San Francisco

You can read a little bit about what I’m going to talk about in my posts

Enterprise build and Test in the Cloud with Selenium I
and
Enterprise build and Test in the Cloud with Selenium II, probably a 3rd part coming after ApacheCON.

Interview in JavaHispano

Posted on by Carlos Sanchez
Reply

JavaHispano, the portal for the Spanish speaking Java community has published a podcast (in Spanish if you haven’t figured out yet 😉 ) where Abraham Otero interviews me, covering a little bit of my past, how I got involved with open source, the Apache Software Foundation, Eclipse, and other projects, how do they work, my personal view on the status of open source, the job market and a comparison of the USA vs Spain in many aspects for people working in the technology world.

Upgrading Struts 2.0.x to 2.1.x

Posted on by Carlos Sanchez
Reply

Now that Struts 2.1.2 is officially out you may want to upgrade. I’ve found the upgrading instructions at the Struts wiki to be extremely useful, and pretty much you’ll need to read the whole thing.

Two things that I’ve contributed to the wiki:

Ensure no dependencies in the freemarker groupId are used as the latest version used by Struts is now under the org.freeemarker groupId and you’ll get classpath conflicts if you added a freemarker dependency.

In the unit tests you may get a NullPointerException as ActionContext.getContext() does not create a context on demand, I think there are better alternatives to avoid the direct use of getContext, but this snippet will fix the problem.


ConfigurationManager configurationManager = new ConfigurationManager();
configurationManager.addContainerProvider(new XWorkConfigurationProvider());
Configuration config = configurationManager.getConfiguration();
Container container = config.getContainer();

ValueStack stack = container.getInstance(ValueStackFactory.class).createValueStack();
stack.getContext().put(ActionContext.CONTAINER, container);
ActionContext.setContext(new ActionContext(stack.getContext()));

assertNotNull(ActionContext.getContext());

Speaking at CommunityOne on Monday

Posted on by Carlos Sanchez
Reply

Seems that I have been volunteered to do a lightening talk at CommunityOne on Monday at the Moscone as part of the Atlassian session.

Thanks to Cenqua’ Pete for offering free drinks in exchange (he hasn’t realized yet the mistake he has made) and entertainment like last year (hopefully with some differences in critical parts).

See you there.