Trying Rootless Docker with Testcontainers
Rootless Docker is one of the most exciting recent changes in the Docker ecosystem.
It allows you to run the same good old Docker but without having to obtain root privileges on the machine.
Installing Rootless Docker on a fresh VM
Although you can run Rootless Docker-in-Docker, I wanted to try it on a fresh environment.
A few seconds later, I had an Ubuntu VM running on Oracle Cloud to play with:
ubuntu@instance-20200710-1405:~$As you can see, I am logged in as ubuntu user, not root.
Before we proceed to the installation, there is one thing that needs to be installed on the host, as per the Prerequisites section of the docs:
$ sudo apt-get install -y uidmapI guess it shouldn’t be hard to convience your sysadmin to install it anyways.
Now we’re ready to install Rootless Docker:
$ curl -fsSL https://get.docker.com/rootless | sh
# ...
# Docker binaries are installed in /home/ubuntu/bin
# WARN: dockerd is not in your current PATH or pointing to /home/ubuntu/bin/dockerd
# Make sure the following environment variables are set (or add them to ~/.bashrc):
export PATH=/home/ubuntu/bin:$PATH
export DOCKER_HOST=unix:///run/user/1001/docker.sock
#
# To control docker service run:
# systemctl --user (start|stop|restart) docker
#
$ systemctl --user start docker
$ systemctl --user status docker
● docker.service - Docker Application Container Engine (Rootless)
Loaded: loaded (/home/ubuntu/.config/systemd/user/docker.service; disabled; vendor preset: enabled)
Active: active (running) since Fri 2020-07-10 12:39:41 UTC; 51s ago
Docs: https://docs.docker.com
Main PID: 7638 (rootlesskit)
CGroup: /user.slice/user-1001.slice/user@1001.service/docker.service
├─7638 rootlesskit --net=vpnkit --mtu=1500 --slirp4netns-sandbox=auto --slirp4netns-seccomp=auto --disable-host-loopback --port-driver=builtin --copy-up=/etc --copy-up=/run /home/ubuntu/bin/dockerd-rootless.sh --experimental>
├─7647 /proc/self/exe --net=vpnkit --mtu=1500 --slirp4netns-sandbox=auto --slirp4netns-seccomp=auto --disable-host-loopback --port-driver=builtin --copy-up=/etc --copy-up=/run /home/ubuntu/bin/dockerd-rootless.sh --experimen>
├─7659 vpnkit --ethernet /tmp/rootlesskit977878316/vpnkit-ethernet.sock --mtu 1500 --host-ip 0.0.0.0
├─7678 dockerd --experimental --storage-driver=overlay2
└─7695 containerd --config /run/user/1001/docker/containerd/containerd.toml --log-level info
Jul 10 12:39:42 instance-20200710-1405 dockerd-rootless.sh[7678]: time="2020-07-10T12:39:42.061487164Z" level=warning msg="Your kernel does not support cgroup blkio weight"
Jul 10 12:39:42 instance-20200710-1405 dockerd-rootless.sh[7678]: time="2020-07-10T12:39:42.061495439Z" level=warning msg="Your kernel does not support cgroup blkio weight_device"
Jul 10 12:39:42 instance-20200710-1405 dockerd-rootless.sh[7678]: time="2020-07-10T12:39:42.061719048Z" level=info msg="Loading containers: start."
Jul 10 12:39:42 instance-20200710-1405 dockerd-rootless.sh[7678]: time="2020-07-10T12:39:42.064508578Z" level=warning msg="Running modprobe bridge br_netfilter failed with message: modprobe: ERROR: could not insert 'br_netfilter': Operat>
Jul 10 12:39:42 instance-20200710-1405 dockerd-rootless.sh[7678]: time="2020-07-10T12:39:42.145749301Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a prefe>
Jul 10 12:39:42 instance-20200710-1405 dockerd-rootless.sh[7678]: time="2020-07-10T12:39:42.181690135Z" level=info msg="Loading containers: done."
Jul 10 12:39:42 instance-20200710-1405 dockerd-rootless.sh[7678]: time="2020-07-10T12:39:42.188475776Z" level=warning msg="Not using native diff for overlay2, this may cause degraded performance for building images: failed to set opaque >
Jul 10 12:39:42 instance-20200710-1405 dockerd-rootless.sh[7678]: time="2020-07-10T12:39:42.188735172Z" level=info msg="Docker daemon" commit=42e35e61f3 graphdriver(s)=overlay2 version=19.03.11
Jul 10 12:39:42 instance-20200710-1405 dockerd-rootless.sh[7678]: time="2020-07-10T12:39:42.188795144Z" level=info msg="Daemon has completed initialization"
Jul 10 12:39:42 instance-20200710-1405 dockerd-rootless.sh[7678]: time="2020-07-10T12:39:42.204860831Z" level=info msg="API listen on /run/user/1001/docker.sock"As you can see, the DOCKER_HOST isn’t the same as with “normal” Docker, for obvious reasons.
What surprised me is that Docker CLI does not know how to locate it:
$ /home/ubuntu/bin/docker ps
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?Why surprising? Because Docker seems to be using a well known location for the socket - $XDG_RUNTIME_DIR/docker.sock.
You may ask yourself - wtf is XDG_RUNTIME_DIR? At least I did.
Apparently, Rootless Docker relies on its presence, and, after some digging, I discovered this page: https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html
There is a single base directory relative to which user-specific runtime files and other file objects should be placed.
This directory is defined by the environment variable$XDG_RUNTIME_DIR.
Oh, cool, TIL! Maybe future versions of Docker CLI will try to auto-detect.
Meanwhile, we need to pass it explicitly:
$ export DOCKER_HOST=unix://$XDG_RUNTIME_DIR/docker.sock
$ docker info -f '{{ join .SecurityOptions "\n" }}'
name=seccomp,profile=default
name=rootlessAs you can see, there is a new security option reported by Docker, rootless.
Now, the moment of truth:
$ DOCKER_HOST=unix://$XDG_RUNTIME_DIR/docker.sock docker run -it --rm busybox whoami
Unable to find image 'busybox:latest' locally
latest: Pulling from library/busybox
91f30d776fb2: Pull complete
Digest: sha256:9ddee63a712cea977267342e8750ecbc60d3aab25f04ceacfa795e6fce341793
Status: Downloaded newer image for busybox:latest
rootDespite the feature being shipped (experimental, but still), I must say I was positively surprised
to see it running an arbitary container, especially given that busybox will be using root user by default.
But… will it run… Testcontainers’ Ryuk?
$ docker run -it -v $XDG_RUNTIME_DIR/docker.sock:/var/run/docker.sock --rm testcontainers/ryuk:0.3.0
2020/07/10 12:49:19 Pinging Docker...
2020/07/10 12:49:19 Docker daemon is available!
2020/07/10 12:49:19 Starting on port 8080...
2020/07/10 12:49:19 Started!Wow. Just wow. Note that I am mounting Rootless Docker unix socket, and it just works!
Did you say… Testcontainers?
If you’re reading this article, most probably you came here to learn how Testcontainers works with Rootless Docker.
I knew that a change would be required in Testcontainers to make it work, but apparently it is rather trivial: https://github.com/testcontainers/testcontainers-java/pull/2985
The cherry on the cake is that Testcontainers will automatically detect the presence of XDG_RUNTIME_DIR,
which is already set by modern login managers (e.g. pam_systemd).
To demonstrate that, I restated the shell:
$ env | grep DOCKER_
$ One of my favorite modules to test is Kafka.
It is heavyweight, uses networks, etc etc.
So… Can we get… Rootless Kafka?
$ ../../gradlew cleanTest test
> Task :kafka:test
Gradle Test Executor 3 > org.testcontainers.containers.KafkaContainerTest > testExternalZookeeperWithExternalNetwork STARTED
Gradle Test Executor 3 > org.testcontainers.containers.KafkaContainerTest > testExternalZookeeperWithExternalNetwork STANDARD_OUT
13:08:12.044 INFO org.testcontainers.dockerclient.DockerClientProviderStrategy - Loaded org.testcontainers.dockerclient.RootlessDockerClientProviderStrategy from ~/.testcontainers.properties, will try it first
13:08:12.769 INFO org.testcontainers.dockerclient.DockerClientProviderStrategy - Found Docker environment with Rootless Docker accessed via Unix socket (/run/user/1001/docker.sock)
13:08:12.771 INFO org.testcontainers.DockerClientFactory - Docker host IP address is localhost
13:08:12.800 INFO org.testcontainers.DockerClientFactory - Connected to docker:
Server Version: 19.03.11
API Version: 1.40
Operating System: Ubuntu 20.04 LTS
Total Memory: 16009 MB
13:08:13.441 INFO org.testcontainers.DockerClientFactory - Ryuk started - will monitor and terminate Testcontainers containers on JVM exit
13:08:13.441 INFO org.testcontainers.DockerClientFactory - Checking the system...
13:08:13.442 INFO org.testcontainers.DockerClientFactory - ✔︎ Docker server version should be at least 1.6.0
13:08:13.516 INFO org.testcontainers.DockerClientFactory - ✔︎ Docker environment should have more than 2GB free disk space
13:08:13.526 INFO 🐳 [confluentinc/cp-kafka:5.2.1] - Creating container for image: confluentinc/cp-kafka:5.2.1
13:08:13.573 INFO 🐳 [confluentinc/cp-kafka:5.2.1] - Starting container with ID: 45be3c3b937cf4cc3bda7db03cf37b6fa77d5a61b8cb8926011973de60be47df
13:08:13.803 INFO 🐳 [confluentinc/cp-kafka:5.2.1] - Container confluentinc/cp-kafka:5.2.1 is starting: 45be3c3b937cf4cc3bda7db03cf37b6fa77d5a61b8cb8926011973de60be47df
13:08:20.635 INFO 🐳 [confluentinc/cp-kafka:5.2.1] - Container confluentinc/cp-kafka:5.2.1 started in PT8.623S
13:08:20.640 INFO 🐳 [confluentinc/cp-zookeeper:4.0.0] - Creating container for image: confluentinc/cp-zookeeper:4.0.0
13:08:20.758 INFO 🐳 [confluentinc/cp-zookeeper:4.0.0] - Starting container with ID: afed4f9041dc8b0dfcf3f675ef7ba28fd32d7e30d84285707e7a70f483eea3ad
13:08:21.179 INFO 🐳 [confluentinc/cp-zookeeper:4.0.0] - Container confluentinc/cp-zookeeper:4.0.0 is starting: afed4f9041dc8b0dfcf3f675ef7ba28fd32d7e30d84285707e7a70f483eea3ad
13:08:21.188 INFO 🐳 [confluentinc/cp-zookeeper:4.0.0] - Container confluentinc/cp-zookeeper:4.0.0 started in PT0.548S
13:08:21.189 INFO 🐳 [confluentinc/cp-kafka:5.2.1] - Creating container for image: confluentinc/cp-kafka:5.2.1
13:08:21.224 INFO 🐳 [confluentinc/cp-kafka:5.2.1] - Starting container with ID: 0599af42ab33d78f2cc7a16659a5682481627dd7c5e14a5658a8d398af5233c2
13:08:21.547 INFO 🐳 [confluentinc/cp-kafka:5.2.1] - Container confluentinc/cp-kafka:5.2.1 is starting: 0599af42ab33d78f2cc7a16659a5682481627dd7c5e14a5658a8d398af5233c2
13:08:28.873 INFO 🐳 [confluentinc/cp-kafka:5.2.1] - Container confluentinc/cp-kafka:5.2.1 started in PT7.684S
13:08:28.874 INFO 🐳 [alpine:latest] - Creating container for image: alpine:latest
13:08:28.914 INFO 🐳 [alpine:latest] - Starting container with ID: 9a20a13af178a5868291a7512bb856f919461ecf4304ec9f43705308d1a3a268
13:08:29.265 INFO 🐳 [alpine:latest] - Container alpine:latest is starting: 9a20a13af178a5868291a7512bb856f919461ecf4304ec9f43705308d1a3a268
13:08:29.271 INFO 🐳 [alpine:latest] - Container alpine:latest started in PT0.397S
13:08:29.480 INFO org.apache.kafka.clients.consumer.KafkaConsumer - Subscribed to topic(s): messages
13:08:30.285 INFO org.apache.kafka.clients.consumer.KafkaConsumer - Unsubscribed all topics or patterns and assigned partitions
13:08:30.308 INFO org.apache.kafka.clients.producer.KafkaProducer - Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms.
Gradle Test Executor 3 > org.testcontainers.containers.KafkaContainerTest > testExternalZookeeperWithExternalNetwork PASSED
Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.3/userguide/command_line_interface.html#sec:command_line_warnings
BUILD SUCCESSFUL in 21s
8 actionable tasks: 3 executed, 5 up-to-dateApparently we can!
Let’s focus on the most important parts:
13:08:12.769 INFO org.testcontainers.dockerclient.DockerClientProviderStrategy
- Found Docker environment with Rootless Docker
accessed via Unix socket (/run/user/1001/docker.sock)Although we did not set DOCKER_HOST, Testcontainers was able to detect the environment - this is one of my favorite features of Testcontainers!
13:08:12.771 INFO org.testcontainers.DockerClientFactory
- Docker host IP address is localhostThe host IP was correctly detected as localhost.
Will it work when you run rootless docker-in-docker?
It will 😎
13:08:12.800 INFO org.testcontainers.DockerClientFactory - Connected to docker:
Server Version: 19.03.11
API Version: 1.40
Operating System: Ubuntu 20.04 LTS
Total Memory: 16009 MB
13:08:13.441 INFO org.testcontainers.DockerClientFactory
- Ryuk started - will monitor and terminate Testcontainers containers on JVM exitIt should not surprise us that Ryuk works since we already checked it, but still good to know :)
Limitations
At this stage you must have a question: “Okay, Kafka works, nice, but what are the limitations? Which Testcontainers’ features will NOT work?”.
I was as curious as you are! So I went ahead and added a CI job to test all Testcontainers’ Core tests.
You won’t believe what I saw The result was more than satisfying - only one test failed, the one that tests the OOM killer! 🎉
After checking the daemon logs, the reason was clear:
error=“cgroups: memory cgroup not supported on this system”Luckily, this isn’t a big deal, we do not implicitly apply the memory limits anywhere, and the only reason the test was failing is because it expected the OOM killer to terminate the container.
After testing core, I went ahead and played with a few other modules, but even DB2 passed:
$ ../../gradlew test
> Task :db2:test
Gradle Test Executor 4 > org.testcontainers.jdbc.db2.DB2JDBCDriverTest > test[0 - jdbc:tc:db2://hostname/databasename] STARTED
Gradle Test Executor 4 > org.testcontainers.jdbc.db2.DB2JDBCDriverTest > test[0 - jdbc:tc:db2://hostname/databasename] PASSED
Gradle Test Executor 4 > org.testcontainers.junit.db2.SimpleDb2Test > testWithAdditionalUrlParamInJdbcUrl STARTED
Gradle Test Executor 4 > org.testcontainers.junit.db2.SimpleDb2Test > testWithAdditionalUrlParamInJdbcUrl PASSED
Gradle Test Executor 4 > org.testcontainers.junit.db2.SimpleDb2Test > testSimple STARTED
Gradle Test Executor 4 > org.testcontainers.junit.db2.SimpleDb2Test > testSimple PASSEDAnd trust me, DB2 is a big beast running in a privileged mode!
Shut Up And Take My Money!
Well, if you really want to give the Testcontainers project some money, you can sponsor us :)
http://github.com/sponsors/testcontainers
As always, this exciting “feature” will be coming in the next release of Testcontainers, 1.15.0, very soon.
In Testcontainers, we truly believe that it will help adopting Docker in some CI environments, and, after the analysis, I think Docker team did an amazing job implementing the rootless mode without breaking the compatibility and keeping it the same Docker as we know it (at least from the API point of view 😅).