As a JVM guy, I got used to “write once, run everywhere” paradigm. But recently I got an interesting challenge to solve.
I was working on a debugging tool for Project Reactor and part of it is implemented as a native library for Java, using JNI technology.

I did some C++ in the past but never thought how hard is it, to cross-compile a library for 3 major platforms (Windows, Linux and Mac), both locally and on CI environment (from where we perform the releases). And, since Googling didn’t really help, I decided to share my findings here.

First attempt - Clang, LLVM and all that Jazz

Of course I started with “half-VM” solution, LLVM with Bitcode. You compile your C++ into something they call “Bitcode”, sounds almost like Java’s bytecode, right? Nope.

Apparently, Clang’s Bitcode is not really cross-platform, and even if you manage to compile into a “cross-platform” Bitcode, you still have to link it. And, despire the fact that everyone is talking about “Clang” as an “easy to cross-compile” tool, my experience was far from “easy” word.
I will just leave it here: https://clang.llvm.org/docs/CrossCompilation.html

Second attempt - “Hey Google, what about Docker?”

My first attempt (as well as my whining about C++, toolchains, Clang vs GCC, etc…) was observed by others who were sitting in DockerCon EU 2018’s speakers room, and Sune @sirlatrom Keller was like: “Why don’t you run it in Docker?”. Although the idea was nice, I wasn’t sure that I can run everything inside a container without changing the development workflow. But…

If you Google for “c++ Docker cross-compile”, most probably you will find this image:
https://github.com/dockcross/dockcross

When I clicked the link, I had this warm “A-ha, solved with Docker!” feeling. But, unfortunatelly, I haven’t figured out how to use it with my use case, plus it is lacking Mac OS X support.

The Best Place to Hide a Dead Body is Page Two of Google

Apparently, there is another project:
https://github.com/multiarch/crossbuild

It contains a rich set of toolchains, provides “all-in-one” image (although it might be a bad thing for some, having to download ~1Gb image) and super easy to use! Here is an example:

$ docker run -v $PWD:/workdir -e CROSS_TRIPLE=windows multiarch/crossbuild make

$ file helloworld
helloworld: PE32+ executable (console) x86-64, for MS Windows

Changing CROSS_TRIPLE to darwin (or non-alias triple x86_64-apple-darwin) will produce a binary for Mac OS X. Simple, huh?

CMake & cross-compilation

I am using CMake to compile my C++ projects, and there were a few properties I had to set depending on platform to make it work.

Mac OS X

If you’re building a library, most probably you expect it to have .dylib suffix (so that the JVM can load it with System.load). But, since we’re running inside Linux environment, I was getting .so instead.
Setting -DCMAKE_SYSTEM_NAME=Darwin fixed the problem.

Also, unfortunatelly, the Mac toolchain inside the image is a bit old, and we need to force LibC++:
-DCMAKE_CXX_FLAGS=-stdlib=libc++

Windows

By default, you will get linkage error with something like Unknown flag "-rdynamic". To fix it, you need to hint CMake with:
-DCMAKE_SYSTEM_NAME=Windows.

JNI & cross-compilation

Since I was aiming to build a JNI library, I had to deal with platform-specific headers. There is JNI module in CMake but it doesn’t help because even if we install JDK inside the image it will only contain Linux headers.
Luckily, there is AdoptOpenJDK project where you can get OSS friendly builds of OpenJDK, including the headers for each platform. Here is a snippet I was using to include them:

include_directories(src/main/headers)
if(WIN32)
    include_directories(src/main/headers/win32)
endif()
if(APPLE)
    include_directories(src/main/headers/darwin)
endif()
if(UNIX AND NOT APPLE)
    include_directories(src/main/headers/linux)
endif()

Bonus: Gradle with dockerized toolchain

🤪 Don’t ask why, but I was also using Gradle to run the build (the project is multi-module and most of the code is in Java, not C++). However, it simply delegates to CMake with com.cisco.external-build plugin. Apparently, you can easily run the build inside Docker with it!

First, you need to prepare a fake toolchain:

$ bat toolchain/*
───────┬────────────────────────────────────────────────────────────────────────────
       │ File: toolchain/cmake
───────┼────────────────────────────────────────────────────────────────────────────
   1   │ `dirname "$0"`/run_in_docker.sh `basename "$0"` $@
───────┴────────────────────────────────────────────────────────────────────────────
───────┬────────────────────────────────────────────────────────────────────────────
       │ File: toolchain/make
───────┼────────────────────────────────────────────────────────────────────────────
   1   │ `dirname "$0"`/run_in_docker.sh `basename "$0"` $@
───────┴────────────────────────────────────────────────────────────────────────────
───────┬────────────────────────────────────────────────────────────────────────────
       │ File: toolchain/run_in_docker.sh
───────┼────────────────────────────────────────────────────────────────────────────
   1   │ docker run -i --rm \
   2   │     -v $PROJECT_ROOT_DIR:$PROJECT_ROOT_DIR \
   3   │     -w $PWD \
   4   │     -e CROSS_TRIPLE \
   5   │     multiarch/crossbuild \
   6   │     "$@"
───────┴────────────────────────────────────────────────────────────────────────────

Here we simply delegate cmake and make to multiarch/crossbuild Docker container.
Note -e CROSS_TRIPLE flag, it is something like “pass host’s CROSS_TRIPLE into a container”.

Now you need to tell External Build Plugin to use our toolchain instead of system’s one and make it build for 3 platforms at once. Here is my build.gradle:

plugins {
    id 'com.cisco.external-build' version '1.14'
}

model {
    platforms {
        osx {
            operatingSystem 'osx'
            architecture 'x86-64'
        }

        linux {
            operatingSystem 'linux'
            architecture 'x86-64'
        }

        windows {
            operatingSystem 'windows'
            architecture 'x86-64'
        }
    }

    components {
        MyLibrary(com.cisco.gradle.externalbuild.ExternalNativeLibrarySpec) {
            targetPlatform "osx"
            targetPlatform "linux"
            targetPlatform "windows"

            buildConfig(com.cisco.gradle.externalbuild.tasks.CMake) {
                cmakeExecutable file("toolchain/cmake") // (1)
                executable file("toolchain/make") // (2)
                environment = [
                        'PROJECT_ROOT_DIR': rootProject.projectDir,
                        'CROSS_TRIPLE': binary.targetPlatform.name, // (3)
                ]

                // See "CMake & cross-compilation" section
                switch (binary.targetPlatform.operatingSystem.name) {
                    case "osx":
                        cmakeArgs(
                                '-DCMAKE_SYSTEM_NAME=Darwin',
                                '-DCMAKE_CXX_FLAGS=-stdlib=libc++',
                        )
                        break;
                    case "windows":
                        cmakeArgs(
                                '-DCMAKE_SYSTEM_NAME=Windows',
                        )
                        break;
                }

                cmakeRoot '.'
                workingDir file("${buildDir}/ext/${binary.targetPlatform.name}")

                inputs.dir 'toolchain'
                inputs.dir 'src'
                inputs.file 'CMakeLists.txt'
                outputs.dir workingDir
            }
        }
    }
}

(1), (2): here we override cmake and make commands, this is where the magic happens. Otherwise, it will use system’s.
(3): we use target platform’s name as CROSS_TRIPLE env variable.

Result

Now we can easily build our JNI library for Windows, Mac and Linux, on a single machine with Docker instead of a farm of hard to maintain CI nodes.