Nothing new under the sun, but this is a bit of a “clever” hack which I think it is worth discussing.
It is often the case we have to compile software for a target on which it is impractical, or outright impossible, to perform such computation.
This is usually referred as cross-compilation or, more generally, cross-building.
However, if our target system is Linux, one can leverage the absurd degree of stability in its binary interface.
With some tricks, we can significantly improve our developer experience.
qemu-user
Most modern Linux distribution have a hidden magical power: they can run executables compiled for virtually any other architecture.
Most of the heavy work is performed by qemu which aside from the emulating the target architecture, converts system calls so that they can be run natively by the native kernel..
Enabling this feature is usually as easy as just installing a few packages.
On debian and similar systems, sudo apt install qemu-system-xxx qemu-user is generally sufficient.
Make sure to replace xxx with the desired architecture. You can add as many as you want.
One issue with this approach, is that only binaries which have been statically compiled can run out of the box.
For those assuming dynamic linking to system libraries is available, we have to implement a bit of a workaround.
This is because you would be expected to have all dependencies for the target architectures already cross-compiled and available in some location.
Also, the application loader can no longer run transparently, and you are forced to write something like:
qemu-xxx -L /usr/local/xxx/lib ./app
Continuing down this path is surely possible, but I would argue there is a better solution to that.
chroot
chroot is a second super-power we get in Linux almost for free.
Basically, we are able to convince the current shell that its root is no longer / but somewhere else in the filesystem, allowing us to implement a very thin (and unsafe) container.
The nice thing is that the new root can contain binaries and libraries for whatever architecture we want.
The background service which was intercepting the execution of binaries, and associating it to qemu-user for not native ones is still running, which means opening a shell in this chroot will lead to a mostly seamless experience.
The main issue we discussed earlier was about dynamic libraries, but now once they are installed within our chroot everything just works as intended.
As for the image on which we can run chroot, it depends on our application. It can be something custom we constructed via buildroot, or a minimal alpine image designed for docker.
cross-compile
The last ingredient in the mix is cross compilation. You can pick your compiler of choice, but I want to make a case for clang/llvm in this specific scenario.
Unlike gcc, for which different packages are needed for different architecture, each of which comes with its own set of libraries and build-utils, clang allows us to generate code from and to any platform out of the box.
The idea is that all headers and libraries will not be installed as part of the main system as we would do with gcc, but as part of the chroot image.
Actually, in my specific scenario I have two: one used for building and one only for runtime; this way we can package our software for distribution, without shipping toolchains and development dependencies.
An example can be better than a thousand words, so here is a possible way to replicate what has been presented so far.
These instructions will assume we desire a riscv64 environment to work with:
sudo apt install qemu-system-riscv qemu-user clang clang++
# In this case we are using a recent alpine image
wget https://dl-cdn.alpinelinux.org/alpine/v3.21/releases/riscv64/alpine-minirootfs-3.21.3-riscv64.tar.gz
mkdir -p chroot-dev
tar -xvzf alpine-minirootfs-3.21.3-riscv64.tar.gz -C chroot-dev
# Otherwise internet connection from within will be broken out of the box
echo "nameserver 8.8.8.8" > chroot-dev/etc/resolv.conf
# We also create the environment we will use for distribution
cp -r chroot-dev chroot
Now we will get inside the images to install some basic packages we need:
# Optionally we can have a specific cpu model defined in here like `QEMU_CPU=cortex-a76 \`
# unshare is used so that we don't need to be root to run the chroot.
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" HOME=/root USER=root TERM=xterm \
unshare -mr /sbin/chroot ./chroot-dev /bin/sh
You might want to create a small script to make the process of opening a shell inside chroot-dev a bit easier.
Once inside:
apk add gcc g++ libstdc++-dev
While in chroot:
apk add libstdc++
Now let’s create a simple source file:
#include <iostream>
using namespace std;
int main(){
std::cout<<"Hello world from RISCV64!\n";
return 0;
}
We can be compiled as follows:
clang++ main.cpp -o main --target=riscv64-alpine-linux-musl --sysroot=./chroot-dev/ -fuse-ld=lld
mv main ./chroot/root/
and back to chroot we can run it in all its glory:
~/main
If we have full applications in our “container” we want to run from the outside, we could wrap the command to run them into a bash script or a *.Desktop file.
This is it! I hope you found it informative 🙂.