This document is targeted at people cross-compiling lots of large Open Source software projects and who want to scale and generalize that effort.
Usually people understand cross-compilation as compiling a program for a CPU architecture different from the one used on the machine where the compilation is done. This is accomplished using a cross-compiler toolchain and cross-compiled libraries and specifying these in the Makefiles that build the software.
This is easy with the low-level software and compilation tools because they are required to support cross-compilation. It's a very different matter with Open Source software in general, especially with end-user applications and e.g. GUI toolkits.
Most of the Open Source software projects use a 'configure' script to configure their software for compilation. This script is produced by tools called 'autoconf' and 'automake', which process 'm4' macros written by the application developer. The script will generate the Makefiles that are used for building the software and a 'config.h' header file containing defines for features found on the build system.
Configure is meant to ease configuring the software for compilation and its default assumption is that the software will be run in the same environment in which it was compiled in and run from the place where it was installed to.
Autoconf provides application developers certain macros to check out features in the system (see MacKenzie, David & Elliston, Ben: Creating Automatic Configuration Scripts ). The problematic ones are:
Tries to compile, link and run given test code for some feature. Test programs return zero for success.
Tries to compile and link test code for library function existence.
Tries to compile and link test code using certain library.
This does the same as AC_TRY_LINK, but 'configure' tries to do linking from all of the system library paths, not just from ones given in CPPFLAGS, CFLAGS and CXXFLAGS environment variables.
Higher level software is traditionally compiled natively on Unix systems. Therefore, most of the application developers haven't taken into account cross-compilation when using these macros, so 'configure' ends up cross-compiling test code and trying to run it on build host which breaks the configuring. Configure can also find libraries, headers and versions of those that are only present on the build host, not on the target, which will fail either compilation of the program or running it on the target.
The worst thing is that you don't have any idea whether 'configure' found the correct values or not, unless you go manually through tens of thousands lines of output it produces (or the binary fails when it's run). Even if you've checked the standard things, application developers can and have written also their own m4 test macros and test programs for 'configure' to run.
When software is configured, the installation path is often written into sources. This installation path is often included also into numerous configuration files, dynamically loaded libraries etc. And the software doesn't work if it doesn't find it's configuration files in the specified place!
You usually need 'root' user rights to be able to install software into same path on the build host that it will be run on the target. Then you have in your build host system directories a program that doesn't run on that machine, and to test and run the program on the target device, you have to first track all of its files.
Configure itself is not a problem. It solves very well the problem it was designed for; configuring software for system of which the application developer (who made the software) doesn't know anything, but which the configure script can find out information about. The problem is how to limit and be sure about what it finds, how to let it test whether the found functionality actually works as expected and to make it install the software into correct place, both for the build host testing and the target device.
Current solution in many projects using already existing Open Source software is to configure and compile it natively. On real embedded development this solution is not optimal. It does not scale and embedded devices, even developer editions, are usually much slower than developer's x86 workstations and they have space constraints.
We have solved the problem by sandboxing the software configuration, compilation, building and testing into an environment that is exactly the same (where it matters to software being configured) as on the target device.
Within the sandbox user can select the CPU target which software compiled inside the sandbox will see as the native one. This sandbox is our cross-compilation SDK. It provides developer a self-contained environment containing everything needed for configuring, compiling and testing the software on the target environment on his normal desktop . Most of the time developers will need to use only the native (x86) CPU target which provides on the build host a Linux environment identical to that of the target device.
A company wanting to publish a SDK for it's device, can configure and compile this sandbox for their requirements and distribute it in binary format for the application developers. Sandbox can also contain device specific proprietary device drivers and libraries. In addition to the sandbox a complete company SDK will contain GUI tools, documentation etc.
Linux From Scratch project has compiled a lot of information about making a (sandboxed) Linux distribution from scratch and provides that information also in a book form. "Target environment" section will give an overview of these details. However, our main aim is cross-compilation and cross-configuration and we need to be able to use also other C-libraries than the standard glibc C-library so we have a few additional problems.
Creation of SDK sandbox can be divided into following parts which are compiled and configured in successive steps:
Tools needed by configure scripts + CPU transparency + chroot + SDK configuration / package management. These are common to all SDKs and needed only for x86.
Cross-compiler. There has to be a separate compiler toolchain for each of the CPU target / kernel major version / C-library combination (see Gcc cross-dependency mess).
Post-install part of the SDK tools.
Kernel / UML.
Debug tools + target package management.
Building, testing and documenting target framework. This is often company specific.
Low level target software arbitrating hardware access. These can be e.g. Busybox, GUI servers (Nano-X, TinyX version of XFree86 etc).
Other target libraries and services included into SDK. Some of this "middle-ware" may be proprietary.
Normal software design, edit and version management tools used outside the sandbox.
First three steps bootstrap the SDK sandbox and produce what we call a Scratchbox (i.e. Linux From Scratch -sandbox). Scratchbox utilities are compiled outside the SDK using the regular compiler (gcc) on the build host. Post-install part will be done when SDK is run the first time. Usually this is done by the person maintaining the SDK and normal developers get a fully installed binary version of the SDK.
After first three steps have been done, initial sandbox is ready for compiling rest of the software inside them, using the compiler(s) produced in step two.
Some of the debug tools might be used outside the sandbox to connect to debug utilities inside the sandbox.
Implementation section will give you more detailed explanation and reasoning for the above steps. Please get back here after reading it.
All the targets inside the SDK sandbox use the same host-compiled utilities (needed by 'configure' and other configuration utilities) which are run only on the build host. The compiler toolchain and libraries are specific to each of the sandbox target CPUs / C-library combination.
As the binaries produced by the PC target x86-toolchain can be run on your x86-desktop, we call this target the native target.
Target device will have two target systems:
Root image and packages produced as the result of the cross-compilation and removing of unnecessary files. This is the system that will be put on target devices when they are ready. We call it the product target system.
One used by the cross-compilation sandbox on the build host. This is used only in the product development, inside the sandbox. We call it the development target system.
Ideally you would have two target devices. One is used for developing the software and unit testing it with the development target system and second one is where you do all the integration and IPC tests for the final system using the produced product target system.
Development target system is the one you use for CPU transparency.
Our build host machine is a x86 Linux desktop machine running RedHat 8 with glibc2 as the shared C-library and comes with all the normal desktop Linux libraries and utilities installed.
We're tried both uClibc and glibc as the shared C-libraries and dynamically linked Busybox for the command line tools. Target CPU is ARM.
First problem of sandboxing 'configure' is that it needs a lot of (POSIX standard) software, which won't be available on the embedded target environment.
Second problem is that many 'configure' scripts test for features by compiling programs (with the supplied cross-compiler tool chain) and then running them. This doesn't work if program is compiled for a different CPU architecture than on which 'configure' tries to run it.
First you have to compile special (see Dynamic vs. Static Linking) versions of the utilities that 'configure' needs for configuring the software on the build host. They need to be separately compiled because when you use them inside the SDK sandbox, they won't be able to use the normally installed libraries from your desktop. In Scratchbox this is accomplished by using a separate GCC spec file which tells the linker that the libraries for the utilities reside in a special directory which is same while you compile the tools outside the SDK and once you run them inside it.
For utilities which install a lot of external (e.g. configuration or locale) files, installation can be harder. They work inside the sandbox if they will be there in the same directory path as you had installed them on the build host . We used the /scratchbox/ path as the installation prefix for these utilities on the build host and then copied this directory under the sandbox directory.
Normally 'configure' requires following standard GNU Unix utilities:
gcc and C-library
Busybox contains most of the things included into fileutils, textutils and findutils packages, but you still need to install at least following utilities from them: install, comm, split and find. If you don't use Busybox, then you need to install also following packages:
Autoconf and some of the configure files will additionally need:
Perl is painful, because it has its own configuration system. Either it won't install or run correctly for uClibc, depending on whether you've compiled it outside or inside the sandbox. Its configuration system will need manual editing to get it working.
More complete dependencies between the tools are listed in the following diagram.
Because also testing is done inside the sandbox target environment, developers will need to have debugging tools or stubs there. Some good tools are:
Because we don't want to configure, compile and maintain for each of the sandboxes CVS, IDE tools, GUI builders, code editors and other utilities used in creating and maintaining the source files, these utilities are run outside sandbox (see diagram in Proposed solution / Sandboxing).
This means that sandbox needs an easy and synchronized way to access the sources you maintain outside the sandbox(es).
A utility which traces for given program what kernel calls it does and what are their arguments and return values. This works almost on any CPU that Linux kernel work on.
A symbolic source code debugger. You need to recompile your program to include debug information to fully benefit from GDB. For each CPU, you need to compile GDB server (containing network debug stubs) and normal x86-version of GDB which understands that CPU . Server is run inside the CPU specific sandbox or on the target, and GDB on the build host. This way you can have those huge debug binaries only on the build host and the GDB server on the target can use stripped versions of them. Additional benefit is that you can use any of the GUI GDB front-ends on the build host to run GDB.
A dynamic library for testing all kinds of programming errors by running the program on a simulated x86-CPU. For example, Valgrind can detect stack, heap and kernel call memory access errors, memory leaks, threading, and bad x86-CPU cache utilization. It is only for the x86 target as CPU simulation slows the program execution significantly and consumes more memory.
There are two ways to limit the software configuration and compilation environment to the sandbox.
UML, the User Mode Linux is a Linux kernel that can be run inside another Linux instance. You point it to an image file containing the root filesystem and it boots from there. It's a great way to test new kernel versions and debug drivers. In the latest 2.6 Linux kernel UML is a standard compilation option.
In our case UML with the root image containing the above described "Target system" will provide the sandbox. Sandbox will also need to access the source files created and maintained outside it, this can be done by exporting the source directory with NFS from the build host, and mounting this inside the UML.
Currently UML has some minor problems and it's not included in the stable kernel. As UML is included in the 2.6 kernel, these issues should be fixed pretty soon. UML and NFS overhead will make compilation significantly slower.
Chroot is a kernel facility and user-space utility to limit program into given directory in the filesystem. Chrooted programs can't access any files outside this directory, this includes devices and dynamic libraries. Using symbolic links doesn't work either because program can't access the file pointed to by the link.
If you don't need or want to use UML, you can use 'chroot' to limit the configure environment to what will be on the target device. Just 'chroot' the shell which you use to do the compilation. Using chroot is faster than using UML and SDK tools can be on a normal build host directory instead of a separate root image.
If you want to do testing also in the chroot environment, you need to be able to use device files. To accomplish this, you can either:
'mknod ' the device files your target system requires into /dev/ -directory in the sandbox. For the device major and minor numbers, look into the /dev/ - directory of your build host, they are same on all the Linux versions. Note that several programs (e.g. ssh) will act funny if they can't access certain "system" devices like /dev/zero .
Or just copy your whole /dev/ directory inside the sandbox. In Scratchbox /dev/ is copied inside the SDK and that copy is then mounted with -bind option inside to each SDK users' own Scratchbox sandbox.
If you want regular users to be able to use 'chroot' after you've setup the sandbox, install the chroot-uid.c program as suid root. Note that users can then use this program to fool other suid root programs. It's mainly intended for users who already have root privileges, but would rather do development as a normal user. In the long run UML is a better alternative, once it is complete.
When doing testing in the sandboxed environment, you might need to access services such as X11 which are not inside the sandbox.
Easiest way would be to run Xnest with the same display attributes as your target system. Then you set the DISPLAY variable host part 127.0.0.1 and use same display number as you gave to Xnest. This should be as fast as using unix socket for the X11 communication.
UML offers network interface to the host machine. You can use X11 through the network. The downside is that you can't use shared memory extension which will slow significantly image operations. See tutorial .
UML offers also access to devices in case you have full X11 inside your sandbox. For this the sandbox user will need access to the required devices.
With chroot you don't necessarily need to go through the network for service access, you just need to get the server Unix domain socket available inside the sandbox. Most of the servers put their socket into /tmp/ directory.
In Scratchbox we've mounted the system /tmp/ directory with the -bind option to each users own Scratchbox sandbox.
If you have a network enabled device with CPU compatible to your target device and enough memory, Linux provides a mechanism which can be used to make the CPU architecture transparent to the software we're configuring.
Linux kernel includes support for running miscellaneous binary formats transparently to the user and user-space programs. This support is in a module called binfmt_misc which can also be compiled statically to the kernel. You configure it by echoing following values to the '/proc/sys/fs/binfmt_misc/register ' file:
Name of the binary format
How to identify the format
"Interpreter" that should be invoked to run the binary
See ARM identification for binfmt_misc for an example.
In our case the "interpreter" will be a program that executes the binary on the target device and forwards the standard input, output and error streams, environment variables and execution error code between the build host and target. This program has to be compiled so that it works in the sandbox also when it's configured for the target CPU.
'ssh' can be used on build host and 'sshd' on the target for doing the above. SSH keys required for automating this can be on the NFS directory and these keys shouldn't be used for anything else. Unfortunately, with ssh you will need to transfer the environment variables separately.
To automate all this, we've developed 'sbrsh / sbrshd ' client-server software that automates all the required interaction (NFS mounting, environment variable transfer, I/O redirection, error returning, chrooting) between build host and target device.
You can get the binary CPU architecture signature for binfmt_misc configuration from the '/usr/share/magic ' file used by the 'file ' utility.
You need to have run-time environment similar to the sandbox on the target device (or device having the same CPU as the target). This includes the shell and dynamic libraries, not the utilities needed by the 'configure'.
This requires networking as you need to NFS-mount the directory where you're doing the source configuration on the build host, into same place (in directory hierarchy) on the target device as it's on the build host sandbox. This way target will have the same cross-compiled programs as build host, and any files that the executed program writes on the target device, will be in the same place on the build host sandbox.
This setup is called the "Development target environment".
How the above works:
'configure' tests the system and won't find any libraries and include files located outside the sandbox.
'configure' creates and cross-compiles a test program and tries to run it.
Kernel binfmt_misc module will note a registered binary format being 'executed', and runs it using the given interpreter.
"interpreter" will run the binary on the target device from NFS mounted development directory and redirect all of it's input and output between target and build host.
Result is that proper files are updated through NFS and program input and output forwarded correctly so that everything works for 'configure' as it were doing things natively.
This works both with the chroot and User Mode Linux and solves the second problem.
Because CPU is transparent to the sandbox, testing the cross-compiled binaries can be done in the pre-configured target CPU sandbox without manually logging to the target and copying the files. This requires NFS to be set up correctly (including the firewall settings of the host machine). When doing the tests, it's better check that you're doing them using the correct CPU target environment. Note that you can do only tests that don't require processes to communicate with each other this way. For integration tests you need the target product system environment.
A single development target device can be enough for a whole developer group for the target configuration / compilation and testing phase.
You can do development for the target environment with SDK sandbox containing only the x86 target toolchain.
You need the target device and cross-compilation support only for testing CPU architecture specific features and of course making a target specific binary release of your software. For this it's enough that you have a device with the correct CPU, network interface supported by Linux and enough RAM. It doesn't need to be the exact device for which you're doing the development for, the same CPU (family) is enough (e.g. iPAQ for StrongArm target).
Some of the target specific kernel device driver testing can be done using x86 emulated device drivers in User Mode Linux and a x86-sandbox. This should be much easier than native testing when your final target has limited resources and/or no network.
If you don't have a target device with networking capabilities or are otherwise lacking a target device, you can still do something with the software 'configure' scripts. Having the above described SDK will deal with the issues of 'configure' finding wrong libraries, versions, header files etc.
Then you have to fix just everything on configure scripts that tries to run cross-compiled binaries. Some possible ways of dealing with this:
Ifdeffing (the m4 way) those lines in the scripts and providing the required values otherwise e.g. by using configure cache values and environment values.
For certain things you can e.g. compile the programs on x86 and make configure to use your pre-compiled version of the test program. You still have to check the output for CPU architecture specific differences, with which our solution above deals with.
Make the binfmt_misc solution to catch all the cross-compiled program runs and return a correct value.
In our opinion this is suitable way for fixing only a small number of software build systems.
We tested two ARM emulators to see whether they could be used instead of a real device with target CPU.
Neither of these emulators could be used with the sandbox, either because their emulation of the target CPU was incomplete, they were too slow (several orders of magnitude) or they needed an environment which would have made the interaction with the sandbox environment too difficult.
For other CPUs (e.g. Motorola 68xxx family) there are other emulators, but we would assume the problems to be similar and in most cases our solution to be most convenient.
GDB debugger contains an ARM emulator, ARMulator .
SoftWare ARM . Can run ARM binaries as you were running normal x86 binaries (i.e. doesn't need separate operating and disk images) but it's very slow.
Cross-developing OSS projects poses the following major challenges:
cross-compilation: compiling a program for a CPU architecture different from the one used on the machine where the compilation is done
cross-configuration: setting up build environment configuration so that is reliably mirrors the target environment.
Scratchbox deals with these challenges by sandboxing the software configuration, compilation, building and testing process into an environment that is exactly the same as on the target device. There are two major problems when configuration is sandboxed inside an isolated build environment:
configure needs a lot of POSIX software, which won't be available on the embedded target environment.
many configure scripts test for features by compiling programs and then running them, which normally does not work if the program is cross-compiled for a different CPU architecture.
Scratchbox solves the first problem by supplying a full development toolchain that can be run on the build host. Build utilities are not required on the target device at all.
The second problem is handled by Scratchbox "CPU transparency" using a Linux kernel feature to detect binaries that are compiled for a different CPU architecture. Any test source files required by the configuration process are cross-compiled for the target platform. When they are run, Scratchbox effectively commands the target device to run the binary instead, and correct results are communicated back to the configuration script on the build host.
First we though that by compiling SDK tools statically we would avoid the problem that dynamic C-library inside the SDK is different and that it can be for different CPU architecture than the one which SDK tools are compiled against.
However, that didn't work because glibc compiled programs can't be compiled completely statically, nss (which resolves the user names and groups for 'ls' for instance) and pam (checks user password and other information in /etc/passwd) components get their functionality from runtime loaded dynamic libraries which in turn depend on dynamic glibc library. Recompiling glibc would have had the disadvantage that we don't know what name service and security module functionality developer would need in compiled into glibc and then SDK tool compilation would have included also glibc compilation which we would like to avoid.
Finally we solved this by forcing host linker to hardcode the dynamic library load path into SDK tool binaries and copying all the dynamic library dependencies (+ nss and pam modules) inside SDK into that hardcoded library location from the SDK build host. That way SDK tools don't find dynamic libraries intended for the target target software and target software configure scripts don't find host libraries.
Building of the base C and C++ libraries in integrated into GCC build. This is very awkward from the cross-compilation point of view because then dynamic applications compiled with gcc will depend from the C-library against which gcc itself was compiled with.
E.g. gcc v3.2 will compile stdlibc++ library (containing STL etc.) along with the g++ compiler. So, this library will now depend from the C-library with which g++ was compiled with and C++ applications compiled against that library will then also depend from C-library with which g++ was compiled with. Because libstdc++ is over 1MB, using it statically is not an option as libgcc.a is for gcc. Why can't they release gcc base C-libraries as a separate package which is always released with the gcc package?
Because of above, you can't build compilers like this: compile gcc / g++ for each target CPU, compile C-libraries, compile C++ libraries, compile applications.
Instead you have to do it like this: compile first-phase gcc, compile C-library with that gcc version, re-compile whole gcc toolchain with the resulting C-library, then compile applications.
The result is that you need a separate toolchain for each CPU / C-library combination. It would have been nice to keep gcc toolchains as part of the SDK and only C / C++ -libraries target specific.
You should note that because C-library implements the kernel syscall interface, C-library always depends from the target kernel, or headers for the same (major) version of kernel. You can't just blindly use kernel headers from your host, Linux kernel user space API changes between major kernel versions.
So actually you need a separate toolchain for each kernel major version / CPU / C-library combination.
For 99% of the projects this is not a problem. For a generic cross-compilation SDK it is.
ARM identification for binfmt_misc
# register arm_runner to run ARM ELF binaries
\xff:/bin/arm_runner: > /proc/sys/fs/binfmt_misc/register
When you're cross-compiling for the target CPU, some of the binaries in the SDK sandbox are compiled for the target device CPU and some of the tools needed by the 'configure' are compiled for the build host CPU (in our case "build host" means x86 and "target" ARM binaries).
Here's is how we have divided the binaries in the SDK sandbox and what the different directories are supposed to contain:
Contains SDK (x86) tools which are used by the configure and other build systems and used for configuring the Scratchbox.
Dynamic and static system libraries for the target copied from the corresponding target compiler toolchain.
Everything compiled inside the sandbox for the target, should be installed into /usr directory. /usr is a link to target specific binary directory.
Build-tools (built by target software packages) that are intended not to be run on the target.
The sources to configure and compile are under each developers home directory in the sandbox.
Contains configuration files (user IDs, terminal and shell setting etc) for the host tools . Target software configuration files should go to /usr/etc/. When rootimage is is created, its /etc directory contents come from a special module that takes care of the target system setup.
SDK cross-compilers + build tools and the libraries & files required by them. These binaries can be run only on the host system. The dynamic library path in the binaries is hard-coded to /scratchbox/host_shared/lib/ directory (see Dynamic vs. static linking).
Contains all the targets installed to the SDK. Top directories from the currently selected target directory are linked to the sandbox root directory.
Mount -bind'ed from a copy of system /dev directory.
Mounted inside SDK just like the system /proc.
Mount -bind'ed from the system /tmp so that sockets can be shared with services on the desktop.
Any other directories under the root directory contain target device specific files. It's possible to switch the target on the fly, this will just link directories to ones suitable for selected target. After changing target, the previous compiled object files etc. under /home should be cleaned before starting to compile things for the new target.
The name of the current Scratchbox compilation target (CPU) is shown on the sandbox shell prompt.