Cross-Compiling Rust Apps II: Linux Subtrees and Linker Shenanigans

Enabling dbus_mpris

Now that we’ve got a working cross-compile setup for the base feature set in our app, let’s turn on the additional features that we need for our project.

In addition to spotifyd’s default features, I also need dbus_mpris, which provides metadata about what’s playing to other applications.

So, let’s build with the feature enabled:

cross build --target=armv7-unknown-linux-gnueabihf --features=dbus_mpris
...
error: failed to run custom build command for `libdbus-sys v0.2.1`

Caused by:
  process didn't exit successfully: `/target/debug/build/libdbus-sys-132519d4200a73a4/build-script-build` (exit code: 101)
  --- stderr
  thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Failure { command: "\"pkg-config\" \"--libs\" \"--cflags\" \"dbus-1\" \"dbus-1 >= 1.6\"", output: Output { status: ExitStatus(ExitStatus(256)), stdout: "", stderr: "Package dbus-1 was not found in the pkg-config search path.\nPerhaps you should add the directory containing `dbus-1.pc\'\nto the PKG_CONFIG_PATH environment variable\nNo package \'dbus-1\' found\nPackage dbus-1 was not found in the pkg-config search path.\nPerhaps you should add the directory containing `dbus-1.pc\'\nto the PKG_CONFIG_PATH environment variable\nNo package \'dbus-1\' found\n" } }', /cargo/registry/src/github.com-1ecc6299db9ec823/libdbus-sys-0.2.1/build.rs:6:70

Oh, right; same problem as before. pkg-config is failing, probably because dbus isn’t installed. We know how to fix that! Let’s add the package libdbus-1-dev¹ to our install in the Docker container and build again.

FROM rustembedded/cross:armv7-unknown-linux-gnueabihf-0.2.1

RUN apt-get update
RUN dpkg --add-architecture armhf && \
    apt-get update && \
    apt-get install --assume-yes \
      libssl-dev:armhf \
      libasound2-dev:armhf \
      libdbus-1-dev:armhf

ENV PKG_CONFIG_LIBDIR=/usr/lib/arm-linux-gnueabihf/pkgconfig

Now build the container and the application:

docker build -t crossbuild:local .
# We clean beforehand out of good hygiene -- changes to the container
# might change the way some deps are resolved, and `cross` doesn't
# handle that automatically
cargo clean
cross build --target=armv7-unknown-linux-gnueabihf --features=dbus_mpris

Oh, we got a grody-looking linker error right at the end:

Compiling spotifyd v0.2.24 (/project)
error: linking with `arm-linux-gnueabihf-gcc` failed: exit code: 1
|
= note: "arm-linux-gnueabihf-gcc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-Wl,--eh-frame-hdr" "-L" "/rust/lib/rustlib/armv7-unknown-linux-gnueabihf/lib"
...[approximately 200 lines of text]...
"-Wl,-Bdynamic" "-lssl" "-lcrypto" "-ldbus-1" "-lasound" "-lutil" "-ldl" "-lutil" "-ldl" "-lrt" "-lpthread" "-lgcc_s" "-lc" "-lm" "-lrt" "-lpthread" "-lutil" "-ldl" "-lutil" "--sysroot=/build/armhf-sysroot"
= note: /usr/lib/gcc-cross/arm-linux-gnueabihf/5/../../../../arm-linux-gnueabihf/bin/ld: warning: libsystemd.so.0, needed by /usr/lib/arm-linux-gnueabihf/libdbus-1.so, not found (try using -rpath or -rpath-link)
        /usr/lib/arm-linux-gnueabihf/libdbus-1.so: undefined reference to `sd_listen_fds@LIBSYSTEMD_209'
        /usr/lib/arm-linux-gnueabihf/libdbus-1.so: undefined reference to `sd_is_socket@LIBSYSTEMD_209'
        collect2: error: ld returned 1 exit status

libsystemd.so.0 not found? Ok, weird; I don’t know how they’re connected. Fine, let’s just install libsystemd-dev?

FROM rustembedded/cross:armv7-unknown-linux-gnueabihf-0.2.1

RUN apt-get update
RUN dpkg --add-architecture armhf && \
    apt-get update && \
    apt-get install --assume-yes \
      libssl-dev:armhf \
      libasound2-dev:armhf \
      libdbus-1-dev:armhf \
      libsystemd-dev:armhf

ENV PKG_CONFIG_LIBDIR=/usr/lib/arm-linux-gnueabihf/pkgconfig

Rebuild:

$ docker build -t crossbuild:local .
$ cargo clean
$ cross build --target=armv7-unknown-linux-gnueabihf --features=dbus_mpris
...
Compiling spotifyd v0.2.24 (/project)
    Finished dev [unoptimized + debuginfo] target(s) in 1m 32s

Success! Now just copy it across to the Raspberry Pi and run it:

pi@boombox:~ $ ./spotifyd --version
./spotifyd: error while loading shared libraries: libssl.so.1.0.0: cannot open shared object file: No such file or directory

Oh.

Ok?

But… wasn’t I building with libssl before?

Let’s download the old and new compiled versions of spotifyd and try to figure out what’s going on. At some point in my travels I read that you’re supposed to use ldd to verify that things are linked ‘how you would expect them’. That’s going to be hard, given that I don’t know what I’m doing and therefore have no expectations, but lets run it anyway:

$ ldd old-binary
linux-vdso.so.1 (0x7ee61000)
/usr/lib/arm-linux-gnueabihf/libarmmem-${PLATFORM}.so => /usr/lib/arm-linux-gnueabihf/libarmmem-v7l.so (0x761be000)
libasound.so.2 => /usr/lib/arm-linux-gnueabihf/libasound.so.2 (0x760dd000)
libdl.so.2 => /lib/arm-linux-gnueabihf/libdl.so.2 (0x760ca000)
librt.so.1 => /lib/arm-linux-gnueabihf/librt.so.1 (0x760b3000)
libpthread.so.0 => /lib/arm-linux-gnueabihf/libpthread.so.0 (0x76089000)
libgcc_s.so.1 => /lib/arm-linux-gnueabihf/libgcc_s.so.1 (0x7605c000)
libc.so.6 => /lib/arm-linux-gnueabihf/libc.so.6 (0x75f0e000)
/lib/ld-linux-armhf.so.3 (0x76f41000)
libm.so.6 => /lib/arm-linux-gnueabihf/libm.so.6 (0x75e8c000)

Yep, no idea what this means.

$ ldd new-binary
linux-vdso.so.1 (0x7efb8000)
/usr/lib/arm-linux-gnueabihf/libarmmem-${PLATFORM}.so => /usr/lib/arm-linux-gnueabihf/libarmmem-v7l.so (0x75c4b000)
libssl.so.1.0.0 => not found
libcrypto.so.1.0.0 => not found
libdbus-1.so.3 => /lib/arm-linux-gnueabihf/libdbus-1.so.3 (0x75bfa000)
libasound.so.2 => /usr/lib/arm-linux-gnueabihf/libasound.so.2 (0x75b19000)
libdl.so.2 => /lib/arm-linux-gnueabihf/libdl.so.2 (0x75b06000)
librt.so.1 => /lib/arm-linux-gnueabihf/librt.so.1 (0x75aef000)
libpthread.so.0 => /lib/arm-linux-gnueabihf/libpthread.so.0 (0x75ac5000)
libgcc_s.so.1 => /lib/arm-linux-gnueabihf/libgcc_s.so.1 (0x75a98000)
libc.so.6 => /lib/arm-linux-gnueabihf/libc.so.6 (0x7594a000)
/lib/ld-linux-armhf.so.3 (0x76ef1000)
libm.so.6 => /lib/arm-linux-gnueabihf/libm.so.6 (0x758c8000)
libsystemd.so.0 => /lib/arm-linux-gnueabihf/libsystemd.so.0 (0x7582b000)
liblzma.so.5 => /lib/arm-linux-gnueabihf/liblzma.so.5 (0x757fa000)
liblz4.so.1 => /usr/lib/arm-linux-gnueabihf/liblz4.so.1 (0x757ce000)
libgcrypt.so.20 => /lib/arm-linux-gnueabihf/libgcrypt.so.20 (0x756f4000)
libgpg-error.so.0 => /lib/arm-linux-gnueabihf/libgpg-error.so.0 (0x756ca000)

Ok, now that I see the two things side-by-side, I can in fact ascertain that I did have expectations: I expected libssl to be linked from the start. This appears to be telling me that I went through all the effort of building it into my toolchain and it wasn’t even used? How does that even work?

Linker Detectives

The answer lies in the way that the binary is linked. The problem is, convincing Cargo to tell us what it’s asking the linker to do is convoluted².

Let’s switch to the Nightly Rust Compiler so we can investigate using a handful of more exotic, unstable options:

# Tell Rust to use the less stable "nightly" compiler by default,
# which has bonus compiler options
$ rustup default nightly
# spotifyd has an override for the rust toolchain in this file...
$ cat rust-toolchain
stable
# ...so let's delete it.
rm rust-toolchain

The unexpected behaviour was when we were building without --feature=dbus_mpris, so let’s build without that feature, but with an unstable switch to print the linker arguments³:

cross rustc --target=armv7-unknown-linux-gnueabihf -- -Z print-link-args

The last thing that happens in the build process is a truly horrific link command. For reference, the full command is here – but here’s a heavily summarised version:

"arm-linux-gnueabihf-gcc"
    "-Wl,--as-needed"
    "-Wl,-z,noexecstack"
    "-Wl,--eh-frame-hdr"
    "-L" "/rust/lib/rustlib/armv7-unknown-linux-gnueabihf/lib"
    [a whole bunch of files matching /target/armv7-unknown-linux-gnueabihf/debug/deps/spotifyd-$HASH.rcgu.o]
    "-o" "/target/armv7-unknown-linux-gnueabihf/debug/deps/spotifyd-5db2b974482c40d9"
    "/target/armv7-unknown-linux-gnueabihf/debug/deps/spotifyd-5db2b974482c40d9.k44uzd5y4kgye2x.rcgu.o"
    "-Wl,--gc-sections"
    "-pie"
    "-Wl,-zrelro"
    "-Wl,-znow"
    "-nodefaultlibs"
    [a whole bunch of "-L" $BUILD_OUTPUT_DIRECTORY entries]
    "-Wl,-Bstatic"
    [a whole bunch of files matching /target/armv7-unknown-linux-gnueabihf/debug/deps/lib$CRATE_NAME-$HASH.rlib]
    "-Wl,--start-group"
    [a whole bunch of files matching /rust/lib/rustlib/armv7-unknown-linux-gnueabihf/lib$CRATE_NAME-$HASH.rlib]
    "-Wl,--end-group"
    "/rust/lib/rustlib/armv7-unknown-linux-gnueabihf/lib/libcompiler_builtins-1fbf992051cf5302.rlib"
    "-Wl,-Bdynamic"
    "-lasound" "-lutil" "-ldl" "-lutil" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-lutil"

There’s a lot in here that I don’t understand, but two things are surprising to me:

• There’s no -lssl in the last line, which is required to link the openssl shared system library
• In fact, all references to openssl-sys are gone from the link command. 🤔

It looks like the Rust Compiler is smart enough to determine that the openssl-sys crate, while being a build dependency, isn’t actually used anywhere, and then removes the linkage to the library and its native dependencies entirely⁴ ✨.

Getting the right version of libssl

All of that was (maybe?) good to know, but it hasn’t helped with our actual problem – that the version of libssl we linked against (libssl.so.1.0.0) isn’t available on our target system.

So, what version is installed on the Raspberry Pi?

$ find /usr/lib -name 'libssl*'
/usr/lib/arm-linux-gnueabihf/libssl.so.1.1
# more results...

Ah, right, so the version of libssl in my Raspberry Pi distribution (I’m running Buster) is newer than the one I’m compiling against.

Googling the error message (“error while loading shared libraries: libssl.so.1.0.0: cannot open shared object file: No such file or directory”) reveals a lot of people all dealing with the same problem, and a lot of other people suggesting that they download a .deb package from another linux distribution and install that alongside the preinstalled, newer libssl.so.1.1. This is maybe the most expedient solution for closed-source programs, where we can’t control the linkage, but feels sloppy for something that we have the source for. So instead, let’s somehow convince the openssl-sys rust crate to link against the Raspberry Pi’s libssl.

Building a mini-distribution with multistrap

What we need to do, then, is ensure that a copy of libssl.so.1.1 (and anything libssl.so.1.1 requires) is available on the host system, for the target architecture. There’s a couple of ways of doing this that come to mind:

• Switch our build host to the roughly the same distribution as what’s running on the Raspberry Pi, and then continue with our current strategy of installing the required packages for the target architecture. The advantage of this is that it’s less fiddling, if you’re able to change the host distribution easily (because e.g. it’s running in a VM / container) and are confident that the steps for setting up a cross-build toolchain haven’t changed much.
• Setting up a minimal and separate Raspberry Pi distribution within a subdirectory of our host system. This has the advantage of being able to make use of the dependency graph of the target linux distribution, but has the disadvantages of requiring specific tooling, and of meaning that we’ll need to do a bit of work to tell our existing build tools where to find this system.
• Downloading and unpacking the required packages manually. I… honestly can’t see any big advantages to this approach, but I guess it works as a catch-all / last-resort.⁵

Of the three, I picked the second – setting up a target distribution within the host. We’re going to do it with a program called multistrap⁶.

So, back to modifying our Dockerfile. Let’s install multistrap on our host system, and then use it to install a minimal Raspberry Pi installation:

FROM rustembedded/cross:armv7-unknown-linux-gnueabihf-0.2.1

# Install multistrap
RUN apt-get update && apt-get install multistrap --assume-yes
# Put the target system here in /sysroot-armhf
RUN mkdir /sysroot-armhf
# We'll explain this in just a second
COPY multistrap-config /
RUN multistrap -a armhf -f multistrap-config /sysroot-armhf

We also need to define a config file for multistrap, which we’ve named multistrap-config:

[General]
# target architecture
arch=armhf
# Don't bother with authentication. I'd love to support this, but
# couldn't get it working! If you know how, please get in touch.
noauth=true
# Unpack the packages
unpack=true
# Tidy up afterwards (makes the container smaller)
cleanup=true
# Both of these refer to the 'Raspbian' stanza, below
bootstrap=Raspbian
aptsources=Raspbian

[Raspbian]
# Required packages for our build
packages=libasound2-dev libdbus-1-dev libssl-dev libc6-dev
source=http://raspbian.raspberrypi.org/raspbian/
# distribution version name
suite=buster

We can now open a shell inside that container to check that everything is installed as expected:

$ docker build -t crossbuild:local .
$ docker run -it crossbuild:local bash
# inside the container
$ find /sysroot-armhf/ -name '*.so' | grep -E 'lib(ssl|asound|dbus-1)'
/sysroot-armhf/usr/lib/arm-linux-gnueabihf/libssl.so
/sysroot-armhf/usr/lib/arm-linux-gnueabihf/libasound.so
/sysroot-armhf/usr/lib/arm-linux-gnueabihf/libdbus-1.so

Nice.

Finding libraries in the target distribution

Let’s now modify our container so that pkg-config knows where to find things again. Let’s set the PKG_CONFIG_LIBDIR at the end of our Dockerfile (just like last time), and also a new variable, PKG_CONFIG_SYSROOT_DIR, which allows us to specify that everything should happen relative to the target system directory:

#...
ENV PKG_CONFIG_LIBDIR_armv7_unknown_linux_gnueabihf=/sysroot-armhf/usr/lib/arm-linux-gnueabihf/pkgconfig
ENV PKG_CONFIG_SYSROOT_DIR_armv7_unknown_linux_gnueabihf=/sysroot-armhf

Now let’s try rebuilding:

$ docker build -t crossbuild:local .
$ cargo clean && cross build --target=armv7-unknown-linux-gnueabihf --features=dbus_mpris
# ...
The following warnings were emitted during compilation:

warning: build/expando.c:2:33: fatal error: openssl/opensslconf.h: No such file or directory
warning: compilation terminated.

error: failed to run custom build command for `openssl-sys v0.9.53`
#...

Ugh, ok. There’s a whole bunch of information printed in this output, including relevant environment variables⁷ and the command that failed:

"arm-linux-gnueabihf-gcc" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-march=armv7-a" "-I" "/sysroot-armhf/usr/include" "-Wall" "-Wextra" "-E" "build/expando.c"

So, let’s track down that missing header file:

$ docker run -it crossbuild:local bash
# in container...
$ find /sysroot-armhf -name 'opensslconf.h'
/sysroot-armhf/usr/include/arm-linux-gnueabihf/openssl/opensslconf.h

Alright. We can see that the build command was trying to import "openssl/opensslconf.h", and that that file wasn’t available on the import path. Note that the path /sysroot-armhf/usr/include is on the import path already (from the -I option in the command).⁸

Let’s start by just trying to add the other import path to the C compiler flags. Looking at the output of the failed build step indicates that the variable CFLAGS_armv7_unknown_linux_gnueabihf might be useable for this:

process didn't exit successfully: `/target/debug/build/openssl-sys-dae80554025aa995/build-script-main` (exit code: 101)
--- stdout
# ...
TARGET = Some("armv7-unknown-linux-gnueabihf")
HOST = Some("x86_64-unknown-linux-gnu")
CC_armv7-unknown-linux-gnueabihf = None
CC_armv7_unknown_linux_gnueabihf = Some("arm-linux-gnueabihf-gcc")
CFLAGS_armv7-unknown-linux-gnueabihf = None
CFLAGS_armv7_unknown_linux_gnueabihf = None
TARGET_CFLAGS = None
CFLAGS = None
# ...

So let’s add that to the end of the Dockerfile:

# ...
ENV CFLAGS_armv7_unknown_linux_gnueabihf='-I /sysroot-armhf/usr/include/arm-linux-gnueabihf'

… and try rebuilding again.

$ docker build -t crossbuild:local .
$ cargo clean
$ cross build --target=armv7-unknown-linux-gnueabihf --features=dbus_mpris

error: failed to run custom build command for `backtrace-sys v0.1.32`

Caused by:
  process didn't exit successfully: `/target/debug/build/backtrace-sys-cd1a565274849227/build-script-build` (exit code: 1)
  --- stdout
  # ...
  cargo:warning=In file included from /usr/arm-linux-gnueabihf/include/features.h:367:0,
  cargo:warning=                 from /usr/arm-linux-gnueabihf/include/errno.h:28,
  cargo:warning=                 from src/libbacktrace/alloc.c:35:
  cargo:warning=/sysroot-armhf/usr/include/arm-linux-gnueabihf/sys/cdefs.h:482:49: error: missing binary operator before token "("
  cargo:warning= #if __GNUC_PREREQ (4,8) || __glibc_clang_prereq (3,5)
  [1000 more lines of C errors]

Ok, it appears that we’ve run into something really, really nasty here in the build for backtrace-sys, a different crate. I’m not entirely sure what the cause is, but I can see that there’s includes from the cross-compile toolchain on the host system including things from the target distribution, and that feels like a bad idea. Especially because the C compiler on the host system in this case is older, and so if it’s pulling in header files for a newer compiler, then there’s a chance it would be running into incompatibilities.

Either way, we could try and fix this mess⁹, or we could try a different approach.

Hack it!

Our problem is:

• We need to keep GCC’s header files separate from the library headers, so we don’t accidentally end up including things across GCC versions
• We need to somehow ensure that openssl/opensslconf.h is available on the include path for our target system.

So… why don’t we just symlink it? Normally, this would violate assumptions that Debian makes to support multiple architectures, but because this is all happening in a distribution-in-a-subdirectory that we’re only using for our target architecture anyway, I think it’s ok to violate that assumption.

So, let’s add this to the Dockerfile:

RUN ln -s /sysroot-armhf/usr/include/arm-linux-gnueabihf/openssl/opensslconf.h \
          /sysroot-armhf/usr/include/openssl/

And when we build, we’ve moved on to linker errors again!!

Linker errors! Again!

This time, we’re getting:

error: linking with `arm-linux-gnueabihf-gcc` failed: exit code: 1
[enormous link command]

  = note: /usr/lib/gcc-cross/arm-linux-gnueabihf/5/../../../../arm-linux-gnueabihf/bin/ld: cannot find /lib/arm-linux-gnueabihf/libc.so.6
          /usr/lib/gcc-cross/arm-linux-gnueabihf/5/../../../../arm-linux-gnueabihf/bin/ld: cannot find /usr/lib/arm-linux-gnueabihf/libc_nonshared.a
          /usr/lib/gcc-cross/arm-linux-gnueabihf/5/../../../../arm-linux-gnueabihf/bin/ld: cannot find /lib/arm-linux-gnueabihf/ld-linux-armhf.so.3
          collect2: error: ld returned 1 exit status

Alright. That’s hella weird because all of these libraries are indeed on the system, just… on those paths within the target system, not within the host system:

$ docker run -it crossbuild:local find / -name libc.so.6
/usr/arm-linux-gnueabihf/lib/libc.so.6
/lib/x86_64-linux-gnu/libc.so.6
/sysroot-armhf/lib/arm-linux-gnueabihf/libc.so.6

Some quick googling reveals that this should be fixable by telling the gcc linker to look in a different system root. We can tell cargo to tell rustc to tell gcc to tell the linker (😅) to do that by setting the CARGO_TARGET_<triple>_RUSTFLAGS environment variable in the Dockerfile:

#...
ENV CARGO_TARGET_ARMV7_UNKNOWN_LINUX_GNUEABIHF_RUSTFLAGS="-C link-arg=--sysroot=/sysroot-armhf"

Linker errors! Again again!

It feels like maybe (maybe) we’re getting closer now, because after a rebuild, our linker errors have changed:

error: linking with `arm-linux-gnueabihf-gcc` failed: exit code: 1
[enormous link command]

= note:
/sysroot-armhf/usr/lib/arm-linux-gnueabihf/libdbus-1.a(libdbus_1_la-dbus-sysdeps-unix.o): In function `_dbus_listen_systemd_sockets':
(.text+0x1c78): undefined reference to `sd_listen_fds'
/sysroot-armhf/usr/lib/arm-linux-gnueabihf/libdbus-1.a(libdbus_1_la-dbus-sysdeps-unix.o): In function `_dbus_listen_systemd_sockets':
(.text+0x1cb0): undefined reference to `sd_is_socket'
/sysroot-armhf/usr/lib/arm-linux-gnueabihf/libdl.a(dlsym.o): In function `dlsym':
(.text+0xc): undefined reference to `__dlsym'
/sysroot-armhf/usr/lib/arm-linux-gnueabihf/libdl.a(dladdr.o): In function `dladdr':
(.text+0x0): undefined reference to `__dladdr'
/sysroot-armhf/usr/lib/arm-linux-gnueabihf/libpthread.a(nptl-init.o): In function `__pthread_initialize_minimal_internal':
/build/glibc-FUvrFr/glibc-2.28/nptl/nptl-init.c:434: undefined reference to `_dl_pagesize'
/build/glibc-FUvrFr/glibc-2.28/nptl/nptl-init.c:434: undefined reference to `_dl_init_static_tls'
/build/glibc-FUvrFr/glibc-2.28/nptl/nptl-init.c:434: undefined reference to `_dl_wait_lookup_done'
/sysroot-armhf/usr/lib/arm-linux-gnueabihf/libpthread.a(nptl-init.o): In function `__pthread_get_minstack':
/build/glibc-FUvrFr/glibc-2.28/nptl/nptl-init.c:443: undefined reference to `_dl_pagesize'
/sysroot-armhf/usr/lib/arm-linux-gnueabihf/libpthread.a(pthread_create.o): In function `__pthread_create_2_1':
/build/glibc-FUvrFr/glibc-2.28/nptl/pthread_create.c:697: undefined reference to `_dl_stack_flags'
/sysroot-armhf/usr/lib/arm-linux-gnueabihf/libpthread.a(pthread_getattr_np.o): In function `pthread_getattr_np':
/build/glibc-FUvrFr/glibc-2.28/nptl/pthread_getattr_np.c:210: undefined reference to `_dl_pagesize'
/sysroot-armhf/usr/lib/arm-linux-gnueabihf/libpthread.a(unwind.o): In function `unwind_stop':
/build/glibc-FUvrFr/glibc-2.28/nptl/unwind.c:72: undefined reference to `__pointer_chk_guard_local'
collect2: error: ld returned 1 exit status

Alright. This is also all very opaque to me, but here’s what I can figure out:

• Symbols prefixed with two underscores are usually internal symbols
• A lot of the symbols that aren’t being found are in static libraries (.a files).

I started googling these error messages, and in the end it was a StackOverflow post for the libpthread errors that gave the first clue:

In short, the problem appears to be due to trying to statically link libpthread against a dynamic libc. Dynamically linking libpthread should make the error go away, and I suspect that the error would also go away if both libpthread and libc were statically linked.

Yes! That seems like a plausible explanation. But, how do we convince the linker to use the dynamic version of the library? According to the ld man page:

On systems which support shared libraries, ld may also search for files other than libnamespec.a. Specifically, on ELF and SunOS systems, ld will search a directory for a library called libnamespec.so before searching for one called libnamespec.a. (By convention, a “.so” extension indicates a shared library.)

Ok, so if there’s a matching .so file on the search path, then it should get priority over the .a file. That’s definitely the case though? We can verify it with:

docker run -it crossbuild:local bash
# inside container
ls -l /sysroot-armhf/usr/lib/arm-linux-gnueabihf

… which clearly indicates that the .so files exist, but also, they’re all red…

… oh, because they point to files that don’t exist! The -dev packages that we installed through multistrap all ship absolute symlinks, which break if you install them anywhere other than / (the host system root).

It turns out there’s no one-liner to fix this, but there are lots of people in similar scenarios looking for similar things. I ended up adapting this shell script:

# Find all absolute symlinks recursively in the current directory
# (find command) and process them line by line (`while read`)
find . -lname '/*' | while read -r link; do
  # Get the (broken) path from the symlink.
  abs_target=$(readlink "$link")
  # Prepend that with the current working directory and overwrite
  # the old link with the prepended version.
  ln -fs "$(pwd)$abs_target" "$link"
done

Let’s save that in a file called relink.sh and then modify our container build to run it. This needs to run after multistrap – we need to fix what multistrap installed – but before that line we added to fix the pesky opensslconf.h header, because relink.sh just assumes that every absolute symlink is broken:

# ...
COPY multistrap-config /
RUN multistrap -a armhf -f multistrap-config /sysroot-armhf
# --- new ---
COPY relink.sh /
RUN cd /sysroot-armhf && /relink.sh
# --- end ---
RUN ln -s /sysroot-armhf/usr/include/arm-linux-gnueabihf/openssl/...

Now, let’s cross our fingers (🤞) and build again:

cross build --target=armv7-unknown-linux-gnueabihf --features=dbus_mpris

Ok, everything compiles! But if we’ve learned anything, it’s that we can’t guarantee that the binary is linked correctly without trying it out¹⁰, so let’s copy the binary to the Raspberry Pi and run it again:

pi@boombox:~ $ ./spotifyd --version
spotifyd 0.2.24

… and we’re done 🥳

Summing up

Well, what a process. Turns out cross-compiling is still hard! There’s lots of pitfalls to watch out for, and you still need to become relatively familiar with the rust compiler, the gcc toolchain, and linux to make it work. Hopefully these posts have given you some tools for diagnosing your own gnarly cross-compilation issues.

For reference, I’ve put the complete, tidied and commented Dockerfile and associated scripts into the crossbuild-spotifyd repo on GitHub. If you’ve got comments, commiserations, or answers to anything I left unsolved, I’d love to hear from you!