I cross compile with debian x86_64, and target both aarch64 and armv6l by using --sysroot to point to a complete filesystem (I.E /cache/arm64 or /cache/armel) for the target architectures. I use qemu-debootstrap to create /cache/arm64 and /cache/armel debian systems, include the necessary *-dev packages on those systems, and then include those libraries for the compiler by simply declaring the entire cached system the sysroot (--sysroot=/cache/arm64). This works cleanly for gcc-8, but I'm having a problem cross-compiling with clang-7 using address-sanitizer.
For example, with aarch64 the linker complains:
/usr/bin/aarch64-linux-gnu-ld: cannot find /usr/lib/llvm-7/lib/clang/7.0.1/lib/linux/libclang_rt.asan-aarch64.a: No such file or directory
libclang_rt.asan-aarch64.a comes from libclang-common-7-dev:
dpkg -S /usr/lib/llvm-7/lib/clang/7.0.1/lib/linux/libclang_rt.asan-aarch64.a
libclang-common-7-dev: /usr/lib/llvm-7/lib/clang/7.0.1/lib/linux/libclang_rt.asan-aarch64.a
This package and file is installed on the arm64 system. Using clang -v I can see the linker is trying to use --whole-archive /usr/lib/llvm-7/lib/clang/7.0.1/lib/linux/libclang_rt.asan-aarch64.a but I need it to use --whole-archive /cache/arm64/usr/lib/llvm-7/lib/clang/7.0.1/lib/linux/libclang_rt.asan-aarch64.a. The --sysroot option is changing all my -L options in the linker, but not this --whole-archive option.
I can manually change the linker command to complete compilation. However, I have further problems.
It seems not to use /cache/arm64/lib/aarch64-linux-gnu/, but instead uses /lib/aarch64-linux-gnu/ on the host system. If I rsync all the /cache/arm64/lib/aarch64-linux-gnu/ files to /lib/aarch64-linux-gnu/ on the host, and also manually change the linker command, then it "succeeds", except the binary still fails to run when I chroot into the target system:
# chroot /cache/arm64 /bin/bash
# ./binary
==13339==Sanitizer CHECK failed: /build/llvm-toolchain-7-whi4O5/llvm-toolchain-7-7.0.1/projects/compiler-rt/lib/sanitizer_common/sanitizer_procmaps_common.cc:83 ((data_.proc_self_maps.mmaped_size)) > ((0)) (0, 0)
My motivation for this --sysroot approach is merely that it's simple. It avoids having to dpkg --add-architecture arm64 and load the libraries onto the host. Further, in the case of libclang_rt.asan-aarch64.a in libclang-common-7-dev, I couldn't load libclang-common-7-dev:arm64 if I wanted to; It conflicts with the clang package:
apt install libclang-common-7-dev:arm64
Reading package lists... Done
Building dependency tree... Done
The following additional packages will be installed:
gcc-8-base:arm64 libbsd0:arm64 libbsd0 libc6:arm64 libedit2:arm64 libffi6:arm64 libgcc1:arm64
libidn2-0:arm64 libllvm7:arm64 libstdc++6:arm64 libtinfo6:arm64 libunistring2:arm64 zlib1g:arm64
Suggested packages:
glibc-doc:arm64 locales:arm64
The following packages will be REMOVED:
clang-7 libclang-common-7-dev
The following NEW packages will be installed:
gcc-8-base:arm64 libbsd0:arm64 libc6:arm64 libclang-common-7-dev:arm64 libedit2:arm64 libffi6:arm64
libgcc1:arm64 libidn2-0:arm64 libllvm7:arm64 libstdc++6:arm64 libtinfo6:arm64 libunistring2:arm64
zlib1g:arm64
My clang-7 options: -v -fPIC -O1 -target aarch64-linux-gnu --sysroot=/cache/arm64 -fsanitize=address -fno-omit-frame-pointer -fno-optimize-sibling-calls'
Also of note: I compile source to assembly, assembly to object, then object to binary. I perform these steps to retain the intermediate files mostly out of curiosity. When I do this NOT as a cross-compilation, I have no issues. So, I shouldn't think it matters, but I'll mention it just in case.