Why does the same executable use different RUNPATHs for different library lookups?

user_8304467 0
Viewed 287

I have a Linux C++ program that builds & links (with -Wl,--no-undefined, no less) but fails to load all of its libraries when I try to run it. I've been trying to inspect its dependencies with ldd, but I can't figure out why it's tripping over the library that's failing:

$ ldd standalone_test
   ...  lots of libraries that are found ...
libPythia6.so => not found

Now, this library is in a place where it ought to be found according to the RUNPATH embedded in the executable:

$ readelf -d ./standalone_test | grep "RUNPATH"
 0x000000000000001d (RUNPATH)            Library runpath: [/home/jeremy/code/NOvARwgt/cmake-build-debug/src:/home/jeremy/code/genie-2.12.2/lib:/opt/genie/GENIESupport/pythia6/v6_424/lib:/opt/root/lib:/cvmfs/nova-development.opensciencegrid.org/novasoft/releases/development/lib/Linux2.6-GCC-debug]
$ ls /opt/genie/GENIESupport/pythia6/v6_424/lib
liblund.a  libPythia6.so  pydata.o

(and both the executable standalone_test and the library libPythia6.so are ELF64, so I don't think it's a processor architecture mismatch or something like that.)

I've tried digging further into what's going on during library loading, and I've noticed that different RUNPATHs are reported for the loading of different libraries under the same executable (some of these are 'chain' loads where they're requested by a library that was requested by the executable):

$ LD_DEBUG=libs,files ./standalone_test 2>&1 | ack RUNPATH
  # this one is the 'correct' RUNPATH (the one from the executable)
 19553:  search path=/opt/root/lib/tls/haswell/x86_64:/opt/root/lib/tls/haswell:/opt/root/lib/tls/x86_64:/opt/root/lib/tls:/opt/root/lib/haswell/x86_64:/opt/root/lib/haswell:/opt/root/lib/x86_64:/opt/root/lib        (RUNPATH from file ./standalone_test)
 # next one has been 'augmented' with processor architecture subdirs
 19553:  search path=/home/jeremy/code/NOvARwgt/cmake-build-debug/src/tls/haswell/x86_64:/home/jeremy/code/NOvARwgt/cmake-build-debug/src/tls/haswell:/home/jeremy/code/NOvARwgt/cmake-build-debug/src/tls/x86_64:/home/jeremy/code/NOvARwgt/cmake-build-debug/src/tls:/home/jeremy/code/NOvARwgt/cmake-build-debug/src/haswell/x86_64:/home/jeremy/code/NOvARwgt/cmake-build-debug/src/haswell:/home/jeremy/code/NOvARwgt/cmake-build-debug/src/x86_64:/home/jeremy/code/NOvARwgt/cmake-build-debug/src:/home/jeremy/code/genie-2.12.2/lib/tls/haswell/x86_64:/home/jeremy/code/genie-2.12.2/lib/tls/haswell:/home/jeremy/code/genie-2.12.2/lib/tls/x86_64:/home/jeremy/code/genie-2.12.2/lib/tls:/home/jeremy/code/genie-2.12.2/lib/haswell/x86_64:/home/jeremy/code/genie-2.12.2/lib/haswell:/home/jeremy/code/genie-2.12.2/lib/x86_64:/home/jeremy/code/genie-2.12.2/lib:/opt/genie/GENIESupport/pythia6/v6_424/lib/tls/haswell/x86_64:/opt/genie/GENIESupport/pythia6/v6_424/lib/tls/haswell:/opt/genie/GENIESupport/pythia6/v6_424/lib/tls/x86_64:/opt/genie/GENIESupport/pythia6/v6_424/lib/tls:/opt/genie/GENIESupport/pythia6/v6_424/lib/haswell/x86_64:/opt/genie/GENIESupport/pythia6/v6_424/lib/haswell:/opt/genie/GENIESupport/pythia6/v6_424/lib/x86_64:/opt/genie/GENIESupport/pythia6/v6_424/lib:/opt/root/lib:/cvmfs/nova-development.opensciencegrid.org/novasoft/releases/development/lib/Linux2.6-GCC-debug/tls/haswell/x86_64:/cvmfs/nova-development.opensciencegrid.org/novasoft/releases/development/lib/Linux2.6-GCC-debug/tls/haswell:/cvmfs/nova-development.opensciencegrid.org/novasoft/releases/development/lib/Linux2.6-GCC-debug/tls/x86_64:/cvmfs/nova-development.opensciencegrid.org/novasoft/releases/development/lib/Linux2.6-GCC-debug/tls:/cvmfs/nova-development.opensciencegrid.org/novasoft/releases/development/lib/Linux2.6-GCC-debug/haswell/x86_64:/cvmfs/nova-development.opensciencegrid.org/novasoft/releases/development/lib/Linux2.6-GCC-debug/haswell:/cvmfs/nova-development.opensciencegrid.org/novasoft/releases/development/lib/Linux2.6-GCC-debug/x86_64:/cvmfs/nova-development.opensciencegrid.org/novasoft/releases/development/lib/Linux2.6-GCC-debug      (RUNPATH from file ./standalone_test)
    ... more variants of these two ...
 # this one contains only a single entry from the real runpath??
 19553:  search path=/opt/root/lib      (RUNPATH from file ./standalone_test)

In the last case, the one where the library isn't located and the executable fails with a cannot open shared object file: No such file or directory error, it moves on to looking in the system search path as well.

My question is: how are these variants on RUNPATH concocted? I gather that they're modified by the various libraries that are requesting them somehow, but I can't seem to find any explanations out there on the googlewebs. (Maybe I'm searching for the wrong thing?) It seems like if I understood what was going on here, I'd be able to work backwards to understanding why the library isn't being found in the first place.

shared-libraries linux
1 Answers

My question is: how are these variants on RUNPATH concocted?

Unlike older RPATH, the RUNPATH applies only when searching for direct dependencies of the binary.

That is, if a.out has RUNPATH of /foo, and NEEDED of libfoo.so (located in /foo), then libfoo.so will be found. But if libfoo.so itself depends on libbar.so (also located in /foo), and if libfoo.so does not have RUNPATH, then libbar.so will not be found.

This behavior promotes "every ELF binary should be self-sufficient". In the case above, libfoo.so is not self-sufficient (needs libbar.so but doesn't say where to find it).

If you use RPATH instead, the path there would apply to every search, and libbar.so will be found. You can achieve this with -Wl,--disable-new-dtags when linking a.out.

Employed_Russian_506170
Related