Running builds (gcc, make, cmake) inside identical docker, from same source gives binaries with different hashes

user_9073206 0
Viewed 176

I run into "interesting" issue. I am doing build automation for our project in Bamboo. I was quite sure that I am done and then someone asked: why builds from Friday night and Saturday night (no code changes) are different? I have 2 remote builders. Both have same OS installed (Ubuntu) and updated at to the latest at the same time. Both has the same libraries installed. The compilation is inside docker (image imported from same source) So I started looking into this and got to the point where I can observer the following:

  • running compilation twice from same source on the same machine, starting new docker for each compilation produces identical (as per sha1sum) binaries.
  • running compilation twice from same source on two different machines, produces different binaries. Source is in the folder of the same name and it is mounted into docker on the path with same name as well.
  • running compilation on 1 PC with some HW spec, then taking the disk and connecting it to the different PC (with different HW spec) and running compilation from the same source again. Resulting binaries were identical.

Is it possible that somehow the code depends on exact OS image? or any other OS attribute? Any other idea?

It feels like chasing ghosts ...

EDIT: After trying to go step by step, I narrowed this down to cmake. That means: cmake made on two different machines differ (going to start 1-by-1 diffing now). If I put results of cmake of several machines and compile with make from there, I am always getting same binaries. So I believe that the problem is the way cmakefiles are written not compilation itself.

EDIT2: I now know that this is Qt 5.2.1 rcc being non-deterministic issue. During cmake rcc is run and among others calculates some hasheh. Difference in those hashes is what is causing the whole thing to non-deterministic. If I do cmake and take content of it to 3 different machines and run compilation (make) there I got 100% identical (as per sha1sum) results. I think that settles it here. Now I need either convince project to upgrade to newer version of Qt where I know how to make rcc deterministic or I need to find how to make rcc deterministic in 5.2.1.

cmake gcc makefile c c++
2 Answers

If you started with identical OS images, then your tests suggest a post-image configuration step introduced a change. Review that configuration. A trivial example would be the hostname. I would diff any intermediate build artifacts, for instance, config.h or whatever cmake writes its output. Try run strings on two binaries (that differ), then diff that output to see if you learn something.

https://tests.reproducible-builds.org/debian/index_issues.html might be very instructive, too.

Allan_Wind_97060

The results could be coincidence. Linking requires that all the required functions go into the executable, but the order is generally not constrained. This could very well depend on the order in which the object files appear on disk.

MSalters_154160
Related