Get scons to distinguish an empty and a non-existing source while rebuilding

user_3052438 0
Viewed 204

While building my program it is important to distinguish between files that don't exists and files that are empty. However, it appears that scons treats them the same and neglect to rebuild a target when a source file changed from one of these states to the other one.

Step by step example:

Step 0:

SConstruct

foo = Command('foo', [], 'echo $TARGET is not created here!')
bar = Command('bar', foo, 'touch $TARGET ; test -f $SOURCE')
Default(bar)

Result:

scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
echo foo is not created here!
foo is not created here!
touch bar ; test -f foo
scons: *** [bar] Error 1
scons: building terminated because of errors.

My interpretation:

The command for foo fails to create the file but it doesn't raise and error so the command for bar is run. It checks if foo exists and returns an error. Build fails (everything as expected so far).

Step 1:

SConstruct:

foo = Command('foo', [], 'touch $TARGET')
bar = Command('bar', foo, 'touch $TARGET ; test -f $SOURCE')
Default(bar)

Result:

scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
touch foo
touch bar ; test -f foo
scons: done building targets.

My interpretation:

foo is rebuilt because it has changed. This time it creates an empty file. bar is rebuild because it failed before. It succeeds this time. The build is successful (still as expected).

Step 2:

SConstruct

foo = Command('foo', [], 'echo $TARGET is not created here!')
bar = Command('bar', foo, 'touch $TARGET ; test -f $SOURCE')
Default(bar)

Result:

scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
echo foo is not created here!
foo is not created here!
scons: `bar' is up to date.
scons: done building targets.

My interpretation:

foo is rebuilt because it has changed again (was restores to the previous version). The file foo doesn't exist any longer because scons removes files before building them and the command fails to recreate it. bar is not rebuilt because scons doesn't seem to detect a change in the source file. The build is successful while it shouldn't!

How can I force scons to rebuild bar in the last step?

The solution should scale well to "foo" commands that produce many files, list of which is generated programmatically in SConscript and cannot be hard-coded.

scons rebuild
3 Answers

The default way SCons determines is file has hanged is to compare MD5 signatures of the previous and the current versions of the file. The signature for a non-existent file is calculated from 0 bytes of data just like for an empty file so SCons doesn't see a difference between them. This is normally OK, especially that not creating the target file isn't an entirely legitimate use of SCons. However, we can make it work by supplying different function that decides if files are different.

Such function in SCons is called a Decider. There are three of them provided out of the box. The default one uses MD5. The second one uses timestamp. The third one uses MD5 but only if the timestamp is different.

In this case, timestamp could perhaps work because it is 0 for a non-existent file. However, it would generate false-positives when timestamp changes and the contents of the file do not.

Instead, we can supply our own decider which will do exactly what we want it to:

from os import path

env = DefaultEnvironment()
decider_env = env.Clone()

def decide_if_changed(dependency, target, prev_ni, repo_node=None):
    csig = dependency.get_csig() # it has to be called every time or the value won't be in `prev_ni` for the next check
    return not path.isfile(dependency.abspath) or not hasattr(prev_ni, 'csig') or prev_ni.csig != csig
decider_env.Decider(decide_if_changed)

foo = env.Command('foo', [], 'echo $TARGET is not created here!')
bar = decider_env.Command('bar', foo, 'touch $TARGET ; test -f $SOURCE')
Default(bar)

This custom decider is similar to the default implementation based on MD5 except it also reports a change if the file doesn't exist. This should cover the problem described in the question.

The new decider is assigned to a clone of the default environment. This way we have control over which target uses it. In this case only bar uses the non-default decider.

user_30524380

By the way, scons does now distinguish between emtpy and nonexistent, that's a fairly recent change (commit 3b7f8b4ce0, github.com/SCons/scons/pull/3833).

user_50035700

If you're saying "how can I force SCons to do xyz?", then you're understanding of SCons is incomplete.

SCons will only build targets which are out of date.

Unless..

You use AlwaysBuild(target) see: https://scons.org/doc/production/HTML/scons-man.html#f-AlwaysBuild

It also seems like you never want foo to be removed before it's (re)built?

Then you should use Precious(target) see: https://scons.org/doc/production/HTML/scons-man.html#f-Precious

Also.. it's bad form to call a builder with an empty source.

How would SCons ever know if it's out of date?

For your example what causes foo to be (re)built?

user_826810
Related