How to handle parallel make invocations with multiple ordered goals

user_1773798 0
Viewed 1127

GNU make allows 1) parallel execution and 2) specifying several goals in the same invocation:

make -j4 clean all

But, as GNU make parallelizes the goals, some race conditions can occur. Illustration:

$ cat Makefile
clean:
    @sleep 1 && rm -f foo

all: foo
    @sleep 2 && cat foo

foo:
    @echo '$@' > $@
$ make -j4 clean ; make -j4 all
foo
$ make -j4 clean all
cat: foo: No such file or directory
Makefile:5: recipe for target 'all' failed
make: *** [all] Error 1

Is there a nice way to force an order between the goals, but still benefit from the parallel acceleration for each goal? In the above example it would be nice to wait until clean completes before all starts in order to avoid race conditions.

As shown, the separate make invocations work as expected but this is not 100% satisfactory:

  1. Some goals can be invoked simultaneously, some others cannot. Completely forbidding multiple goals can thus be considered as too restrictive. But identifying all valid and invalid combinations is tricky and error prone.
  2. To completely avoid the problem, one could warn all potential users of such a Makefile that multiple goals invocations are not supported in parallel mode, but this warning will inevitably be overlooked by some users.
  3. Race conditions do not always cause errors. Some could apparently work seamlessly but produce erroneous results.
gnu-make makefile
2 Answers

IMHO this seems to root in the problem that in a programming language (in this case the shell) we are able to formulate dependencies which are of fundamentally different nature than the ones that make can handle. In your example there is a dependency of clean on the non-existence of foo, while all has the inverse dependency. If you make both targets active at the same time, this seems to surpass make's theoretical foundation - I don't know if there exists a sensible theory that can handle such relations. All that I could come up with is the explicit formulation:

.PHONY: all clean

clean:
    @sleep 1
    rm -f foo

all: foo $(filter clean,$(MAKECMDGOALS))
    @sleep 2
    cat foo

foo: $(filter clean,$(MAKECMDGOALS))
    @echo Creating $@
    @echo '$@' > $@

I think this is an interesting problem for sure.

user_18194520

I found a workaround (but I am not 100% convinced that it is the best solution and that it has no hidden drawbacks). The idea is to use the MAKECMDGOALS GNU make variable and the conditionals to force the serialization of multiple goals:

ifeq ($(words $(MAKECMDGOALS)),1)
.PHONY: all clean

clean:
    @sleep 1 && rm -f foo

all: foo
    @sleep 2 && cat foo

foo:
    @echo '$@' > $@
else
.NOTPARALLEL:

%:
    @$(MAKE) $@
endif

Of course, the condition of the conditional could be more sophisticated, like, for instance, testing if one of the goals matches clean...

user_17737980
Related