Submakes run the same rule multiple times

user_9936880 0
Viewed 260

I have some software built using parallel multi-level makefiles and I see that when my main Makefile runs two separate targets from a submakefile that have the same dependency, this dependency is run twice simultaneously and an error is created. Consider the following main Makefile in the project root folder:

TARGETS = t1 t2 t3 t4 t5 t6 t7 t8

.PHONY: all $(TARGETS) clean

all: $(TARGETS)

$(TARGETS): 
    @echo Making $@
    @sleep 1
    $(MAKE) -C folder s$@

clean:
    @echo Making $@
    $(MAKE) -C folder clean

and the sub-makefile folder/Makefile:

SUBTARGETS = st1 st2 st3 st4 st5 st6 st7 st8

$(SUBTARGETS): dep
    @echo Making $@
    @sleep 1
    @touch $@

dep:
    @echo Making $@
    @sleep 1
    @echo bla >> dep

clean:
    rm -f $(SUBTARGETS)
    rm -f dep
    rm -f dep2dump

Then running make -j8 in the root folder will run targets t1...t8 in parallel, which will then run subtargets st1...st8, which all depend on dependency dep. From the shell output and the contents of the dep file (8 lines) it is obvious that the dep rule is run 8 times, as if the 8 implications of folder/Makefile are completely independent. I thought submakes coordinated when running in parallel and that they would avoid running the same target twice, but it seems this is not the case. Can anyone suggest a correct way to solve such a case? If eventually this is an unavoidable weakness of make, what alternative build tools should I look into? Thanks

EDIT: The answers by MadScientist and Renaud Pacalet are useful but don't exactly solve my problem because they both require that the author of the top-level makefile has knowledge about the internals of the sub-makefile. I have not explained this requirement explicitly in my original post though.

So to give more details, the use case I am trying to solve is that where the source code in path folder/ is a separate project, eg. a collection of utilities st1...st8 where all (or some) of them have a dependency on library dep, internal to the utilities project in folder. Then I want to be able to use this sub-project (as seamlessly as possible) in various master projects, each of them using just a (possible different) subset of the utilities st1...st8. Additionally, the master project may contain many targets t1...t8, each depending on a different subset of st1...st8, as shown in my example above. Targets t1...t8 need to be able to run separately, building only the required dependencies from subproject (so make t1 only builds st1, etc), thus having to build all st1...st8 for each one of t1...t8 is not desired. On the other hand they also need to be able to run in parallel, eg. by running make all.

Ideally I would not want the author of each master makefile to have to know about internals of sub-project, nor have to include in the sub-makefile all the possible combinations of st1...st8 so that each master project can call just ONE of these to avoid the parallel build issue.

So far I have in mind but not tested the following imperfect solutions:

  1. As Renaud suggested, use something like flock to at least ensure that the multiple runs of dep (by separate sub-make instances) won't happen simultaneously. Cons: requires extra tool (flock or similar) to be installed + dep runs multiple times, so extra work is needed to avoid doing the actual compilation over and over again, otherwise just eat the performance cost.
  2. Include the sub-makefile in the master makefile so that everything runs in one make instance. This requires makes the sub-makefile able to work regardless of the path of the master makefile that includes it. No big issue. Cons: merging / including two makefile from different authors can open a can of worms, i.e. variables with same name, etc.
  3. Modify sub-makefile as described in (2) + In the main project create another makefile, eg. utils.make, that contains a rule for the targets of sub-makefile needed and includes the sub-makefile. So utils.make will be (assuming this master project only needs st1, st5 and st7:
utils: st1 st5 st7
include foldes/Makefile

Then the master makefile will have a utils-ext rule as dependency of each of t1...t8 that will be:

utils-ext:
$(MAKE) -f rules.make utils

to build all the utils needed. This keeps the two main makefiles separate but has all utils / subtargets built when building any single one of t1...t8, which is suboptimal.

gnu-make makefile
2 Answers

You could try to move the dep dependency to your top Makefile:

.PHONY: all $(TARGETS) clean dep

all: $(TARGETS)

$(TARGETS): dep
    @echo Making $@
    @sleep 1
    $(MAKE) -C folder s$@

dep:
    $(MAKE) -C folder s@
user_17737980

The only decent solution to your problem is to have ONE instance of make build all the sub-directory targets you want. Having the parent make invoke multiple sub-makes in parallel in the same directory, unless every invocation uses a completely disjoint set of targets, is a guaranteed fail situation. So if you have multiple things you want to do in the submake you should collect them all in one invocation of the sub-make and let the sub-make's parallelism handle it for you.

You could do something like this:

TARGETS = t1 t2 t3 t4 t5 t6 t7 t8

.PHONY: all $(TARGETS) clean

all: $(TARGETS)

$(TARGETS): .submake ;

.submake:
        $(MAKE) -C folder $(addprefix s,$(MAKECMDGOALS))

Then in the sub-make add this so that when invoked with no arguments it builds everything:

all: $(SUBTARGETS)

Here, if you run make then the sub-make is invoked with no arguments and builds all the things in parallel. If you invoke make t1 t2 then the submake is invoked with the arguments st1 st2.

Alternatively, you can re-architect your makefiles so that you don't use recursive make at all, and one instance of make knows all the different rules and dependency relationships.

user_9395570
Related