makefile rules being applied when they shouldn't

Question

I have a makefile (pasted below) for some C++ code, which has the annoying problem that when I do make clean the dependency files are compiled (before being removed again), which makes make clean too slow. The dependency rules are

$(DEPENDDIR)%.d: %.cpp
    @mkdir -p $(DEPENDDIR)
    $(CXX) -M -MG -MT $(OBJECTDIR)$*.o $(CXXFLAGS) $< > $@

Can anyone see what is the problem?

I tried moving the dependencies to the part where the objects are compiled, ie:

$(OBJECTDIR)%.o: %.cpp
    @mkdir -p $(OBJECTDIR) # $(dir $@)
    @echo " "
    $(CXX) -M -MG -MT $@ $(CXXFLAGS) $< \
        -MF $(patsubst $(OBJECTDIR)%.o, $(DEPENDDIR)%.d, $@)

but then the header file version.hpp wasn't created (and compilation failed)

KERNEL := $(shell uname -s)

PROGNAME=nextsim.exec

CXX = g++

# setting the C++ standard according to the gcc compiler version (from gcc-5.2, the default is C++14)
ifeq ($(shell echo `$(CXX) -dumpversion | cut -f1-2 -d.` \>= 5.2 | sed -e 's/\.//g' | bc),1)
    CXXFLAGS += -std=c++14
else
    CXXFLAGS += -std=c++11
endif

# add g++ option flags
CXXFLAGS += -ftemplate-depth-256 -Wno-inline \
        -fPIC -fopenmp \
        -DHAVE_CONFIG_H -D_MULTITHREADING_
ifdef NEXTSIM_COMPILE_VERBOSE
    CXXFLAGS += -v
endif

ifdef USE_OASIS
    CXXFLAGS += -DOASIS
    CXXFLAGS += -I $(NEXTSIMDIR)/modules/oasis/include
    LDFLAGS += -lgfortran
    LDFLAGS += -L $(NEXTSIMDIR)/lib -loasis
    CHAN = MPI1
    #LIBPSMILE = $(OASIS_DIR)/lib/libpsmile.${CHAN}.a $(OASIS_DIR)/lib/libmct.a $(OASIS_DIR)/lib/libmpeu.a $(OASIS_DIR)/lib/libscrip.a
endif

ifneq (,$(strip $(filter DEBUG Debug debug PROFILE Profile profile,$(NEXTSIM_BUILD_TYPE))))
    #ifeq ($(NEXTSIM_BUILD_TYPE),$(filter $(NEXTSIM_BUILD_TYPE),Debug debug))
    CXXFLAGS := $(filter-out -O3 -pthread,$(CXXFLAGS))
    CXXFLAGS += -g -O0 -DNDEBUG
ifneq (,$(strip $(filter PROFILE Profile profile,$(NEXTSIM_BUILD_TYPE))))
    CXXFLAGS += -DWITHGPERFTOOLS
endif
ifneq ($(KERNEL),Linux)
    CXXFLAGS += -Wl,-no_pie
endif
else
    CXXFLAGS += -O3 -pthread
endif

# add include paths
CXXFLAGS += -I $(NEXTSIMDIR)/core/include
CXXFLAGS += -isystem $(NEXTSIMDIR)/contrib/bamg/include # suppress annoying compilation warnings from -I
CXXFLAGS += -isystem $(NEXTSIMDIR)/contrib/mapx/include # suppress annoying compilation warnings from -I
# CXXFLAGS += -I $(NEXTSIMDIR)/contrib/interp/include

ifdef USE_ENSEMBLE
    CXXFLAGS += -DENSEMBLE
    CXXFLAGS += -I $(NEXTSIMDIR)/modules/enkf/perturbation/include
endif

ifdef USE_AEROBULK
        CXXFLAGS += -I $(AEROBULK_DIR)/include
        CXXFLAGS += -DAEROBULK
endif

# openmpi
CXXFLAGS += -I $(OPENMPI_INCLUDE_DIR)/

# petsc
CXXFLAGS += -I $(PETSC_DIR)/include

# boost
CXXFLAGS += -I $(BOOST_INCDIR)/ -I .

# netcdf
CXXFLAGS += -I $(NETCDF_DIR)/include

# gmsh
CXXFLAGS += -I $(GMSH_DIR)/include/gmsh

CXXFLAGS += -I /opt/local/include

ifeq ($(KERNEL),Linux)
    #CXXFLAGS += -std=c++0x -std=c++11 -pedantic -ftemplate-depth-256 -Wno-inline -fPIC -g -lm -pthread -v #-MMD -MP -lm -pthread -v

else

    ifeq ($(CXX),clang)
    CXXFLAGS += -stdlib=libc++
    endif

    CXXFLAGS += -I /usr/local/include #-I /opt/local/include/openmpi-mp

    #LDFLAGS += -Wl,-rpath,/usr/local/lib #-Wl,-rpath,/opt/local/lib/openmpi-mp
    #LDFLAGS += -L /usr/local/lib #-L /opt/local/lib/openmpi-mp -lmpi_cxx -lmpi -ldl -lstdc++ -lpthread

    ifeq ($(CXX),clang)
    LDFLAGS += -stdlib=libc++
    endif

endif

LDFLAGS += -L /usr/local/lib

LDFLAGS += -Wl,-rpath,$(OPENMPI_LIB_DIR)/
ifndef MACHINE_HEXAGON
    LDFLAGS += -L $(OPENMPI_LIB_DIR)/ -lmpi_cxx -lmpi -ldl -lstdc++ #-lpthread
else
    LDFLAGS += -L $(OPENMPI_LIB_DIR)/ -lmpichcxx -lmpich -ldl -lstdc++ #-lpthread #-lssl -luuid -lpthread -lrt
        LDFLAGS += -Wl,-rpath,$(BLAS_LAPACK_DIR)/lib
        LDFLAGS += -L $(BLAS_LAPACK_DIR)/lib -lsci_gnu_mp
endif

LDFLAGS += -Wl,-rpath,$(NETCDF_DIR)/lib -L $(NETCDF_DIR)/lib -lnetcdf_c++4

LDFLAGS += -Wl,-rpath,$(BOOST_LIBDIR)
LDFLAGS += -L $(BOOST_LIBDIR) -lboost_program_options -lboost_filesystem -lboost_system -lboost_serialization -lboost_mpi -lboost_date_time

LDFLAGS += -Wl,-rpath,$(PETSC_DIR)/lib
LDFLAGS += -L $(PETSC_DIR)/lib -lpetsc

LDFLAGS += -Wl,-rpath,$(NEXTSIMDIR)/lib
LDFLAGS += -L $(NEXTSIMDIR)/lib -lbamg
#LDFLAGS += -L $(NEXTSIMDIR)/lib -linterp
LDFLAGS += -L $(NEXTSIMDIR)/lib -lmapx
#LDFLAGS += -L $(NEXTSIMDIR)/lib -loasis

ifdef USE_ENSEMBLE
    LDFLAGS += -L $(NEXTSIMDIR)/lib -lpseudo2D
    LDFLAGS += -lgfortran
endif

ifdef USE_AEROBULK
        LDFLAGS += -L $(AEROBULK_DIR)/lib -laerobulk_cxx -laerobulk
        LDFLAGS += -lgfortran
endif

ifneq (,$(strip $(filter DEBUG Debug debug PROFILE Profile profile,$(NEXTSIM_BUILD_TYPE))))
#ifeq ($(NEXTSIM_BUILD_TYPE),$(filter $(NEXTSIM_BUILD_TYPE),Debug debug))
    LDFLAGS += -Wl,-rpath,/opt/local/lib
ifneq (,$(strip $(filter PROFILE Profile profile,$(NEXTSIM_BUILD_TYPE))))
    LDFLAGS += -L /opt/local/lib -lprofiler
endif
endif

LDFLAGS += -L $(NEXTSIMDIR)/lib -lnextsim

OBJECTDIR=$(NEXTSIMDIR)/objs/
DEPENDDIR=$(NEXTSIMDIR)/.deps/
BINARYDIR=bin/

# C++ files
CXXSRCDIR=.
CXXHDRDIR=.
CXXSRC=$(wildcard $(CXXSRCDIR)/*.cpp)
# We must exclude the version.hpp file from the list of header files because otherwise we get a circular dependency
CXXHDR=$(filter-out $(CXXHDRDIR)/version.hpp, $(wildcard $(CXXHDRDIR)/*.hpp))

OBJS=$(CXXSRC:%.cpp=$(OBJECTDIR)%.o)
DEPS=$(CXXSRC:%.cpp=$(DEPENDDIR)%.d)

# Rules to always execute.
.PHONY: exec clean mrproper all cleanall mrproperall

# Default action.
exec: $(PROGNAME)

# Create a header file with the git version
version.hpp: version.sh $(CXXSRC) $(CXXHDR)
    $(SHELL) -x $<

# Delete the object files.
clean:
    @echo " "
    $(RM) $(OBJS) $(DEPS)
    @echo " "

mrproper: clean
    $(RM) $(BINARYDIR)$(PROGNAME)
    @echo " "

# Rule for making the actual target
lines="=========="
Lines=$(lines)$(lines)$(lines)$(lines)$(lines)$(lines)$(lines)$(lines)
$(PROGNAME): $(OBJS) #$(CCOBJS)
    @mkdir -p $(BINARYDIR)
    @echo " "
    @echo $(Lines)$(Lines)
    @echo "Creating executable: $(BINARYDIR)$(PROGNAME)"
    @echo $(Lines)$(Lines)
    @echo " "
    $(CXX) $(CXXFLAGS) -o $(BINARYDIR)$@ $^ $(LDFLAGS)
    @echo " "
    @echo $(Lines)$(Lines)
    @echo "Created executable: $(BINARYDIR)$(PROGNAME)"
    @echo $(Lines)$(Lines)
    @echo " "

# Rules for object files from cpp files
$(OBJECTDIR)%.o: %.cpp
    @mkdir -p $(OBJECTDIR) # $(dir $@)
    @echo " "
    $(CXX) -o $@ -c $< $(CXXFLAGS)

# Make dependancy rules
$(DEPENDDIR)%.d: %.cpp
    @mkdir -p $(DEPENDDIR)
    $(CXX) -M -MG -MT $(OBJECTDIR)$*.o $(CXXFLAGS) $< > $@

# The compilation depends on this Makefile.
$(OBJS): Makefile

# Make everything
all:
    cd ..; $(MAKE) all


# Clean everything
cleanall:
    cd ..; $(MAKE) clean

# Properly clean everything
mrproperall:
    cd ..; $(MAKE) mrproper

# Properly clean & recompile
fresh:
    cd ..; $(MAKE) fresh

-include $(DEPS)

user_8710344 · Answer

It's being built because you're using -include $(DEPS).

See Including Other Makefiles

If an included makefile cannot be found in any of these directories, a warning message is generated, but it is not an immediately fatal error; processing of the makefile containing the include continues. Once it has finished reading makefiles, make will try to remake any that are out of date or don’t exist. See How Makefiles Are Remade. Only after it has tried to find a way to remake a makefile and failed, will make diagnose the missing makefile as a fatal error.

It then goes on to say:

If you want make to simply ignore a makefile which does not exist or cannot be remade, with no error message, use the -include directive instead of include, like this: -include filenames…

This acts like include in every way except that there is no error (not even a warning) if any of the filenames (or any prerequisites of any of the filenames) do not exist or cannot be remade.

Admittedly, this is can likely be interpreted in different ways. I just verified that it does indeed try to remake files on a -include line that have an explicit rule to be remade:

all:
        @echo building $@

foo.d :
        @echo building $@
        @touch $@

-include foo.d bar.d

Which gives:

 tmp> make all
 building foo.d
 building all

So, this leave an interesting case of how to solve your issue. Having a separate rule for the dependency files does increase your compile time -- you're now parsing each source file twice -- once to generate the .d and once to compile. This is not good. I believe that combining the rule for dependency and .o is in your best interest. Then make the subset of $(OBJS) that rely on version.hpp explicitly dependent on it, and you should be good.

user_4850040 · Answer

You ought to make the -include $(DEPS) line conditional - exclude it from the makefile when MAKECMDGOALS contains only the "clean" targets:

ifneq(,$(filter-out clean distclean clobber,$(MAKECMDGOALS)))
-include $(DEPS)
endif

As an aside, beware of assuming that cd will succeed - use && after it - or, if invoking Make in a different directory, just use its -C option.

makefile rules being applied when they shouldn't

2 Answers