inconsistent behaviour grouped target in Make/Makefile with/without dependencies

user_16736384 0
Viewed 39

Edited the post to clarify my issue.

Short explanation: in directory data/raw there're 15 .csv files. You can see their names in RAW_NAMES.

print-% :
    @echo "$*=$($*)"

RAW = data/raw
PROCESS = data/process
RAW_NAMES = circuits constructor_results constructor_standings constructors driver_standings\
drivers lap_times pit_stops qualifying races results seasons sprint_results status
RAW_FILES = $(addsuffix .csv, $(addprefix $(RAW)/,$(RAW_NAMES)))

$(RAW_FILES) :  
    wget -nc --no-check-certificate -P $(RAW) http://ergast.com/downloads/f1db_csv.zip
    unzip -d $(RAW) $(RAW)/f1db_csv.zip

data/process/roundtimes.csv : code/create_roundtimes.R $(RAW_FILES)
    R -e "source('code/create_roundtimes.R')"

data/process/fastest_rounds.csv : code/fastest_rounds.R data/process/roundtimes.csv
    R -e "source('code/fastest_rounds.R')"

Let's say I remove 3 files (constructor_results.csv, constructor_standings.csv, constructors.csv) and I run the command:

make -n data/process/fastest_rounds.csv

wget -nc --no-check-certificate -P data/raw http://ergast.com/downloads/f1db_csv.zip
unzip -d data/raw data/raw/f1db_csv.zip "constructor_results.csv"
wget -nc --no-check-certificate -P data/raw http://ergast.com/downloads/f1db_csv.zip
unzip -d data/raw data/raw/f1db_csv.zip "constructor_standings.csv"
wget -nc --no-check-certificate -P data/raw http://ergast.com/downloads/f1db_csv.zip
unzip -d data/raw data/raw/f1db_csv.zip "constructors.csv"
R -e "source('code/create_roundtimes.R')"
R -e "source('code/fastest_rounds.R')"

Now, I try to make the following code work:

$(RAW)/f1db_csv.zip :
    wget -nc --no-check-certificate -P $(RAW) http://ergast.com/downloads/f1db_csv.zip

$(RAW_FILES) : $(RAW)/f1db_csv.zip
    unzip -d $(RAW) $(RAW)/f1db_csv.zip "$(@F)"

But now the output is this:

make -n data/process/fastest_rounds.csv

unzip -j data/raw/f1db_csv.zip -d data/raw "circuits.csv"
unzip -j data/raw/f1db_csv.zip -d data/raw "constructor_results.csv"
unzip -j data/raw/f1db_csv.zip -d data/raw "constructor_standings.csv"
unzip -j data/raw/f1db_csv.zip -d data/raw "constructors.csv"
unzip -j data/raw/f1db_csv.zip -d data/raw "driver_standings.csv"
unzip -j data/raw/f1db_csv.zip -d data/raw "drivers.csv"
unzip -j data/raw/f1db_csv.zip -d data/raw "lap_times.csv"
unzip -j data/raw/f1db_csv.zip -d data/raw "pit_stops.csv"
unzip -j data/raw/f1db_csv.zip -d data/raw "qualifying.csv"
unzip -j data/raw/f1db_csv.zip -d data/raw "races.csv"
unzip -j data/raw/f1db_csv.zip -d data/raw "results.csv"
unzip -j data/raw/f1db_csv.zip -d data/raw "seasons.csv"
unzip -j data/raw/f1db_csv.zip -d data/raw "sprint_results.csv"
unzip -j data/raw/f1db_csv.zip -d data/raw "status.csv"
R -e "source('code/create_roundtimes.R')"
R -e "source('code/fastest_rounds.R')"

or when I use:

$(RAW)/f1db_csv.zip :
    wget -nc --no-check-certificate -P $(RAW) http://ergast.com/downloads/f1db_csv.zip

$(RAW_FILES) &: $(RAW)/f1db_csv.zip
        unzip -j $(RAW)/f1db_csv.zip -d $(RAW) "$(@F)"

the output is: make -n data/process/fastest_rounds.csv

unzip -j data/raw/f1db_csv.zip -d data/raw "circuits.csv"
R -e "source('code/create_roundtimes.R')"
R -e "source('code/fastest_rounds.R')"

So, you see what I'm getting at? Whenever a rule is called upon because of a dependency changed, I want to extract the specific dependencies, that triggered the rule, out of the group. So if those 3 constructor files where removed I want to be able to single them out and do something with them.

makefile bash
1 Answers

Your last option using grouped targets is correct, but you can't use $@ because that expands only to the specific file that make is trying to build.

You should just use $(RAWFILES) if you want to pass all the targets to the command:

$(RAW)/f1db_csv.zip :
        wget -nc --no-check-certificate -P $(@D) http://ergast.com/downloads/$(@F)

$(RAW_FILES) &: $(RAW)/f1db_csv.zip
        unzip -o -d $(<D) -j $< $(RAWFILES)

ETA

The problem is that when unzip extracts a file, it also extracts the timestamp of the file. So, if the file had a timestamp of June 2020 when it was zipped up, when you extract it it also has a timestamp of June 2020. It doesn't have a timestamp of June 2022, when you extracted it.

Because make works completely on timestamps, the only way it knows whether an extracted file needs to be updated or not is by comparing it to the timestamp of the zip file: if the timestamp is newer then we know we extracted the file since the zip file was updated. If not, we assume that the zip file is newer and we need to re-extract the file.

You want to either add the -D option to your unzip command (see the man page), or else use the poor-man's method such as:

$(RAW_FILES) : $(RAW)/f1db_csv.zip
        unzip -d $(RAW) $(RAW)/f1db_csv.zip "$(@F)"
        touch $@

to update the timestamp of the file you just extracted.

Ways you can figure this out: you can check the timestamps on the files, since that's what make uses. You can use make -d to see why make decides to rebuild files.

user_9395570
Related