How to skip progress when curl response size equals zero byte in shell script?

user_6719857 0
Viewed 135

I have created a script that uses curl to retrieve a CSV file from a public site and copy it to my server.

#!/bin/sh
INCOMING="http://www.example.jp/csv/ranking.csv"
OUTPUT="/var/www/html/csv/ranking.csv"

curl -s $INCOMING > $OUTPUT

My boss ordered that if the CSV file retrieved from the site is 0 bytes, it should not be overwritten.

I heard that, then I wrote a script that looks like this.

#!/bin/sh

INCOMING="http://www.example.jp/csv/ranking.csv"
OUTPUT="/var/www/html/csv/ranking.csv"

INCOMING_LENGTH=$(curl -L -s -o /dev/null -w '%{size_download}\n' $INCOMING)

if [ "$INCOMING_LENGTH" -ne "0" ]; then
  curl -s $INCOMING > $OUTPUT
fi

What I want to ask is, how to skip csv output progress when curl response size equals zero by a single curl command.

In the script above, I am running the curl command twice, so even if I check the file size in the first curl command and make sure it is not zero bytes, if the file size is zero in the second curl command, it will generate a zero-byte CSV file.

This should rarely happen, but my boss is a perfectionist, so I want to eliminate the element of poking around.

Thank you!

sh curl linux
1 Answers

I think the solution is the following...

Get the Content-Length size from your HEAD request using curl. This does not perform a download, only sends a request to get the Headers:

GET_HEADER_NODATA=`curl -s -L -I https://my-url/file.csv | grep Content-Length | awk '{print $2}' | tr -d $'\r' `

The above command extracts the size of the Content-Length and removes some characters at the end.

Now you compare if the csv file is empty you downloaded it

if [ $GET_HEADER_NODATA -eq 0 ]; then
    echo "do here what you want because the file size is 0"
fi

All this will work if your web server sends the Content-Length in his header.

Good luck!

user_39744130
Related