make wget overwrite a file if already exist and download it everytime regardless it being changed on remote server

user_8863970 0
Viewed 1908

When I do wget twice, it does not overwrite the file but instead appends a .1 to the name.

$ wget https://cdn.sstatic.net/askubuntu/img/logo.png
...
Saving to: ‘logo.png’
...

$ wget https://cdn.sstatic.net/askubuntu/img/logo.png
...
Saving to: ‘logo.png.1’
...

I want wget to overwrite the logo.png file:

  • Regardless if it exists already
  • Regardless if it isn't changed on remote server, want it to be re-downloaded again (can't use -N flag)
  • Regardless if file size is same, it is downloaded everytime with wget and replaced.
  • Does not append .1 at the end of name. Keeps the same name evertime (logo.png)
  • Can't use cURL or something else.
  • Preferably not using the -O flag as i want to keep the original name of the file.

Is there still a way to do it? I searched but couldn't find an example?

wget
3 Answers

There is a way to do this with wget options but it's a bit of a hack:

wget --page-requisites --no-host-directories --cut-dirs=1000 URL

Explanation:

  • --page-requisites forces a download, clobbering existing files, but creates a tree hierarchy
  • --no-host-directories prevents wget from creating a top dir named after the host in the URL
  • --cut-dirs=1000 cuts the 1000 first directory components, effectively putting the downloaded file in the current directory

Another less hacky solution is to create a bash function for this:

wget_clobber() {
    local url=${1:?First parameter is a URL}
    wget --output-document="${url##*/}" "$url"
}

Explanation: we just use the --output-document (or -O) to force wget to write to the file named after the last part of the URL (${url##*/} is equivalent to $(basename "$url") and you can use the latter as well).

user_72334230

Download in a new temporary directory and then move the files to target directory.

DIR=$(mktemp --directory);
wget --directory-prefix="$DIR" "$URL";
mv "$DIR/"* .
user_19215460

wget doesn't let you overwrite an existing file unless you explicitly name the output file on the command line with option -O.

wget --backups=1 https://cdn.sstatic.net/askubuntu/img/logo.png

This is not what you requested but might work. Renames original file with .1 suffix and writes new file

user_113703470
Related