How to handle all wget transport exceptions

nora 0
Viewed 12

I've a script that will download 200k websites index source web pages.

I used wget for that.

wget $i -T5 --tries=3 --no-check-certificate

I included --no-check-certificate, --tries=3 and -T5 to avoid my script being blocked on any http request which requires manual intervention which i don't want to do each time!

But sadly, my script got stuck on "Awaiting http response" many times for hours (this is the only message I remember when it's stuck).

Could anyone tell me what are the other flags that I should include to instruct my script to pass to the next website if something like that happened?

I'm sure there is more exceptions, so could you send me a modified version of my wget command which instructs my script to continue if it got stuck.

wget automation command-line linux
1 Answers

But, sadly my script got stuck on "Awaiting http response" many times for hours (this is the only message i remember when it's stuck).

For case which is not dealt with by -T5 setting of your script I suggest adding timeout for example

timeout 300 wget "https://www.example.com" -T5 --tries=3 --no-check-certificate

will terminate run if it takes longer than 300 seconds. Feel free to adjust that value, but remember that it take such number of seconds only for cases where wget is unable to download or ignore resource by itself.

Daweo0
Related