make the bash script to be faster

bintaaaaang 0
Viewed 63

I have a fairly large list of websites in "file.txt" and wanted to check if the words "Hello World!" in the site in the list using looping and curl.

i.e in "file.txt" :

blabla.com
blabla2.com
blabla3.com

then my code :

#!/bin/bash
put() {
printf "list : "
read list
run=$(cat $list)
}
put
scan_list() {
for run in $(cat $list);do
if [[ $(curl -skL ${run}) =~ "Hello World!" ]];then
printf "${run} Hello World! \n"
else
printf "${run} No Hello:( \n"
fi
done
}
scan_list

this takes a lot of time, is there a way to make the checking process faster?

bash
1 Answers

Use xargs:

% tr '\12' '\0' < file.txt | \
    xargs -0 -r -n 1 -t -P 3 sh -c '
    if curl -skL "$1" | grep -q "Hello World!"; then
      echo "$1 Hello World!"
      exit
    fi
    echo "$1 No Hello:("
  ' _
  • Use tr to convert returns in the file.txt to nulls (\0).
  • Pass through xargs with -0 option to parse by nulls.
  • The -r option prevents the command from being ran if the input is empty. This is only available on Linux, so for macOS or *BSD you will need to check that file.txt is not empty before running.
  • The -n 1 permits only one file per execution.
  • The -t option is debugging, it prints the command before it is ran.
  • We allow 3 simultaneous commands in parallel with the -P 3 option.
  • Using sh -c with a single quoted multi-line command, we substitute $1 for the entries from the file.
  • The _ fills in the $0 argument, so our entries are $1.
James Risner0
Related