Entering input to a website and generating output for all files in folder using Google Chrome [MacOS; Terminal]

Viewed 33

As a minimum reproducible example, please navigate to: https://www.genome.jp/kegg/tool/map_pathway3.html

At that URL, enter in hsa00010 to the "Select KEGG pathway map: " field, then into the "Enter Data" field, cut-and-paste the following:

#hsa CLP/CMP    Blast_phase All
226 #FFFFFF #6495ED,#FF0000
229 #FFFFFF #6495ED,#FF0000
230 #FF0000 #6495ED,#FF0000
51071   #FFFFFF #6495ED,#FF0000
2203    #FFFFFF #6495ED,#FF0000
8789    #CBDBF9 #6495ED,#FF0000
2539    #FFFFFF #6495ED,#FF0000
132158  #FFFFFF #6495ED,#FF0000
2821    #FFFFFF #6495ED,#FF0000
9563    #FF6D6D #6495ED,#FF0000
414328  #FFFFFF #6495ED,#FF0000
5211    #97B8F3 #6495ED,#FF0000
5213    #E5EDFC #6495ED,#FF0000
5214    #97B8F3 #6495ED,#FF0000
5226    #FFFFFF #6495ED,#FF0000
25796   #FFFFFF #6495ED,#FF0000
5236    #E5EDFC #6495ED,#FF0000
55276   #6495ED #6495ED,#FF0000
5631    #FFFFFF #6495ED,#FF0000
221823  #FFFFFF #6495ED,#FF0000
5634    #FF6D6D #6495ED,#FF0000
64080   #FFFFFF #6495ED,#FF0000
9104    #FFFFFF #6495ED,#FF0000
6120    #FFFFFF #6495ED,#FF0000
729020  #FFFFFF #6495ED,#FF0000
22934   #FFFFFF #6495ED,#FF0000
6888    #B1CAF6 #6495ED,#FF0000
7086    #7DA6F0 #6495ED,#FF0000
8277    #E5EDFC #6495ED,#FF0000
84076   #E5EDFC #6495ED,#FF0000

Finally, select the "color specification" and "negative-zero-positive" options, and hit the 'exec' button.

Doing so should lead you to this URL: https://www.genome.jp/kegg-bin/show_pathway?162660760057905/hsa00030.1.args

With some of the boxes colored.

At that same URL, there is a "save image" button. I want to hit that button and save each image. My problem is, I would like to generate a few hundred files this way.

Now, I have all the pathway codes (e.g. hsa00030) stored as file names (e.g. hsa00030.txt) in a directory on my Mac. Each .txt file has input just like the 4 columns above.

So, I could open each file, type in the pathway code, then cut-and-paste the file contents and generate them one at a time. Because I have a few hundred files, doing this manually would be very time-consuming.

What I'd like to ask is, can this process be automated?

In the past, I have used Google Chrome to conduct simpler tasks of a related kind, but I am not sure how to proceed when I have a form to fill out, bubbles to tick, etc.

Thank you.

1 Answers

Okay here is what I came up with. Make sure you have wget installed. It takes in the HSA number as a first input and the color codes file as a second. It then generates a color codes string to build the cURL. The result of the cURL is a redirect to the correct URL. This then pulls the HTML and cuts out the temporary file that gets generated when you navigate to this page. It then uses wget to get the png.

After saving this to downloadhsa.sh file and then chmod +x, you would run with ./downloadhsa.sh hsa00010 colorcodes.txt

#!/bin/sh
HSA=$1
COLOR_CODES_FILE=$2

if [[ ${HSA} == "" ]]; then
  echo "No HSA number given"
  exit 1
fi

if [[ ${COLOR_CODES_FILE} == "" ]]; then
  echo "No color codes file given"
  exit 1
fi
COLOR_CODES=$''
i=0
while IFS=$'\n' read -r line; do
  ((i >= 1 )) && COLOR_CODES+=$'\r\n'$line
  ((i++))
done <${COLOR_CODES_FILE}

printf ${COLOR_CODES}
echo "Generating file for ${HSA}..."
BASE_URL="https://www.genome.jp"
IMAGE_URL=$(curl -L 'https://www.genome.jp/kegg-bin/mcolor_pathway' \
  -H 'Connection: keep-alive' \
  -H 'Cache-Control: max-age=0' \
  -H 'sec-ch-ua: " Not;A Brand";v="99", "Google Chrome";v="91", "Chromium";v="91"' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'Origin: https://www.genome.jp' \
  -H 'Upgrade-Insecure-Requests: 1' \
  -H 'DNT: 1' \
  -H 'Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryV72KHjGaYHrBOToa' \
  -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36' \
  -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9' \
  -H 'Sec-Fetch-Site: same-origin' \
  -H 'Sec-Fetch-Mode: navigate' \
  -H 'Sec-Fetch-User: ?1' \
  -H 'Sec-Fetch-Dest: document' \
  -H 'Referer: https://www.genome.jp/kegg/tool/map_pathway3.html' \
  -H 'Accept-Language: en-US,en;q=0.9' \
  --data-raw $'------WebKitFormBoundaryV72KHjGaYHrBOToa\r\nContent-Disposition: form-data; name="map"\r\n\r\n'"${HSA}"$'\r\n------WebKitFormBoundaryV72KHjGaYHrBOToa\r\nContent-Disposition: form-data; name="unclassified"\r\n\r\n#hsa CLP/CMP Blast_phase All\r\n'"${COLOR_CODES}"$'\r\n------WebKitFormBoundaryV72KHjGaYHrBOToa\r\nContent-Disposition: form-data; name="s_sample"\r\n\r\n\r\n------WebKitFormBoundaryV72KHjGaYHrBOToa\r\nContent-Disposition: form-data; name="mapping_list"; filename=""\r\nContent-Type: application/octet-stream\r\n\r\n\r\n------WebKitFormBoundaryV72KHjGaYHrBOToa\r\nContent-Disposition: form-data; name="mode"\r\n\r\ncolor\r\n------WebKitFormBoundaryV72KHjGaYHrBOToa\r\nContent-Disposition: form-data; name="numericalType"\r\n\r\nmm\r\n------WebKitFormBoundaryV72KHjGaYHrBOToa\r\nContent-Disposition: form-data; name="minColor"\r\n\r\n#ffff00\r\n------WebKitFormBoundaryV72KHjGaYHrBOToa\r\nContent-Disposition: form-data; name="maxColor"\r\n\r\n#ff0000\r\n------WebKitFormBoundaryV72KHjGaYHrBOToa\r\nContent-Disposition: form-data; name="negativeColor"\r\n\r\n#00ff00\r\n------WebKitFormBoundaryV72KHjGaYHrBOToa\r\nContent-Disposition: form-data; name="zeroColor"\r\n\r\n#ffff00\r\n------WebKitFormBoundaryV72KHjGaYHrBOToa\r\nContent-Disposition: form-data; name="positiveColor"\r\n\r\n#ff0000\r\n------WebKitFormBoundaryV72KHjGaYHrBOToa--\r\n' \
  --compressed | grep "id=\"pathwayimage\"" | cut -d " " -f 2 | cut -d "\"" -f 2)

echo "Downloading image ${HSA} from ${BASE_URL}${IMAGE_URL}..."
if [[ ${IMAGE_URL} == "" ]]; then
  echo "No URL returned for which to fech an image"
  exit 1
fi

wget ${BASE_URL}${IMAGE_URL}
Related