Easiest way to automate https requests

user_8792857 0
Viewed 110

I need to automate the submission of a lot of requests to a cloud-based database interface (Intelex). There isn't any way to submit certain actions in bulk, but all that is necessary to submit an individual request is for an authenticated user to attempt to open the web link. So, to restore a record with a given GUID all I need to do is open the page "https://.../restore/[GUID]". The webpage will load and display the now-restored record, but I don't actually need it to even finish loading - simply requesting the page is enough for the server to perform the action. I expect the server will either kick me out or drop some of the requests if I send them too quickly, but it has no issues with processing them as fast as I can navigate and open links in a new browser tab (about 1 per second).

What is the easiest way to submit these requests for a few thousand web addresses?

https automation
2 Answers

I can see couple of ways here. Each are pretty much easy.

  1. Store your variable part of URL in a file. Then use the tool like CURL and loop through file content calling CURL with the URL you have built on each iteration. If you're using Linux you can do something like Looping through the content of a file in Bash, for Windows check this post: How do you loop through each line in a text file using a windows batch file?

  2. Use JMeter tool (jMeter loop through all values in CSV). This will even aloww you to parallelize your queries.

user_83438430

Assume guid.txt contains:

[GUID]
[GUID]
[GUID]
:
[GUID]

Then you can run:

cat guid.txt | parallel --delay 1s wget https://.../restore/{}

To automatically find the optimal --delay use:

cat guid.txt |
  parallel --retries 5 --delay 1sauto wget https://.../restore/{}

(Requires GNU Parallel version 20210122).

user_3630280
Related