parameters for async GET requests with HTTP.jl

user_12671149 0
Viewed 166

I want to set up a function that sends async GET requests to a list of URLs. For an example, I want to open the google home page with the search bar containing a letter of the alphabet.

using HTTP

endpoint = "https://www.google.com/?q="
letters = vcat('a':'z')

function query_sync()    
    for i in letters
        HTTP.request("GET", "$endpoint/?letter=$i")
        println(i)
    end
end

@time query_sync()
# 3.537998 seconds (7.75 k allocations: 1.795 MiB)

function query_async()    
    @sync for i in letters
        @async begin
            HTTP.request("GET", "$endpoint/?letter=$i")
            println(i)
        end
    end
end

@time query_async()
# 0.399345 seconds (8.95 k allocations: 1.993 MiB, 1.17% compilation time)

So far so good. But if I have rate limits or want to set the number of concurrent connections, what options do I have?

I know that there are connection pool options such as connection_limit and pipeline_limit but it just makes me feel that I've implemented a hackey solution to an async call instead of using the package properly. Will those connection pool options take into account the @async macro, or is it part of the module's larger server functions, unrelated to my use case?

Some considerations I've had if the Connection Pool options aren't for my use case: using channels to limit the availability of URLs to the function, therefore limiting the concurrency. And good old fashion rate limiting by checking calls per unit of time and throttling.

julia
1 Answers

Use asynmap with the ntasks parameter:

asyncmap(letters; ntasks=8) do i
    HTTP.request("GET", "$endpoint/?letter=$i")
    println(i)
end

asyncmap "...uses multiple concurrent tasks to map f over a collection (or multiple equal length collections). For multiple collection arguments, f is applied elementwise. ntasks specifies the number of tasks to run concurrently. Depending on the length of the collections, if ntasks is unspecified, up to 100 tasks will be used for concurrent mapping..."

user_99577100
Related