I have created an API endpoint on AWS Lambda which I invoke using a POST request (with JSON data input) to get the result as a response (as a JSON data output).
Now I have 1 million data.tables that I need to process using the API. One API execution roughly takes 600ms and AWS Lambda allows upto 3000 requests per second. For faster execution I want to use all 64 cores in my system to generate 64 concurrent requests but I don't want these requests to wait till they get a response (the result) before they generate the next 64 requests and so on (Basically I want to hit the 3000 request threshold). For this I need to generate requests asynchronously with each request not waiting for it's response and once the response is generated to append it to a large list or data.table
I've looked into the curl, Rcurl, future and doFuture package documentations but couldn't find anything that solves this problem. Any help would by appreciated
I found a similar question on this post but the answer wasn't complete.
Example pseudo code to be run inside an asynchronous foreach (or similarly functioning function):
output = foreach(i = 1:n) %dopar%
{
x = input[i]
body = toJSON(x)
url = "https://exampleURL.amazonaws.com/dev/LambdaTest"
response = as.data.table(fromJSON(content(POST(url,
body = body,
content_type_json()))))
return(response)
}
Here data is the data.table containing n subsets that need to be passed individually.