Python Request takes a long time to run, is my code written poorly or am I using the wrong library?

Viewed 32

I am using an API endpoint with a GET HTTP method, provided by an E-Mail service provider that gives back a list of all active subscribers (emails) on a given list within the account. Part of the issue is that the list of subscribers is paginated - meaning if you want the entire list of active subscribers, you have to alter your code to sift through pages. There are roughly 100k emails on this list with approximately 100 pages (1000 results per page). The code I wrote works fine and produces a list of just the 100k emails, however the code takes ~2 minutes to retrieve the entire list.

I plan on having this code run in an AWS Lambda function, and it could be triggered as many as 30 times a day. I'm concerned about the processing power of the lambda function and possible timeout issues. I'm terribly new to this (if not evident already) so I'm not really sure what I'm asking, but I'm wondering if this code is written so poorly that it's causing the issue, or if it's something else entirely. I tried utilizing the Request library's "Session" method, but perhaps there's something else better for this use-case.

url = 'API URL ENDPOINT'
basic = HTTPBasicAuth('API KEY', '')

r = requests.get(url, auth=basic)
jsonResponse = r.json()
pages = jsonResponse['NumberOfPages']

emails = []
page_number = 1
while page_number <= pages:
    payload = {'page': page_number}
    s = requests.Session()
    r = s.get(url, params=payload, auth=basic)
    jsonResponse = r.json()
    for result in jsonResponse['Results']:
        emails.append(result['EmailAddress'])
    page_number = page_number + 1
0 Answers
Related