[Github]Is there any limit with downloading hundreds of raw files from public repo via python requests?

user_17145190 0
Viewed 264

Scenario: the users of my app can retrieve the list of (up to 1000) files in a directory of a public repo, and iterate through each of them to download the raw version (which is generally 500KB or less).

Is there any sort of limit with this workflow? I'm using the following python code:

url = 'https://api.github.com/repos/'+owner+'/'+repo+'/contents/Scenes?ref=main'
r = requests.get(url)

for i in r.json():
    raw_url = i['download_url']
    f = urllib.request.urlopen(raw_url)
    code = f.read()
    ### do my operations with the file 
    f.close()
github download python
1 Answers

There is limit on how many API hits you make in a given time.. For github the limit for basic auth API hits is 5,000 requests per hour.

Good idea would be to recognize that and put a time.sleep(1) between requests, which will offcourse take time to execute but will ensure you do not end up crossing the hourly limit

More details

import time
url = 'https://api.github.com/repos/'+owner+'/'+repo+'/contents/Scenes?ref=main'
r = requests.get(url)

for i in r.json():
    raw_url = i['download_url']
    f = urllib.request.urlopen(raw_url)
    code = f.read()
    ### do my operations with the file 
    f.close()
    time.sleep(1)
user_41789130
Related