How to get more than 1000 search results with API Github

Viewed 1831

I want to list the most star Github repo that were created in the last 30 days, but to display more than 1000 search, I have this error message

{ "message": "Only the first 1000 search results are available", "documentation_url": "https://developer.github.com/v3/search/" }

do you have a solution for this problem?

thank you in advance

3 Answers

zifan is right. You can create a query per day for the last 30 days; or two queries per day (one each 12 hours); and so forth. The lower the interval, the more the query calls. At the same time, the more the repositories you catch.

Below an example in Python. It runs a curl call, so you can easily translate it to different languages.

import requests
from datetime import datetime, timedelta

URL = f'https://api.github.com/search/repositories?q=is:public created:SINCE..UNTIL'
HEADERS = {'Authorization': 'token <PASTE_HERE_GITHUB_ACCESS_TOKEN>'}

since = datetime.today() - timedelta(days=30)  # Since 30 days ago
until = since + timedelta(days=1)   # Until 29 days ago 

while until < datetime.today():
    day_url = URL.replace('SINCE', since.strftime('%Y-%m-%dT%H:%M:%SZ')).replace('UNTIL', until.strftime('%Y-%m-%dT%H:%M:%SZ'))
    r = requests.get(day_url, headers=HEADERS)
    print(f'Repositories created between {since} and {until}: {r.json().get("total_count")}')

    # Update dates for the next search
    since = until
    until = since + timedelta(days=1)

Of course, the number of repositories might still be too large. In that case, try

  1. to use pagination;
  2. to reduce the interval SINCE..UNTIL, as well as the timedelta;
  3. to add further filters in the query, for example: exclude archived and forked repositories, get repositories with a minimum number of stars only, and so forth.

Take a look here for an example. Here is a Python tool to collect repositories from Github: https://github.com/radon-h2020/radon-repositories-collector

Run into the same issue. I think the only way work around is to split your query into small query which returns less than 1k result. (refer to github search limit results)

Related