Is it possible to fetch programming language of file from GitHub?

user_6857902 0
Viewed 548

I am preparing Python script which will diff files from git commits log, and write difference into pretty printed html file.

I would like to color source code similarly to what is done on GitHub diff, and for that I would like to use pygments. There is only one problem, it will need information about language in which source code is written, thus I must fetch it somehow.

I was wondering if there is possibility to fetch it directly from local copy of repository (preferred) or GitHub itself. GitHub is using linguist so the work is already done why not use that? Moreover when we enter specific language link from below language bar information on GitHub UI interface you can clearly see all the files which were recognised as written in selected language.

I saw some information about .gitattributes file that you can set language for files in there, but I would need to set it manually, and after project update I would have to update that file with all added files.

github git
1 Answers

As soon I finished writting this question I found GitHub REST API. And even though I have not found function that could give me file language as is. I figured out that it is possible to combine two GitHub REST API methods:

I will describe it below based on python implementation.

  1. List all languages from project and store them into python dictionary
  2. Iterate through repository languages and perform find with language parameter in the query (as the parameter q is required I decided that character which should be in any file is "space" thus we provide %20 as string we are looking for)
  3. If total_count equals to 1 it means that we found our file with language provided we can return language name
import requests
import json


def check_file_lang(filename: str, repo: str):
    response = requests.get("https://api.github.com/repos/{}/languages".format(repo))
    lang = json.loads(response.text)
    for lang in lang.keys():
        response = requests.get("https://api.github.com/search/code?q=%20+filename:{}+repo:{}+language:{}".format(filename,
                                                                                                       repo,
                                                                                                       lang))
        search_result = json.loads(response.text)
        if search_result['total_count'] == 1:
            return lang

f = "Gruntfile.js"
r = "jquery/jquery"
file_lang = check_file_lang(f, r)
print(file_lang)

One problem with this is when repository have many language it is possible that this request could not be completed. As GitHub limits number of request per minute (unauthenticated==10 and authenticated==30) it is possible that to many request would be necessary to check all languages from repository. GitHub REST API Rate limit Moreover even if repository have only one language it still takes 2 requests per file, meaning that you can check at most accordingly 5 or 15 files per minute.

Following that I though that it would be better to return all files from repository based on language provided but GitHub limits queries GitHub REST API Timeouts and incomplete results meaning for big projects your data could be truncated thus you couldn't know all files from repository.

def list_all_files_by_language(repo: str):
    response = requests.get("https://api.github.com/repos/{}/languages".format(repo))
    lang = json.loads(response.text)
    ret_dict = {}
    for lang in lang.keys():
        response = requests.get("https://api.github.com/search/code?q=%20+repo:{}+language:{}".format(repo,
                                                                                                      lang))
        search_result = json.loads(response.text)
        if 'items' in search_result.keys():
            ret_dict[lang] = search_result['items']
    return ret_dict

Solution has its flaws but it works. If someone would found anything better I am eager to try it.

For now I accept this answare because it fits my needs, as GitHub Enterprise (which is my use case) does not have mentioned limitations.

user_68579020
Related