Google Scraping Results Difference - Google Custom Search API vs Google Selenium Search

user_11250327 0
Viewed 161

I wrote two scraping codes for Google search results.

Google web-site chrome search (selenium) / Google custom API search (free, 100 quires limit)

When two codes are executed, the conditions are the same but the results are different(Maybe I am wrong?)

Google Web site selenium "url"

GoogleUrl = f'https://www.google.com/search?q={SearchKeyword}+intext:{MUSTKeyword}&hl=en&rlz=&start={pageNum}'

This is the Google API parameter that I use.

response = service.cse().list(
        q=searchterm_in,
        exactTerms=incl_searchterm_in,
        siteSearch=sitesearch_in,
        cx=my_cse_id,
        num=10,
        start=index,
    ).execute()

I thought "search keywords" = "q", "MUSTKeyword" = "exact terms".Google API doesn't seem to have "intext:" parameters in the official document.(https://developers.google.com/custom-search/v1/reference/rest/v1/cse/list), So I tried the "exactTerms" but I was able to get the results more accurately with the Selenium Google search results than the API.

Question: How do I set or add parameters in the Google API to import the same results into Google Selenium Search? What replaces "intext:" in Google API search parameters?

google-custom-search web-scraping python
1 Answers

That search modifier (intext:) just needs to be part of the text of the search. Here's getting allintitle results from the same API:

k_search = 'allintitle:"' + k + '"'
search_results = google_search(k_search)
search_totalresults = search_results['searchInformation']['totalResults']

def google_search(search_text):
    try:
        service = build("customsearch", "v1", developerKey=my_api_key)
        res = service.cse().list(q=search_text, cx=my_cse_id).execute()
        return res
    except Exception as e:
        strtext = 'Exception Occured: ' + str(e)
        CustomLog(strtext, "ERROR")
user_151956630
Related