How to get whole URL of google search results instead of just website name through request and beautifulsoup in Python?

Viewed 31

Here's the code I'm using:

def get_rank(keyword,  country="in", language="en"):
    urls=[]
    r=requests.get(f"https://www.google.com/search?q={keyword}&gl={country}&hl={language}")
    soup=BeautifulSoup(r.text, 'lxml')
    urls=[re.search("(http[s]?://[a-z0-9-.]*)",ele.a['href']).group(1) for ele in soup.find_all('div', class_="Gx5Zad fP1Qef xpd EtOod pkphOe")]
    return urls

Output: https://apple.com

Required Output: https://apple.com/something/

0 Answers
Related