Unable to parse some information from a webpage using search keywords

user_7180194 0
Viewed 98

I've created a script to parse some information related to some songs from a website. When I try with this link or this one, I get my scrpt working flawlessly. What I could understand is that when I append my search keyword after this portion https://www.billboard.com/music/, I get the desired page having information.

However, things go wrong when I try with these keywords 1 Of The Girls or Al B. Sure! or Ashford & Simpson and so on.

I can't figure out how to append the above keywords after the base link https://www.billboard.com/music/ to locate the pages with information.

Script I've tried with:

import requests
from bs4 import BeautifulSoup

LINK = "https://www.billboard.com/music/Adele"

res = requests.get(LINK)
soup = BeautifulSoup(res.text,"lxml")
scores = [item.text for item in soup.select("[class$='-history__stats'] > p > span")]
print(scores)

Result I'm getting (as expected):

['4 No. 1 Hits', '6 Top 10 Hits', '13 Songs']

Result located in that page is just after the chart history like the following:

enter image description here

How can I fetch some information from a webpage using critical search keywords?

web-scraping python-3.x python
1 Answers

I don't know all use cases but the obvious pattern I have seen for cases mentioned is that special characters are stripped (without leaving whitespace in their place) out, words are lower-case and then spaces replaced with "-". The tricky bit may be the definition and handling of special characters.

e.g.

https://www.billboard.com/music/ashford-simpson

https://www.billboard.com/music/al-b-sure

https://www.billboard.com/music/1-of-the-girls

You could start with writing something to perform those string manipulations and then test the response code. Perhaps see if there is any form of validation in js files.

EDIT:

Multiple blanks between words becomes single blank before being replaced with "-" ?

Answer developed with @Mithu for preparing search terms:

import re
keywords = ["Y?N-Vee","Ashford & Simpson","Al B. Sure!","1 Of The Girls"]
spec_char = ["!","#","$","%","&","'","(",")","*","+",",",".","/",":",";","<","=",">","?","@","[","]","^","_","`","{","|","}","~",'"',"\\"]

for elem in keywords:
    refined_keywords = re.sub('-+','-' , ''.join(i.replace(" ","-") for i in elem.lower() if i not in spec_char))
    print(refined_keywords)
user_62412350
Related