Webscraping - don't display the text part of the html code

Viewed 101

I'm having an issue when I try to webscrape a website using Selenium library through python. The point is that I want to get some info about songs that are collected into this site: https://bandcamp.com/?g=all&s=top&p=0&gn=0&f=all&w=0.

However, when I try to extract the text from the corresponding html code, the process return an empty list.

If I look at the html code from my browser (Chrome), I'll see the text part, but when I look at the same code in python, the text part does not appear.

Here is my code:

browser = webdriver.Chrome()
browser.get("https://bandcamp.com/?g=all&s=top&p=0&gn=0&f=all&w=0")

name_song = browser.find_elements_by_css_selector("a.item-title")
name_artist = browser.find_elements_by_css_selector("a.item-artist")

genre = browser.find_elements_by_class_name("item-genre")
print(name_song, name artist, genre)

When I print the three variables, I get the html code but there's nothing that I can extract from it. How can I solve this problem? Thanks a lot in advance for your help.

My goal is to get "Apocalypticists" and "Kriegsmachine" and "metal", each assigned to one different variable.

That's the webpage of the site and the corresponding html code

3 Answers

You were so close. You just need to induce WebDriverWait for the desired elements to be visible and store the WebElements in three different Lists and iterate over them to print the desired text and you can use the following solution:

  • Code Block:

    from selenium import webdriver
    from selenium.webdriver.common.by import By
    from selenium.webdriver.support.ui import WebDriverWait
    from selenium.webdriver.support import expected_conditions as EC
    
    options = webdriver.ChromeOptions() 
    options.add_argument("start-maximized")
    options.add_argument('disable-infobars')
    browser = webdriver.Chrome(chrome_options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
    browser.get("https://bandcamp.com/?g=all&s=top&p=0&gn=0&f=all&w=0")
    name_song = WebDriverWait(browser, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "a.item-title")))
    name_artist = WebDriverWait(browser, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR,"a.item-artist")))
    genre = WebDriverWait(browser, 20).until(EC.visibility_of_all_elements_located((By.XPATH,"//a[@class='item-artist']//following::span[1]")))
    for song, artist, gen in zip(name_song, name_artist, genre):
        print("%s song is by %s and is of %s genre" % (song.text, artist.text, gen.text))
    
  • Console Output:

    Apocalypticists song is by Kriegsmaschine and is of metal genre
    The Path song is by Carbon Based Lifeforms and is of ambient genre
    Christmas Time Is Here (N & S America Edition) song is by Khruangbin and is of funk genre
    Christmas Time Is Here (Excluding N & S America) song is by Khruangbin and is of funk genre
    Snailchan Adventure song is by Ujico*/Snail's House and is of electronic genre
    O God who avenges, shine forth. Rise up, Judge of the Earth; pay back to the proud what they deserve. song is by the body and is of metal genre
    T-Rex EP song is by Ben Prunty and is of soundtrack genre
    Woodland Womp (24bit 96kHz) song is by Kalya Scintilla and is of electronic genre
    

You just need to go into each element to get what you want. Your code above returns three list of selenium element onjects. Each object has attributes that you can access, and one of those attributes is .text

If I run the code above, I can then access name_song:

[<selenium.webdriver.remote.webelement.WebElement (session="83853054732fa0a5bfbc8a7e32a1e05b", element="0.4629143928625561-1")>,...

But, if I want to get only the text from those elements, I can call the text attribute from each:

[i.text for i in name_song]

['Apocalypticists',
 'The Path',
 'Christmas Time Is Here (N & S America Edition)',
 'Christmas Time Is Here (Excluding N & S America)',
 'Snailchan Adventure',
 'O God who avenges, shine forth. Rise up, Judge of the Earth; pay back to the proud what they deserve.',
 'T-Rex EP',
 'Woodland Womp (24bit 96kHz)']

Then indexing into that list:

[i.text for i in name_song]
'Apocalypticists'

Element object will not give the innerText value. You need to call element.text to get it. browser.find_elements_by_class_name("item-genre") is returning 23 element. the locator has to be changed as well to get the appropriate 8 element.

browser = webdriver.Chrome()
browser.get("https://bandcamp.com/?g=all&s=top&p=0&gn=0&f=all&w=0")

name_song = browser.find_elements_by_css_selector("a.item-title")
name_artist = browser.find_elements_by_css_selector("a.item-artist")
genre = browser.find_elements_by_css_selector("span.item-genre")

for i in range(len(name_song)-1):
  print(name_song[i].text)
  print(name_artist[i].text)
  print(genre[i].text)
Related