How to scrape text file from children of children sublinks with Beautifulsoup?

Bbrown44 0
Viewed 37

I am trying to scrape all the rap lyrics text documents and place them into an array from the following webpage (https://ohhla.com/all.html).

I am having a challenge figuring out how to write a script that will go to the lowest sublink and pull the text into an array.

My code:

import requests
from bs4 import BeautifulSoup

base_site = "https://ohhla.com/all.html"

response = requests.get(base_site)

relative_urls = [a['href'] for a in soup.select('a[href]') if soup.select('a[href]') != None]

relative_urls = relative_urls[31:]

from urllib.parse import urljoin

full_urls = [urljoin(base_site, url) for url in relative_urls

each URL in the full_urls has sublinks that have the text file that I want to pull.

Does anyone have ideas on how to best approach writing a list comprehension?

beautifulsoup python
1 Answers

First you need to separate anonymous and famous artists, parsing for them is different. I will show on the example of anonymous. But you will need to figure out how to store this information yourself. The easiest way seems to me to make a folder hierarchy.

response = requests.get('https://ohhla.com/all.html')
soup = BeautifulSoup(response.text, 'lxml')
links = ['https://ohhla.com/' + link.get('href') for link in soup.find('pre').find_all('a', {'href': True})]
anonymous_links = [link for link in links if 'anonymous' in link]
famous_links = [link for link in links if 'anonymous' not in link]

Output for famous:

['https://ohhla.com/YFA_natedogg.html', 'https://ohhla.com/YFA_2pac.html', 'https://ohhla.com/YFA_goodie.html', 'https://ohhla.com/YFA_liks.html', 'https://ohhla.com/YFA_asaprocky.html', 'https://ohhla.com/YFA_azealia.html', 'https://ohhla.com/YFA_eminem.html', 'https://ohhla.com/YFA_beastieboys.html', 'https://ohhla.com/YFA_otk.html', 'https://ohhla.com/YFA_kane.html', 'https://ohhla.com/YFA_goodie.html/', 'https://ohhla.com/YFA_bigkrit.html', 'https://ohhla.com/YFA_bigl.html', 'https://ohhla.com/YFA_littlebrother.html', 'https://ohhla.com/YFA_bigpun.html', 'https://ohhla.com/YFA_bigsean.html', 'https://ohhla.com/YFA_bizmark.html', 'https://ohhla.com/YFA_blackeyedpeas.html', 'https://ohhla.com/YFA_talib.html', 'https://ohhla.com/YFA_BOB.html', 'https://ohhla.com/YFA_icet.html', 'https://ohhla.com/YFA_krs.html', 'https://ohhla.com/YFA_brand.html', 'https://ohhla.com/YFA_cypress.html', 'https://ohhla.com/YFA_cypress.html#misc', 'https://ohhla.com/YFA_ugk.html', 'https://ohhla.com/YFA_getoboys.html', 'https://ohhla.com/YFA_busta.html', 'https://ohhla.com/YFA_camron.html', 'https://ohhla.com/YFA_can_i_bus.html', 'https://ohhla.com/YFA_cnn.html', 'https://ohhla.com/YFA_goodie.html', 'https://ohhla.com/YFA_j5.html', 'https://ohhla.com/YFA_chiefkeef.html', 'https://ohhla.com/YFA_childishgambino.html', 'https://ohhla.com/YFA_koolkeith_two.html', 'https://ohhla.com/YFA_E40.html', 'https://ohhla.com/YFA_common.html', 'https://ohhla.com/YFA_coflow.html', 'https://ohhla.com/YFA_breeze.html', 'https://ohhla.com/YFA_coup.html', 'https://ohhla.com/anonymoys/cymphoni/', 'https://ohhla.com/YFA_cypress.html', 'https://ohhla.com/YFA_eminem.html', 'https://ohhla.com/YFA_dabrat.html', 'https://ohhla.com/YFA_danjamowf.html', 'https://ohhla.com/YFA_danny.html', 'https://ohhla.com/YFA_dpg.html', 'https://ohhla.com/YFA_dls.html', 'https://ohhla.com/YFA_devinthedude.html', 'https://ohhla.com/YFA_camron.html', 'https://ohhla.com/YFA_ludacris.html', 'https://ohhla.com/YFA_willsmith.html', 'https://ohhla.com/YFA_djkhaled.html', 'https://ohhla.com/YFA_three6.html', 'https://ohhla.com/YFA_rundmc.html', 'https://ohhla.com/YFA_dmx.html', 'https://ohhla.com/YFA_dpg.html', 'https://ohhla.com/YFA_oddfuture.html', 'https://ohhla.com/YFA_mfdoom.html', 'https://ohhla.com/YFA_drake.html', 'https://ohhla.com/YFA_koolkeith_two.html', 'https://ohhla.com/YFA_drdre.html', 'https://ohhla.com/YFA_koolkeith_two.html', 'https://ohhla.com/YFA_df.html', 'https://ohhla.com/YFA_E40.html', 'https://ohhla.com/YFA_oddfuture.html', 'https://ohhla.com/YFA_coflow.html', 'https://ohhla.com/YFA_eminem.html', 'https://ohhla.com/YFA_epmd.html', 'https://ohhla.com/YFA_eve.html', 'https://ohhla.com/YFA_houseofpain.html']

Now you need function to get links to txt:

def get_tracks(url):
    response = requests.get(url)
    soup = BeautifulSoup(response.text, 'lxml')
    links = [url + link.get('href') for link in soup.find_all('a')][5:]
    artist_name = url[:-1].split('/')[-1]
    albums = {}
    for link in links:
        response = requests.get(link)
        soup = BeautifulSoup(response.text, 'lxml')
        album = link[:-1].split('/')[-1]
        tracks = [link + _link.get('href') for _link in soup.find_all('a')][5:]
        albums[album] = tracks
    return {artist_name: albums}

Now everything is ready to get all the albums\lyrics. I will display the first 100 artists as an example:

artists = []
for artist in anonymous_links[:100]:
    artists.append(get_tracks(artist))
print(artists)

OUTPUT is so long, just image: enter image description here

Now that you have links to texts, you can get them using:

BeautifulSoup(requests.get(url).text, 'lxml').find('pre').get_text()
userXeFr0
Related