How to list out all the h2, h3, and p tags then create a dataframe to store them

Admin 1
Viewed 188

I had given a website to scrape all of the key items

But the output I got is only for one item using BeautifulSoup4. So wonder if I need to use anything like soup.findall to extract all the key items in a list from the website.

from urllib.request import urlopen
from bs4 import BeautifulSoup
import re

url=
html = urlopen(url)
soup = BeautifulSoup(html, 'html.parser')

column= soup.find(class_ = re.compile('columns is-multiline'))
print(column.prettify())

position = column.h2.text
company = column.h3.text
city_state= column.find_all('p')[-2].text
print (position, company, city_state)

Thank you.

web-scraping beautifulsoup url python
2 Answers

You need to use find_all to get all the elements like so. find only gets the first element.

titles = soup.find_all('h2', class_='title is-5')
companies = soup.find_all('h3', class_='subtitle is-6 company')
locations = soup.find_all('p', class_='location')
# loop over locations and extract the city and state
for location in locations:
   city = location.split(', ')[0]
   state = location.split(', ')[1]
user_46041210

Try this:

from urllib.request import urlopen
from bs4 import BeautifulSoup

url = 'https://realpython.github.io/fake-jobs/'
html = urlopen(url)
soup = BeautifulSoup(html, 'html.parser')

positions = [pos.text for pos in soup.find_all('h2')]

companies = [com.text for com in soup.find_all('h3')]

city_state0 = []
city_state1 = []
for p in soup.find_all('p', {'class' : 'location'}):
    city_state0.append(p.text.split(',')[0].strip())
    city_state1.append(p.text.split(',')[1].strip())

    
df = pd.DataFrame({
    'city_state1': city_state0,
    'city_state2': city_state1,
    'companies' : companies,
    'positions' : positions
})
print(df)

Output:

enter image description here

user_17405770
Related