Webscraping does not find all classes in Python

user_11331843 0
Viewed 260

I am trying to extract user information and date from particular website using bs4 Python, but my code is not finding all the classes from the website.

code is as follows

url = "https://www.expeditionforum.com/threads/distance-indication-feature.34452/"
page = requests.get(url)
soup = BeautifulSoup(page.text, 'html.parser')
title = soup.find('h1')
date=soup.findAll('a',attrs={"class":"datePermalink"})
name=soup.findAll('a',attrs={"class":"username"})

It is able to detect h1 tag but not others. Can you please suggest what am I doing wrong? Thank you in advance

web-scraping beautifulsoup python
1 Answers
from bs4 import BeautifulSoup
import requests

r = requests.get(
    "https://www.expeditionforum.com/threads/distance-indication-feature.34452/")

soup = BeautifulSoup(r.content, 'html.parser')

table = soup.findAll("div", class_="messageUserInfo")
dates = soup.findAll("a", class_="datePermalink")

for item1, item2 in zip(table, dates):
    print("UserName: {:<15}, Date: {}".format(
        item1.a.img.get('alt'), item2.text))

Output:

UserName: jgudnason      , Date: Jan 19, 2018 at 11:54 AM
UserName: ExpeditionAndy , Date: Jan 19, 2018 at 7:29 PM
UserName: zarga          , Date: Jan 19, 2018 at 10:34 PM
UserName: ExpeditionAndy , Date: Jan 19, 2018 at 11:01 PM
UserName: dlcorbett      , Date: Jan 20, 2018 at 7:06 AM
UserName: 17LimitedExpy  , Date: Jan 20, 2018 at 7:12 AM
UserName: AmpForE        , Date: Jan 21, 2018 at 3:07 AM
UserName: dlcorbett      , Date: Jan 21, 2018 at 3:40 AM
UserName: zarga          , Date: Jan 21, 2018 at 12:13 PM
UserName: jgudnason      , Date: Jan 21, 2018 at 12:29 PM
user_76589850
Related