AttributeError: 'Response' object has no attribute 'txt' - Python Web Scraping

user_13817882 0
Viewed 2807

I was developing a new project to kinda move away from the most basic stuff I could do and I decided to look into web scraping. My idea was to use SteamStatus to check Steam's current status and have my script print it. For the first one, I was going with the Steam Store's status, and I've wrote the following code:

import requests
import bs4

res = requests.get('https://www.steamstatus.io/')
res.raise_for_status

SteamStatus = bs4.BeautifulSoup(res.txt, 'html.parser')
type(SteamStatus)

storeStatus = SteamStatus.select('#statustables > div.statustable.left > div > div:nth-child(1) > div.statusrow_status.store-status')
print(str(storeStatus))

With this, I'm getting the following error:

Traceback (most recent call last):
  File "C:/Users/a864/PycharmProjects/automation/steam status/webpage.py", line 8, in <module>
    SteamStatus = bs4.BeautifulSoup(res.txt, 'html.parser')
AttributeError: 'Response' object has no attribute 'txt'

From what I've searched and found, this would be a problem of outdated versions of the requests module but I've already made sure I have the latest version (2.24.0)

python-requests web-scraping beautifulsoup python
2 Answers

Welcome to SO!

As specified in the previous answers, the error relates to the use of the wrong attribute .txt - in spite of the correct .text.

As a final note, the page you are trying to scrape is loaded with javascript, hence requests is not the package you are looking for. See below for a rough solution using selenium webdriver

from selenium import webdriver
from bs4 import BeautifulSoup

driver = webdriver.Firefox() # initialize the driver

driver.get('https://www.steamstatus.io/') # go to the page

source = driver.page_source # extract the source

SteamPage = BeautifulSoup(source, 'html.parser')

SteamStatus = SteamPage.findAll('div', {'class' : 'statusrow'})
for s in SteamStatus:
    print(s.findNext('div', {'class' : 'statusrow_name'}).text) # print the row name
    print(s.findNext('div', {'class' : 'statusrow_status'}).text) # and the uploaded value
user_51401990

As the exception is telling you, you are attempting to refer to a nonexistent attribute. The Response exposes a .text attribute, not a .txt attribute.

user_52493070
Related