Python web-scraping: problem with soup.select

ryantl 0
Viewed 47

I'm developing a python script to scrape data from a specific site: https://finance.yahoo.com/quote/AUDUSD%3DX/history?p=AUDUSD%3DX

I'm using BeautifulSoup. The interesting data on this page are : enter image description here

I'm using soup.select method this time, the class name is W(100%) M(0) and my code is as below:

import requests
import pandas as pd
from bs4 import BeautifulSoup

url = "https://finance.yahoo.com/quote/AUDUSD%3DX/history?p=AUDUSD%3DX"

soup = BeautifulSoup(requests.get(url).content, "html.parser")
table = soup.select(table:has(-soup-contains("W(100%) M(0)")))
print(table)

And this does not generate the result I want.

I have also tried this way:

import requests
import pandas as pd
from bs4 import BeautifulSoup

url = "https://finance.yahoo.com/quote/AUDUSD%3DX/history?p=AUDUSD%3DX"

soup = BeautifulSoup(requests.get(url).content, "html.parser")
table = soup.select("W(100%) M(0)")
print(table)

And there is error as shown below

Traceback (most recent call last):
  File "/Users/ryanngan/PycharmProjects/Webscraping/seek.py", line 8, in <module>
    table = soup.select("W(100%) M(0)")
  File "/Users/ryanngan/PycharmProjects/Webscraping/venv/lib/python3.9/site-packages/bs4/element.py", line 1973, in select
    results = soupsieve.select(selector, self, namespaces, limit, **kwargs)
  File "/Users/ryanngan/PycharmProjects/Webscraping/venv/lib/python3.9/site-packages/soupsieve/__init__.py", line 144, in select
    return compile(select, namespaces, flags, **kwargs).select(tag, limit)
  File "/Users/ryanngan/PycharmProjects/Webscraping/venv/lib/python3.9/site-packages/soupsieve/__init__.py", line 67, in compile
    return cp._cached_css_compile(pattern, ns, cs, flags)
  File "/Users/ryanngan/PycharmProjects/Webscraping/venv/lib/python3.9/site-packages/soupsieve/css_parser.py", line 218, in _cached_css_compile
    CSSParser(
  File "/Users/ryanngan/PycharmProjects/Webscraping/venv/lib/python3.9/site-packages/soupsieve/css_parser.py", line 1159, in process_selectors
    return self.parse_selectors(self.selector_iter(self.pattern), index, flags)
  File "/Users/ryanngan/PycharmProjects/Webscraping/venv/lib/python3.9/site-packages/soupsieve/css_parser.py", line 985, in parse_selectors
    key, m = next(iselector)
  File "/Users/ryanngan/PycharmProjects/Webscraping/venv/lib/python3.9/site-packages/soupsieve/css_parser.py", line 1152, in selector_iter
    raise SelectorSyntaxError(msg, self.pattern, index)
soupsieve.util.SelectorSyntaxError: Invalid character '(' position 1
  line 1:
W(100%) M(0)

How can I scrape the above data using the soup.select method? Thank you very much.

beautifulsoup web-scraping
1 Answers

Using direct class selectors (e.g. .W(100%)) breaks because it's invalid CSS selector syntax.

However, you can get around this using contains syntax which is expressed through attribute*=partial:

import requests
from bs4 import BeautifulSoup

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
}
response = requests.get(
    "https://finance.yahoo.com/quote/AUDUSD%3DX/history?p=AUDUSD%3DX",
    headers=headers
)
# select any element where class contains "W(100%)" and class contains "M(0)":
soup = BeautifulSoup(response.text)
table = soup.select('[class*="W(100%)"][class*="M(0)"]')
Granitosaurus0
Related