BeautifulSoup findAll return empty list

user_19655353 0
Viewed 32

So i wanted to get all "b" tags with "price" class form this website, but soup.findAll return empty list with 0 elements. But when I try the same thing, but with local html file, everything just works fine.

I've searched all over the internet, but nothing seems to help.

The code itself:

from bs4 import BeautifulSoup
from urllib.request import Request, urlopen

url = 'https://warframe.market/items/nami_skyla_prime_blueprint'

req = Request(url, headers={'User-Agent': 'Mozilla/5.0'})
webpage = urlopen(req).read()
soup = BeautifulSoup(webpage, 'html.parser')
tags = soup.findAll('b')
print(tags)
web-scraping urllib beautifulsoup html python
1 Answers

The data you see on the page is loaded from external URL via JavaScript (so beautifulsoup doesn't see it). To load the data into pandas dataframe you can use this example:

import requests
import pandas as pd

api_url = (
    "https://api.warframe.market/v1/items/nami_skyla_prime_blueprint/orders"
)

df = pd.DataFrame(requests.get(api_url).json()["payload"]["orders"])
df = pd.concat([df, df.pop("user").apply(pd.Series)], axis=1)
print(df.to_markdown(index=False))

Prints:

platform region order_type creation_date platinum quantity last_update visible id reputation region avatar last_seen ingame_name id status
pc en sell 2018-03-24T13:19:44.000+00:00 5 1 2018-03-24T13:19:44.000+00:00 True 5ab6507049ef000fcf1920a9 0 en 2022-07-30T18:00:20.943+00:00 lastcrist 62e56f91330ca40377b19895 offline
pc en sell 2019-01-10T18:21:06.000+00:00 14 1 2022-07-26T18:45:05.000+00:00 True 5c378d12af045700a4291477 3 en 2022-07-27T11:04:01.133+00:00 kekxqt 5a0628ddc611360011c7a66e offline
pc en sell 2019-12-09T07:11:43.000+00:00 35 1 2021-12-02T04:32:53.000+00:00 True 5dedf3afc72155005cf4028c 968 en user/avatar/55f4e189b66f831701dbff77.png?0fe7b910face5429601e7085d289f869 2022-07-30T16:18:02.366+00:00 Sahysa 55f4e189b66f831701dbff77 offline
pc en sell 2020-02-04T03:29:02.000+00:00 13 1 2022-07-29T23:22:45.000+00:00 True 5e38e4fe7b0275005e6d29a9 4 en 2022-07-30T06:40:38.150+00:00 YellowSnowBoi 5b3c268c0c3535012e9b6ee3 offline
pc en sell 2020-03-29T20:14:39.000+00:00 15 1 2022-07-24T18:14:30.000+00:00 True 5e8101aff3441505a6b292c6 19 en user/avatar/5b2931d4eb069f04986adb6d.png?822ece14d8951796338ef3739d5c6b2c 2022-07-29T20:55:01.617+00:00 xblq 5b2931d4eb069f04986adb6d offline
pc ko sell 2020-11-07T09:15:58.000+00:00 9 1 2020-11-07T09:15:58.000+00:00 True 5fa665ced5b8ca0262d04cf0 0 ko 2022-07-30T17:33:05.154+00:00 blazespirit 5a4380778c474d007384823b ingame
pc en sell 2021-02-07T12:42:27.000+00:00 15 1 2021-12-26T14:35:01.000+00:00 True 601fe033ec5e7e01d47859a1 3 en user/avatar/600fcf3dc59d3703156a15e3.png?5f9404dcf7c36b3200d2eaab5ed9dba9 2022-07-30T10:26:53.816+00:00 pavelfizek 600fcf3dc59d3703156a15e3 offline
pc en buy 2021-02-12T14:34:49.000+00:00 10 6 2022-07-20T21:11:53.000+00:00 True 60269209039a2702632da531 412 en user/avatar/598d8bf2d3ffb648a838abba.png?dfe50b7091f0e8c44b63976a700eebd9 2022-07-30T10:02:13.214+00:00 Xnidior 598d8bf2d3ffb648a838abba offline
pc en sell 2021-02-13T03:24:21.000+00:00 10 1 2022-07-23T06:48:55.000+00:00 True 602746651c9894026f386fb0 2 en 2022-07-28T02:31:25.058+00:00 Reaper575 5cd904444532a609fef941db offline
pc en sell 2021-03-13T18:03:01.000+00:00 15 1 2022-07-22T14:35:07.000+00:00 True 604cfe55171b1903005aa375 23 en user/avatar/5c1cc13e6ad4a90015110a59.png?3eca6f848f62560068b7c0f17f141d6d 2022-07-27T11:36:30.529+00:00 -CG-Luthix 5c1cc13e6ad4a90015110a59 offline
pc ru buy 2021-04-03T05:56:22.000+00:00 5 1 2022-04-07T10:38:12.000+00:00 True 606803861a02f00165d2965c 2 ru user/avatar/624ebb41596f2f00881ce522.png?c1556e2a317fe8dfbd7e86ba051e371a 2022-07-27T14:19:19.788+00:00 1FioLet1 624ebb41596f2f00881ce522 offline
pc en sell 2021-06-11T13:42:09.000+00:00 20 1 2021-06-11T13:42:09.000+00:00 True 60c368311970de00c952915f 0 en 2022-07-30T02:14:54.187+00:00 NEXUREX21 6039915229f04b0128def8fa offline

...

user_100359850
Related