BeautifulSoup find_all returns only first 50 tags

user_11730549 0
Viewed 178

I am trying to parse html data from this website with BeautifulSoup, but strangely enough, it returns only the first 50 tags that it finds.

When I search the html code through Google DevTools I get 115 matches for the class name that I am looking for.

url='https://wolt.com/az/aze/baku/restaurant/mcdonalds-nnrimanov-ms'
html=urllib.request.urlopen(url).read()
soup = BeautifulSoup(html,'html.parser')

modules=soup.find_all('div',attrs={'class':'MenuItem-module__content___mNrbB'})
print(len(modules))

Output:

>>> 50

I have tried parsing other pages on this website, and still get only 50 results back.

I have also used the answer from Beautiful Soup findAll doesn't find them all to tweak my code. Any help is appreciated!

web-scraping beautifulsoup html-parsing python
3 Answers

The data is loaded from external URL in Json format. To get all 115 items you can use next example:

import json
import requests

url = "https://restaurant-api.wolt.com/v4/venues/slug/mcdonalds-nnrimanov-ms/menu?unit_prices=true&show_weighted_items=true"

data = requests.get(url).json()

# ucomment this to print all data:
# print(json.dumps(data, indent=4))

for i, item in enumerate(data["items"], 1):
    print("{:<4} {}".format(i, item["name"]))

Prints:

...

106  Latte (200 ml)
107  Milk Chocolate (300 ml)
108  Milk Chocolate (200 ml)
109  Ketchup
110  Mayonnaise
111  Barbecue Sauce
112  Sweet Sauce
113  Mustard Sauce
114  1000 Island Sauce
115  Sweet Chilli
user_100359850

When you first access the url, the page will load the first 50 elements with class MenuItem-module__content___mNrbB by default. The rest of them, along with some other content is being loaded dynamically by some scripts which execute in page. Requests is not able to execute Javascript (and BeautifulSoup is just the html parser).

You can test this by disabling Javascript in browser, and reloading the page: you will only find 50 elements in Dev Tools, corresponding to class MenuItem-module__content___mNrbB.

If you want to get all 119 elements corresponding to that class, you will need to inspect the network calls made by javascript - you can see those calls in Network tab, Dev tools, and you can try and scrape those endpoints (this was already clarified in another response).

I hope this is addressing your concerns as to why BeautifulSoup can only find 50 elements, and you understand that particular url/page behaviour now.

user_194751850

You can't use BeautifulSoup for this task. You don't even need to grab the content of this page.

The link is https://wolt.com/en/aze/baku/restaurant/mcdonalds-nnrimanov-ms and the real link is https://restaurant-api.wolt.com/v4/venues/slug/mcdonalds-nnrimanov-ms/menu?unit_prices=true&show_weighted_items=true

So in fact you can use this piece of code for grabbing the json file.

import urllib
import json
url='https://wolt.com/az/aze/baku/restaurant/mcdonalds-nnrimanov-ms'
link='https://restaurant-api.wolt.com/v4/venues/slug/'+url.replace('/', ' ').strip().split(' ')[-1]+'/menu?unit_prices=true&show_weighted_items=true'
jsn=urllib.request.urlopen(link).read()
dic=json.loads(jsn)
print(dic['items'])
'''
Useful properties:
'alcohol_percentage': percentage of alcohol
'baseprice': price with no discount
'description' and 'name': as it says
'no_contact_delivery_allowed': something special in COVID times
'times': available times(a list)
each element in 'times': 'available_days_of_week': a list, as it means
'''
user_192701680
Related