How to scarpe li tag under ul?

tanmay saxena 0
Viewed 30
beautifulsoup web-scraping html python
1 Answers

Content is served dynamically based on data of an additional XHR request, so you have to call this instead. You can inspect this by taking a look into devtools of browser on XHR tab.

Example

Instead of appending only the obvious to a list of dicts you could also iterate all detailpages while requesting them.

from bs4 import BeautifulSoup
import requests, string

data = []
for letter in list(string.ascii_uppercase):
    result = requests.post(f'https://www.parliament.lk/members-of-parliament/directory-of-members/index2.php?option=com_members&task=all&tmpl=component&letter={letter}&wordfilter=&search_district=')
    
    for e in result.json():
        #result = requests.get(f"https://www.parliament.lk/en/members-of-parliament/directory-of-members/viewMember/{e['mem_intranet_id']}")
        data.append({
            'url':f"https://www.parliament.lk/en/members-of-parliament/directory-of-members/viewMember/{e['mem_intranet_id']}",
            'id':e['mem_intranet_id'],
            'name':e['member_sname_eng']
        })

data

Output

[{'url': 'https://www.parliament.lk/en/members-of-parliament/directory-of-members/viewMember/3266',
  'id': '3266',
  'name': 'A. Aravindh Kumar'},
 {'url': 'https://www.parliament.lk/en/members-of-parliament/directory-of-members/viewMember/50',
  'id': '50',
  'name': 'Abdul Haleem'},
 {'url': 'https://www.parliament.lk/en/members-of-parliament/directory-of-members/viewMember/3325',
  'id': '3325',
  'name': 'Ajith Rajapakse'},
 {'url': 'https://www.parliament.lk/en/members-of-parliament/directory-of-members/viewMember/3296',
  'id': '3296',
  'name': 'Akila Ellawala'},
 {'url': 'https://www.parliament.lk/en/members-of-parliament/directory-of-members/viewMember/3355',
  'id': '3355',
  'name': 'Ali Sabri Raheem'},...]
HedgeHog0
Related