How to index through <li of a <ul given by a class name

user_15217462 0
Viewed 68

I have an html code similar to the following

<div>
<div>
    <div>
        <div>
            <div>
                <div>
                <div>
                    <section >
                        <article>
                            <div>
                                <div>
                                <ul    class='hover p-0  job-search-key-kgm6qi exy0tjh1'>
                                    <li     <a....</a>    </li>
                                    <li     <a....</a>    </li>
                                    <li     <a....</a>    </li>
                                    <li     <a....</a>    </li>
                                    <li     <a....</a>    </li>
                                    <li     <a....</a>    </li>
                                <ul/>
                                <div/>
                            <div/>
                        <article/>
                    <section/>
                <div/>
                <div/>
            <div/>
        <div/>
    <div/>
<div/>

Also, here's an image of the real code enter image description here

I want to get all the links that are contained in the <li.... of the <ul where the class is named 'hover p-0 job-search-key-kgm6qi exy0tjh1' I tried to use the following code, but it gives me an empty list

import requests
from bs4 import BeautifulSoup


page = requests.get('https://www.glassdoor.com/Job/united-states-business-intelligence-analyst-jobs-SRCH_IL.0,13_IN1_KO14,43_IP3.htm?includeNoSalaryJobs=true&pgc=AB4AAoEAPAAAAAAAAAAAAAAAAd1d%2Bb4AbwECARFcBgSsznIZFPk4fW3sDR3jdfmJewv2%2BLA8TQi8YFhbqVFBx1lHzaVOYjw909IQ%2BIC38g6D%2F3KlCEAfpAkJoqF3AECpEZLOSE%2FDN2rQ33BRLDrpbxe6MwndhK4pkphs5Qq5uzHuklUQKM%2BrUQAA')


soup = BeautifulSoup(page.text, 'html.parser')


results = soup.find_all('ul', attrs={'class':'hover p-0  job-search-key-kgm6qi exy0tjh1'})


print(results)
web-scraping beautifulsoup html python
1 Answers

You need to give requests a proper header, and also to revisit Beautiful Soup syntax (afaik you cannot select multiple classes at once, separated by spaces, in selector). The following code works:

import requests
from bs4 import BeautifulSoup

headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.60 Safari/537.17'}

r = requests.get('https://www.glassdoor.com/Job/united-states-business-intelligence-analyst-jobs-SRCH_IL.0,13_IN1_KO14,43_IP3.htm?includeNoSalaryJobs=true&pgc=AB4AAoEAPAAAAAAAAAAAAAAAAd1d%2Bb4AbwECARFcBgSsznIZFPk4fW3sDR3jdfmJewv2%2BLA8TQi8YFhbqVFBx1lHzaVOYjw909IQ%2BIC38g6D%2F3KlCEAfpAkJoqF3AECpEZLOSE%2FDN2rQ33BRLDrpbxe6MwndhK4pkphs5Qq5uzHuklUQKM%2BrUQAA', headers=headers)
soup = BeautifulSoup(r.text, 'html.parser')
results = soup.select('li.eigr9kq0')
for result in results:
    print(result.text)
user_194751850
Related