Beautiful soup glassdoor pages

user_8281211 0
Viewed 108

I have a glassdoor link that I'm trying to access through request.get() https://www.glassdoor.com/Job/jobs.htm?sc.generalKeyword=%22teaching%22&sc.locationSeoString=new+york&locId=1132348&locT=C

I noticed that when I click on the next page a lo_IP{page_number}.htm is added. For example: https://www.glassdoor.com/Job/jobs.htm?sc.generalKeyword=%22teaching%22&sc.locationSeoString=new+york&locId=1132348&lo_IP4.htm for page 4.

but when I go directly to that link (page 4 for example), it does not take me to page 4. Is there a way to go to the nth number of page?

    pages= 2
 
    for x in range(1, pages):
        page_url = "https://www.glassdoor.com/Job/jobs.htm?sc.generalKeyword=%22teaching%22&sc.locationSeoString=new+york&locId=1132348&lo_IP{}.htm".format(x)
        headers = {"User-Agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36"}
        page = requests.get(page_url, headers=headers)
        soup = BeautifulSoup(page.content, 'html.parser')
beautifulsoup python
1 Answers

By:

<li class="page">
    <a href="/Job/jobs.htm?sc.generalKeyword=%22teaching%22&amp;sc.locationSeoString=new+york&amp;locId=1132348&amp;locT=C&amp;p=4">
        <span class="link">4</span>
    </a>
</li>

https://www.glassdoor.com/Job/jobs.htm?sc.generalKeyword=%22teaching%22&sc.locationSeoString=new+york&locId=1132348&locT=C&p=4 will goto the page 4.

And logically &p=n will goto nth page. So to get the nth page

url = f'https://www.glassdoor.com/Job/jobs.htm?sc.generalKeyword="teaching"&sc.locationSeoString=new+york&locId=1132348&locT=C&p={n}'

Origin website is working by JS. It just request data and update both url and page. So https://www.glassdoor.com/Job/jobs.htm?sc.generalKeyword=%22teaching%22&sc.locationSeoString=new+york&locId=1132348&lo_IP4.htm is just something that it put on the url.

user_103300050
Related