Web Scraping - Get to Page 2

user_10765369 0
Viewed 184

How to I get to page two of the data sets? No matter what I do, it only returns page 1.

import bs4
from urllib.request import urlopen as uReq
from bs4 import BeautifulSoup as soup

myURL = 'https://jobs.collinsaerospace.com/search-jobs/'

uClient = uReq(myURL)
page_html = uClient.read()
uClient.close()

page_soup = soup(page_html, "html.parser")
container = page_soup.findAll("section", {"id":"search-results"}, {"data-current-page":"4"})


for child in container:
    for heading in child.find_all('h2'):
        print(heading.text)
web-scraping beautifulsoup python
3 Answers

The site actually uses JSON to return the HTML containing all of the entries. The API for this allows a page number to be specified and also the number of records to be returned for each page, increasing this will further increase the speed.

The JSON that is returned contains 3 keys. Filter information, the results HTML and a flag to indicate if jobs were returned. This last entry can be used to signal when you have reached the end of the pages.

You might want to look at the very popular Python requests library which simplifies generating the correct URLs for you and is also fast.

import bs4
import requests
from bs4 import BeautifulSoup as soup

params = {            
    "CurrentPage" : 1,
    "RecordsPerPage" : 100,
    "SearchResultsModuleName" : "Search Results",
    "SearchFiltersModuleName" : "Search Filters",
    "SearchType" : 5,
}

myURL = 'https://jobs.collinsaerospace.com/search-jobs/results'
page = 1
more_jobs = True

while more_jobs:
    print(f"\nPage {page}")
    params['CurrentPage'] = page
    req = requests.get(myURL, params=params)
    json = req.json()
    page_soup = soup(json['results'], "html.parser")
    container = page_soup.findAll("section", {"id":"search-results"}, {"data-current-page":"4"})

    for child in container:
        for heading in child.find_all('h2'):
            print(heading.text)

    more_jobs = json['hasJobs']  # Did this return any jobs?
    page += 1
user_49857330

try this :

import bs4
from urllib.request import urlopen as uReq
from bs4 import BeautifulSoup as soup


for letter in range(10):

    myURL = 'https://jobs.collinsaerospace.com/search-jobs/'+ str(letter) + ' '

    uClient = uReq(myURL)
    page_html = uClient.read()
    uClient.close()

    page_soup = soup(page_html, "html.parser")
    container = page_soup.findAll("section", {"id":"search-results"}, {"data-current-page":"4"})


    for child in container:
        for heading in child.find_all('h2'):
            print(heading.text)

output: of 3 first pages:

0
SYSTEMS / APPLICATIONS ENGINEER
Data Scientist
Sr Engineer, Drafter/Product Definition
Finance and Accounting Intern
Senior Software Engineer - CT3
Intern Manufacturing Engineer
Staff Eng., Reliability Engineering
Software Developer
Configuration Management Specialist
Disassembler I--2nd Shift
Disassembler I--3rd Shift
Manager, Supplier Performance
Manager, Supplier Performance
Assoc Eng, Mfg Engrg-Ops, ME P1
Manager, Supplier Performance
1
Assembly Operator (UK7014) 1 1 1 1
Senior Administrator (DF1040) 1 1 1
Tester 1
Assembler 1
Assembler 1
Finisher 1
Painter 1
Technician 1 Manufacturing/Operations
Assembler 1 - 1st Shift
Supply Chain Analyst 1
Assembler (W7006) 1
Assembler (W7006) 1
Supplier Quality Engineer 1
Supplier Inspection Engineer 1
Assembler 1 - 1st Shift
2
Assembler I-FAA-2
Senior/Business Analyst-2
Operational Technical Support Level 2
Project Engineer - 2 – EMU Program
Line & Surface Plate Inspector Class 2
Software Engineer (LVL 2) - Embedded UAV Controls
Software Engineer (LVL 2 / JAVA) - Air Combat Training
Software  Engineer (Level 2) - Mission Simulation & Training
Electrical Engineer (LVL 2) - Mission Systems Design Tools
Quality Inspector II
GET/PGET
GET/PGET
Production Supervisor - 2nd shift
Software Developer
Trainee Operator/ Operator
user_97170920

Try the following script to get the results from whatever pages you are interested in. All you need to do is change the range as per your requirement. I could have defined a while loop to exhaust the whole content but that is not the question you asked.

import requests
from bs4 import BeautifulSoup

link = 'https://jobs.collinsaerospace.com/search-jobs/results?'

params = {
'CurrentPage': '',
'RecordsPerPage': 15,
'Distance': 50,
'SearchResultsModuleName': 'Search Results',
'SearchFiltersModuleName': 'Search Filters',
'SearchType': 5
}
for page in range(1,5):  #This is where you change the range to get the results from whatever page you want
    params['CurrentPage'] = page
    res = requests.get(link,params=params)
    soup = BeautifulSoup(res.json()['results'],"lxml")
    for name in soup.select("h2"):
        print(name.text)
user_91897990
Related