how to get to know the last page index number for my post request for the given website when it changes dynamically

user_10369901 0
Viewed 146

the given website enter link description here loads its table data dynamically whenever the scrollbar reaches to the window end point. it also generates a new post request with new Page Index number to load the new data for the table. I want to sent multiple post requests so that I could load all table data and then to scrape it . how can I know what would be the last index number which should be sent along with my request.

import requests
from bs4 import BeautifulSoup


url = 'https://www.ungm.org/Public/Notice/Search'

payload = {
      "PageIndex": 0,
      "PageSize": 15,
      "Title": "",
      "Description": "",
      "Reference": "",
      "PublishedFrom": "",
      "PublishedTo": "12-Jul-2020",
      "DeadlineFrom": "12-Jul-2020",
      "DeadlineTo": "",
      "Countries": [],
      "Agencies": [],
      "UNSPSCs": [],
      "NoticeTypes": [],
      "SortField": "DatePublished",
      "SortAscending": False,
      "isPicker": False,
      "NoticeTASStatus": [],
      "IsSustainable": False,
      "NoticeDisplayType": None,
      "NoticeSearchTotalLabelId": "noticeSearchTotal",
      "TypeOfCompetitions": []
    }

I want this line should be executed the number of times eqaul to max of page index.

soup = BeautifulSoup( requests.post(url, json=payload).content, 'html.parser' )
beautifulsoup python
1 Answers

To know the last page index, you can select all rows from the post result. If the result is zero (no rows are selected, you are on your last page).

For example:

import requests
from bs4 import BeautifulSoup


url = 'https://www.ungm.org/Public/Notice/Search'

payload = {
  "PageIndex": 0,
  "PageSize": 15,
  "Title": "",
  "Description": "",
  "Reference": "",
  "PublishedFrom": "",
  "PublishedTo": "12-Jul-2020",
  "DeadlineFrom": "12-Jul-2020",
  "DeadlineTo": "",
  "Countries": [],
  "Agencies": [],
  "UNSPSCs": [],
  "NoticeTypes": [],
  "SortField": "DatePublished",
  "SortAscending": False,
  "isPicker": False,
  "NoticeTASStatus": [],
  "IsSustainable": False,
  "NoticeDisplayType": None,
  "NoticeSearchTotalLabelId": "noticeSearchTotal",
  "TypeOfCompetitions": []
}

page = 0
while True:
    print('Page {}...'.format(page))

    payload['PageIndex'] = page
    soup = BeautifulSoup( requests.post(url, json=payload).content, 'html.parser' )

    rows = soup.select('.tableRow')

    # no more data, exit!
    if not rows:
        break

    page += 1

Prints:

Page 0...
Page 1...
Page 2...
Page 3...

...

Page 49...
Page 50...
Page 51...
user_100359850
Related