Python beautifulsoup extract all urls from a website search results. New Python beginner

There is 754 book, i show example with 35. To get all the books, at the end of the url change 35 to 754

import pandas as pd
import requests

url = 'https://cdm20045.contentdm.oclc.org/digital/api/search/collection/p20045coll17/searchterm/Integrated%20Data%20Infrastructure%20(IDI)/field/projeb/mode/exact/conn/and/maxRecords/35'
response = requests.get(url)
books = []
for book in response.json()['items']:
    books.append({
        'link': ('https://cdm20045.contentdm.oclc.org' + book['itemLink']).replace('singleitem', 'digital'),
        'title': book['metadataFields'][0]['value'],
        'subjec': book['metadataFields'][1]['value'],
        'date': book['metadataFields'][2]['value'],
        'publis': book['metadataFields'][3]['value']
    })
df = pd.DataFrame(books)
print(df.to_string())

OUTPUT:

                                                                           link                                                                                                                                                                                                       title                                                                                              subjec     date                                                                                              publis
0   https://cdm20045.contentdm.oclc.org/digital/collection/p20045coll17/id/1181                                                                                                                                  The Future of Work in New Zealand - An Empirical Examiniation (MAA2019-95)                                                                                 Business Practices;     2021                                                                  Auckland University of Technology;
1    https://cdm20045.contentdm.oclc.org/digital/collection/p20045coll17/id/993                                                                                                                                           In Fairness to Our Schools: Better measures for better outcomes\n                                                                                           Education     2019                                                                          The New Zealand Initiative
2   https://cdm20045.contentdm.oclc.org/digital/collection/p20045coll17/id/1029                                                              Dementia case-finding and prevalence estimation using routinely collected health data in the Integrated Data Infrastructure (IDI) (MAA2020-12)                                                                                             Health;     2020                                                                              University of Auckland
3   https://cdm20045.contentdm.oclc.org/digital/collection/p20045coll17/id/1218                                                                                                                          Investigating linkage bias in the IDI using education and census data (MAA2020-69)                                                                           Meta-research; Education;  2020-12                                                                             University of Auckland;
4    https://cdm20045.contentdm.oclc.org/digital/collection/p20045coll17/id/757                                                                                                                  Explaining Ethnic Differences in Student Success at University in New Zealand [MAA2018-09]                                                                              Education and training     2018                                                                   Auckland University of Technology
5   https://cdm20045.contentdm.oclc.org/digital/collection/p20045coll17/id/1357                                                                                                                                          Health and social harms from alcohol: what does NZ's data tell us?                                                                                             Health;  2022-03                                                                                University of Otago;
6   https://cdm20045.contentdm.oclc.org/digital/collection/p20045coll17/id/1268                                                                                                                 'What about the Menz?' Low employer attachment and ineligibility for partner parental leave                                                            Income and Work; People and Communities;  2021-08                                                          Auckland Council; Social Wellbeing Agency;
7    https://cdm20045.contentdm.oclc.org/digital/collection/p20045coll17/id/936                                                                                                Migrant Networks, Brain Waste and its Economic Impacts: Evidence from Immigrants in New Zealand [MAA2019-31]                                                                  Employment; People and Communities     2019                                                                              University of Auckland
8   https://cdm20045.contentdm.oclc.org/digital/collection/p20045coll17/id/1306                                                                                                                                                                      Pacific uptake of temporary work visas                                            Employment; Business financials; People and Communities;  2020-05  NZIER; Ministry of Business, Innovation & Employment, MBIE; Ministry of Foreign Affairs and Trade;
9    https://cdm20045.contentdm.oclc.org/digital/collection/p20045coll17/id/123                                                                                                                                                             Firm Productivity Growth and Skill (MAA2012-16)                                                                 Income and work; Business practices     2015                                                            Motu Economic and Public Policy Research
10   https://cdm20045.contentdm.oclc.org/digital/collection/p20045coll17/id/380                                                                                                                                The Productivity Costs of Four Health Conditions in New Zealand [MAA2016-59]                                                                             Health; Income and work     2016                                                                                 University of Otago
11   https://cdm20045.contentdm.oclc.org/digital/collection/p20045coll17/id/732                                                                                               Differential labour supply effects of pension eligibility to beneficiaries and non-beneficiaries [MAA2018-50]                                                      Income and Work; Benefits and Social Services;     2018                                                                  Auckland University of Technology;
12   https://cdm20045.contentdm.oclc.org/digital/collection/p20045coll17/id/189                                                                                                                                                                    Intergenerational Analyses Using the IDI                                                                              People and communities     2017                                                                COMPASS (The University of Auckland)
13   https://cdm20045.contentdm.oclc.org/digital/collection/p20045coll17/id/198                                                                                                                         School to work: What matters? Education and Employment of Young People Born in 1991                                                                              Education and training     2016                                                                               Ministry of Education
14   https://cdm20045.contentdm.oclc.org/digital/collection/p20045coll17/id/975                                                                                                                                        Equity Index of socioeconomic disadvantage in education [MAA2019-85]                                            Education; Benefits and Social Services; Income and Work  2019-12                                                                               Ministry of Education
15  https://cdm20045.contentdm.oclc.org/digital/collection/p20045coll17/id/1225                                                                                                                                                                        Linkage bias in the IDI (MAA2020-58)                                                                                      Meta-research;  2020-11                                                                                University of Otago;
16   https://cdm20045.contentdm.oclc.org/digital/collection/p20045coll17/id/431                                                                                    The relationship between exposure to the natural environment and children's health at different life stages [MAA2017-11]                                                                                              Health     2017                                                                                   Massey University
17  https://cdm20045.contentdm.oclc.org/digital/collection/p20045coll17/id/1025                                                                                                                      Who are the 1M and 1X? Police engagement with citizens in mental distress (MAA2020-08)                                                                                    Justice; Health;     2020                                                                              University of Auckland
18  https://cdm20045.contentdm.oclc.org/digital/collection/p20045coll17/id/1073                                                                                                                                      Who received the Wage Subsidy and Wage Subsidy Extension? (MAA2018-48)                                                                       Benefits and Social Services;     2020                                                                Ministry of Social Development, MSD;
19   https://cdm20045.contentdm.oclc.org/digital/collection/p20045coll17/id/929                                                                                                                Factsheet IDI first stage - Exploring work-related claims difference by Maori and non-Maori                                  Business Practices; Health; Income and Work; People and Communities  2019-02                                                                                Worksafe New Zealand
20  https://cdm20045.contentdm.oclc.org/digital/collection/p20045coll17/id/1138                                                                                      Measuring Commute Patterns Over Time: Using administrative data to identify where employees live and work (MAA2018-55)                                                                              Transport; Employment;     2020                             Motu Economic and Public Policy Research; New Zealand Transport Agency;
21   https://cdm20045.contentdm.oclc.org/digital/collection/p20045coll17/id/835                                                                                                                                                          Evaluating the Family Start Programme [MAA2018-87]        People and Communities; Health; Education; Employment; Benefits and Social Services; Justice  2018-12                                                    Ministry for Vulnerable Children Oranga Tamariki
22   https://cdm20045.contentdm.oclc.org/digital/collection/p20045coll17/id/980  Access to primary health care services for people in Canterbury with poor access: improving our understanding of people who are unenrolled or tenuously enrolled with a general practice team [MAA2019-51]                                                                                              Health  2019-11                                                                 Pegasus Health (Charitable) Limited
23   https://cdm20045.contentdm.oclc.org/digital/collection/p20045coll17/id/996                                                                                                                                         Intergenerational Analyses Using the IDI:\nAn update\n [MAA2016-53]                                                                                          Population  2020-03                                                                    COMPASS, University of Auckland;
24  https://cdm20045.contentdm.oclc.org/digital/collection/p20045coll17/id/1351                                                                                                                                                                          Individualised Funding in Aotearoa                                                                       Benefits and Social Services;     2020                                                                               Nicholson Consulting;
25    https://cdm20045.contentdm.oclc.org/digital/collection/p20045coll17/id/39                                                                                   Comparing the Household Economic Survey to administrative records: an analysis of income and benefit receipt (MAA2015-27)                                                                        Benefits and Social Services     2017                                                                                New Zealand Treasury
26  https://cdm20045.contentdm.oclc.org/digital/collection/p20045coll17/id/1320                                                                                                                 International students and graduates and their impact on the NZ housing market (MAA2017-31)                                                                    Housing; Education and Training;     2021                                                                           Universities New Zealand;
27  https://cdm20045.contentdm.oclc.org/digital/collection/p20045coll17/id/1002                                                                                                              The expression, experience and transcendence of low-skill in Aotearoa New Zealand (MAA2019-91)                                                                             Education and Training;     2019                                                                   Auckland University of Technology
28   https://cdm20045.contentdm.oclc.org/digital/collection/p20045coll17/id/394                                                                                                                           Comparison of NZDep2013 with Index of Multiple Deprivation (IMD2013) [MAA2017-70]                                                           Health; Income and Work; Housing; Justice     2017                                                                                 University of Otago
29  https://cdm20045.contentdm.oclc.org/digital/collection/p20045coll17/id/1005                                                                                                                                    Accessibility of Disability Support Services funding in NZ (MAA2019-102)                                                                                             Health;     2019                                                                               Nicholson Consulting;
30  https://cdm20045.contentdm.oclc.org/digital/collection/p20045coll17/id/1135                                         Understanding Parental education and health of Pacific families: Background and study protocol: Parental Education and Pacific Health, study protocol  (MAA2018-47)                                                        Education; People and Communities; Children;     2020                                                                                University of Otago;
31    https://cdm20045.contentdm.oclc.org/digital/collection/p20045coll17/id/47                                                                                                          Evaluation of the Impact of the Youth Service: Youth Payment and Young Parent Payment (MAA2013-16)                                                                        Benefits and Social Services     2017                                                                                New Zealand Treasury
32    https://cdm20045.contentdm.oclc.org/digital/collection/p20045coll17/id/63                                                                                                                  Using IDI data to estimate fiscal impacts of better social sector performance (MAA2013-16)                                                                        Benefits and Social Services     2016                                                                                New Zealand Treasury
33  https://cdm20045.contentdm.oclc.org/digital/collection/p20045coll17/id/1040                                                                                                                                                                      Māori Student Transitions (MAA2020-35)                  Education and Training; Benefits and Social Services; Income and Work; Employment;     2020                                                                            Social Wellbeing Agency;
34    https://cdm20045.contentdm.oclc.org/digital/collection/p20045coll17/id/64                                                                           Financial wellbeing of older workers following injury: research utilising Statistics New Zealand’s Integrated Data Infrastructure                                                                             Health; Income and work     2017                                                                                 University of Otago

@bmcculley,already has stated that the required data is loaded dynamically via JS and Bs4 can render JS. So you have two options: use either selenium (more complex) or use API. As they are using API, so you can esily grab the required data from API as GET methosd as Json format data which is the robust way.

I've made the pagination using for loop and range function.

Example:

import requests
import pandas as pd
api_url = 'https://cdm20045.contentdm.oclc.org/digital/api/search/collection/p20045coll17/searchterm/Integrated%20Data%20Infrastructure%20(IDI)/field/projeb/mode/exact/conn/and/page/{page}/maxRecords/30'

data = []
for page in range(1,27):
    r = requests.get(api_url.format(page=page))
    for link in r.json()['items']:
        url = 'https://cdm20045.contentdm.oclc.org' + link['itemLink'].replace('singleitem','digital')
        data.append({
            'URL':url
            })

df = pd.DataFrame(data)
print(df)

Outout:

                                   URL
0    https://cdm20045.contentdm.oclc.org/digital/co...
1    https://cdm20045.contentdm.oclc.org/digital/co...
2    https://cdm20045.contentdm.oclc.org/digital/co...
3    https://cdm20045.contentdm.oclc.org/digital/co...
4    https://cdm20045.contentdm.oclc.org/digital/co...
..                                                 ...
749  https://cdm20045.contentdm.oclc.org/digital/co...
750  https://cdm20045.contentdm.oclc.org/digital/co...
751  https://cdm20045.contentdm.oclc.org/digital/co...
752  https://cdm20045.contentdm.oclc.org/digital/co...
753  https://cdm20045.contentdm.oclc.org/digital/co...

[754 rows x 1 columns]