request next page but url with no page numbers

user_15274265 0
Viewed 37

I want to scrape a url and get all links from 100 pages. The problem is that the url doesn't change to be formated with page number. I am able to get the first 10 links but I didn't find a way to get the links from page 2 till page 100. Is there any way to do so without page number ? or filter 100 results per page? I am using python requests and beautifulsoup

my code so far:

import requests
from bs4 import BeautifulSoup

project = []

url = 'https://unicorner.news'
r = requests.get(url)
soup = BeautifulSoup(r.text, "html.parser")

links = []
categories = soup.findAll('a','utils_postLink__m_2J6')
for category in categories:
    links.append(category['href'])

how to continue?

python-requests python
1 Answers

You can directly access the article data by calling the website's data API.

Try this:

url = "https://unicorner.news/_next/data/XpX_u4UJUUahh9Sa-aCE4"
res = requests.get(url)
data = res.content
df = pd.DataFrame(json.loads(data)["pageProps"]["articles"])
df.head()

Print out:

image   oneliner    name    date    url     description
0   https://img.mailinblue.com/2970250/images/rnb/...   Automate repetitive tasks with one click    Bardeen     July 25, 2022   https://sh1.sendinblue.com/3g6fxt972pxpfe.html...   Bardeen is a no-code workflow automation tool....
1   https://img.mailinblue.com/2970250/images/rnb/...   Bringing commands and search to any app     CommandBar  July 18, 2022   https://sh1.sendinblue.com/3g63ap972pxpfe.html...   CommandBar is a search bar that enables users ...
2   https://img.mailinblue.com/2970250/images/rnb/...   Automate copy-and-pasting between tabs  Magical     July 11, 2022   https://sh1.sendinblue.com/3g5z2zx72pxpfe.html...   Magical allows you to quickly transfer data be...
3   https://img.mailinblue.com/2970250/images/rnb/...   Advertising without cookies     SegmentStream   July 4, 2022    https://sh1.sendinblue.com/3g5uval72pxpfe.html...   SegmentStream provides businesses marketing an...
4   https://img.mailinblue.com/2970250/images/rnb/...   YouTube creator school  creator now     June 27, 2022   https://sh1.sendinblue.com/3g5qnl972pxpfe.html...   creator now is an online learning platform and...

If you just want the links do this:

df.url.values

Print out:

array(['https://sh1.sendinblue.com/3g6fxt972pxpfe.html?t=1658778042',
       'https://sh1.sendinblue.com/3g63ap972pxpfe.html?t=1658167180',
       'https://sh1.sendinblue.com/3g5z2zx72pxpfe.html?t=1657713951',
       'https://sh1.sendinblue.com/3g5uval72pxpfe.html?t=1657178365',

...

user_71170030
Related