Getting Error 404 while scraping using beautiful soup, even though the site exists

Viewed 390

I am trying to scrape some data from OLX, there are multiple pages, so I have added loops. My code works for 1st 4 iterations but after i = 3 I get error 404, even though the sites exist beyond i=3 they give 404 btw I can open them on the browser. I tried changing user agent but didn't work. Site for page 4: https://www.olx.com.pk/mobiles_c1411/q-samsung-galaxy-s10?page=4&sorting=asc-price Please help

import csv
from urllib.request import urlopen as ureq
from bs4 import BeautifulSoup as soup
import time
import requests


headers = {'USER-AGENT': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.143 Safari/537.36'}


for i in range(0, 8, 1):

    page_html = requests.get('https://www.olx.com.pk/mobiles_c1411/q-samsung-galaxy-s10?page='+ str(i) + '&sorting=asc-price', headers=headers, timeout=10)
    print(page_html.status_code)
0 Answers
Related