Scraping ajax pages using python, but after requesting for a few times it returns fake values

Viewed 54

I am trying to scrape shopee item information using python.

Take https://shopee.com.my/All%20in%20one%20pc%20Intel%20core%20I3/I5/I7%20Dual-core%208G%20RAM%20128%20gb%20SSD%20With%20optical%20drive%20CD%2023.8%20Inch%20computer%20Office%20Desktop%20All-in-one%20desktop%20Support%20WiFi-i.206039726.5859069631 as an example.

As it is using ajax, I am trying to extract item info from: https://shopee.com.my/api/v2/item/get?itemid=5859069631&shopid=206039726

After a few requests, i found out that the json it responded turned out to be fake values (such as its actual rating is 4.78 but it returns 0.24).

I tried to solve this problem by changing headers and ip/proxy, but still not working.

Is there any other way that can potentially solve this?


def get_info(url,itemurl):
    
    
    requests.adapters.DEFAULT_RETRIES = 5 
    s = requests.session()
    s.keep_alive = False

    try:
        
        fake_ua=UserAgent()
        headers = {'User-Agent':fake_ua.random, 

                   'Accept': '*/*',
                    'Accept-Language': 'en-US,en;q=0.5',
                    'X-Shopee-Language': 'en',
                    'X-Requested-With': 'XMLHttpRequest',
                    'X-API-SOURCE': 'pc',
                    'If-None-Match-': '55b03-2ff39563c299cbdc937f8ab86ef322ab',
                    'DNT': '1',
                    'Referer': referer,
                    'TE': 'Trailers'}
        ip = get_daili()
        proxies = {"proxies":{"https":ip}}

        
        response = requests.get(url, headers = headers, proxies = proxies, verify=False)
        #response = requests.request("GET", url, headers=headers, data=payload)
        if response.status_code == 200:
            shop_info = response.json()
    except requests.ConnectionError as e:
        print(f' {url} error', e.args)    

    shop_name = shop_info['data']['name']    
    followers = shop_info['data']['follower_count']   
    ratinggood = shop_info['data']['rating_good'] 
    ratingbad = shop_info['data']['rating_bad'] 
    ratingnormal = shop_info['data']['rating_normal'] 



    try:
        fake_ua=UserAgent() 
        headers = {'User-Agent':fake_ua.random, 

                   'Accept': '*/*',
                    'Accept-Language': 'en-US,en;q=0.5',
                    'X-Shopee-Language': 'en',
                    'X-Requested-With': 'XMLHttpRequest',
                    'X-API-SOURCE': 'pc',
                    'If-None-Match-': '55b03-2ff39563c299cbdc937f8ab86ef322ab',
                    'DNT': '1',
                    'Referer': referer,
                    'TE': 'Trailers'} 
        ip = get_daili()
        proxies = {"proxies":{"https":ip}}
        response = requests.get(itemurl, headers = headers, proxies = proxies, verify=False)
        #response = requests.request("GET", itemurl, headers=headers, data=payload)
        if response.status_code == 200:
            item_info = response.json()
    except requests.ConnectionError as e:
        print(f' {url} error', e.args)    

    #print(json.dumps(item_info, indent=4))


    print(itemurl)
1 Answers

I think it's algorithm to protect their API service, so people can't abuse their server.

Maybe you can try using python selenium and Selenium Wire to capture the data.

Related