Python crawler can crawl the first ten pages of content, but cannot crawl the eleventh page

Viewed 16

The reason for the error The ['data']['data'] returned by res.json is empty, but after verifying that the data in the json on the website is not empty, I don't know why Supplement 1. I am using the proxy ip, the ip should not be a problem, I copied the 2json from the browser, and it should be fine

    for i in range(20):
        data1 = {"data":{"mainSearchPcConditionForm":{"city":"410","dq":"410","pubTime":"","currentPage":i,"pageSize":40,"key":"腾讯","workYearCode":"","compId":"","compName":"","compTag":"","industry":"","salary":"","jobKind":"","compScale":"","compKind":"","compStage":"","eduLevel":""},"passThroughForm":{"ckId":"shttcss0i616qc3696ppu5weu9ssgjha","scene":"page","skId":"5oaletiaf2hqtupemfkgkudx2irjpfz9","fkId":"txeirfi9jt95qqxmqkynw132xi9rrueb","sfrom":"search_job_pc"}}}

        res = requests.post(
            url='https://apic.liepin.com/api/com.liepin.searchfront4c.pc-search-job',
            json=data1,
            headers=headers,
        )
        # print(res.text)
        # 解析json
        print(res.status_code)
        if res.status_code!=200:
            print("多页面出错")
            break
        print("test140")
        # print(res.text)
        result = res.json()
        print(result)
        result=result['data']['data']['jobCardList']
        print(result)
        data = data + self.get_one_page_data(result)
    # 返回所有获取到的招聘信息   一个list
    return data'''
0 Answers
Related