How to accept cookies with requests

glitch_123 0
Viewed 52

I am trying to scrape a webpage using python but in order to scrape the webpage , I need to accept cookies on the webpage.

The code which I have tried is

URL = "https://www.howoge.de/wohnungen-gewerbe/wohnungssuche.html"
    with open('cookies') as f:
        j = json.load(f)

    session = requests.Session()
    for cookie in j: session.cookies.set(cookie['name'], cookie['value'])

    r = session.get(URL)

though this is not raising any error but still not accepting the cookies.

Here are my cookies:

[
    {
        "domain": ".howoge.de",
        "expirationDate": 1694266885,
        "hostOnly": false,
        "httpOnly": false,
        "name": "__cmpcpcu10543",
        "path": "/",
        "sameSite": "no_restriction",
        "secure": true,
        "session": false,
        "storeId": null,
        "value": "__51_54__"
    },
    {
        "domain": ".howoge.de",
        "expirationDate": 1694266885,
        "hostOnly": false,
        "httpOnly": false,
        "name": "__cmpconsent10543",
        "path": "/",
        "sameSite": "no_restriction",
        "secure": true,
        "session": false,
        "storeId": null,
        "value": "BPfEJk4PfEJk4AfHIBDEDXAAAAAAAA"
    },
    {
        "domain": "www.howoge.de",
        "expirationDate": 1696858880,
        "hostOnly": true,
        "httpOnly": false,
        "name": "__cmpcc",
        "path": "/",
        "sameSite": "no_restriction",
        "secure": true,
        "session": false,
        "storeId": null,
        "value": "1"
    },
    {
        "domain": ".howoge.de",
        "expirationDate": 1694266885,
        "hostOnly": false,
        "httpOnly": false,
        "name": "__cmpcvcu10543",
        "path": "/",
        "sameSite": "no_restriction",
        "secure": true,
        "session": false,
        "storeId": null,
        "value": "__s974_U__"
    },
    {
        "domain": "www.howoge.de",
        "hostOnly": true,
        "httpOnly": false,
        "name": "PHPSESSID",
        "path": "/",
        "sameSite": null,
        "secure": false,
        "session": true,
        "storeId": null,
        "value": "8pnd5h5up4v4rjh498if7hedac"
    }
]

What should be the best approach to tackle this problem?

beautifulsoup session-cookies cookies python-requests python
2 Answers

You don't need cookies or anything else.

Try this:

import requests

api_url = "https://www.howoge.de/?type=999&tx_howsite_json_list[action]=immoList"

request_payload = {
    "tx_howsite_json_list[page]": "1",
    "tx_howsite_json_list[limit]": "12",
    "tx_howsite_json_list[lang]": "",
    "tx_howsite_json_list[rent]": "",
    "tx_howsite_json_list[area]": "",
    "tx_howsite_json_list[rooms]": "egal",
    "tx_howsite_json_list[wbs]": "all-offers",
}

response = requests.post(api_url, data=request_payload).json()

for item in response["immoobjects"]:
    print(f'{item["title"]} - {item["rent"]}')

Output:

Rüdickenstraße 23, 13053 Berlin - 1174.31
Rüdickenstraße 23, 13053 Berlin - 1174.31
Rotkamp 4, 13053 Berlin - 1428.25
Rotkamp 6, 13053 Berlin - 617.41
Rotkamp 6, 13053 Berlin - 1147.71
Rotkamp 6, 13053 Berlin - 1147.71
Rotkamp 6, 13053 Berlin - 565.12
Frankfurter Allee 218, 10365 Berlin - 513.85
Frankfurter Allee 218, 10365 Berlin - 501.6
Frankfurter Allee 218, 10365 Berlin - 513.85
Frankfurter Allee 218, 10365 Berlin - 717
Frankfurter Allee 218, 10365 Berlin - 890.6
baduker0

You do not need cookies nor headers. Try this to get a clean dataframe of the listings:

data = {
    'tx_howsite_json_list[page]': '1',
    'tx_howsite_json_list[limit]': '12',
    'tx_howsite_json_list[lang]': '',
    'tx_howsite_json_list[rent]': '',
    'tx_howsite_json_list[area]': '',
    'tx_howsite_json_list[rooms]': 'egal',
    'tx_howsite_json_list[wbs]': 'all-offers',
}

response = requests.post('https://www.howoge.de/?type=999&tx_howsite_json_list[action]=immoList', data=data)
df = pd.DataFrame(json.loads(response.content)["immoobjects"])

df.head()

    uid     title   image   district    rent    area    rooms   wbs     features    coordinates     icon    link    favorite    notice
0   19335   Rüdickenstraße 23, 13053 Berlin     /fileadmin/promos/downloadedImages/266355f2369...   Alt-Hohenschönhausen    1174.31     77  3   nein    [Balkon/Loggia, Fußbodenheizung, Zentralheizun...   {'lat': '52.5600754', 'lng': '13.5089916'}  icon-Figures-haus_full  /wohnungen-gewerbe/wohnungssuche/detail/1771-1...   False   Schöne 3-Zimmer-Wohnung
1   19336   Rüdickenstraße 23, 13053 Berlin     /fileadmin/promos/downloadedImages/266355f2369...   Alt-Hohenschönhausen    1174.31     77  3   nein    [Balkon/Loggia, Fußbodenheizung, Zentralheizun...   {'lat': '52.5600754', 'lng': '13.5089916'}  icon-Figures-haus_full  /wohnungen-gewerbe/wohnungssuche/detail/1771-1...   False

If you want to get the listings from the following pages change the value of 'tx_howsite_json_list[page]': '1', in data.

petezurich0
Related