I recently got into using the requests library.
What I am doing: Posting a request to a url, parsing the response html for an element. Here is what my current code looks like:
import requests
client = requests.Session()
client.get(URL) #the url of the site
csrftoken = self.client.cookies["csrftoken"] # the payload requires the token
r = client.post(url=URL, data=dict(csrftoken=csrftoken, download=link))
for line in r.iter_lines():
if b"id_to_be_located" in line:
print(line)
I'm not getting the desired response. I did some digging are here is what I found out.
- The response I get is of the original
URLnot the one that I'm supposed to get after the request. - Tried:
stream=Trueandallow_redirects=Trueno luck. Cross-checking the post url, the payload - Finally I tried comparing the bare cookies of the python session and a browser session, turns out I have a cookie missing, namely
__cf_bm.
Is there a way to generate that specific cookie? Obviously, I can try using selenium, but that's way too much of a small task. I don't know if I should be writing the name of the website, I'm open to questions and suggestions. Thanks