Extract media files from cache using Selenium

Question

I'm trying to download some videos from a website using Selenium.

Unfortunately I can't download it from source cause the video is stored in a directory with restricted access, trying to retrieve them using urllib, requests or ffmpeg returns a 403 Forbidden error, even after injecting my user data to the website.

I was thinking of playing the video in its entirety and store the media file from cache.

Would it be a possibility? Where can I find the cache folder in a custom profile? How do I discriminate among files in cache?

EDIT: This is what I attempted to do using requests

import requests


def main():

    s = requests.Session()

    login_page = '<>'
    login_data = dict()
    login_data['username'] = '<>'
    login_data['password'] = '<>'

    login_r = s.post(login_page)

    video_src = '<>'

    cookies = dict(login_r.cookies) # contains the session cookie

    # static cookies for every session
    cookies['_fbp'] = 'fb.1.1630500067415.734723547'
    cookies['_ga'] = 'GA1.2.823223936.1630500067'
    cookies['_gat'] = '1'
    cookies['_gid'] = 'GA1.2.1293544716.1631011551'
    cookies['user'] = '66051'

    video_r = s.get(video_src, cookies=cookies)
    print(video_r.status_code)



if __name__ == '__main__':
    main()

The print() function returns:

403

This is the network tab for the video:

user_14888730 · Answer

Regarding video_r = s.get(video_src, cookies=cookies) Have you try to stream the response ? which send correct byte-range headers to download the video. Most websites prevent downloading the file as "one" block.

with open('...', 'wb') as f:
     response = s.get(url=link, stream=True)
            
     for chunk in response.iter_content(chunk_size=512):
         if chunk:  # filter out keep-alive new chunks
             f.write(chunk)

You can send a head request before if you want, in that way you can create a progress bar, you will retrieve the full content length from header.

Also a 403 is commonly use by anti-bot system, may be your selenium is detected.

user_16894364 · Answer

You blocked because you forgot about headers. You must use:

s.get('https://httpbin.org / headers', headers ={'user-agent': })

or:

s.headers.update({'user-agent': })

before sending a request

Extract media files from cache using Selenium

2 Answers