Memory error while using selenium (Python)

user_14254196 0
Viewed 954

Recently, I'm trying to scrape a waterfall website through Selenium (request doesn't work and I can't get the API).

However, I came across a problem that I can't solve: Memory error.

Because using Selenium is really ram consuming, I had tried to appoint my USB function as cache through the following code

from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument(r'--user-data-dir=F:\OuoC')
options.add_argument('--incognito')
options.add_argument('--disable-gpu')
driver = webdriver.Chrome(executable_path=r'C:\Users\user\Desktop\chromedriver.exe',options = options)

Yet, this failed to work. When I open my USB, though I saw the files added by selenium, chrome driver still consumes my computer's built in ram, and the files in USB weren't refreshing either(I had pressed f5).

Besides, my Anaconda prompt also shown the following message:

[10320:11384:0202/231331.035:ERROR:device_event_log_impl.cc(211)] [23:13:31.035] USB: usb_device_handle_win.cc:1049 Failed to read descriptor from node connection: 
(0x1F)
[I 23:15:25.741 NotebookApp] Saving file at /play/ques.ipynb
selenium-webdriver selenium python
1 Answers
options.add_argument(r'--user-data-dir=F:\OuoC')

is not a step to set cache but to set the profile user-data , this has nothing to do with RAM . RAM is used to store data temporarly for faster processing as it supports faster input and output signals.

so,

   options.add_argument(r'--user-data-dir=F:\OuoC')

will just use your usb instead of your hard disk. You can use this method if your harddisk space is less.

For saving RAM few things you could do is ,

  1. Use readyboost in windows 10

https://answers.microsoft.com/en-us/windows/forum/windows_10-hardware-winpc/readyboost-for-windows-10/9aa6f86f-f77d-4d81-9853-8bae72a58098

  1. Reduce execution time between steps by adding sleep

Slowing down chrome usage frees' up RAM , when execution is fast it takes up more RAM

  1. Close chrome browser and restart browser in between your tests , use

This will clear RAM that is used by current chrome instance

Try these steps

user_67936370
Related