I have a script in Playwright 1.23.1 with Python 3.10.2 running in Ubuntu LTS 20.4 to extract data from a page, the scripts run for about an hour because it is a lot of data. The problem is that sometimes the execution freezes and I can't catch it and I have to give the SIGINT sign (CTRL + C) to stop the execution. I don't know if it is a problem of Playwright or it is of the page that it is browsing. This page is really not that good, it's slow and have some problems in general.
My code is something like this:
from playwright.sync_api import TimeoutError as PlaywrightTimeoutError
from playwright.sync_api import Error as PlaywrightError
with sync_playwright() as spw:
# connect to the page
browser, page = connect(spw)
# sign in with credentials
sign_in(username, page)
data_code = 1
while True:
try:
# Extract the data from the page using playwright methods
# including select_option, frame_locator, evaluate, click, and reload
end_signal = extract_data(page, data_code)
if end_signal:
break
except (TimeoutError, PlaywrightTimeoutError, PlaywrightError) as e:
print(f"Error is: {e}")
page.reload(timeout=90000, wait_until='domcontentloaded')
data_code += 1
print("Exit cycle")
That's the basic structure of my code, the problem is that sometimes when trying to use click, select_option, or reloading the execution freezes. There are some prints inside extract_data but they don't appear, and sometimes print(f"Error is: {e}") does not print, so it does reach it, therefore, my conclusion is that the execution is freezed.
I have researched a little about Playwright debug, but in general I can't find the problem, and this error happens sometimes not always, so I can't repliacte it for a proper debug with playwights. The logs at least let me identify where it freezes, but nothing else so far.
Hope someone can help me, thanks.