Page is crashing from tab crash when calling multiple APIs - Selenium with flask on heroku

Viewed 157

I am new to python flask with selenium in use and I have hosted the Flask API on Heroku that scrapes my college's website to retrieve attendance data and return it. The API has been hosted using the Gunicorn WSGI and the API is working as expected. But when multiple API calls are there at the same time, it gives an error that says that the page has crashed from tab crash.

ERROR in app: Exception on /attend [POST]
Traceback (most recent call last):
File "/app/.heroku/python/lib/python3.9/site-packages/flask/app.py", line 2073, in wsgi_app
response = self.full_dispatch_request()
File "/app/.heroku/python/lib/python3.9/site-packages/flask/app.py", line 1518, in full_dispatch_request 
rv = self.handle_user_exception(e)
File "/app/.heroku/python/lib/python3.9/site-packages/flask/app.py", line 1516, in full_dispatch_request
rv = self.dispatch_request()
File "/app/.heroku/python/lib/python3.9/site-packages/flask/app.py", line 1502, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
File "/app/app/main.py", line 29, in attend
driver.get("https://academia.srmist.edu.in/#View:My_Attendance")
File "/app/.heroku/python/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 436, in get
self.execute(Command.GET, {'url': url})
File "/app/.heroku/python/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 424, in execute
self.error_handler.check_response(response)
File "/app/.heroku/python/lib/python3.9/site-packages/selenium/webdriver/remote/errorhandler.py", line 247, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: session deleted because of page crash
from tab crashed
(Session info: headless chrome=97.0.4692.99)

The first API call crashes but the second API call succeeds and returns the data.

The starting code for the function is:

app = Flask(__name__)
app.config['JSON_SORT_KEYS'] = False
import os

os.environ['DISPLAY'] = ':0'

@app.route('/attend',methods=["POST"])
def attend():
    options = webdriver.ChromeOptions()
    options.add_argument('--disable-dev-shm-usage')
    driver = webdriver.Chrome(ChromeDriverManager().install())
    input_json = request.get_json(force=True)
    id = input_json["id"]
    password = input_json["pass"]
    driver.get("https://academia.srmist.edu.in/#View:My_Attendance")
...

I have tried adding chrome options like "--disable-dev-shm-usage" and "--no-flags" but that doesn't seem to have any effect on the tab crash whatsoever. I want the API to be able to have multiple calls and send data to all of them even when they are called simultaneously. I also kept in mind using a selenium grid but I was unsure whether it was going to work.

The build packs I am using for the Heroku app are:

https://github.com/heroku/heroku-buildpack-chromedriver
https://github.com/heroku/heroku-buildpack-google-chrome
1 Answers

I encountered the SAME exact error (your question is the only other instance of it I can find online) and it's been driving me to insanity the last few days. I solved it by creating a new, separate, driver instance for each .get call and closing it after usage before the next call.

Can't say much for if it'll suit your needs, but Heroku stopped throwing all exceptions.

Related