I've built a web scraper in Python that I'm trying to put on an AWS Lambda function. Right now, I'm using the requests-html package to get the rendered html. I've created a Lambda Layer that installs all the necessary packages, but when calling the render() method from requests-html, it installs Chromium to a directory that is read-only on AWS Lambda. I've also looked into using Selenium + urllib3, but this requires updating the PATH environment variable for the Lambda configuration in a way I'm not sure I could do.
So my question is, how best to get rendered HTML for an AWS lambda function? Should I be using javascript/node.js instead of Python?