I have a question regarding the usage of EFS as and additional memory location for lambda. I am using python along with pandas to perform some tests on my files. And it works great if the files are not that large, but if the files exceed 2-3 GB lambda dies because of the memory limitation (using both max memory and time of lambda). Files are originally located at S3 and I was wondering would it be possible to use EFS in this scenario? If so what would be required for this solution. Would I need to transfer files from S3 to EFS in order to open them?, or is there an better solution where I can directly load the files from S3 to EFS and open them with pandas. And also there is the timeout limitation, but I hope that wont be an issues if the lambda would be faster with EFS.
1 Answers
As far as I know pandas requires the whole file to fit into memory.
In principle you can fit larger files into memory in Lambda, since you can now configure Lambda functions with up to 10GB of RAM.
That's doesn't translate to you being able to read a 10GB file from S3 and create a dataframe out of it, because in order for pandas to parse the data, it needs to either be stored on disk (of which there's only 500MB available to you) or in memory.
If you download the file into memory, it also takes up it's size in system memory and then you can create a pandas data frame from that. Pandas data structures are probably larger than the raw bytes of the file, so my guess is that you can load a file from S3 into memory and turn that into a data frame that is about 30-40% of the memory capacity of the lambda function.
If you store that file on EFS, you'll be able to fit more into memory, because pandas can read the bytes from disk, so you could probably squeeze out a few more gigabytes. These I/O operations take time however and Lambda is limited to at most 15 minutes of runtime. You probably also need to write that data somewhere which takes time as well.
Bottom line: Loading larger files than that into Lambda is probably not a good idea. If you can, break up the dataset into smaller chunks and have lambda functions work on them in parallel or choose a service like Athena or EMR or Glue ETL, which are built to handle that stuff.
Related