I have a process that I am currently running on EC2.
This is part of my web application, and I am relying on Elastic Load Balancer to automatically spin up more instances as more users need to run the process.
The thing is, each process execution has different context, therefore, every process execution uses different amount of memory in the instances they are running.
The memory usage can widely vary, some executions using only a couple GB, while some executions require up to 100GB of memory in extreme cases.
Without knowing up-front how much memory each execution would use, how do I pick the right EC2 instance type(s) to run my process?
Currently, I am just using one of the most powerful instance types, just to be able to accommodate the executions that use up a lot of memory.
This is a problem because these executions make up only a marginal portion of all executions and thus my instances are rarely using the full amount of memory available to them, wasting a lot of money just to keep them running.
What are some strategies I can consider to optimize my process execution where the memory usage can vary significantly?