I first want to thank you everyone for taking a look at my question here. Im a newbie to the automation world and I have a task to automate a process from a client. I will first tell you the task and then the questions I have. To be truthful, I'm trying to get some ideas on how I can do this with least cost involved to the client and since this is my first time running a real use case through containers and cloud please forgive me if im asking the wrong questions here.
Task at hand
there are zip files created in a azure blob storage which should be imported to a legacy application in windows and by using RPA (power automate or we can use a suitable other RPA software) we unzip and import the file, then click some buttons in the application to generate 30 different csv files. at the end these files are zipped back and put into another azure blob with a naming convention. so to scale this task, we have to have x number of vms available to process each zip file.
What I researched so far
having 20 (or x number of) idle machines in cloud and use a webhook to check the blob storage and start a machine to run the power automate flow upon having a zip file is easy to develop but not scalable.
Then I thought about creating a docker image with the application installed and having the startup changed to run the power automate flow. which then on starting the container, will run the power workflow and do the necessary and at the end will shutdown the container after storing the zip file in the azure blob. I can scale it via a service(which service i'm not sure) that will spin up x number of containers depending on the number of zip files available. the client thinks the zip count will be 10 for the moment and will increase to 100 in the future.
I really like the second option. But I want your expert opinion on this before I start to work.
- what services would you use differently?
- Is there a better way?
- What should I pay attention to?
- How to make sure 2 containers wont import the same zip file?
- What do you think about the docker image? will it be too big?
- Cost wise is there a better option?
NOTE : im more into azure so appreciate if you can point me to services in azure for this.