I am trying to deploy and orchestrate an ETL pipeline within Kubernetes.
I have a source file of xml documents from various sources that are processed and tagged with an NLP model. The model is available to me as a docker image.
The challenge I am facing in a Kubernetes deployment. Depending upon the size of the files I want to split them in batches and spin up multiple pods in parallel.
How can I specify a Kubernetes deployment where we can spin up multiple pods depending upon the number of folders in an S3 location with each pod processing all files with the folder it is assigned?
Regards,