I am new to the whole Kubernetes thing. Is there a way for the Kubernetes cluster to call a lambda function with the job parameters once it is done with the execution of the job?
The idea is to not poll the cluster from time to time to get the status of the job.
There are multiple ways you can do this.
The easiest way I can think of is to call the AWS Lambda trigger directly from your job, one example is to have the AWS CLI installed in your container and use an IAM role tied to a Kubernetes services account. Then after your job is finished you can call:
$ aws lambda invoke \
--function-name my-job-function \
--payload '{ "yourkey": "yourvalue" }' \
myresponse.json
You could tie it all with a single script where you call your job first and then the AWS CLI command in the containers.
Another option is to use a separate pod that runs a service that monitors the state of your Kubernetes job, but this would be polling. For example, you can poll your K8s job(s) with something like kubectl or directly the K8s API and if it sees that it's status Success: 1 (kubectl get job <job-name> o=yaml) then it calls the AWS CLI (or AWS API) to trigger the lambda function.
The gist is basically, there are several ways of doing it and it will depend on your type of job/opinion that will best fit your needs.