I need all the core nodes on an EMR to contain a keystore file in the /usr/local/spark/conf/ directory. This particularly becomes challenging when core node resizing goes on because any newly brought up core node will not have the keystore file and won't even have the /usr/local/spark/conf/ directory. I need to automate the process of populating this directory with the keystore file on any newly brought up core node.
I've created a shell script to create the /usr/local/spark/conf directory and then populate that with the keystore file by fetching it from Amazon S3. The problem is getting this shell script to automatically run on any newly brought up EMR core node.
mkdir -p /usr/local/spark/conf/
cd /usr/local/spark/conf/
aws s3 cp s3://my_bucket/certs/cacerts .
aws s3 cp s3://my_bucket/certs/keystore.jks .