Is there a way to run bootstrap action after application software (eg: spark) is installed? My purpose is to replace one of the AWS spark jar with my customized spark jar and this must be done after AWS install it original spark.
The reason I don't do this via normal step is because I want to perform this replacement on all nodes of the cluster, not just on the master node.
Though this question is old, I got it to work and adding answer for people who still need help with this. As in comment above, my configuration with which its tested is:
Release label:emr-5.30.1 Hadoop distribution:Amazon 2.8.5 Applications:spark 2.4.5
The issue really is that Bootstrap on AWS EMR gets executed BEFORE your software like Spark are installed.
Solution: Do it in two steps
Add a bootstrap script to run your second script as a background process. This completes the bootstrap as well as gives you more ability to WAIT for installations to happen. Following example script takes two parameters, S3 bucket path and environment prefix and shows how they are passed along to the next script.
#!/bin/bash
aws s3 cp $1/$2/some/path/to/second/script/emr_second_script.sh .
chmod +x emr_second_script.sh
nohup ./emr_second_script.sh $1 $2 &>/dev/null &
Write your second script, that first one is running in background. In this script check for the installation directory to appear otherwise wait/sleep. Once that condition is met, do your required magic. Following example script takes two parameters, S3 bucket path and environment prefix.
#!/bin/bash
while [ ! -e /lib/spark/jars/ ]
do
echo "Spark installation not found, waiting..."
sleep 15
done
echo "Spark installation found."
# Do your actual scripting work here. e.g. Copy jars from S3.
enter code here
sudo aws s3 cp $1/$2/path/to/my/jar/my-jar-1.0.jar --region us-east-1 /lib/spark/jars
exit(0)
You can find the same solution to solve slightly different problem here.