I am trying to submit a pyspark job from the terminal after SSh-ing into the AWS EMR master node. The .py file is on the s3 bucket.
I tried -
spark-submit s3://mybucketname/sparkapplication.pyspark-submit --master yarn --deploy-mode cluster --jars /home/hadoop/mysql-connector-java-5.1.45/mysql-connector-java-5.1.45-bin.jar --driver-class-path mysql-connector-java-8.0.19.jar s3://eeslemrtest/mysql_spark.py
Both of the above spark-submits throw the same error. Is there some other method to submit a code that's uploaded on the s3?
EDIT- The s3 bucket is not open to all traffic. I have a Access Key ID and a Secret Access Key. I'm guessing I'll have to use those to gain access to my s3 bucket but how do I do it while I'm submitting a spark job?