I am trying to install requirements.txt through "Add steps"(command-runner.jar) in EMR triggered through Airflow.
and it is giving and error Error:
Cannot run program "pip" (in directory "."): error=2, No such file or directory
My Airflow Code :
SPARK_STEPS = [
{
'Name': 'Copy S3 to EMR',
"ActionOnFailure": "CANCEL_AND_WAIT",
'HadoopJarStep': {
"Jar": "command-runner.jar",
"Args": ['aws' ,'s3', 'cp' ,'s3://datahub-pyspark/pitchbook/', '.', '--recursive'],
},
},
{
'Name': 'Upgrade Pip',
"ActionOnFailure": "CANCEL_AND_WAIT",
'HadoopJarStep': {
"Jar": "command-runner.jar",
"Args": ['pip3','install','--upgrade','pip'],
},
},
{
'Name': 'Execute Requirements',
"ActionOnFailure": "CANCEL_AND_WAIT",
'HadoopJarStep': {
"Jar": "command-runner.jar",
"Args": ['pip','install','-r','requirements.txt'],
},
},
{
'Name': 'Spark-Submit Command',
"ActionOnFailure": "CANCEL_AND_WAIT",
'HadoopJarStep': {
"Jar": "command-runner.jar",
"Args": [
'spark-submit', '--py-files', 'config.zip,jobs.zip', 'main.py']
,
},
}
]
Name': 'Execute Requirements' is failing so I have 'Upgrade Pip' also.Please give me how to remove this error?
Thanks, Xi