awsglue.utils.GlueArgumentError: argument --JOB_NAME is required

Viewed 9798

When I run the Glue job boilerplate in AWS Glue using Python, I get an error

import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job

glueContext = GlueContext(SparkContext.getOrCreate())
spark = glueContext.spark_session

args = getResolvedOptions(sys.argv, ['JOB_NAME'])
job = Job(glue_context)
job.init(args[‘JOB_NAME’], args)

Here's the error

in getResolvedOptions
File "/usr/lib64/python2.7/argparse.py", line 1733, in parse_known_args
namespace, args = self._parse_known_args(args, namespace)
File "/usr/lib64/python2.7/argparse.py", line 1957, in _parse_known_args
self.error(_('argument %s is required') % name)
awsglue.utils.GlueArgumentError: argument --JOB_NAME is required

This follows my code and then

job.commit()

But AWS says do not provide JOBNAME. What am I doing wrong? And also what does getResolved do? I tried reading AWS documentation but couldn't get anything.

5 Answers

Can you provide the reference link for the sample script. I am sure AWS would have explained that you should start job using a script or a lambda, etc. In such case you would specified something as follows:

response = client.start_job_run(
             JobName = 'my_test_Job',
             Arguments = {
               '--day_partition_key':   'partition_0',
               '--hour_partition_key':  'partition_1',
               '--day_partition_value':  day_partition_value,
               '--hour_partition_value': hour_partition_value } )

Then in your glue job, you can access these variables as follows:

args = getResolvedOptions(sys.argv,
                          ['JOB_NAME',
                           'day_partition_key',
                           'hour_partition_key',
                           'day_partition_value',
                           'hour_partition_value'])

So, somewhere you have pass the variable/value for job name. Wise, try to run the job by setting a parameters by name JOB_NAME and pass a value.

Reference for above code sample : https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-crawler-pyspark-extensions-get-resolved-options.html

Let us know.

This may be a late reply, but solved same error for me. To pass parameters to Glue script: pass job parameters while adding/editing job inside Security configuration, script libraries, and job parameters (optional) --> Job parameters (key value), for example:

key = --JOB_NAME    value= my_job
key = --input       value= 'xyz'

To use the parameter in the Glue script:

import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job

## @params: [job_name]
sc = SparkContext()
glueContext = GlueContext(SparkContext.getOrCreate())
spark = glueContext.spark_session
job = Job(glueContext)
args = getResolvedOptions(sys.argv, ['JOB_NAME','input'])
job.init(args['JOB_NAME'], args)
print("Job name is:{}, and input: {}".format(args["JOB_NAME"],args["input"]))

job.commit()

also as per my understanding, it has nothing to do with dev environment, also while adding/editing Glue job in dashboard parameters can be passed without creating lambda function.

You cannot run the following command in dev environment.

args = getResolvedOptions(sys.argv, ['JOB_NAME'])

I have stumbled upon this solution while having a similar issue as the OP, but with the AWS Glue Python Shell Job. Swati's answer above helped.

If someone is trying to pass parameters to AWS Glue Python Shell Job, below is a template that worked for me:

key = --JOB_NAME    value= my_job
key = --input       value= 'xyz'

AWS Glue Python Shell Job script:

import sys
from awsglue.utils import getResolvedOptions

args = getResolvedOptions(sys.argv, ['JOB_NAME','input'])

print("Job name is:{}, and input: {}".format(args["JOB_NAME"],args["input"]))
Related