AWS Glue Scala Job (from S3 bucket) throws ClassNottFound

user_2379259 0
Viewed 1171

I'm trying to follow the AWS Glue documentation to develop a scala program and create a new Glue Job. Following have been my steps so far

  1. Built a sample Scala program as guided by https://docs.aws.amazon.com/glue/latest/dg/glue-etl-scala-example.html.

  2. Bundled the scala main class into a jar-with-dependencies assembly file and uploaded it to S3 under a /bin folder

  3. Launched AWS Glue Service on AWS Management Console

  4. Under "jobs" clicked on Add Job and setup the following

    • Name:
    • IAM Role : Role that has access to S3, Glue, etc
    • Type: Spark
    • Glue Version: Spark 3.1, Scala 2 (Glue Version 3.0)
    • This job runs as : "An existing Script that you provided"
    • Script file Name: FQCN for the scala main class
    • S3 path where the script is stored : s3 link to the jar with dependencies jar file
    • Temporary directory: did not change

  5. Click on "Next"

  6. Save the job and clicked on Run job

  7. After a while it shows the following error in the cloudwatch logs

2021-12-14 02:24:50,558 ERROR [main] glue.ProcessLauncher (Logging.scala:logError(73)): User class 'xxx.xxx.xxxx.GlueJob' is not initialized. java.lang.ClassNotFoundException: xxx.xxx.xxxx.GlueJob

where am I going wrong?

aws-glue apache-spark
1 Answers

I was able to go past the above error and run scala job on glue. To answer the question above, I would say it was more of my understanding of how glue executes scala. Outside of AWS, a spark job lifecycle follows the following steps

code development -> compile -> bundle with libraries -> deploy -> invoke main class

The exception with glue is the first step of COMPILE ! Glue expects the following

  1. The main class is in a separate file (default GlueApp.scala) on s3
  2. All the relevant libraries and files are accordingly passed in "Security configuration, script libraries and job parameters" section enter image description here

When the job is started, Glue compiles the main class with all the relevant libraries and any external libraries passed to it.

The learning for me has been NOT TO BUNDLE the main class into the jar.

If there is any other better way, please feel free to add. Thanks

user_23792590
Related