HadoopDataSource: Skipping Partition {} as no new files detected @ s3:

user_14549398 0
Viewed 700

So, I have an S3 folder with several subfolders acting as partitions (based on the date of creation). I have a Glue Table for those partitions and can see the data using Athena.

Running a Glue Job and trying to access the Catalog I get the following error:

HadoopDataSource: Skipping Partition {} as no new files detected @ s3:...

The line that gives me problems is the following:

glueContext.getCatalogSource(database = "DB_NAME", tableName = "TABLE_NAME", redshiftTmpDir = "", transformationContext = "datasource0").getDynamicFrame().toDF()

I'll want at every point to access all the data in those S3 subfolders as it is updated regularly.

I'm thinking the problem is the Glue Job Bookmark not detecting new files, but this is not running directly as part of a Job but as part of a library used by a Job.

Removing "transformationContext" or changing its value to empty hasn't worked.

aws-glue-data-catalog aws-glue apache-spark scala
1 Answers

So the Hadoop output you are getting is not an error but just a simple log that the partition is empty.

But the partition that is getting logged, {}, seems to be off. Can you check that?

In addition, could you run the job with bookmark disabled, to make sure that this is not the cause of the problem?

I also found this unresolved GitHub issue, maybe you can comment there too, so that the issue gets some attention.

user_126381180
Related