AWS Glue Job doesn't recognize columns

Question

I have a Job in AWS Glue that tries to do an ETL with a database (PostgreSQL RDS) and load the data extracted to S3, but when i run the job it returns an Error saying the columns don't exist.

Error:

An error occurred while calling o97.getDynamicFrame. ERROR: column "id_banner" does not exist.

I have already make the connection with the RDS.

All the columns are in Upercase and i can't rename it in the database because it is not Scalable in my company (So many tables and the production application already consumes these tables).

Here is my script:

import sys
import os
import logging
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
from pyspark import SparkConf

## @params: [JOB_NAME]
args = getResolvedOptions(sys.argv, ['JOB_NAME'])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)

logging.basicConfig(level=os.environ.get("LOGLEVEL", "INFO"),
                    format='[%(filename)s:%(lineno)s - %(funcName)10s() ] %(asctime)-15s %(message)s',
                    datefmt='%Y-%m-%d:%H:%M:%S')
logger = logging.getLogger(__name__)

logger.info('Create dynamicframe by reading data from admin.article table in Hub')
db_username = "xxxxxxxx"
db_password = "xxxxxxxx"
table_name = "TB_BANNER"

#define connection options
connection_options = {  
    "url": "jdbc:postgresql://xxxxxxxxxxx.amazonaws.com:5432/xxxxxxxx",
    "dbtable": table_name,
    "user": db_username,
    "password": db_password,
}

#glue dynamicframe
datasource0 = glueContext.create_dynamic_frame_from_options(
    connection_type="postgresql", 
    connection_options=connection_options,
    transformation_ctx = "datasource0"
    )


logger.info('Writing data to S3')
dataSink0 = glueContext.getSink(
    path = "s3://glue-etl/tb_banner/", 
    connection_type = "s3", 
    updateBehavior = "UPDATE_IN_DATABASE",  
    partitionKeys = [], 
    compression = "snappy", 
    enableUpdateCatalog = True, ##Data Catalog is to be updated during the job run
    transformation_ctx = "DataSink0"
)

dataSink0.setCatalogInfo(catalogDatabase = "HubdeSaude", catalogTableName = "TB_BANNER") ##create a new table in Glue catalog

dataSink0.setFormat("glueparquet")

dataSink0.writeFrame(datasource0)

logger.info('Completed writing data to S3')


job.commit()

Note: The column shown (id_banner) in the Error message is a Primary Key columns and is UpperCase in the database.

AWS Glue Job doesn't recognize columns

0 Answers