AWS Glue creates struct when long is present

I have a Glue job that is reading a bunch of JSON files from S3, creates a DynamicFrame and writes the output on another S3 bucket.

From those files I am selecting a field id. Up until recently, that was a small number and could fit into a Spark IntegerType (max: 2147483647). Now I am starting to see larger numbers that should be of type LongType.

Creating the DynamicFrame in Glue:

data_source = glueContext.create_dynamic_frame.from_options(
    connection_type="s3",
    connection_options={"paths": paths},
    format='json',
    transformation_ctx='data_source',
    format_options={"jsonPath": "$[*]"}
)

Reading the selected fields:

df = SelectFields.apply(frame=data_source, paths=['id', ...])
df.toDF().printSchema()

The problem: Glue creates a struct instead of a long and that messes the whole job.

root
 |-- id: struct (nullable = true)
 |    |-- int: int (nullable = true)
 |    |-- long: bigint (nullable = true)

Example: for an id with value 5.000.000.000.000 the struct is [, 5000000000000] and for an id with value 1.000.000 the struct is [1000000,].

Question: how do I prevent Glue for doing this? I want to parse all to a long type.

Thanks!

Update

It seems that some records were processed even if the value of the id was clearly long. So it seems that not always (?) Glue creates the struct.

Another update

It seems that it's an expected behavior of Glue whenever it feels it cannot determine the real data type. That's why there is this class ResolveChoice to allow you to select a default value. https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-crawler-pyspark-transforms-ResolveChoice.html

Using this class to choose the type I want solved my issue. For example if I have an id = [long=None, int=10] or id = [long=10, int=None] (for the sake of the example), using df = ResolveChoice.apply(frame = raw_df, choice = "cast:long") will give me for both cases id=10 and type long.

Hope this helps others! ;)