How does AWS Athena react to schema changes in S3 files?

user_2583604 0
Viewed 2982

What happens when after creating the table in AWS Athena for files on S3, the structure of the files on S3 change?

For eg:

  1. If the files previously had 5 columns when the table was created and later the new files started getting 1 more column: a) at the end? b) in between?

  2. What happens when some columns are not available in new files?

  3. What happens when the columns remain the same but the column order changes?

  4. Can we alter Athena tables to adjust to these changes?

amazon-athena amazon-web-services amazon-s3 schema
2 Answers

It depends on the files format you are using and the setup (if the schema is by field order or by field name). All the details are here: https://docs.aws.amazon.com/athena/latest/ug/handling-schema-updates-chapter.html Take a big note that if the data is nested or in arrays, it will completely break your data, to quote from this page:

Schema updates described in this section do not work on tables with complex or nested data types, such as arrays and structs.

user_55517670
Related