I created a glue crawler to get a table schema from compressed json files in s3 bucket. And used a glue job to read the compressed files, transform the data and write into DynamoDB. After it wrote some data into table, it throws "AmazonDynamoDBException: The provided key element does not match the schema". I think it might be something wrong with keys mapping. The table schema that crawler gets is like:
{
"item": {
"clockID": {
"N": "string"
},
"ID": {
"S": "string"
},
"retryTime": {
"NULL": "boolean",
"S": "string"
},
"updatedTime": {
"S": "string"
},
"createdTime": {
"S": "string"
},
"eventName": {
"S": "string",
"NULL": "boolean"
},
"eventState": {
"S": "string"
},
"events": {
"M": {
"0c26d418-b426-40be-a12d-e6fd6e63d81d": {
"M": {
"status": {
"S": "string"
}
}
}
},
"NULL": "boolean"
}
}
}
The Partition key of DynamoD table is ID and Sort key is clockID. Some of fields in the schema like retryTime, eventName, events have NULL or string value, but no idea on how to map this situation. Here is my mapping function:
Mapped = ApplyMapping.apply(frame = Source, mappings = [
("item.ID.S", "string", "ID", "string"),
("item.clockID.N", "string", "clockID", "long"),
("item.retryTime.S", "string", "retryTime", "string"),
("item.retryTime.NULL", "boolean", "retryTime", "string"),
("item.updatedTime.S", "string", "updatedTime", "string"),
("item.createdTime.S", "string", "createdTime", "string"),
("item.eventName.S", "string", "eventName", "string"),
("item.eventName.NULL", "boolean", "eventName", "string"),
("item.eventState.S", "string", "eventState", "string"),
("item.events.M", "string", "events", "map"),
("item.events.NULL", "boolean", "events", "map")],
transformation_ctx = "Mapped")
Links that I followed: Cross-account replication with Amazon DynamoDB Import Script