How do I configure an XML crawler for AWS Glue?
I receive 20 data files from a govt source each week. They are about 500Mb each.
The XSD, and an examination of a file, shows that they are quite complex. The main element type (i.e. direct descendant of the root element) has required elements, optional elements, repeatable elements, etc. Each of these child elements itself can have multiple levels of descendents.
I created and ran a crawler with the default settings. The resulting table includes a column definition something like this:
"cols": {
"FieldSchema": [
{
"name": "nametext",
"type": "string",
"comment": ""
},
{
"name": "_type",
"type": "string",
"comment": ""
}
]
},
The nametext element is just one of the child types in the XSD. I don't know why Glue picked it up. More importantly, shouldn't Glue have picked up much more?
I tried configuring a classifier with type = XML, setting the row tag to the type of the direct descendants of my root element, but that did not change the output.
I see that I can edit the schema manually, but that seems like recreating the wheel, given that I have an XSD, and the crawler is supposedly able to imply a schema from a file. I also ran the crawler against a whole directory of similar files but the result was much the same.