I am trying to use AWS Glue to crawl a data set and make it available to query in Athena. My data set is a delimited text file using ^ to separate columns. Glue is not able to infer the schema for this data as the CSV classifier only recognises comma (,), pipe (|), tab (\t), semicolon (;), and Ctrl-A (\u0001). Is there a way of updating this classifer to include non standard delimeters? The option to build custom classifiers only seems to support Grok, JSON or XML which are not applicable in this case.
1 Answers
You will need to create a custom classifier using the custom Grok pattern and use that in the crawler. Suppose your data is like below with four fields:
qwe^123^22.3^2019-09-02
To process the above data, your custom pattern will look like below:
%{NOTSPACE:name}^%{INT:class_num}^%{BASE10NUM:balance}^%{CUSTOMDATE:balance_date}
Please let me know if that worked for you.
Related