Hive column with special character in parquet

Viewed 109

I have some server logs data which contains a nested column with special character '/'. The data is generated in json format and I have convert it with spark to parquet format for analytics convenience.

When I am tring to create a table in hive, I got error as "FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.IllegalArgumentException: Error: : expected at the position 355 of '...' but '/' is found."

The key error for this is the nested column declaration, something as below

create external table parquet_test
(
    ...,
    kubernetes struct<
                ...
                labels:struct<app:string,region:string,`app_kubernetes_io/name`:string>,
                ...
                >,
    ...
) STORED AS PARQUET;

I know that in json serde with 'org.openx.data.jsonserde.JsonSerDe', I can set a mapping property in SERDEPROPERTIES with ('mapping.app_kubernetes_io_name' = 'app_kubernetes_io/name'), in which way the issue of column with special character '/' can be solved by mapping. I wonder if there is a similar way in parquet that I can use to solve this kind of problem. Or is there other way to solve this?

0 Answers
Related