I understand that Spark supports partition discovery, where directories names follow fixed pattern of column_name=column_value. Example from the linked webpage:
path -> to -> table -> gender=male -> country=US -> data.parquet
I would like to use this functionality, but unfortunately the file structure that I'm currently working on does not follow this pattern. I can't change it and converting it is not feasible. In my case the directories are purely column values, as in:
path -> to -> table -> male -> US -> data.parquet
Ideally I would like to configure Spark to indicate that the sub-directories of "table" directory contain "gender", followed by "country" sub-directory and so on.
I'm using Spark 2.11 from Java. I'm also using Parquet files.