I am looking into building a customized Spark SQL CSV inferschema to identify custom datatypes (those that are not identified with the inferschema option). I am using a StringIndexer to get unique categorical values to identify the custom data types.
String[] categoricalStringValues = new StringIndexer().setInputCol(columnName).fit(dataframe).labels();
// Code to identify/guess the most likely custom datatype
// some examples: currency ($12.00), percent (24.05%) etc
The process is painfully slow. I have found the CSVInferSchema in scala code. Is there a way in which this class can be invoked or used or interpreted to develop a code to identify custom datatypes?