pyspark how to check if given spark dataframe has created with inferSchema=True

user_9850636 0
Viewed 485

I'm developing an API, and one of the functions should receive a spark data frame, and use the schema in the spark data frame to perform actions.

As there is no guarantee that the user will create the data frame and set the infer Schema to True, my function will receive a data frame that has no inferred schema, so I would like to throw an error if that occurs.

Can I check whether the data frame has been created with inferSchema=True? Or Is there a pyspark function that infer the schema after the data frame has been created?

Since inferSchema=False all the columns are StringType, I can't reject the data frame based on this, since this is a possible schema for some data frame.

pyspark apache-spark-sql
2 Answers

No that's not how it works. Spark dataframe always has a schema, no matter if the schema came from "inferred" or by defined. The only way to "suspect" a dataframe has inferSchema=False is – like you said – all columns are StringType. So one option here is your API can throw a warning instead of an error, if you see all columns are string

user_34415100

When inferSchema is set to false, Spark will read all columns with String Data Type. For instance it will also read ISO formatted Date column as String. But if its set to true, Spark is smart enough to determine correct data types.

So try inspecting the dataframe by doing dataFrame.printSchema() and check if the above condition holds.

user_93892930
Related