In pyspark how to check the format a pyspark was read in? Delta vs parquet

user_715858 0
Viewed 276

I have function that reads in file which could either be in delta or parquet format.

def getData(filename,fileFormat)
      if data_format == "parquet":
        return spark.read.parquet(filename)
      elif data_format == "delta":
        return spark.read.format("delta").load(filename)

I then use the returned pyspark.sql.dataframe.DataFrame in some analysis function

def someAnalyticalFunction(df)
  if df == "parquet":  #I know this isnt right but how do i check the data format?
     #do some analysis
  elif df == "delta"
     #do some slight different analysis

Is there away that i can check in the analysis function, what format the dataframe (df) the was read in?

pyspark azure-databricks apache-spark-sql python
1 Answers

You can't do that with Spark, but you can use dbutils.fs to check if delta metadata file exists

user_34415100
Related