pySpark check if dataframe exists

user_12368632 0
Viewed 4731

Is there a way to check if a dataframe exists in pySpark?

I know in native python, to check if dataframe exists: 

exists(df_name) && is.data.frame(get(df_name))

How can this be done in pySpark? Since command exists throws an error.

pyspark dataframe exists python
3 Answers

It is same as Petel code. You can import the dataframe type.

 from pyspark.sql import DataFrame

 df= sc.parallelize([
 (1,2,3), (4,5,7)]).toDF(["a", "b", "c"])

 if df is not None and isinstance(df,DataFrame):
      #<some operation>
      print("dataframe exists")
user_86658430

try this: df_name is not None and isinstance(df_name, DataFrame)

user_48588600

I think you want to know if df_name is defined and pointing to a DataFrame. None of the answers above handle the case where df_name is not set. This does:

from pyspark.sql import DataFrame

try:
  if df_name is not None and isinstance(df_name,DataFrame):
    print('df_name exists')

except NameError as error:
  print('df_name does not exist and not defined')
user_30411280
Related