Azure Databricks - Reading Parquet files into DataFrames

user_85952 0
Viewed 1898

Am newbie with Python ... trying to read parquet files from Databricks, but when the file is empty is throwing error. How can i check filesize before reading it into DataFrame. Code below:

%python

##check if file is empty ???
##if not empty read
##else do something else

try:
   parquetDF =              
   spark.read.parquet("wasbs://XXXXX@XXXX.blob.core.windows.net/XXXX/2019-10- 11/account.parquet")
except:
   print('File is Empty !!!')
parquet azure-databricks dataframe python
1 Answers

For now am doing handing this as below 

%python
import pandas as pd
data = {
    'Dummy': ['Dummy'], 
}
parquetDF = pd.DataFrame(data)
try:
  parquetDF = spark.read.parquet("wasbs://XXXXX@XXXXX.blob.core.windows.net/XXXXX/2019-10-11/account.parquet")
except:
  print('Empty File!!!')
if (parquetDF.columns[0] == 'Dummy'):
  print('Do Nothing !!!!')
else:
  print('Do Something !!!')

Creating Dummy DataFrame, then trying to load the DataFrame with parquet Data. If any exceptions / source file is empty DF will not be loaded. Then check if the DF is loaded or not and process accordingly.

Also tried to read filesize, but getting exception 'No such file or directory'

%python
import os
statinfo = os.stat("wasbs://XXXXX@XXXXX.blob.core.windows.net/XXXXX/2019-10-11/account.parquet")
statinfo
user_859520
Related