Spark createDataFrame is getting stuck (during spark-submit)

user_13548763 0
Viewed 227

I am trying to create a spark dataframe from a list. The list has 1 or 2 elements So far I tried the 4 methods below, but strangely my application is getting stuck at spark.createDataFrame(). Can someone please point out what I am doing wrong?

Update: These methods are working when tried in pyspark shell, but getting stuck during spark-submit

Method 1

schema = StructType([
StructField('TABLE_NAME', StringType(), True)
])
rdd = spark.sparkContext.parallelize(table_name_list)
df = spark.createDataFrame(rdd,schema)
df.show(truncate=False)

Method2

column = ["TABLE_NAME"]
df = spark.createDataFrame(data=table_name_list, schema = column)
df.show(truncate=False)

Method3

df = spark.createDataFrame([table_name_list], StringType())
df.show(truncate=False)

Method 4

pd_df_tb_nm=pd.DataFrame({'TABLE_NAME':table_name_list})
print(pd_df_tb_nm) #printing pd df data
df_tb_nm=spark.createDataFrame(pd_df_tb_nm) 
df.show(truncate=False)
pyspark pandas apache-spark dataframe python
2 Answers

Here is an simple example of creating a dataframe from a single list of strings

The following should work when calling the script using spark-submit.

from pyspark.sql import types

def main():

    arr = ["apple", "banana", "cherry", "pear","grape"]
    arr = [[x] for x in arr] 

    arr_schema = (types.StructType([
            types.StructField('TABLE_NAME', types.StringType()),
            ]))

    df = spark.createDataFrame(data, schema=arr_schema)
    df.show()

if __name__ == '__main__':
    spark = SparkSession.builder.appName('example code').getOrCreate()
    spark.sparkContext.setLogLevel('WARN')
    sc = spark.sparkContext
    main()
user_147222970

This issue is coming only when I set master as YARN using config("spark.master", "yarn"). In local mode all of the above methods and the one suggested here are working seamlessly)**

user_135487630
Related