I am using, input_file_name() to get the s3 file paths from a sql, but it's returning null when the table contains parquet files. I tried the same thing on a table with csv files and single partition and this worked (Don't know if this could be a reason) If, so how should i proceed?
dynamic_frame1= glueContext.create_dynamic_frame.from_catalog(database = database_name, table_name = table_name)
dataframe1 = dynamic_frame1.toDF().withColumn("filename", input_file_name())
dataframe1.createOrReplaceTempView("some_temp")
sql = """select $PATH, * from some_temp """
paths = spark.sql(sql)