I was trying to solve a problem related to PySpark Dataframe operations but every time it used to fail certain test cases. So I decided to check one by one and I found that when the data was retrieved from the parquet file that was created during the solution, then it is not in the order in which it was saved. I am adding my code below and also attaching the screenshots of failure causes and data frames. Please help with this.`# Put your code here
from pyspark.sql import SparkSession
from pyspark.sql import *
from pyspark.sql.types import *
from pyspark.sql.functions import *
spark = SparkSession.builder.appName("Netflix analysis").config("spark.some.config.option", "some-value").getOrCreate()
columns=["ID","Name","Age","Area of Interest"]
data=[("1","Jack",22,"Data Science"),("2","Luke",21,"Data Analytics"),("3","Leo",24,"Micro Services"),("4","Mark",21,"Data Analytics")]
rdd = spark.sparkContext.parallelize(data)
df=rdd.toDF(columns)
df.coalesce(1)
df.show()
dfnew=df.describe("Age")
dfnew.write.parquet("Age")
dfnew.coalesce(1)
dfnew.show()
df2=df.select("ID","Name","Age").orderBy("Name",ascending=False)
df2.write.parquet("NameSorted")
df2.coalesce(1)
df2.show()
parDF1=spark.read.parquet("NameSorted")
parDF2=spark.read.parquet("Age")
parDF1.show()
parDF2.show()
This was the error shown to me
enter image description here ` This is what I got when I tried to check what was happening. The first table is the one that I saved and the second table is the when which i got when i read it from the saved parquet file.
