How can I generate the same UUID for multiple dataframes in spark?

user_1864552 0
Viewed 215

I have a df that I read from a file

import uuid

df = spark.read.csv(path, sep="|", header=True)

Then I give it a UUID column

uuidUdf= udf(lambda : str(uuid.uuid4()),StringType())
df = df.withColumn("UUID",uuidUdf())

Now I create a view

view = df.createOrReplaceTempView("view")

Now I create two new dataframes that take data from the view, both dataframes will use the original UUID column.

df2 = spark.sql("select UUID from view")
df3 = spark.sql("select UUID from view")

All 3 dataframes will have different UUIDs, is there a way to keep them the same across each dataframe?

pyspark azure-databricks python
1 Answers

Spark uses a lazy evaluation mechanism, where the computation is invoked when you call show or other actions. This means every time you call an action, the uuid is recalculated. To avoid this you need to cache the df before you call createOrReplaceTempView, here is what you should do

import uuid

df = spark.read.csv(path, sep="|", header=True)
uuidUdf= udf(lambda : str(uuid.uuid4()),StringType())
df = df.withColumn("UUID",uuidUdf())

df.cache()

view = df.createOrReplaceTempView("view")

df2 = spark.sql("select UUID from view")
df3 = spark.sql("select UUID from view")
user_96287930
Related