Let's say I write some pyspark that looks like
from pyspark.sql import SparkSession
import pyspark.sql.functions as F
spark = SparkSession.builder.master("local") \
.appName("localapp") \
.getOrCreate()
df = spark.createDataFrame([
('alice', 23),
('bob', 48)
],
['name', 'age']
)
new_df = (
df
.withColumn('new_name', F.lower(F.col('name')))
.withColumn('double_age', F.col('age') * 2)
.withColumn(
'new_name_double_age',
F.concat_wd(' ', F.col('new_name'), F.col('double_age').cast('string'))
)
)
The dependencies between columns would be
name -> new_name
\
-> new_name_double_age
/
age -> double_age
Is there some attribute of new_df that I could leverage in order to construct the graph above?