Is it possible to map out the dependencies between columns in PySpark?

Viewed 89

Let's say I write some pyspark that looks like

from pyspark.sql import SparkSession
import pyspark.sql.functions as F
spark = SparkSession.builder.master("local") \
            .appName("localapp") \
            .getOrCreate()
df = spark.createDataFrame([
   ('alice', 23),
   ('bob', 48)
  ],
  ['name', 'age']
)
new_df = (
  df
  .withColumn('new_name', F.lower(F.col('name')))
  .withColumn('double_age', F.col('age') * 2)
  .withColumn(
    'new_name_double_age',
    F.concat_wd(' ', F.col('new_name'), F.col('double_age').cast('string'))
  )
)

The dependencies between columns would be

name -> new_name
                 \
                  -> new_name_double_age
                 /
age -> double_age

Is there some attribute of new_df that I could leverage in order to construct the graph above?

0 Answers
Related