Im having a ETL pipeline written in pyspark. I can autogenerate a DAG for this pipeline. However a DAG does not state anything about primary keys and entity relations (one-to-one, one-to-many etc.). Is it possible to automatically create ERD diagrams from pyspark dataframes? Do helper libraries exist for this?
0 Answers
Related