I am trying to get the metadata from a bucket/prefix on S3 using datahub but i am getting error:
{logger:26} - Please set env variable SPARK_VERSION.
The datahub s3 documentation mentions
Profiles are computed with PyDeequ, which relies on PySpark. Therefore, for computing profiles, we currently require Spark 3.0.3 with Hadoop 3.2 to be installed and the SPARK_HOME and SPARK_VERSION environment variables to be set.
Where should i install Spark and SPARK_VERSION? In the container? I have Spark locally installed.