How to add database prefix for unittest python using spark

Viewed 133

I am writing unittest for a python class. I want to add database prefix because the original class has it. when i create a temporary view for the same. i am getting this error. Any pointers on how to specify the database prefix

Error

 pyspark.sql.utils.AnalysisException: It is not allowed to add database prefix `union` for the TEMPORARY view name.

Code:

 columns = ["company_id", "item_id", "ingest_date", "attachable_id"]
        data = [
            (100, 1, datetime(2021, 1, 1, 0, 0, 0, 0), 900),
            (100, 2, datetime(2021, 1, 1, 0, 3, 0, 0), 903),
            (100, 2, datetime(2021, 1, 1, 0, 1, 0, 0), 902),
            (200, 1, datetime(2021, 1, 1, 0, 1, 0, 0), 900)
        ]
        rowData = map(lambda x: Row(*x), data)
        spark = SparkSession.builder.appName('SparkByExamples.com').getOrCreate()
        df = spark.createDataFrame(rowData, columns)
        df.createOrReplaceTempView("union.items_vw")
        select_query = "SELECT count(*) from union.items_vw"
        batch_df = spark.sql(select_query)
        print(batch_df.show())
    ```
0 Answers
Related