I have a Parquet file saved as a Table in Spark, which is copied to Trino as a Hive Table (also in parquet format).
My problem is that Superset's SQL Labs, which uses the Trino database is showing misplaced values. I'm querying the Spark Table and it is showing correct values, but SQL Labs is not. Furthermore, I am seeing that the incorrect rows are valid, just misplaced (probably from some other rows).
All this happened when I upgraded my Spark version from 3.1 to 3.2. Our current hunch is that spark has some leftover cache that is being read by Trino, but clearing Sparks's cache using spark.catalog.clearCache() did not yield any difference.
Example Spark Table:
+----------+----------+----------+
| Column 1 | Column 2 | Correct? |
+----------+----------+----------+
| ABC | ABC | Yes |
| DEF | DEF | Yes |
| GHI | GHI | Yes |
| JKL | JKL | Yes |
+----------+----------+----------+
What I see in SQL Labs:
+----------+----------+----------+
| Column 1 | Column 2 | Correct? |
+----------+----------+----------+
| ABC | GHI | No |
| DEF | DEF | Yes |
| GHI | ABC | No |
| JKL | JKL | Yes |
+----------+----------+----------+
Edit:
Versions:
Spark: 3.2
Trino in Superset: 363
PySpark: 3.2.1
Superset 1.3.1