Querying Delta.io tables with Power BI returns the "The table has no visible columns"

user_16034953 0
Viewed 336

Has anyone managed to query delta (delta.io) tables with Power BI Desktop? Setup: Spark 3+ on Kubernetes, Delta.io 0.8 or 1.0, Hive 1.2 or 2.3.7 on postgresql 12.5 (AWS RDS), Thrift Server, S3 for table location. Tested both with managed and external Hive tables.

Connection to the thrift server is fine. Queries form Power BI on parquet tables are fine. Querying delta tables is fine only with Tableau Desktop. But querying delta tables with Power BI returns the message: "DataSource.Error: The table has no visible columns and cannot be queried."

Comparing the Thrift server log (Debug Level) between Tableau and Power BI shows that:

  • Tableau and Power BI both created a Delta snapshot:

INFO Snapshot: [tableId=6f41c252-8e40-4d10-813f-f987e7029f54] DELTA: Done

INFO DeltaLog: Returning initial snapshot ... which shows the table schema

  • This log DEBUG Analyzer$ResolveReferences: Resolving ... appears for each table columns along with the column names for Tableau only.
  • This log INFO SparkGetColumnsOperation: Listing columns 'catalog : null, schemaPattern : xxx, tablePattern : xxx, columnName : null' with xxx appears only for Power Bi
delta-lake powerbi apache-spark hive
1 Answers

For those who are still searching…

PowerBI cannot display columns in a delta table because of the issue that is comming from Spark. There is a discussion around it in the delta-io repository.

As a workaround, you can create views (and query them in PowerBi) or use Apache Kyuubi instead of Spark Thrift Server.

user_194965670
Related