what is the maximum data size allowed to read from ADX cluster table using kusto-spark connector?

usertPl41 0
Viewed 46

I was trying to pull some table data from the ADX cluster in another tenant to Databricks workspace, but when I try to read the data, some data were missing, and row count does not match with the original table. The data size is very large, Would that be the cause of the issue? or Is there any better way for pulling ADX data to the databricks notebook instead of using the kusto-spark connector? this is the code which I have used

kustoDf = spark.read. \
    format("com.microsoft.kusto.spark.datasource"). \
    option("kustoCluster",        kustoOptions["kustoCluster"]). \
    option("kustoDatabase",       kustoOptions["kustoDatabase"]). \
    option("kustoQuery",          kustoOptions["kustoTable"]). \
    option("kustoAadAppId",       kustoOptions["kustoAadAppId"]). \
    option("kustoAadAppSecret",   kustoOptions["kustoAadAppSecret"]). \
    option("kustoAadAuthorityID", kustoOptions["kustoAadAuthorityID"]). \
    load()

Am I doing something wrong? or is there something missing in the code?

Thanks in advance!

azure-data-explorer azure-databricks apache-spark
1 Answers

The Kusto query limits the number of records returned to the client to 500,000 and the overall data size for other records to 64 MB.

When either of these limits is exceeded, the query fails with a "partial query failure".

I tried to repro the same with the same scenario as given by you. Here is my approach.

I have used the same spark-connector for ADX making connection.

val KustoSparkTestAppId = "b93b3cde-46a9-a2c8-cb0b9c8bdf1e"
val KustoSparkTestAppKey = "fbv8Q~UyoK93P.nSjUg8Wz67dkc"

val appId = KustoSparkTestAppId
val appKey = KustoSparkTestAppKey
val authorityId = "72f988bf-86f1-41af-011db47" 
val cluster = "utrolicadxcluster.eastus"
val database = "demodb"
val table = "StringAndIntTable"

enter image description here

But when I’m trying to read the data, I’m getting below error.

NoClassDefFoundError: scala/Product$class

enter image description here

I’m not sure why this error is coming. So, Alternatively, I have tried the same with Azure Data Explorer tool.

Firstly, I queried a file which has a size of 9.1 MB and the result of query was good as expected. Refer below.

enter image description here

Next, when I run the same query on the table with size 545 MB and some 2 million rows, the query takes quiet long to gives below error and return only 500,000 records.

Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 500000 records, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..

Refer below image.

enter image description here

So, you need to take care of the record size when reading data from Data Explorer using Kusto query. The size of the should not exceed 64 MB and maximum records 500000.

Utkarsh Pal0
Related