How to read in large number of rows from Kusto database table using C#

user_13539062 0
Viewed 128

I am using Kusto.Data methods to create a query client object and read table from my database on Kusto cluster. However I am unable to read large no of rows(when in millions).in that case the Ireader object throws an exception: *

Kusto.Data.Exceptions.KustoServicePartialQueryFailureException: 'Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003).

The documentation says:

When the Streaming flag is enabled (as is the default), the SDK does not buffer all response data in memory; instead, it "pulls" the data from the service when the caller requests it. Therefore, it is essential that in this case the caller properly disposes of the data (such as IDataReader) once it is done reading the data, as the network connection to the service is held open unnecessarily.

if data is being pulled when reader is reading the rows, why is my memory exceeding. Also if there is an alternative way to read millions of way, please share

kql azure-data-explorer c#
1 Answers

As the documentation states:

A query result set has exceeded the internal ... limit is a kind of partial query failure that happens when the query's result has exceeded one of two limits:

  • A limit on the number of records (record count limit, set by default to 500,000)
  • A limit on the total amount of data (data size limit, set by default to 67,108,864 (64MB))

Add OptionNoTruncation to your code

clientRequestProperties.SetOption(Kusto.Data.Common.ClientRequestProperties.OptionNoTruncation, true);

(Alternatively you could use OptionTruncationMaxSize and/or OptionTruncationMaxRecords)

user_63364790
Related