How to set spark Kusto connector write polling interval?

user_1645645 0
Viewed 76

I'm writing data to Kusto using azure-kusto-spark.

I see this write has high latency. On seeing debug logs from Spark cluster, I see KustoConnector does polling on write. I believe there is default long polling time interval value. Is there a way to configure it to lower time interval value?

In azure-kusto-spark codebase I see this piece of code which I think is responsible for polling.

 def finalizeIngestionWhenWorkersSucceeded(
...
                DelayPeriodBetweenCalls,
                (writeOptions.timeout.toMillis / DelayPeriodBetweenCalls + 5).toInt,
                res => res.isDefined && res.get.status == OperationStatus.Pending,
                res => finalRes = res,
                maxWaitTimeBetweenCalls = KDSU.WriteMaxWaitTime.toMillis.toInt)
                .await(writeOptions.timeout.toMillis, TimeUnit.MILLISECONDS)
....

Not sure about understanding it.

azure-data-explorer apache-spark-sql apache-spark azure scala
1 Answers

The polling operations is just to check whether the data was inserted to Kusto. It has a max timeout but this is not impacting the latency. I believe the latency is coming from the time you have in your Kusto DB batching ingestion policy - see details here https://docs.microsoft.com/en-us/azure/data-explorer/kusto/management/batchingpolicy By default it's 5 minutes, so you might want to reduce this time if it won't have any negative impact (see doc). Note that the policy should be changed on DB, not table, the reason being the following: the connector creates a temporary table in the same Kusto DB and first inserts into this temporary table. So even if you change the policy of your destination table, it'll still take at least 5min to write to the temporary.

user_191927270
Related