How to ingress stream data with Azure Data Exploere SDK

Viewed 29

We are needing to write some software, that receives events one at a time, and we need to ingress them into ADX. We are struggling to understand how the Kusto Client is meant to be utilized.

public void SaveEvent(Object event)
{

 var _kcsb = new KustoConnectionStringBuilder(Uri).WithAadApplicationKeyAuthentication(
            applicationClientId: "{}",
            applicationKey: "{}",
            authority: TenantId);

using var ingestClient = KustoIngestFactory.CreateQueuedIngestClient(_kcsb);

//// Create your custom implementation of IRetryPolicy, which will affect how the ingest client handles retrying on transient failures
IRetryPolicy retryPolicy = new NoRetry();
//// This line sets the retry policy on the ingest client that will be enforced on every ingest call from here on
        
((IKustoQueuedIngestClient)ingestClient).QueueOptions.QueueRequestOptions.RetryPolicy = retryPolicy;

var ingestProperties = new KustoIngestionProperties(DatabaseName, TableName)
        {
            Format = DataSourceFormat.json,
            IngestionMapping = new IngestionMapping { IngestionMappingKind = Kusto.Data.Ingestion.IngestionMappingKind.Json, IngestionMappingReference = MappingName }
        };

 // Build the stream

 var stream = new MemoryStream();
        using var streamWriter = new StreamWriter(stream: stream, encoding: Encoding.UTF8, bufferSize: 4096, leaveOpen: true);
        using var jsonWriter = new JsonTextWriter(streamWriter);
        packet.Id = DateTime.UtcNow.Ticks;
        var serializer = new JsonSerializer();
        serializer.Serialize(jsonWriter, event);
        streamWriter.Flush();
        stream.Seek(0, SeekOrigin.Begin);
       
// Tell the client to ingest this
await ingestClient.IngestFromStreamAsync(data, ingestProperties);

}

Now I have several concerns with this. We are calling this function 300 to 500 times a second. I believe the custom client has built in batching, but do we not then need to use a singleton instance of the custom client?

Next thing is that I am creating a steam per event and then calling ingerss. This feels wrong? is there no way I can setup the custom client etc, and then just enqueue each event into the custom client as we receiver them?

0 Answers
Related