Best way to process raw CDN logs stored in Azure Blob Storage to an Azure SQL Database

user_8919569 0
Viewed 78

I have an Azure CDN and all my raw logs (every request made to the CDN) goes to my Azure Blob Storage formatted as "JSON Lines". (these are AzureCdnAccessLog)

I am looking to process those logs (every 10-20 minutes) to count the number of times each resource has been accessed, when and from which platform etc... and output the result to my Azure SQL Database.

What is the best solution to batch process those logs on Azure?

Thank you in advance for your aswers.

azure-storage azure-sql-database azure
1 Answers

Sounds like Azure function Blob storage trigger is the best candidate. But keep in mind that azure function has limitation for how long it can run

[FunctionName("BlobTriggerCSharp")]        
public static void Run([BlobTrigger("container/{name}")] Stream myBlob, string name, ILogger log)
{
    log.LogInformation($"C# Blob trigger function Processed blob\n Name:{name} \n Size: {myBlob.Length} Bytes");
}

The other option would be Azure Data Factory, Here is more information about how to import from blob

user_11002480
Related