I have created an Azure Stream Analytics job with a stream input as an event hub and reference input as JSON file from Azure Blob Storage. I have used following expression as Path pattern for my reference input JSON file: reference_data/guest-preferences/guest-preference-{date}-{time}.json. I have one data factory pipeline which creates a new file at the above location every 12 hours
This reference JSON file has more than 50 Million records (approx 2.5 GB). My Stream Analytics job query simply makes a Left join between input and reference data. I have set streaming units as 12.
Now the question is: when I am starting my Stream Analytics job it takes 15 minutes to start and after 12 hours when a new reference file is loaded it takes 30 minutes.
So is there any way to reduce this time?
Thanks for reading.