Acronyms
- ADLA : Azure Data Lake Analytics
- ADLS : Azure Data Lake Storage
I've been trying to figure out my best option for storing the input data for an ADLA U-SQL job in either ADLS (gen 1) or a general purpose storage blob container, and have been finding that the ADLA job's extract performance is abysmal when reading from ADLS.
For one test, I uploaded the same dataset to ADLS Gen 1 and a general purpose blob container, and ran the same ADLS U-SQL job that varied only in where they sourced their input data. Input fileset elimination (by date) pruned the input set to 67 input files, totalling 1.59MB (yes, MB - lots of small files in this synthetic data set)
The ADLA jobs source their data from ADLS (either adl: or swebhdfs: protocols - made almost no difference), or general purpose blob storage (wasb: protocol).
From ADLS, the extract phase took almost 13 minutes, while general purpose blob account took a mere 18 seconds. The remainder of the job time was pretty much the same for all input sources (as you'd expect).
Should I be seeing this degree of performance difference when my input data set is in ADLS as opposed to a general purpose storage account?
My ADLA script looks like this - there's nothing exotic in it:
//General purpose blob wasb: protocol
DECLARE @InputPath = @"wasb://{container}@{account}.blob.core.windows.net/events/{date:yyyy}/{date:MM}/{date:dd}/{date:HH}_{*}";
//ADLS adl: protocol
//DECLARE @InputPath = @"adl://{adls-account}.azuredatalakestore.net/events/{date:yyyy}/{date:MM}/{date:dd}/{date:HH}_{*}";
//ADLS swebhdfs: protocol
//DECLARE @InputPath = @"swebhdfs://{adls-account}.azuredatalakestore.net/events/{date:yyyy}/{date:MM}/{date:dd}/{date:HH}_{*}";
DECLARE @StartDate = DateTime.Parse("2018-02-25T00:00:00+00:00");
DECLARE @EndDate = DateTime.Parse("2018-02-27T23:59:59+00:00");
@tsvArchiveRecords =
EXTRACT
TenantId int?,
UserId long?,
// Snip more fields for brevity
date DateTime //virtual column
FROM @InputPath
USING Extractors.Tsv(skipFirstNRows: 1);
@logs =
SELECT TenantId,
UserId,
// Snip more fields for brevity
FROM @tsvArchiveRecords
WHERE date BETWEEN @StartDate AND @EndDate;
//snip more processing and outputs for brevity