ADLS Gen 1 poor performance vs General Purpose Storage Blob for ADLA U-SQL job

Viewed 188

Acronyms

  • ADLA : Azure Data Lake Analytics
  • ADLS : Azure Data Lake Storage

I've been trying to figure out my best option for storing the input data for an ADLA U-SQL job in either ADLS (gen 1) or a general purpose storage blob container, and have been finding that the ADLA job's extract performance is abysmal when reading from ADLS.

For one test, I uploaded the same dataset to ADLS Gen 1 and a general purpose blob container, and ran the same ADLS U-SQL job that varied only in where they sourced their input data. Input fileset elimination (by date) pruned the input set to 67 input files, totalling 1.59MB (yes, MB - lots of small files in this synthetic data set)

The ADLA jobs source their data from ADLS (either adl: or swebhdfs: protocols - made almost no difference), or general purpose blob storage (wasb: protocol).

From ADLS, the extract phase took almost 13 minutes, while general purpose blob account took a mere 18 seconds. The remainder of the job time was pretty much the same for all input sources (as you'd expect).

Should I be seeing this degree of performance difference when my input data set is in ADLS as opposed to a general purpose storage account?

My ADLA script looks like this - there's nothing exotic in it:

//General purpose blob wasb: protocol
DECLARE @InputPath = @"wasb://{container}@{account}.blob.core.windows.net/events/{date:yyyy}/{date:MM}/{date:dd}/{date:HH}_{*}";

//ADLS adl: protocol
//DECLARE @InputPath = @"adl://{adls-account}.azuredatalakestore.net/events/{date:yyyy}/{date:MM}/{date:dd}/{date:HH}_{*}";

//ADLS swebhdfs: protocol
//DECLARE @InputPath = @"swebhdfs://{adls-account}.azuredatalakestore.net/events/{date:yyyy}/{date:MM}/{date:dd}/{date:HH}_{*}";

DECLARE @StartDate = DateTime.Parse("2018-02-25T00:00:00+00:00");
DECLARE @EndDate   = DateTime.Parse("2018-02-27T23:59:59+00:00");

@tsvArchiveRecords =
    EXTRACT 
            TenantId int?,
            UserId long?,
           // Snip more fields for brevity
            date DateTime //virtual column
    FROM @InputPath
    USING Extractors.Tsv(skipFirstNRows: 1);

@logs =
    SELECT TenantId,
            UserId,
            // Snip more fields for brevity
    FROM @tsvArchiveRecords
    WHERE date BETWEEN @StartDate AND @EndDate;

//snip more processing and outputs for brevity
0 Answers
Related