Reviewing some Azure Log Analytics logs and I see that each time my Python Azure Function downloads a blob from Azure Storage, there is an initial 32MB chunk, then all subsequent GetBlob actions are 4MB chunks.
How can I increase this number to reduce the amount of time my Function executes?
Example Python that downloads blob from storage (Azure Function):
def load_blob_to_memory(blob_client):
blob_data = blob_client.download_blob().readall()
blob_bytes = io.BytesIO(blob_data)
return blob_bytes
Example Log Analytics showing ResponseBodySize:
- Query:
//==================================================//
// Assign variables
//==================================================//
let varStart = ago(2d);
let varEnd = now();
let varStorageAccount = 'stgtest';
let varIngressContainerName = 'cont-test';
let varFileName = 'test.csv';
let varSep = '/';
let varSampleUploadUri = strcat('https://', varStorageAccount, '.dfs.core.windows.net', varSep, varIngressContainerName, varSep, varFileName);
let varSampleDownloadUri = replace(@'%2F', @'/', replace(@'.dfs.', @'.blob.', tostring(varSampleUploadUri)));
//==================================================//
// Filter table
//==================================================//
StorageBlobLogs
| where TimeGenerated between (varStart .. varEnd)
and AccountName == varStorageAccount
//and StatusText == varStatus
and split(Uri, '?')[0] == varSampleUploadUri
or split(Uri, '?')[0] == varSampleDownloadUri
| summarize
count() by OperationName,
TimeGenerated,
UserAgent = tostring(split(UserAgentHeader, '(')[0]),
FileName = tostring(split(tostring(parse_url(url_decode(Uri))['Path']), '/')[-1]),
DownloadChunkSize = format_bytes(ResponseBodySize, 2, 'MB'),
StatusCode,
StatusText
| order by TimeGenerated asc
- Output:
6/9/2021, 6:24:22.226 PM GetBlob azsdk-python-storage-blob/12.8.1 Python/3.8.10 test.csv 32 MB 206 Success 1
6/9/2021, 6:24:22.442 PM GetBlob azsdk-python-storage-blob/12.8.1 Python/3.8.10 test.csv 4 MB 206 Success 1
6/9/2021, 6:24:22.642 PM GetBlob azsdk-python-storage-blob/12.8.1 Python/3.8.10 test.csv 4 MB 206 Success 1
6/9/2021, 6:24:22.780 PM GetBlob azsdk-python-storage-blob/12.8.1 Python/3.8.10 test.csv 4 MB 206 Success 1
There is a max_concurrency parameter for the download_blob() method of BlobClient class, but I'm not sure if it requires full async/await rewrite of the code.
EDIT 1: Thank you @Guarav. This increased the default to 32MB.
def create_blob_client(credentials):
blob_client = BlobClient.from_blob_url(
event.get_json()["blobUrl"],
credentials,
max_single_get_size = 64*1024*1024,
max_chunk_get_size = 32*1024*1024
)
return blob_client