My task is to get some data from a postgres database and then write this to blob storage and then return the URI.
As the data contains up to 14 million rows, I have been using binary export to get the data out like so
var fileText = new StringBuilder();
fileText.AppendLine(string.Join(',', "Column1", "Column2", "Column3", "Column4"));
using var connection = CreateYellowBrickConnection();
connection.Open();
using var reader =
connection.BeginBinaryExport(
$"COPY {tableName} (Column1,Column2,Column3,Column4) TO STDOUT (FORMAT BINARY)");
while (reader.StartRow() > 0)
{
fileText.AppendLine(string.Join(',', reader.Read<long>(), reader.Read<string>(), reader.Read<string>(), reader.Read<int>()));
}
return await _blobHelper.WriteToBlobAndReturnUri(application, retailerId, containerName, fileText, fileName);
This can take some time (obs depending on amount of data but up to say 14 mill rows). To upload the blob it calls
public async Task<string> WriteToBlobAndReturnUri(string application, int retailerId, string containerName, StringBuilder fileText, string fileName)
{
try
{
GetBlobContainerService(application, retailerId, containerName);
_azureServicesFactory.CloudBlobContainerService.GetCloudBlockBlobService($"{fileName}.csv");
await _azureServicesFactory.CloudBlockBlobService.UploadTextAsync(fileText.ToString());
}
catch (Exception ex)
{
_logger.LogInformation(ex.ToString());
throw;
}
return _azureServicesFactory.CloudBlockBlobService.Uri() + GetSasBlobToken(_azureServicesFactory.CloudBlockBlobService);
This all works fine locally with Microsoft Azure Storage Explorer (tried up to 14 mill rows) but once I put this on a testing environment with larger files say 300,000 KB (5 mil rows) when it does the uploadtextAsync it basically fails silently and I get no response error messages in app insights.
I tried chunking this into blocs using basically the microsoft documentation
//var blob = new BlobContainerClient(connectionString, containerName);
//BlockBlobClient blobClient = blob.GetBlockBlobClient($"{fileName}.csv");
////ArrayList blockIDArrayList = new ArrayList();
//byte[] buffer;
//var bytesLeft = ms.Length - ms.Position;
//while (bytesLeft > 0)
//{
// if (bytesLeft >= blockSize)
// {
// buffer = new byte[blockSize];
// await ms.ReadAsync(buffer, 0, blockSize);
// }
// else
// {
// buffer = new byte[bytesLeft];
// await ms.ReadAsync(buffer, 0, Convert.ToInt32(bytesLeft));
// }
// using (var stream = new MemoryStream(buffer))
// {
// string blockID = Convert.ToBase64String
// (Encoding.UTF8.GetBytes(Guid.NewGuid().ToString()));
// blockIDArrayList.Add(blockID);
// await blobClient.StageBlockAsync(blockID, stream);
// }
// bytesLeft = ms.Length - ms.Position;
//}
//string[] blockIDArray = (string[])blockIDArrayList.ToArray(typeof(string));
//await blobClient.CommitBlockListAsync(blockIDArray);
But this was slower when timing locally. Is there anyway of making this more efficient and can anything be adjusted in the azure portal that help with this (just wondered if I am missing something) or is this just not a great implementation? Any suggestions would be much appreciated.