How to upload the Stream from an HttpContent result to Azure File Storage

Question

I am attempting to download a list of files from urls stored in my database, and then upload them to my Azure FileStorage account. I am successfully downloading the files and can turn them into files on my local storage or convert them to text and upload them. However I lose data when converting something like a pdf to a text and I do not want to have to store the files on the Azure app that this endpoint is hosted on as I do not need to manipulate the files in any way.

I have attempted to upload the files from the Stream I get from the HttpContent object using the UploadFromStream method on the CloudFile. Whenever this command is run I get an InvalidOperationException with the message "Operation is not valid due to the current state of the object."

I've tried converting the original Stream to a MemoryStream as well but this just writes a blank file to the FileStorage account, even if I set the position to the beginning of the MemoryStream. My code is below and if anyone could point out what information I am missing to make this work I would appreciate it.

public DownloadFileResponse DownloadFile(FileLink fileLink)
{
    string fileName = string.Format("{0}{1}{2}", fileLink.ExpectedFileName, ".", fileLink.ExpectedFileType);
    HttpStatusCode status;
    string hash = "";
    using (var client = new HttpClient())
    {
        client.Timeout = TimeSpan.FromSeconds(10); // candidate for .config setting
        client.DefaultRequestHeaders.Add("User-Agent", USER_AGENT);

        var request = new HttpRequestMessage(HttpMethod.Get, fileLink.ExpectedURL);
        var sendTask = client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead);
        var response = sendTask.Result; // not ensuring success here, going to handle error codes without exceptions

        status = response.StatusCode;
        if (status == HttpStatusCode.OK)
        {
            var httpStream = response.Content.ReadAsStreamAsync().Result;

            fileStorage.WriteFile(fileLink.ExpectedFileType, fileName, httpStream);
            hash = HashGenerator.GetMD5HashFromStream(httpStream);
        }
    }

    return new DownloadFileResponse(status, fileName, hash);
}

public void WriteFile(string targetDirectory, string targetFilePath, Stream fileStream)
{
    var options = SetOptions();
    var newFile = GetTargetCloudFile(targetDirectory, targetFilePath);
    newFile.UploadFromStream(fileStream, options: options);
}

public FileRequestOptions SetOptions()
{

    FileRequestOptions options = new FileRequestOptions();

    options.ServerTimeout = TimeSpan.FromSeconds(10);
    options.RetryPolicy = new NoRetry();

    return options;
}

public CloudFile GetTargetCloudFile(string targetDirectory, string targetFilePath)
{
    if (!shareConnector.share.Exists())
    {
        throw new Exception("Cannot access Azure File Storage share");
    }

    CloudFileDirectory rootDirectory = shareConnector.share.GetRootDirectoryReference();
    CloudFileDirectory directory = rootDirectory.GetDirectoryReference(targetDirectory);

    if (!directory.Exists())
    {
        throw new Exception("Target Directory does not exist");
    }

    CloudFile newFile = directory.GetFileReference(targetFilePath);

    return newFile;
}

user_1335354 · Answer

Had the same problem, the only way it worked is by reading the coming stream (in your case it is httpStream in DownloadFile(FileLink fileLink) method) to a byte array and using UploadFromByteArray (byte[] buffer, int index, int count) instead of UploadFromStream

So your WriteFile(FileLink fileLink) method will look like:

public void WriteFile(string targetDirectory, string targetFilePath, Stream fileStream)
{
  var options = SetOptions();
  var newFile = GetTargetCloudFile(targetDirectory, targetFilePath);

  const int bufferLength= 600;
  byte[] buffer = new byte[bufferLength];
  // Buffer to read from stram This size is just an example

  List byteArrayFile = new List(); // all your file will be here
  int count = 0;

  try
  {
      while ((count = fileStream.Read(buffer, 0, bufferLength)) > 0)
         {
            byteArrayFile.AddRange(buffer);               
         }                    
         fileStream.Close();
   }

   catch (Exception ex)
   {
       throw; // you need to change this
   }

   file.UploadFromByteArray(allFile.ToArray(), 0, byteArrayFile.Count);
   // Not sure about byteArrayFile.Count.. it should work
}

user_5320661 · Answer

I had a similar problem and got to find out that the UploadFromStream method only works with buffered streams. Nevertheless I was able to successfully upload files to azure storage by using a MemoryStream. I don't think this to be a very good solution as you are using up your memory resources by copying the content of the file stream to memory before handing it to the azure stream. What I have come up with is a way of writing directly to an azure stream by using instead the OpenWriteAsync method to create the stream and then a simple CopyToAsync from the source stream.

CloudStorageAccount storageAccount   = CloudStorageAccount.Parse( "YourAzureStorageConnectionString" );
CloudFileClient     fileClient       = storageAccount.CreateCloudFileClient();
CloudFileShare      share            = fileClient.GetShareReference( "YourShareName" );
CloudFileDirectory  root             = share.GetRootDirectoryReference();
CloudFile           file             = root.GetFileReference( "TheFileName" );

using (CloudFileStream fileWriteStream = await file.OpenWriteAsync( fileMetadata.FileSize, new AccessCondition(),
                                                                    new FileRequestOptions { StoreFileContentMD5 = true },
                                                                    new OperationContext() ))
{
    await fileContent.CopyToAsync( fileWriteStream, 128 * 1024 );
}

How to upload the Stream from an HttpContent result to Azure File Storage

3 Answers