RavenDB PeriodicExport Error "RequestEntityTooLarge"

Viewed 96

We created PeriodicExport for RavenDB database. We try to upload backup files to Azure BLOB Container.

There are our settings: enter image description here

In the Azure BLOB Container I can see incremental backup files. But I do not see full backups files.

Also I can see next error form the Raven Alerts:

Label Description
Title Error in Periodic Export
Message Status code: RequestEntityTooLarge <?xml version="1.0" encoding="utf-8"?><Error><Code>RequestBodyTooLarge</Code><Message>The request body is too large and exceeds the maximum permissible limit. RequestId:8b013757-401e-0014-4965-b7e992000000 Time:2021-10-02T08:13:54.8562742Z</Message><MaxLimit>67108864</MaxLimit></Error>
Level Error

And there is full exception information:

Raven.Abstractions.Connection.ErrorResponseException: Status code: RequestEntityTooLarge
<?xml version="1.0" encoding="utf-8"?><Error><Code>RequestBodyTooLarge</Code><Message>The request body is too large and exceeds the maximum permissible limit.
RequestId:8b013757-401e-0014-4965-b7e992000000
Time:2021-10-02T08:13:54.8562742Z</Message><MaxLimit>67108864</MaxLimit></Error>

   at Raven.Database.Client.Azure.RavenAzureClient.PutBlob(String containerName, String key, Stream stream, Dictionary`2 metadata) in c:\Builds\RavenDB-Stable-3.0\Raven.Database\Client\Azure\RavenAzureClient.cs:line 93
   at Raven.Database.Bundles.PeriodicExports.PeriodicExportTask.UploadToAzure(String backupPath, PeriodicExportSetup localExportConfigs, Boolean isFullBackup) in c:\Builds\RavenDB-Stable-3.0\Raven.Database\Bundles\PeriodicExports\PeriodicBackupTask.cs:line 397
   at Raven.Database.Bundles.PeriodicExports.PeriodicExportTask.<>c__DisplayClasse.<<TimerCallback>b__a>d__10.MoveNext() in c:\Builds\RavenDB-Stable-3.0\Raven.Database\Bundles\PeriodicExports\PeriodicBackupTask.cs:line 248

Raven DB version is:

{
  "ProductVersion": "3.0.0 / cdc39ac / ",
  "BuildVersion": "3690"
}

It is possible to fix this error?

Any help is much appreciated.


UPDATE

I have found source code of the relevant build version in GitHub repository.

The bug exists in the next method:

public void PutBlob(string containerName, string key, Stream stream, Dictionary<string, string> metadata)
{
    var url = GetUrl(containerName) + "/" + key;

    var now = SystemTime.UtcNow;
    var content = new StreamContent(stream)
                  {
                      Headers =
                      {
                          { "x-ms-date", now.ToString("R") }, 
                          { "x-ms-version", "2011-08-18" },
                          { "x-ms-blob-type", "BlockBlob" },
                          { "Content-Length", stream.Length.ToString(CultureInfo.InvariantCulture) }
                      }
                  };

    foreach (var metadataKey in metadata.Keys)
        content.Headers.Add("x-ms-meta-" + metadataKey.ToLower(), metadata[metadataKey]);

    var client = GetClient(TimeSpan.FromHours(1));
    client.DefaultRequestHeaders.Authorization = CalculateAuthorizationHeaderValue("PUT", url, content.Headers);

    var response = client.PutAsync(url, content).ResultUnwrap();

    if (response.IsSuccessStatusCode)
        return;

    throw ErrorResponseException.FromResponseMessage(response);
}

As you can see this method use Azure API version 2011-08-18. Limitations for this Azure version described in the Microsoft documentation.

Regarding documentation "Maximum blob size via single write operation (via Put Blob)" is exactly 64 MiB (67108864 bytes). So exception provided by Azure is absolutely correct.

Unfortunately Raven Build-3690 has a bug, that was fixed in the next version. Source code with fixes can be reviewed here.

1 Answers

is exists any way to increase "Maximum blob size via single write operation (via Put Blob)" limit for BLOB storage?

The limits can not be change because its designed.

Based on the Microsoft document :

Azure Storage standard accounts support higher capacity limits and higher limits for ingress and egress by request. To request an increase in account limits, contact Azure Support.

As you said , here is the table describes the maximum block and blob sizes permitted by service version. enter image description here

For more information please refer this Azure Blog: Run high scale workloads on Blob storage with new 200 TB object sizes

Related