Slow ElasticSearch Serilog

My C# dotnet core web application uses serilog with elasticsearch which is configured like the code below. Whenever elastic search server has a problem my server's Http queries slow down to being unsusable. Let's say if elasticsearch is down each query takes 15 seconds or so. I assume each query tries to log the message and waits for the elasticserch server to respond. I understand that I can play with elasticsearch timeout but that is not a solution. Is there a better way not to limit the application performance by the logging method ?

  "Serilog": {
"IncludeScopes": true,
"MinimumLevel": {
  "Default": "Information",
  "Override": {
    "Microsoft": "Warning",
    "System": "Warning"
  }
},
"WriteTo": [
  {
    "Name": "File",
    "Args": {
      "path": "....\\log.txt",
      "rollingInterval": "Day",
      "outputTemplate": "{Timestamp:o} [{Level:u3}] {Message}{NewLine}{Exception}"
    }
  },
  {
    "Name": "Elasticsearch",
    "Args": {
      "nodeUris": "http://localhost:9200",
      "typeName": "SomeApp",
      "batchPostingLimit": 50,
      "restrictedToMinimumLevel": "Information",
      "bufferFileSizeLimitBytes": 5242880,
      "bufferLogShippingInterval": 5000,
      "bufferRetainedInvalidPayloadsLimitBytes": 5000,
      "bufferFileCountLimit": 31,
      "connectionTimeout": 5,
      "queueSizeLimit": "100000",
      "autoRegisterTemplate": true,
      "overwriteTemplate": false
    }
  }
],
"Enrich": [
  "FromLogContext"
],
"Properties": {
  "Application": "SomeApp",
  "Environment": "SomeApp.Production"
}
}

The middleware part looks like this

public async Task Invoke(HttpContext context, ILogger logger, ILogContext logContext)
    {
        context.Request.EnableBuffering();
        var stopWatch = new Stopwatch();
        stopWatch.Start();
        try
        {
            await _next.Invoke(context);
        }
        finally
        {
            stopWatch.Stop();
            await LogRequest(context, logger, stopWatch.ElapsedMilliseconds, logContext);
        }
    }
    private async Task LogRequest(HttpContext context, ILogger logger, long elapsedMs, ILogContext logContext){
       logger.Information(LogConstants.RequestMessageTemplate, message);

}