Server sent an invalid nonce when making multiple rapid connections from C# driver

Question

I recently added authentication to my development database, authenticating against the "admin" database, and using a username/password combination in my connection string, e.g. mongodb://username:password@server:27017. Almost immediately I started seeing connections failing to open with an exception showing "Server sent an invalid nonce". To try and mitigate the problem I looked at the lifetime of my IMongoClient objects, and moved from instantiating many such objects to using a Singleton injected into my business services using the Microsoft.Extensions.DependencyInjection libraries. This hasn't alleviated the issue. I set up the MongoClient in my Startup.cs using .AddSingleton(factory => new MongoClient(Configuration["Storage:MongoConnectionString"])). I know that the connection string is correct as it works in MongoDB Compass, and also because the first couple of calls through the driver work successfully; the issue starts occuring when multiple concurrent calls are in progress.

I am using the MongoDB .NET driver, version 2.11.0, under .NET Core 3.1.2. The problem occurs in my local environment running Windows 10, and also my staging environment running inside Docker on VMware Photon.

There are two components of the app that make connnections to MongoDB, both of which are ASP.Net Core applications, one serving an API for interactive usage of my applicaiton, and one running a Quartz scheduler for background processing. I'm running MongoDB 4.4.0 Community inside a Docker container.

My references to include the driver are:

According to this post on the MongoDB Jira site I'm not the first person to experience this issue. Mathias Lorenzen suggested in the issue on Jira that he had reduced the number of errors he encountered with various fixes including recreating the user, using SCRAM-SHA-1, and increasing the maximum number of connections permitted on the server. With these changes in place, the issue still occurs for me.

I'm guessing that the problem is related to threading when used in conjunction with database authentication. I can't, for obvious reasons, put this code into production use by disabling authentication to work around the problem, and equally the use of a synchronous model rather than async seems counter-productive. What steps can I take to try and resolve the authentication issues? Is this likely to be a bug in the Mongo C# driver, or am I simply using it wrong.

Insight on what I can try next, or alternative approaches, would be gratefully received.

Edit: Minimum reproducible example as requested:

using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using Microsoft.Extensions.DependencyInjection;
using MongoDB.Driver;

namespace MongoDbIssueExample
{
    internal class Program
    {
        private static IServiceProvider services;

        private static IServiceProvider BuildDependencyInjector()
        {
            services = new ServiceCollection()
                .AddSingleton()
                .AddSingleton(factory => new MongoClient("mongodb://username:password@server:27017"))
                .BuildServiceProvider();

            return services;
        }

        private static async Task DoSeed()
        {
            var service = services.GetService();
            // Don't do these async as we'll never get any data in...
            service.CreateTestThings().Wait();
            service.CreateOtherTestThings().Wait();
        }

        private static async Task DoTest()
        {
            var service = services.GetService();

            var things = service.GetTestThings();
            var otherThings = service.GetOtherTestThings();

            Task.WaitAll(things, otherThings);
        }

        private static async Task Main(string[] args)
        {
            BuildDependencyInjector();

            await DoTest();
        }
    }

    public class TestThingsService
    {
        private readonly IMongoClient _client;
        private readonly IMongoDatabase _database;
        private readonly IMongoCollection _otherTestThingsCollection;
        private readonly IMongoCollection _testThingsCollection;

        public TestThingsService(IMongoClient client)
        {
            _client = client;
            _database = _client.GetDatabase("Things");
            _testThingsCollection = _database.GetCollection("TestThings");
            _otherTestThingsCollection = _database.GetCollection("OtherTestThings");
        }

        public async Task CreateOtherTestThings()
        {
            for (var item = 1; item <= 10000; item++)
            {
                var testThing = new OtherTestThing {Id = item, Name = $"Other thing no. {item}", WhenCreated = DateTime.UtcNow};
                await _otherTestThingsCollection.ReplaceOneAsync(f => f.Id == item, testThing, new ReplaceOptions {IsUpsert = true});
            }
        }

        public async Task CreateTestThings()
        {
            for (var item = 1; item <= 10000; item++)
            {
                var testThing = new TestThing {Id = item, Name = $"Thing no. {item}", WhenCreated = DateTime.UtcNow};
                await _testThingsCollection.ReplaceOneAsync(f => f.Id == item, testThing, new ReplaceOptions {IsUpsert = true});
            }
        }


        public async Task> GetOtherTestThings()
        {
            return await _otherTestThingsCollection.Find(_ => true).ToListAsync();
        }

        public async Task> GetTestThings()
        {
            return await _testThingsCollection.Find(_ => true).ToListAsync();
        }
    }

    public class OtherTestThing
    {
        public int Id { get; set; }
        public string Name { get; set; }
        public DateTime WhenCreated { get; set; }
    }

    public class TestThing
    {
        public int Id { get; set; }
        public string Name { get; set; }
        public DateTime WhenCreated { get; set; }
    }
}

Requires references as follows:

user_5015647 · Answer

I also hit this issue recently, and was able to use your test code to reproduce the problem locally. I debugged the mongo c# driver and found the issue to be a race condition using a feature designed to piggyback part of the authentication protocol onto an initial server discovery (isMaster) request -- part of that authentication state ends up being stored in a global authenticator object and multiple concurrent authentication requests use the last seen state, which means that one request sees the nonce generated for the other request which is mismatched with the server response, leading to the exception being thrown.

There is a more recent bug report in the mongo bug tracker which contains the details. I have also fixed the issue in this fork of the mongo c# driver by associating the authentication state with the discovery request. I am using a build of the forked mongo driver to work around the issue (you can create one of these by running build --target=Package in the forked repository).

user_1617156 · Answer

This issue has been resolved in a newer version of the MongoDB driver for C#, version 2.11.2.

user_9784697 · Answer

Minimum code is:

var client = new MongoClient(_connection_string);;
var database = client.GetDatabase(Database_name);
var collection = database.GetCollection(Collection_name);
var document_count = collection.EstimatedDocumentCount();

remove the last row, it works correctly.

environment: .Net Core SDK 3.1.401, Mongodb.Driver 2.11.0, using Sharding Mongodb 4.4.

Server sent an invalid nonce when making multiple rapid connections from C# driver

3 Answers