MongoDB Client throws Wait Queue Full exception

I am seeing an odd issue where The .NET client for MongoDB throws a The wait queue for acquiring a connection to server 127.0.0.1:27017 is full. exception.

I have a semaphore that guards any call to MongoDB, with a size of 10. Meaning, there are never more than 10 concurrent calls to Mongo.

The default connection pool size is 100 for the .NET driver, which is more than 10. so 10 concurrent calls should not be an issue.

To replicate this I have the following code, contrived yes, but it makes the issue visible.

I also found this spec for MongoDB https://github.com/mongodb/specifications/blob/master/source/connection-monitoring-and-pooling/connection-monitoring-and-pooling.rst#id94

Is that related? Does each calling thread (thread pool worker in this case) go into the wait queue and try to grab a connection, and if I have more worker threads, even if concurrency level is low, the connections still have to be assigned to this new calling worker thread?

using System;
using System.Threading.Tasks;
using MongoDB.Bson;
using MongoDB.Driver;
using System.Threading;

namespace ConsoleApp58
{
    public class AsyncSemaphore
    {
        private readonly SemaphoreSlim _semaphore;

        public AsyncSemaphore(int maxConcurrency)
        {
            _semaphore = new SemaphoreSlim(
                maxConcurrency,
                maxConcurrency
            );
        }

        public async Task<T> WaitAsync<T>(Task<T> task)
        {
            await _semaphore.WaitAsync();
            //proves we have the correct max concurrent calls
            //  Console.WriteLine(_semaphore.CurrentCount);

            try
            {
                var result = await task;
                return result;
            }
            finally
            {
                _semaphore.Release();
            }
        }
    }
    
    class Program
    {
        public class SomeEntity
        {
            public ObjectId Id { get; set; }
            public string Name { get; set; }
        }

        static void Main(string[] args)
        {
            var settings = MongoClientSettings.FromUrl(MongoUrl.Create("mongodb://127.0.0.1:27017"));
            // settings.MinConnectionPoolSize = 10;
            // settings.MaxConnectionPoolSize = 1000;
            
            // I get that I can tweak settings, but I want to know why this occurs at all?
            // if we guard the calls with a semaphore, how can this happen?

            var mongoClient = new MongoClient(settings);
            var someCollection = mongoClient.GetDatabase("dummydb").GetCollection<SomeEntity>("some");
            var a = new AsyncSemaphore(10);
            
            // is this somehow related ?
            // https://github.com/mongodb/specifications/blob/master/source/connection-monitoring-and-pooling/connection-monitoring-and-pooling.rst#id94


            _ = Task.Run(() =>
                {
                    while (true)
                    {
                        // this bit is protected by a semaphore of size 10
                        // (we will flood the thread pool with ongoing tasks, yes)
                        _ = a.WaitAsync(RunTask(someCollection))
                            //after the task is done, dump the result
                            // dot is OK, else exception message
                            .ContinueWith(x =>
                            {
                                if (x.IsFaulted)
                                {
                                    Console.WriteLine(x.Exception);
                                }
                            });
                    }
                }
            );

            Console.ReadLine();
        }

        private static async Task<SomeEntity> RunTask(IMongoCollection<SomeEntity> pids)
        {
            //simulate some mongo interaction here
            var res = await pids.Find(x => x.Name == "").FirstOrDefaultAsync();
            return res;
        }
    }
}