My team and I have been experiencing some EF Core / SQL pooling issues on a high utilization .NET Core Web App.
Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.
The issue started occurring when EF Core / ApplicationDbContext was being referenced in a Background task on a hosted service (QueuedHostedService). Following the guide at:
I've followed the recommended dependency injection and hosted service guides.
Steps to reproduce
The relevant lines:
Startup.cs
...
services.AddDbContext<ApplicationDbContext>(options =>
options.UseSqlServer(Configuration.GetConnectionString("DefaultConnection")));
...
services.AddScoped<IScopedWorker, ScopedWorker>();
services.AddSingleton<MySingletonThatAddsToBackground>();
// Follow the host service guide from microsoft.
services.AddHostedService<QueuedHostedService>();
services.AddSingleton<IBackgroundTaskQueue, BackgroundTaskQueue>();
MySingletonThatAddsToBackground.cs
public class MySingletonThatAddsToBackground
{
private readonly IBackgroundTaskQueue _taskQueue;
private readonly ILogger _logger;
private readonly CancellationToken _cancellationToken;
public IServiceProvider _services { get; }
public MySingletonThatAddsToBackground(IServiceProvider services, IBackgroundTaskQueue taskQueue,
ILogger<MonitorLoop> logger,
IHostApplicationLifetime applicationLifetime)
{
_services = services;
_taskQueue = taskQueue;
_logger = logger;
_cancellationToken = applicationLifetime.ApplicationStopping;
}
public void DoWorkBackground()
{
// Enqueue a background work item
_taskQueue.QueueBackgroundWorkItem(async token =>
{
try
{
using (var scope = _services.CreateScope())
{
var scopedWorker = scope.ServiceProvider.GetRequiredService<IScopedWorker>();
await scopedWorker.DoWork();
}
}
catch (OperationCanceledException)
{
// Prevent throwing if the Delay is cancelled
}
});
}
}
ScopedWorker.cs
public class ScopedWorker : IScopedWorker
{
private readonly ApplicationDbContext _db;
public ScopedWorker(ApplicationDbContext db)
{
_db = db;
}
public void DoWork()
{
var customers = _db.MyCustomers.ToListAsync();
// Do stuff to customers.
await _db.SaveChangesAsync();
}
}
Discussion
Inside MySingletonThatAddsToBackground, after the using (var scope) is completed, shouldn't the scope be disposed, which then disposes ScopedWorker (scoped), which then disposes the ApplicationDbConext (scoped), and then closes the connection / pool connection?
Is there something I'm not implementing properly to cause the connection pool leak?
Further technical details
EF Core version: 3.1.4 Database provider: Microsoft.EntityFrameworkCore.SqlServer Target framework: .NET Core 3.1.4 Operating system: Windows Server 2016, SQL Server 2016 IDE: 16.6