My server uses MySqlConnector and communicates with a MySQL database on AWS. I store 5 minute chunks of API counters in MySQL. Those API counters are incrementing with each API call, and are handled by ConcurrentDictionary code (which doesn't appear to be a problem). An exception was recently raised by this line of code, which is part of a linq query through MySqlConnector to access a MySQL database table:
await _context.ApiCounts.Where(c => c.ApiName == apiName && c.StartTime >= startTime).ToListAsync();
I've never seen this line fail before, but suddenly one of my servers started throwing the following exception at the line above:
InvalidOperationException: Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct.
at System.ThrowHelper.ThrowInvalidOperationException_ConcurrentOperationsNotSupported
at System.Collections.Generic.Dictionary`2.FindEntry
at System.Collections.Generic.Dictionary`2.TryGetValue
at Remotion.Linq.Parsing.Structure.NodeTypeProviders.MethodInfoBasedNodeTypeRegistry.GetNodeType
at Remotion.Linq.Parsing.Structure.NodeTypeProviders.MethodInfoBasedNodeTypeRegistry.IsRegistered
at System.Linq.Enumerable.Any
at Remotion.Linq.Parsing.Structure.ExpressionTreeParser.GetQueryOperatorExpression
at Remotion.Linq.Parsing.ExpressionVisitors.SubQueryFindingExpressionVisitor.Visit
at System.Linq.Expressions.ExpressionVisitor.VisitBinary
at System.Linq.Expressions.BinaryExpression.Accept
at System.Linq.Expressions.ExpressionVisitor.VisitBinary
at System.Linq.Expressions.BinaryExpression.Accept
at System.Linq.Expressions.ExpressionVisitor.VisitLambda
at System.Linq.Expressions.Expression`1.Accept
at System.Linq.Enumerable+SelectListPartitionIterator`2.ToArray
at System.Linq.Enumerable.ToArray
at Remotion.Linq.Parsing.Structure.MethodCallExpressionParser.Parse
at Remotion.Linq.Parsing.Structure.ExpressionTreeParser.ParseMethodCallExpression
at Remotion.Linq.Parsing.Structure.QueryParser.GetParsedQuery
at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.CompileAsyncQueryCore
at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler+<>c__DisplayClass24_0`1.<CompileAsyncQuery>b__0
at Microsoft.EntityFrameworkCore.Query.Internal.CompiledQueryCache.GetOrAddQueryCore
at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.ExecuteAsync
at Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryable`1.System.Collections.Generic.IAsyncEnumerable<TResult>.GetEnumerator
at System.Linq.AsyncEnumerable+<Aggregate_>d__6`3.MoveNext
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
at BlayFap.Controllers.ServerController+<GetCachedApiCount>d__13.MoveNext (E:\Projects\BlayFap\BlayFap\Controllers\ServerController.cs:342)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
at BlayFap.Controllers.ServerController+<GetApiCount>d__14.MoveNext (E:\Projects\BlayFap\BlayFap\Controllers\ServerController.cs:378)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker+<InvokeActionMethodAsync>d__12.MoveNext
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker+<InvokeNextActionFilterAsync>d__10.MoveNext
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker+<InvokeInnerFilterAsync>d__14.MoveNext
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker+<InvokeNextResourceFilter>d__22.MoveNext
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker+<InvokeFilterPipelineAsync>d__17.MoveNext
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker+<InvokeAsync>d__15.MoveNext
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
All of the other tables can be accessed fine, but from here on any attempt to use the ApiCounts table results in the above exception. I had to actually reboot the affected server to clear the error. It looks like this exception has to do with concurrent edits on a Dictionary, and that Dictionary appears to be in linq code. I'm guessing that the server may have gotten caught in a situation where EF was both updating and writing some sort of data while executing a linq statement, and then got the Dictionary stuck in a weird state, but I'm not sure how to protect against this and where the real issue is. Is this a .NET bug, or my own?
More info: I do not modify the result from ApiCounts in this function, and I do not call SaveChanges. It is, however, possible for code running async to this (in another REST query) to update ApiCounts and call SaveChanges.