Summary
AuditMiddleware runs on every request and, on a cold/expired FusionCache entry, fans out multiple
ISettingsContracts.GetSettingAsync(...) calls that execute concurrently over a single scoped
SettingsDbContext, throwing EF Core's InvalidOperationException: A second operation was started on this context instance.... The GlobalExceptionHandler turns this into HTTP 500, so roughly
30–50% of first-hit requests (and writes) fail intermittently until the cache warms. In a browser it
fires constantly because the page + asset requests arrive concurrently.
Discovered while QA-testing a downstream app (SimpleModule .NET v0.0.38, PostgreSQL).
Repro
- Boot a host on a fresh database.
- Load any page in a browser (or
POST to any endpoint).
- Intermittent
500 {"title":"Internal Server Error","status":500,...}. Reload usually succeeds.
Sequential curl rarely triggers it; concurrent requests reliably do.
Server exception (abridged stack)
fail: Microsoft.EntityFrameworkCore.Query[10100]
An exception occurred while iterating over the results of a query for context type
'SimpleModule.Settings.SettingsDbContext'.
System.InvalidOperationException: A second operation was started on this context instance before a
previous operation completed. This is usually caused by different threads concurrently using the
same instance of DbContext.
at Microsoft.EntityFrameworkCore.Query...SingleOrDefaultAsync[TSource](...)
at SimpleModule.Settings.SettingsService.GetSettingAsync(String key, SettingScope scope, String userId)
at SimpleModule.AuditLogs.Middleware.AuditMiddleware.LoadAllSettingsAsync(ISettingsContracts settings)
at SimpleModule.AuditLogs.Middleware.AuditMiddleware.BuildSettingsAsync(ISettingsContracts settings)
at SimpleModule.AuditLogs.Middleware.AuditMiddleware.LoadSettingsAsync(IServiceProvider services, IFusionCache cache)
at SimpleModule.AuditLogs.Middleware.AuditMiddleware.InvokeAsync(HttpContext context)
fail: SimpleModule.Core.Exceptions.GlobalExceptionHandler[0] Unhandled exception occurred
FusionCache keys seen failing: auditlogs:request-config, setting:System:auditlogs.capture.useragent.
Root cause
AuditMiddleware.LoadAllSettingsAsync reads several audit settings; those reads run concurrently
(or the per-key setting:System:* cache factories run concurrently) against the same request-scoped
SettingsDbContext. EF Core forbids concurrent operations on one context instance. FusionCache's
per-key stampede protection does not help because the concurrency is inside a single factory
invocation across multiple keys/contexts. The settings rows being absent (default values) makes no
difference — the concurrent SingleOrDefaultAsync lookups still collide.
Suggested fixes (any one)
- Await the settings reads sequentially in
LoadAllSettingsAsync (no Task.WhenAll over one context).
- Resolve a transient
SettingsDbContext (or a fresh DI scope) inside each cache factory rather
than sharing the request-scoped one.
- Pre-warm the audit settings caches once at startup (single-threaded) before accepting traffic.
Impact
Severe: the app appears broken on first load of any page and intermittently fails writes, until the
relevant FusionCache entries warm. Affects every downstream app that uses AuditLogs + Settings.
Summary
AuditMiddlewareruns on every request and, on a cold/expired FusionCache entry, fans out multipleISettingsContracts.GetSettingAsync(...)calls that execute concurrently over a single scopedSettingsDbContext, throwing EF Core'sInvalidOperationException: A second operation was started on this context instance.... TheGlobalExceptionHandlerturns this into HTTP 500, so roughly30–50% of first-hit requests (and writes) fail intermittently until the cache warms. In a browser it
fires constantly because the page + asset requests arrive concurrently.
Discovered while QA-testing a downstream app (SimpleModule .NET v0.0.38, PostgreSQL).
Repro
POSTto any endpoint).500 {"title":"Internal Server Error","status":500,...}. Reload usually succeeds.Sequential
curlrarely triggers it; concurrent requests reliably do.Server exception (abridged stack)
FusionCache keys seen failing:
auditlogs:request-config,setting:System:auditlogs.capture.useragent.Root cause
AuditMiddleware.LoadAllSettingsAsyncreads several audit settings; those reads run concurrently(or the per-key
setting:System:*cache factories run concurrently) against the same request-scopedSettingsDbContext. EF Core forbids concurrent operations on one context instance. FusionCache'sper-key stampede protection does not help because the concurrency is inside a single factory
invocation across multiple keys/contexts. The settings rows being absent (default values) makes no
difference — the concurrent
SingleOrDefaultAsynclookups still collide.Suggested fixes (any one)
LoadAllSettingsAsync(noTask.WhenAllover one context).SettingsDbContext(or a fresh DI scope) inside each cache factory ratherthan sharing the request-scoped one.
Impact
Severe: the app appears broken on first load of any page and intermittently fails writes, until the
relevant FusionCache entries warm. Affects every downstream app that uses AuditLogs + Settings.