Skip to content

AuditMiddleware races SettingsDbContext (concurrent GetSettingAsync on one scoped context) → intermittent 500s on any request #232

Description

@antosubash

Summary

AuditMiddleware runs on every request and, on a cold/expired FusionCache entry, fans out multiple
ISettingsContracts.GetSettingAsync(...) calls that execute concurrently over a single scoped
SettingsDbContext
, throwing EF Core's InvalidOperationException: A second operation was started on this context instance.... The GlobalExceptionHandler turns this into HTTP 500, so roughly
30–50% of first-hit requests (and writes) fail intermittently until the cache warms. In a browser it
fires constantly because the page + asset requests arrive concurrently.

Discovered while QA-testing a downstream app (SimpleModule .NET v0.0.38, PostgreSQL).

Repro

  1. Boot a host on a fresh database.
  2. Load any page in a browser (or POST to any endpoint).
  3. Intermittent 500 {"title":"Internal Server Error","status":500,...}. Reload usually succeeds.
    Sequential curl rarely triggers it; concurrent requests reliably do.

Server exception (abridged stack)

fail: Microsoft.EntityFrameworkCore.Query[10100]
  An exception occurred while iterating over the results of a query for context type
  'SimpleModule.Settings.SettingsDbContext'.
System.InvalidOperationException: A second operation was started on this context instance before a
  previous operation completed. This is usually caused by different threads concurrently using the
  same instance of DbContext.
   at Microsoft.EntityFrameworkCore.Query...SingleOrDefaultAsync[TSource](...)
   at SimpleModule.Settings.SettingsService.GetSettingAsync(String key, SettingScope scope, String userId)
   at SimpleModule.AuditLogs.Middleware.AuditMiddleware.LoadAllSettingsAsync(ISettingsContracts settings)
   at SimpleModule.AuditLogs.Middleware.AuditMiddleware.BuildSettingsAsync(ISettingsContracts settings)
   at SimpleModule.AuditLogs.Middleware.AuditMiddleware.LoadSettingsAsync(IServiceProvider services, IFusionCache cache)
   at SimpleModule.AuditLogs.Middleware.AuditMiddleware.InvokeAsync(HttpContext context)
fail: SimpleModule.Core.Exceptions.GlobalExceptionHandler[0] Unhandled exception occurred

FusionCache keys seen failing: auditlogs:request-config, setting:System:auditlogs.capture.useragent.

Root cause

AuditMiddleware.LoadAllSettingsAsync reads several audit settings; those reads run concurrently
(or the per-key setting:System:* cache factories run concurrently) against the same request-scoped
SettingsDbContext. EF Core forbids concurrent operations on one context instance. FusionCache's
per-key stampede protection does not help because the concurrency is inside a single factory
invocation across multiple keys/contexts. The settings rows being absent (default values) makes no
difference — the concurrent SingleOrDefaultAsync lookups still collide.

Suggested fixes (any one)

  • Await the settings reads sequentially in LoadAllSettingsAsync (no Task.WhenAll over one context).
  • Resolve a transient SettingsDbContext (or a fresh DI scope) inside each cache factory rather
    than sharing the request-scoped one.
  • Pre-warm the audit settings caches once at startup (single-threaded) before accepting traffic.

Impact

Severe: the app appears broken on first load of any page and intermittently fails writes, until the
relevant FusionCache entries warm. Affects every downstream app that uses AuditLogs + Settings.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions