Skip to content

chore: refactor concurrency.py to run sync context managers in a single thread#5707

Closed
adriangb wants to merge 14 commits into
fastapi:masterfrom
adriangb:cm-in-same-thread
Closed

chore: refactor concurrency.py to run sync context managers in a single thread#5707
adriangb wants to merge 14 commits into
fastapi:masterfrom
adriangb:cm-in-same-thread

Conversation

@adriangb

@adriangb adriangb commented Nov 28, 2022

Copy link
Copy Markdown
Contributor

I'm hoping that this (maybe) helps resolve issues like MagicStack/asyncpg#978

This change makes sync context managers acquire and hold onto a thread for the entirety of their execution so that we don't split __enter__ and __exit__ calls across threads. I think this will resolve a whole class of intermittent bugs that might crop up for anything using thread locals or context variables.

The main "con" of this approach is that we won't "free" the thread in between __enter__ and __exit__ which means more contention for threads in the thread pool. In other words, despite the 40 thread default limit that anyio imposes it was previously possible to run more than 40 sync context managers as FastAPI dependencies since the 41st would just have to wait for any __enter__ to finish before it could acquire a thread pool. With this change running more than 40 sync context managers will result in a deadlock. I see three options:

  1. Do nothing. If users are running into capacity limiters they should probably be managing their own threads instead of relying on automagic behavior from their web framework.
  2. Raise an error if we hit the capacity limit (instead of deadlocking)
  3. Allow users to pass in their own capacity limit to each APIRoute (maybe with a global one on the FastAPI object as well)
  4. Both 2 and 3

@adriangb adriangb changed the title chore: refactor concurrency.py to run sync context managers in a sing… chore: refactor concurrency.py to run sync context managers in a single thread Nov 28, 2022
@github-actions

Copy link
Copy Markdown
Contributor

📝 Docs preview for commit e24e016 at: https://63851d6f5737095d3317664e--fastapi.netlify.app

@iudeen

iudeen commented Nov 28, 2022

Copy link
Copy Markdown
Contributor

This means more resource optimised DB connections?🤔

(just wondering)

@adriangb

Copy link
Copy Markdown
Contributor Author

Not really. It just means that sync context manager's __enter__ and __exit__ run in the same thread. In fact this might make things worse in some cases because the thread is checked out from the thread pool for the entirety of the context managers life. But I think it also eliminates a whole class of intermittent hard to reason about or figure out bugs.

@iudeen

iudeen commented Nov 28, 2022

Copy link
Copy Markdown
Contributor

Make sense, better house-keeping!

Thanks for your explanation btw! 😅

@github-actions

Copy link
Copy Markdown
Contributor

📝 Docs preview for commit e45a801 at: https://63852343acf33a6f2fa9c78e--fastapi.netlify.app

@adriangb adriangb marked this pull request as ready for review November 28, 2022 21:15
@adriangb

Copy link
Copy Markdown
Contributor Author

marked as ready, cc @tiangolo

Comment thread fastapi/concurrency.py
await anyio.to_thread.run_sync(
cm.__exit__, None, None, None, limiter=exit_limiter
cm: ContextManager[T],
limiter: Optional[anyio.CapacityLimiter] = None,

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is currently unused but IMO exposing it makes sense for future customizability. Happy to remove it if not

@adriangb

Copy link
Copy Markdown
Contributor Author

@kglee0 or @alexted mind trying out this branch? It's a shot in the dark but worth at least confirming the null hypothesis.

@github-actions

Copy link
Copy Markdown
Contributor

📝 Docs preview for commit ac93694 at: https://638663ca7c5147091cf84317--fastapi.netlify.app

@github-actions

github-actions Bot commented Dec 3, 2022

Copy link
Copy Markdown
Contributor

📝 Docs preview for commit 6ba173b at: https://638a9f62bb6c0b319b2736fb--fastapi.netlify.app

@alexted

alexted commented Dec 3, 2022

Copy link
Copy Markdown

The Issue here

@adriangb

adriangb commented Dec 3, 2022

Copy link
Copy Markdown
Contributor Author

Awesome! Did you get to test this branch?

@alexted

alexted commented Dec 3, 2022

Copy link
Copy Markdown

Awesome! Did you get to test this branch?

Yeah, it didn't have any effect.

@github-actions

Copy link
Copy Markdown
Contributor

📝 Docs preview for commit 5ac3f62 at: https://63a1f7af8c940416a38c635d--fastapi.netlify.app

@github-actions

Copy link
Copy Markdown
Contributor

📝 Docs preview for commit 04347ed at: https://63a4d7eaeccde823d1092dc0--fastapi.netlify.app

@jakekaplan

Copy link
Copy Markdown

Is this pr possible to merge sometime soon? Coming here from this thread: MagicStack/asyncpg#978

@adriangb

Copy link
Copy Markdown
Contributor Author

For visibility: #8424 (comment)

@YuriiMotov

YuriiMotov commented Jul 9, 2025

Copy link
Copy Markdown
Member

The test from this PR doesn't fail with current FastAPI version (0.116.0):

import threading
from contextvars import ContextVar

from fastapi import Depends, FastAPI
from fastapi.testclient import TestClient

ctx: ContextVar[str] = ContextVar("ctx")

async def context_same_thread():
    # test both a real world use case and a
    # simpler synthetic check
    ctx.set("foo")
    ident = threading.get_ident()
    yield
    assert ctx.get() == "foo"
    assert ident == threading.get_ident()

app = FastAPI()

@app.get("/check_cm_same_thread")
async def check_cm_same_thread(_=Depends(context_same_thread)):
    return None

def test_sync_runs_in_single_thread():
    client = TestClient(app)
    response = client.get("/check_cm_same_thread")
    response.raise_for_status()

@adriangb, could you please double-check? Seems that it has been fixed.

UPD: shouldn't context_same_thread be sync?

@github-actions

github-actions Bot commented Aug 9, 2025

Copy link
Copy Markdown
Contributor

As this PR has been waiting for the original user for a while but seems to be inactive, it's now going to be closed. But if there's anyone interested, feel free to create a new PR.

@github-actions github-actions Bot closed this Aug 9, 2025
@Kludex Kludex reopened this Aug 10, 2025
@Kludex

Kludex commented Aug 10, 2025

Copy link
Copy Markdown
Member

It's unfair to expect users to reply when review takes so long.

Isn't there a less aggressive strategy? Like pinging the user again a couple of weeks?

@github-actions github-actions Bot removed the waiting label Aug 10, 2025
@YuriiMotov

YuriiMotov commented Aug 13, 2025

Copy link
Copy Markdown
Member

It's unfair to expect users to reply when review takes so long.
Isn't there a less aggressive strategy? Like pinging the user again a couple of weeks?

@Kludex, I agree we should change this.
We will update the script to add more explanations of how we work on external PRs and why we do it this way, also we will add a reminder 1 week before closing PR.

What should we do with this PR now?
@adriangb, are you still interested in this PR?
Could you please take a look at my comment?

@github-actions github-actions Bot added the conflicts Automatically generated when a PR has a merge conflict label Sep 5, 2025
@github-actions

github-actions Bot commented Sep 5, 2025

Copy link
Copy Markdown
Contributor

This pull request has a merge conflict that needs to be resolved.

@tiangolo

Copy link
Copy Markdown
Member

I'm realizing that this was related to a change done a while ago with dependencies with yield, this was solved in 0.74.0 (https://fastapi.tiangolo.com/release-notes/#0740)

The exit code after yield in a dependency was being executed in a different async context (contextvars context), so the value of contextvars was different than the original.

This was because custom middlewares would internally create an AnyIO task group, which creates a new context.

That logic was then refactored once again in 0.106.0 (https://fastapi.tiangolo.com/release-notes/#01060).

And is now refactored again here 😅 : #14099 , which will be released in version 0.118.0


But the most important point is, a test for this use case was added (in the first PR in the version described above) here: https://github.com/fastapi/fastapi/blob/master/tests/test_dependency_contextvars.py

So that should ensure this is handled/fixed and stays handled and fixed. 🚀

Given that, I'll now close this one, but if anyone faces a similar problem again, please create a new discussion with an example to replicate it so we can take a look. 🤓

Thanks all! 🍰

@tiangolo tiangolo closed this Sep 22, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

conflicts Automatically generated when a PR has a merge conflict p2 refactor

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants