FastAPI Instrumentation Bypasses Middleware for Custom Router

[lannuttia]

Describe your environment

OS: Fedora 42
Python: 3.12.11
SDK Version: 1.35.0
API Version: 1.35.0

What happened?

It appears that if I use anything after splunk-opentelemetry version 2.4.0 to auto-instrument my application, the middleware for my application is bypassed when using my custom router. When the application is ran without being auto-instrumented, the middleware is not bypassed and everything works as expected.

Steps to Reproduce

Follow the steps in the README.md for my minimal issue recreation repository

Expected Result

When the application is instrumented, middleware is not bypassed for requests that use a custom API route.

Actual Result

When the application is instrumented, middleware is bypassed for requests that use a custom API route

Additional context

I am unsure of exactly where this issue is coming from but I suspect that it is related to changes made in #3529.

Would you like to implement a fix?

None

[lannuttia]

I got around to running a git bisect today and verified that this issue was introduced in 4d6893e.

EDIT: So just to be clear, my initial assumption that this was introduced in #3529 appears to be incorrect. It appears that this was actually introduced in #3012.

[outergod]

@lannuttia could you maybe check whether #3664 fixes your issue?

[lannuttia]

@outergod I just gave it a look and it does not appear to fix the issue. I still got the following error message that is effectively telling me that the AcceptHeaderVersionMiddleware is not running before the request gets to my custom VersionedAPIRoute.

ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/home/alannutt/Projects/lannuttia/recreate-otel-issue/.venv/lib64/python3.13/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        self.scope, self.receive, self.send
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/alannutt/Projects/lannuttia/recreate-otel-issue/.venv/lib64/python3.13/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/alannutt/Projects/lannuttia/recreate-otel-issue/.venv/lib64/python3.13/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/home/alannutt/Projects/lannuttia/recreate-otel-issue/.venv/lib64/python3.13/site-packages/starlette/applications.py", line 113, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/alannutt/Projects/lannuttia/recreate-otel-issue/.venv/lib64/python3.13/site-packages/opentelemetry/instrumentation/asgi/__init__.py", line 687, in __call__
    span_name, additional_attributes = self.default_span_details(scope)
                                       ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^
  File "/home/alannutt/Projects/lannuttia/opentelemetry-python-contrib/instrumentation/opentelemetry-instrumentation-fastapi/src/opentelemetry/instrumentation/fastapi/__init__.py", line 462, in _get_default_span_details
    route = _get_route_details(scope)
  File "/home/alannutt/Projects/lannuttia/opentelemetry-python-contrib/instrumentation/opentelemetry-instrumentation-fastapi/src/opentelemetry/instrumentation/fastapi/__init__.py", line 444, in _get_route_details
    match, _ = starlette_route.matches(scope)
               ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^
  File "/home/alannutt/Projects/lannuttia/recreate-otel-issue/issue/versioned_api_router.py", line 38, in matches
    if self.is_version_matching(scope):
       ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^
  File "/home/alannutt/Projects/lannuttia/recreate-otel-issue/issue/versioned_api_router.py", line 21, in is_version_matching
    raise HTTPException(
    ...<2 lines>...
    )
starlette.exceptions.HTTPException: 500: Required keys are not present in scope. Are you using the AcceptHeaderVersionMiddleware???
Current Scope is {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.3'}, 'http_version': '1.1', 'server': ('127.0.0.1', 8000), 'client': ('127.0.0.1', 60528), 'scheme': 'http', 'method': 'GET', 'root_path': '', 'path': '/', 'raw_path': b'/', 'query_string': b'', 'headers': [(b'host', b'localhost:8000'), (b'user-agent', b'curl/8.11.1'), (b'accept', b'application/vnd.vendor.v1+json')], 'state': {}, 'app': <fastapi.applications.FastAPI object at 0x7fa8e0fe5be0>}
INFO:     127.0.0.1:60528 - "GET / HTTP/1.1" 500 Internal Server Error

[lannuttia]

I have determined that the issue here is caused by the fact that the OpenTelemetryMiddleware is being injected so that it runs before my AcceptHeaderVersionMiddleware coupled with the fact that the VersionedAPIRoute that I am using has a custom matches method that depends on fields being present on the ASGI scope that are only present after the AcceptHeaderVersionMiddleware has been ran.

What I've determined is that the following sequence of events is occuring:

1. OpenTelemetryMiddleware is injected so that it runs before any user provided middleware.
2. _get_default_span_details is ran when for the HTTP connection scope.
3. _get_default_span_details calls _get_route_details which in turn calls the matches method on the router.
4. The VersionedAPIRoute I use has a custom matches method that depends on fields that don't exist until the AcceptHeaderVersionMiddleware has been ran.
5. The matches method for VersionedAPIRoute is called before AcceptHeaderVersionMiddleware is ran.
6. 💥💥💥💣💣💣💥💥💥

As for how to fix this, I don't really know. It feels like we should try to get the default span details without invoking the matches method on APIRouter subclasses. I'm just not familiar enough with how these things are used to know if that is a feasible solution.