Python BrokenPipeError

Issue Description

After a recent deployment to Vercel production environment, approximately 5% of requests result in BrokenPipeError. The issue appears to be related to connection handling, particularly during concurrent requests. On an older deployment I got no errors. No dependancies have been updated and no real code changes.

Key characteristics:

Requests complete successfully (200 status code) before the error occurs. The error appears in logs but doesn't affect the response to clients. Error occurs across different endpoints, including health checks and OPTIONS requests. More frequent when there are multiple concurrent requests. Cannot be reproduced locally or in Vercel preview deployments.

Additional Context

The application is a API proxy that forwards requests using httpx. All functions are asynchronous. Issue appears to be environment-specific to Vercel production (I cannot reproduce it in preview). Error occurs even on simple endpoints like health checks. Same dependency versions are used as in previous working deployments. Might be caused by this: https://github.com/vercel/vercel/pull/12557 update?

GitHub

[python] add support for streaming by QuiiBz · Pull Request #12557 · vercel/vercel

HTTP handler WSGI handler ASGI handler Logs levels Run all tests Add support for HTTP streaming and other important feature for Python Vercel Functions, opt-in via the VERCEL_FORCE_PYTHON_STRE...

Impact While the errors don't affect client responses (requests complete successfully), they pollute the logs and might indicate underlying issues with connection handling that could impact performance or stability.

Environment

Python 3.9
FastAPI 0.115.3
Starlette 0.41.0
uvicorn  0.30.6
httpx 0.24.0
pipenv
Node 18.x
Other dependencies: aiomysql 0.2.0, loguru 0.7.3, pydantic 2.7.3, sqlalchemy 2.0.37, uvicorn 0.30.6

Error log from POST request:

File "/var/lang/lib/python3.9/socketserver.py", line 683, in process_request_thread
self.finish_request(request, client_address)
File "/var/lang/lib/python3.9/socketserver.py", line 360, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/var/lang/lib/python3.9/socketserver.py", line 747, in __init__
self.handle()
File "/var/lang/lib/python3.9/http/server.py", line 433, in handle
self.handle_one_request()
File "/var/task/vc__handler__python.py", line 203, in handle_one_request
self.handle_request()
File "/var/task/vc__handler__python.py", line 376, in handle_request
loop.run_until_complete(asgi_task)
File "/var/lang/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
return future.result()
File "/var/task/fastapi/applications.py", line 1054, in __call__
await super().__call__(scope, receive, send)
File "/var/task/starlette/applications.py", line 113, in __call__
await self.middleware_stack(scope, receive, send)
File "/var/task/starlette/middleware/errors.py", line 187, in __call__
raise exc
File "/var/task/starlette/middleware/errors.py", line 165, in __call__
await self.app(scope, receive, _send)
File "/var/task/starlette/middleware/base.py", line 189, in __call__
response_sent.set()
File "/var/lang/lib/python3.9/contextlib.py", line 137, in __exit__
self.gen.throw(typ, value, traceback)
File "/var/task/starlette/_utils.py", line 82, in collapse_excgroups
raise exc
File "/var/task/starlette/middleware/base.py", line 188, in __call__
await response(scope, wrapped_receive, send)
File "/var/task/starlette/middleware/base.py", line 225, in __call__
await send({"type": "http.response.body", "body": b"", "more_body": False})
File "/var/task/starlette/middleware/errors.py", line 162, in _send
await send(message)
File "/var/task/vc__handler__python.py", line 368, in send
self.wfile.write(event['body'])
File "/var/lang/lib/python3.9/socketserver.py", line 826, in write
self._sock.sendall(b)
BrokenPipeError: [Errno 32] Broken pipe

Error log from OPTIONS request:

.py", line 1054, in __call__
await super().__call__(scope, receive, send)
File "/var/task/starlette/applications.py", line 113, in __call__
await self.middleware_stack(scope, receive, send)
File "/var/task/starlette/middleware/errors.py", line 187, in __call__
raise exc
File "/var/task/starlette/middleware/errors.py", line 165, in __call__
await self.app(scope, receive, _send)
File "/var/task/starlette/middleware/base.py", line 189, in __call__
response_sent.set()
File "/var/lang/lib/python3.9/contextlib.py", line 137, in __exit__
self.gen.throw(typ, value, traceback)
File "/var/task/starlette/_utils.py", line 82, in collapse_excgroups
raise exc
File "/var/task/starlette/middleware/base.py", line 188, in __call__
await response(scope, wrapped_receive, send)
File "/var/task/starlette/middleware/base.py", line 225, in __call__
await send({"type": "http.response.body", "body": b"", "more_body": False})
File "/var/task/starlette/middleware/errors.py", line 162, in _send
await send(message)
File "/var/task/vc__handler__python.py", line 368, in send
self.wfile.write(event['body'])
File "/var/lang/lib/python3.9/socketserver.py", line 826, in write
self._sock.sendall(b)
BrokenPipeError: [Errno 32] Broken pipe
----------------------------------------
----------------------------------------
Exception occurred during processing of request from ('127.0.0.1', 60408)
+ Exception Group Traceback (most recent call last):
|   File "/var/task/starlette/_utils.py", line 76, in collapse_excgroups
|     yield
|   File "/var/task/starlette/middleware/base.py", line 189, in __call__
|     response_sent.set()
|   File "/var/task/anyio/_backends/_asyncio.py", line 767, in __aexit__
|     raise BaseExceptionGroup(
| exceptiongroup.ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
+-+---------------- 1 ----------------
| Traceback (most recent call last):
|   File "/var/lang/lib/python3.9/socketserver.py", line 683, in process_request_thread
|     self.finish_request(request, client_address)
|   File "/var/lang/lib/python3.9/socketserver.py", line 360, in finish_request
|     self.RequestHandlerClass(request, client_address, self)
|   File "/var/lang/lib/python3.9/socketserver.py", line 747, in __init__
|     self.handle()
|   File "/var/lang/lib/python3.9/http/server.py", line 433, in handle
|     self.handle_one_request()
|   File "/var/task/vc__handler__python.py", line 203, in handle_one_request
|     self.handle_request()
|   File "/var/task/vc__handler__python.py", line 376, in handle_request
|     loop.run_until_complete(asgi_task)
|   File "/var/lang/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
|     return future.result()
|   File "/var/task/fastapi/applications.py", line 1054, in __call__
|     await super().__call__(scope, receive, send)
|   File "/var/task/starlette/applications.py", line 113, in __call__
|     await self.middleware_stack(scope, receive, send)
|   File "/var/task/starlette/middleware/errors.py", line 187, in __call__
|     raise exc
|   File "/var/task/starlette/middleware/errors.py", line 165, in __call__
|     await self.app(scope, receive, _send)
|   File "/var/task/starlette/middleware/base.py", line 189, in __call__
|     response_sent.set()
|   File "/var/lang/lib/python3.9/contextlib.py", line 137, in __exit__
|     self.gen.throw(typ, value, traceback)
|   File "/var/task/starlette/_utils.py", line 82, in collapse_excgroups
|     raise exc
|   File "/var/task/starlette/middleware/base.py", line 188, in __call__
|     await response(scope, wrapped_receive, send)
|   File "/var/task/starlette/middleware/base.py", line 225, in __call__
|     await send({"type": "http.response.body", "body": b"", "more_body": False})
|   File "/var/task/starlette/middleware/errors.py", line 162, in _send
|     await send(message)

Maybe unrelated

I sometimes get this in the log:

/var/task/vc__handler__python.py:346: DeprecationWarning: The loop argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10.
  app_queue = asyncio.Queue(loop=loop)

[packages] fastapi = "==0.115.3" uvicorn = "==0.30.6" [requires] python_version = "3.9" # main.py from fastapi import FastAPI from starlette.middleware.base import BaseHTTPMiddleware import asyncio app = FastAPI() class RequestCounterMiddleware(BaseHTTPMiddleware): def __init__(self, app): super().__init__(app) self.request_count = 0 self.lock = asyncio.Lock() async def dispatch(self, request, call_next): async with self.lock: self.request_count += 1 current_count = self.request_count response = await call_next(request) response.headers["X-Request-Count"] = str(current_count) return response app.add_middleware(RequestCounterMiddleware) @app.get("/") async def root(): return {"message": "Hello World"} @app.head("/health") async def health(): return {"status": "ok"}

File "/var/lang/lib/python3.9/socketserver.py", line 683, in process_request_thread self.finish_request(request, client_address) File "/var/lang/lib/python3.9/socketserver.py", line 360, in finish_request self.RequestHandlerClass(request, client_address, self) File "/var/lang/lib/python3.9/socketserver.py", line 747, in __init__ self.handle() File "/var/lang/lib/python3.9/http/server.py", line 433, in handle self.handle_one_request() File "/var/task/vc__handler__python.py", line 203, in handle_one_request self.handle_request() File "/var/task/vc__handler__python.py", line 376, in handle_request loop.run_until_complete(asgi_task) File "/var/lang/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete return future.result() File "/var/task/fastapi/applications.py", line 1054, in __call__ await super().__call__(scope, receive, send) File "/var/task/starlette/applications.py", line 113, in __call__ await self.middleware_stack(scope, receive, send) File "/var/task/starlette/middleware/errors.py", line 187, in __call__ raise exc File "/var/task/starlette/middleware/errors.py", line 165, in __call__ await self.app(scope, receive, _send) File "/var/task/starlette/middleware/base.py", line 189, in __call__ response_sent.set() File "/var/lang/lib/python3.9/contextlib.py", line 137, in __exit__ self.gen.throw(typ, value, traceback) File "/var/task/starlette/_utils.py", line 82, in collapse_excgroups raise exc File "/var/task/starlette/middleware/base.py", line 188, in __call__ await response(scope, wrapped_receive, send) File "/var/task/starlette/middleware/base.py", line 225, in __call__ await send({"type": "http.response.body", "body": b"", "more_body": False}) File "/var/task/starlette/middleware/errors.py", line 162, in _send await send(message) File "/var/task/vc__handler__python.py", line 368, in send self.wfile.write(event['body']) File "/var/lang/lib/python3.9/socketserver.py", line 826, in write self._sock.sendall(b) BrokenPipeError: [Errno 32] Broken pipe

.py", line 1054, in __call__ await super().__call__(scope, receive, send) File "/var/task/starlette/applications.py", line 113, in __call__ await self.middleware_stack(scope, receive, send) File "/var/task/starlette/middleware/errors.py", line 187, in __call__ raise exc File "/var/task/starlette/middleware/errors.py", line 165, in __call__ await self.app(scope, receive, _send) File "/var/task/starlette/middleware/base.py", line 189, in __call__ response_sent.set() File "/var/lang/lib/python3.9/contextlib.py", line 137, in __exit__ self.gen.throw(typ, value, traceback) File "/var/task/starlette/_utils.py", line 82, in collapse_excgroups raise exc File "/var/task/starlette/middleware/base.py", line 188, in __call__ await response(scope, wrapped_receive, send) File "/var/task/starlette/middleware/base.py", line 225, in __call__ await send({"type": "http.response.body", "body": b"", "more_body": False}) File "/var/task/starlette/middleware/errors.py", line 162, in _send await send(message) File "/var/task/vc__handler__python.py", line 368, in send self.wfile.write(event['body']) File "/var/lang/lib/python3.9/socketserver.py", line 826, in write self._sock.sendall(b) BrokenPipeError: [Errno 32] Broken pipe ---------------------------------------- ---------------------------------------- Exception occurred during processing of request from ('127.0.0.1', 60408) + Exception Group Traceback (most recent call last): | File "/var/task/starlette/_utils.py", line 76, in collapse_excgroups | yield | File "/var/task/starlette/middleware/base.py", line 189, in __call__ | response_sent.set() | File "/var/task/anyio/_backends/_asyncio.py", line 767, in __aexit__ | raise BaseExceptionGroup( | exceptiongroup.ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception) +-+---------------- 1 ---------------- | Traceback (most recent call last): | File "/var/lang/lib/python3.9/socketserver.py", line 683, in process_request_thread | self.finish_request(request, client_address) | File "/var/lang/lib/python3.9/socketserver.py", line 360, in finish_request | self.RequestHandlerClass(request, client_address, self) | File "/var/lang/lib/python3.9/socketserver.py", line 747, in __init__ | self.handle() | File "/var/lang/lib/python3.9/http/server.py", line 433, in handle | self.handle_one_request() | File "/var/task/vc__handler__python.py", line 203, in handle_one_request | self.handle_request() | File "/var/task/vc__handler__python.py", line 376, in handle_request | loop.run_until_complete(asgi_task) | File "/var/lang/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete | return future.result() | File "/var/task/fastapi/applications.py", line 1054, in __call__ | await super().__call__(scope, receive, send) | File "/var/task/starlette/applications.py", line 113, in __call__ | await self.middleware_stack(scope, receive, send) | File "/var/task/starlette/middleware/errors.py", line 187, in __call__ | raise exc | File "/var/task/starlette/middleware/errors.py", line 165, in __call__ | await self.app(scope, receive, _send) | File "/var/task/starlette/middleware/base.py", line 189, in __call__ | response_sent.set() | File "/var/lang/lib/python3.9/contextlib.py", line 137, in __exit__ | self.gen.throw(typ, value, traceback) | File "/var/task/starlette/_utils.py", line 82, in collapse_excgroups | raise exc | File "/var/task/starlette/middleware/base.py", line 188, in __call__ | await response(scope, wrapped_receive, send) | File "/var/task/starlette/middleware/base.py", line 225, in __call__ | await send({"type": "http.response.body", "body": b"", "more_body": False}) | File "/var/task/starlette/middleware/errors.py", line 162, in _send | await send(message)