Debugging QueuePool limit reached timeouts

QueuePool limit of size 20 overflow 10 reached, connection timed out means every connection is checked out and none came back within pool_timeout — and the first thing to establish is whether they are working or leaked, because the fixes are opposite. This is the diagnosis under handling connection leaks and pool exhaustion.

Quick Answer

Four causes, four distinguishing readings Four causes. A genuine traffic spike shows checkedout at the cap and falling back between bursts, with database CPU busy — the pool is doing its job and is simply too small. A leak shows checkedout climbing monotonically and never returning to zero even when traffic stops, which no amount of extra capacity fixes. A slow query shows checkedout at the cap with the database mostly idle, because the connections are waiting rather than working. And a pool sized for four replicas while eight are running shows a database refusing connections rather than a client-side timeout, which is a different error entirely. a traffic spike checkedout returns to 0 between bursts database CPU busy the pool is too small a leak checkedout only ever climbs idle traffic, still climbing capacity will not fix it a slow query checkedout at the cap database mostly idle they are waiting, not working wrong fleet arithmetic the DATABASE refuses connections a different error recompute the budget The distinguishing question is what checkedout does when traffic stops. A spike returns to zero; a leak does not, and that single reading separates the two most common causes.

The error:

sqlalchemy.exc.TimeoutError: QueuePool limit of size 20 overflow 10 reached,
connection timed out, timeout 30.00

The first reading — is this a leak or a spike?

from __future__ import annotations

import asyncio
import logging

from sqlalchemy.ext.asyncio import AsyncEngine

logger = logging.getLogger("db.pool")


async def report_pool(engine: AsyncEngine, interval: float = 10.0) -> None:
    """Scrape the pool on a timer — never inside a request, which biases it."""
    pool = engine.pool
    while True:
        logger.info(
            "pool",
            extra={
                "size": pool.size(),
                "checked_out": pool.checkedout(),
                "checked_in": pool.checkedin(),
                "overflow": pool.overflow(),
            },
        )
        await asyncio.sleep(interval)

If checked_out returns towards zero between bursts, the pool is working and is too small. If it only ever climbs — and stays high when traffic stops — connections are being leaked, and increasing pool_size buys time rather than fixing anything.

The second reading — what does the database think?

SELECT state, count(*), max(now() - state_change) AS longest
FROM   pg_stat_activity
WHERE  datname = current_database() AND application_name = 'orders-api'
GROUP  BY state;

A large idle in transaction count with a longest measured in minutes is a leak, and the number is usually conclusive on its own.

Execution Context & Async Workflow Integration

Under asyncio the pool behaves differently from a threaded server in one important way: a coroutine that cannot get a connection is suspended rather than blocking a thread. That means pool exhaustion does not freeze the process — it quietly accumulates waiting coroutines, all of them counting down towards the same pool_timeout, until they fail together in a burst.

That burst is the shape to recognise. A sudden cluster of TimeoutErrors at the same moment, with no gradual degradation before it, is exactly what a pool that filled up thirty seconds earlier looks like.

Three async-specific leak sources account for most cases:

# 1. A session created without a context manager, on a path that raises
session = factory()                 # checked out on first use
await do_work(session)              # raises → close() is never reached
await session.close()

# Fixed:
async with factory() as session:
    await do_work(session)


# 2. A cancelled task, where cleanup is not shielded
task = asyncio.create_task(handle(job))
task.cancel()                       # cleanup runs only if the code is structured for it


# 3. A background task holding a session created for a request
@app.post("/orders")
async def create(background: BackgroundTasks, session=Depends(get_session)):
    background.add_task(send_receipt, session)     # the session is closed by then

The third is worth stating plainly because it fails in two ways at once: the session has been closed by the dependency's finally before the task runs, and if it had not been, the task would be holding a request-scoped connection for an unbounded time.

The reliable structural fix for all three is that every session is created by a context manager and never handed to anything that outlives it. Grepping for factory() or async_sessionmaker( outside the composition root finds the exceptions in a mature codebase faster than any instrumentation.

The order to look in Four steps, cheapest first. Read the pool counters — checkedout, checkedin and overflow — which tells you whether the pool is saturated at all. Check whether the database is busy: a saturated pool with an idle database means the connections are waiting rather than working. Query pg_stat_activity for sessions idle in transaction, which is the signature of a transaction opened and never closed and is the single most common leak. Only if none of that explains it, capture the stack trace at checkout time so the code holding each connection can be named. That last step is expensive and is rarely needed once the first three have been done. read pool.checkedout() and overflow() is the pool actually saturated? seconds is the database busy? saturated pool, idle server = waiting not working pg_stat_activity: idle in transaction the classic leak signature one query capture checkout stack traces names the code holding them The third step finds most leaks on its own, which is why the expensive fourth step is worth keeping behind a flag rather than running by default.

Resolving Warnings, Errors & Common Mistakes

SymptomRoot CauseProduction Fix
TimeoutError: QueuePool limit ... reached with the database idleConnections are held, not working — a leak.Find the leak; raising pool_size only delays it.
The same error with the database at high CPUThe pool is genuinely too small for the query load.Faster queries first, then more capacity — in that order.
FATAL: remaining connection slots are reservedThe database is out of connections, not the pool.Recompute the fleet budget: (pool_size + max_overflow) × replicas.
Errors arrive in a sudden burst rather than graduallySuspended coroutines all reach pool_timeout together.Expected under async; treat the burst as one event, not many.
checkedout() never falls, even overnightA session opened at start-up and never closed, or a task that leaked one.Look for module-level session creation and for background tasks holding request sessions.
Timeouts only during deploysOld and new replicas both hold pools during the rollout.Size the budget for one generation of overlap.

Advanced: Capturing Where the Connections Went

When the counters say "leak" but nobody can find it, recording the stack at checkout names the code directly. It is expensive, so it belongs behind a flag rather than on by default.

"""Record where each connection was checked out, and report the long-held ones.

Capturing a stack per checkout costs real time — enable it deliberately, during
an investigation, not as standing configuration.
"""
from __future__ import annotations

import time
import traceback

from sqlalchemy import event
from sqlalchemy.ext.asyncio import AsyncEngine

HELD_TOO_LONG_SECONDS = 30.0


def install_checkout_tracking(async_engine: AsyncEngine) -> None:
    engine = async_engine.sync_engine

    @event.listens_for(engine, "checkout")
    def _checkout(dbapi_connection, connection_record, connection_proxy):
        connection_record.info["checked_out_at"] = time.monotonic()
        # Skip the two frames belonging to SQLAlchemy's own event machinery.
        connection_record.info["checked_out_by"] = "".join(
            traceback.format_stack(limit=12)[:-2]
        )

    @event.listens_for(engine, "checkin")
    def _checkin(dbapi_connection, connection_record):
        started = connection_record.info.pop("checked_out_at", None)
        stack = connection_record.info.pop("checked_out_by", None)
        if started is None:
            return
        held = time.monotonic() - started
        if held > HELD_TOO_LONG_SECONDS:
            logger.warning("connection held %.1fs, checked out at:\n%s", held, stack)

Connections that are never returned never reach checkin, so the long-held report alone will not find a true leak. Pair it with a periodic sweep over the records still outstanding:

async def report_outstanding(engine: AsyncEngine, older_than: float = 60.0) -> None:
    """Log every connection checked out for longer than `older_than` seconds."""
    now = time.monotonic()
    for record in list(getattr(engine.pool, "_pool", ()).queue):   # implementation detail
        started = record.info.get("checked_out_at")
        if started and now - started > older_than:
            logger.error(
                "connection outstanding %.0fs, checked out at:\n%s",
                now - started, record.info.get("checked_out_by", "<no stack>"),
            )

The stack points at the code that took the connection, which in practice names the leak within one run. The reliance on a private attribute is deliberate and acceptable for a diagnostic tool that is enabled during an incident and removed afterwards — it is not something to ship permanently.

Once the leak is found, the fix is nearly always structural: a session that was not created by a context manager, or one handed to something that outlives it.

Working, or merely held Two saturation shapes. In the first, every connection is genuinely executing a statement: the database is busy, the queries are slow or numerous, and the fix is to make them faster or to add capacity. In the second, connections are checked out by code that is no longer using them — a session never closed, a transaction never committed, a task cancelled without cleanup — so the database is idle while the pool is empty. Adding capacity to the second case delays the outage by exactly as long as it takes to leak the new connections too, which is why distinguishing them before acting matters. connections are working the database is busy queries are slow or numerous checkedout falls back between bursts fix: faster queries, or more capacity connections are merely held the database is idle sessions never closed checkedout never falls back fix: find the leak — capacity delays it only Raising pool_size against a leak is the most common wrong response, and it converts a quick outage into a slower one that recurs at a less convenient hour.

Sizing the Pool After the Leak Is Fixed

Only once the counters return to zero between bursts is it worth revisiting the size, and the arithmetic is about the fleet rather than about one process.

The budget is (pool_size + max_overflow) × replica_count, and it must fit inside the database's usable connections after the administrative reserve. That calculation and the failure modes when a replica count changes are covered in tuning connection pools for cloud databases; the part that matters here is that a pool timeout is sometimes the right behaviour. A pool sized to its share of the budget will time out under a spike, and the alternative — a larger pool — pushes the failure onto the database, where it affects every service rather than one.

The number that actually determines how large the pool needs to be is hold time, not request rate:

# Concurrency a pool can sustain ≈ pool_size ÷ average hold time
# 20 connections × (1 / 0.03 s) ≈ 660 concurrent requests
# 20 connections × (1 / 0.21 s) ≈  95 concurrent requests

Which means the cheapest capacity increase is usually not more connections but shorter holds: commit before an external call, move a slow serialisation outside the session, and stop holding a connection across work that does not touch the database. A single external HTTP call inside a session boundary can cost an order of magnitude of pool capacity, and removing it is free.

Set pool_timeout deliberately too. Thirty seconds is a long time for a user to wait for a connection that may never come; five to ten gives a faster failure and a clearer signal, and pairs well with a retry at the edge. A timeout that fires quickly and visibly is more useful than one that queues requests until they are irrelevant.

Frequently Asked Questions

Should I just increase pool_size when I see this error?

Only after establishing that connections are working rather than leaked. Against a leak, a larger pool delays the outage by however long it takes to leak the extra connections — and moves the failure to a less convenient time. Against genuine load it is reasonable, provided the fleet budget still fits the database.

What does a negative overflow() mean?

It is reported relative to pool_size, so a value of -20 on a pool of 20 means no overflow connections are in use at all. Values approaching max_overflow mean the pool is drawing on burst capacity; sitting pinned at max_overflow means it is exhausted.

Why do the timeouts all arrive at once?

Because coroutines waiting for a connection are suspended rather than blocked, so they accumulate silently and then reach pool_timeout together. The burst is one event — the pool filling — observed thirty seconds later.

Is pool_pre_ping related to this error?

No. Pre-ping validates a connection at checkout and protects against stale connections; it does nothing about there being no connection to check out. The two errors are frequently confused because both surface as connection problems under load.