Refreshing stale objects with populate_existing

A query returns the object the session already holds rather than the row it just fetched, so use .execution_options(populate_existing=True) to overwrite those attributes from the result — or await session.refresh(obj) for one object, and a fresh session when the whole unit of work should see current data. This guide belongs to session lifecycle and scope management.

Quick Answer

Re-running a query does not refresh objects the session already has. The identity map returns the existing instance and discards the fetched values for attributes that are already loaded.

Before — the second query looks like a refresh and is not:

from sqlalchemy import select

from shop.models import Order


async def poll_status(session, order_id: int) -> str:
    order = await session.scalar(select(Order).where(Order.id == order_id))
    return order.status
# Called twice in one long-lived session: the second call returns the status
# from the first, even though another transaction changed the row.

After — the query overwrites what it returns:

from sqlalchemy import select

from shop.models import Order


async def poll_status(session, order_id: int) -> str:
    order = await session.scalar(
        select(Order)
        .where(Order.id == order_id)
        .execution_options(populate_existing=True)
    )
    return order.status

Or, for a single object already in hand:

await session.refresh(order)                       # one SELECT, overwrites attributes
await session.refresh(order, ["status"])           # just that column
await session.refresh(order, ["lines"])            # and a relationship

populate_existing=True applies to every object the query returns, which is what makes it the right tool for a worker re-reading a batch. refresh() is the targeted version for one object.

Execution Context & Async Workflow Integration

The identity map is a dictionary from primary key to object, per session. It guarantees that within one session there is exactly one Python object per database row, which is what makes the unit of work coherent: two code paths that load the same order get the same instance, and a change made through one is visible to the other.

The consequence is that loading is not refreshing. When a query returns a row whose primary key is already in the identity map, the ORM returns the existing object, and by default it does not overwrite attributes that are already loaded — because those attributes may contain pending changes the session is about to flush. Silently discarding them would lose work.

populate_existing=True says "this query's values win". Attributes are overwritten from the result, pending changes to them are discarded, and eagerly loaded relationships are re-loaded as well. That last part is easy to miss and often the reason to reach for it: a re-query with different loader options only takes effect on existing objects when populate_existing is set.

session.refresh(obj) is the per-object equivalent, issuing a SELECT for that row. refresh() also accepts an attribute list, which is how a single column or a single relationship is reloaded without fetching everything.

session.expire(obj) is the third mechanism and the one to avoid under async. It marks attributes stale without querying, so the next attribute access triggers a load — and lazy I/O from attribute access is exactly what AsyncSession cannot do, so the access raises MissingGreenlet. Expiring is only useful in async code when followed by an explicit refresh(), at which point refresh() alone would have done.

The same reasoning explains expire_on_commit. Its default, True, expires every attribute of every object on commit, which in synchronous code means the next access transparently reloads and in async code means it raises. Setting expire_on_commit=False on the factory is standard for async applications, and its trade-off — objects keep the values they had at commit time, which may be stale — is the subject of using expire_on_commit=False in FastAPI dependencies.

Worth stating plainly: with expire_on_commit=False and a long-lived session, nothing refreshes automatically. That combination is comfortable and does require deciding, per read, whether current data matters.

Resolving Warnings, Errors & Common Mistakes

The disappearing-changes row is worth being explicit about, because it is the reason the default exists. populate_existing is a deliberate instruction that the database wins:

order.status = "cancelled"          # a pending change, not flushed

order = await session.scalar(
    select(Order).where(Order.id == order.id).execution_options(populate_existing=True)
)
order.status                        # the database value; the pending change is gone

That is correct behaviour for "show me what is actually stored", and wrong if the session was mid-way through a unit of work. Refresh at the start of a unit of work, not in the middle.

The deleted-row case has a clean idiom. refresh() raises when the row is gone, which is often exactly what a poller wants to know, but the query form is easier to branch on:

from sqlalchemy import select

from shop.models import Order


async def current_or_none(session, order_id: int) -> Order | None:
    return await session.scalar(
        select(Order).where(Order.id == order_id).execution_options(populate_existing=True)
    )

If the row was deleted by another transaction, that returns None — and note that the stale object remains in the identity map either way, so code holding a reference to it keeps seeing the old values. session.expunge(order) removes it, which is the distinction covered in understanding Session.expunge vs Session.clear.

Advanced: Polling Workers and Set-Based Writes

Two situations produce staleness routinely, and each has an idiomatic answer.

A long-lived worker polling for work. A worker that runs for hours with one session accumulates every row it has ever seen, and its polling query returns those objects unchanged. populate_existing on the poll is the fix; expunging processed objects is the complement, because otherwise the identity map grows without bound:

import asyncio

from sqlalchemy import select

from shop.db import Session
from shop.models import Job


async def worker_loop(poll_interval: float = 1.0) -> None:
    async with Session() as session:
        while True:
            jobs = list(await session.scalars(
                select(Job)
                .where(Job.status == "pending")
                .order_by(Job.run_after)
                .limit(20)
                .execution_options(populate_existing=True)
            ))
            for job in jobs:
                await handle(session, job)
                await session.commit()
                session.expunge(job)          # keep the identity map bounded
            if not jobs:
                await asyncio.sleep(poll_interval)

A session per poll iteration is simpler still, and for most workers it is the better shape: no accumulation, no refresh logic, and one fewer thing to reason about. The version above is for workers that genuinely need session-scoped state across iterations.

After a set-based write. An ORM-enabled update() or delete() changes rows without loading them, so objects the session holds are stale unless the statement synchronises them. The default, synchronize_session="auto", does synchronise — but False, chosen for performance, does not:

from sqlalchemy import update

from shop.models import Account

await session.execute(
    update(Account)
    .where(Account.plan == "trial")
    .values(plan="expired")
    .execution_options(synchronize_session=False)
)
# Any Account object in this session still says "trial".

Three ways out, in order of preference: let it synchronise ("auto" or "fetch"); do the bulk write in a session that holds none of those objects; or re-read with populate_existing afterwards. The first is the default for a reason, and the second is usually the cheapest — a bulk job has no reason to share a session with anything. The mechanics are in using ORM-enabled UPDATE and DELETE statements.

A third case worth naming: re-reading with stronger loader options. Because populate_existing re-loads eager relationships, it is how a second query with selectinload takes effect on objects that were first loaded without it:

from sqlalchemy import select
from sqlalchemy.orm import selectinload

from shop.models import Order

order = await session.scalar(
    select(Order)
    .where(Order.id == order_id)
    .options(selectinload(Order.lines))
    .execution_options(populate_existing=True)     # without this, lines stay unloaded
)

Choosing a Session Lifetime Instead

Almost every staleness question is a session-lifetime question. A session scoped to one unit of work starts with an empty identity map, so every read is current and none of this machinery is needed. The cases that need populate_existing are the cases where a session lives longer than one unit of work — and it is worth asking whether it should.

A web request should have one session, and usually one unit of work, so staleness within it is rarely a concern: the data was read at the start of the request and the request is short. The framework patterns in integrating SQLAlchemy async with FastAPI and Starlette give exactly that.

A background task should open its own session, for the reasons in running background tasks with a fresh AsyncSession in FastAPI — and that session is then short-lived and current by construction.

A worker loop is the genuine long-lived case, and the choice is explicit: a session per iteration (simple, current, one extra checkout per loop) or a long-lived session with deliberate refreshing (fewer checkouts, more to reason about). A session per iteration is the right default; the checkout cost is a fraction of a millisecond against a warm pool.

A CLI or a script that runs for a while is the case where a long-lived session is most tempting and most likely to surprise, because the data can change substantially while it runs. Reading once, at the start, and treating that as a snapshot is often more honest than refreshing piecemeal.

When a long-lived session is the right answer, two habits keep it manageable. Refresh at the boundary of each unit of work rather than at arbitrary reads, so there is one place that decides what "current" means. And expunge objects that are no longer needed, because the identity map holds a strong reference to every object it has ever seen, and a worker that never expunges has a slow memory leak — the counterpart to the connection leak described in fixing garbage collector non-checked-in connection warnings.

Under concurrency, refreshing is also not a substitute for locking. Reading current values tells you what they were a moment ago; if the next step depends on them not changing, that needs SELECT ... FOR UPDATE or a version counter — the approaches in building a job queue with SELECT FOR UPDATE SKIP LOCKED and implementing optimistic locking with version counters.

Frequently Asked Questions

Related

• Session Lifecycle and Scope Management — The parent guide: how long a session should live and what it owns.
• Understanding Session.expunge vs Session.clear — Removing objects from the identity map rather than refreshing them.
• Using expire_on_commit=False in FastAPI dependencies — The commit-time expiry this interacts with.
• Using ORM-enabled UPDATE and DELETE statements — Set-based writes and identity-map synchronisation.