[{"data":1,"prerenderedAt":933},["ShallowReactive",2],{"page-\u002Fasync-engines-dialects-and-connection-pooling\u002Fchoosing-between-asyncpg-and-psycopg-async-drivers\u002Fusing-sqlalchemy-async-with-celery-task-workers\u002F":3},{"id":4,"title":5,"body":6,"description":926,"extension":927,"meta":928,"navigation":126,"path":929,"seo":930,"stem":931,"__hash__":932},"content\u002Fasync-engines-dialects-and-connection-pooling\u002Fchoosing-between-asyncpg-and-psycopg-async-drivers\u002Fusing-sqlalchemy-async-with-celery-task-workers\u002Findex.md","Using SQLAlchemy async with Celery Task Workers: Exact Syntax & Pooling Fixes",{"type":7,"value":8,"toc":919},"minimark",[9,13,60,65,80,339,343,384,696,700,703,764,768,873,877,886,902,915],[10,11,5],"h1",{"id":12},"using-sqlalchemy-async-with-celery-task-workers-exact-syntax-pooling-fixes",[14,15,16,20,21,25,26,29,30,33,34,37,38,41,42,45,46,49,50,53,54,59],"p",{},[17,18,19],"strong",{},"Direct Answer: Core Architecture & Syntax","\nTo run SQLAlchemy 2.0 async sessions inside Celery workers safely, you must decouple the event loop from Celery’s synchronous prefork model and align connection pooling with worker concurrency. Initialize ",[22,23,24],"code",{},"create_async_engine()"," with a ",[22,27,28],{},"pool_size"," that exactly matches your ",[22,31,32],{},"celery --concurrency"," flag, defer engine instantiation until the ",[22,35,36],{},"worker_ready"," signal fires to prevent fork-state corruption, and route all database calls through ",[22,39,40],{},"asyncio.run()"," inside standard ",[22,43,44],{},"@app.task"," decorators. Bind your ",[22,47,48],{},"async_sessionmaker"," to the worker’s isolated event loop to prevent ",[22,51,52],{},"greenlet"," context collisions. Before scaling worker pools, establish baseline connection behavior by reviewing ",[55,56,58],"a",{"href":57},"\u002Fasync-engines-dialects-and-connection-pooling\u002F","Async Engines, Dialects, and Connection Pooling"," to understand how async dialects handle thread-local state and connection lifecycle.",[61,62,64],"h2",{"id":63},"step-1-async-engine-configuration-for-worker-processes","Step 1: Async Engine Configuration for Worker Processes",[14,66,67,68,70,71,74,75,79],{},"Celery’s process forking model breaks pre-initialized async engines. Instantiate the engine lazily or bind it to the ",[22,69,36],{}," signal. Enable ",[22,72,73],{},"pool_pre_ping=True"," to validate idle connections after broker latency spikes. Select your async driver based on your database backend; evaluate performance trade-offs in ",[55,76,78],{"href":77},"\u002Fasync-engines-dialects-and-connection-pooling\u002Fchoosing-between-asyncpg-and-psycopg-async-drivers\u002F","Choosing Between asyncpg and psycopg Async Drivers"," before committing to a dialect.",[81,82,87],"pre",{"className":83,"code":84,"language":85,"meta":86,"style":86},"language-python shiki shiki-themes github-light github-dark","from typing import Final\nfrom sqlalchemy.ext.asyncio import create_async_engine, async_sessionmaker, AsyncSession\n\ndef setup_db_engine(db_url: str, concurrency: int) -> Final:\n return create_async_engine(\n db_url,\n pool_size=concurrency,\n max_overflow=max(2, int(concurrency * 0.2)),\n pool_pre_ping=True,\n pool_recycle=450,\n pool_timeout=30\n )\n\n# Initialize session factory (bind engine after worker_ready signal in production)\nAsyncSessionFactory: Final = async_sessionmaker(\n bind=setup_db_engine(\"postgresql+asyncpg:\u002F\u002Fuser:pass@localhost\u002Fdb\", concurrency=8),\n class_=AsyncSession,\n expire_on_commit=False\n)\n","python","",[22,88,89,108,121,128,154,163,169,182,216,230,243,254,260,265,272,283,311,322,333],{"__ignoreMap":86},[90,91,94,98,102,105],"span",{"class":92,"line":93},"line",1,[90,95,97],{"class":96},"szBVR","from",[90,99,101],{"class":100},"sVt8B"," typing ",[90,103,104],{"class":96},"import",[90,106,107],{"class":100}," Final\n",[90,109,111,113,116,118],{"class":92,"line":110},2,[90,112,97],{"class":96},[90,114,115],{"class":100}," sqlalchemy.ext.asyncio ",[90,117,104],{"class":96},[90,119,120],{"class":100}," create_async_engine, async_sessionmaker, AsyncSession\n",[90,122,124],{"class":92,"line":123},3,[90,125,127],{"emptyLinePlaceholder":126},true,"\n",[90,129,131,134,138,141,145,148,151],{"class":92,"line":130},4,[90,132,133],{"class":96},"def",[90,135,137],{"class":136},"sScJk"," setup_db_engine",[90,139,140],{"class":100},"(db_url: ",[90,142,144],{"class":143},"sj4cs","str",[90,146,147],{"class":100},", concurrency: ",[90,149,150],{"class":143},"int",[90,152,153],{"class":100},") -> Final:\n",[90,155,157,160],{"class":92,"line":156},5,[90,158,159],{"class":96}," return",[90,161,162],{"class":100}," create_async_engine(\n",[90,164,166],{"class":92,"line":165},6,[90,167,168],{"class":100}," db_url,\n",[90,170,172,176,179],{"class":92,"line":171},7,[90,173,175],{"class":174},"s4XuR"," pool_size",[90,177,178],{"class":96},"=",[90,180,181],{"class":100},"concurrency,\n",[90,183,185,188,190,193,196,199,202,204,207,210,213],{"class":92,"line":184},8,[90,186,187],{"class":174}," max_overflow",[90,189,178],{"class":96},[90,191,192],{"class":143},"max",[90,194,195],{"class":100},"(",[90,197,198],{"class":143},"2",[90,200,201],{"class":100},", ",[90,203,150],{"class":143},[90,205,206],{"class":100},"(concurrency ",[90,208,209],{"class":96},"*",[90,211,212],{"class":143}," 0.2",[90,214,215],{"class":100},")),\n",[90,217,219,222,224,227],{"class":92,"line":218},9,[90,220,221],{"class":174}," pool_pre_ping",[90,223,178],{"class":96},[90,225,226],{"class":143},"True",[90,228,229],{"class":100},",\n",[90,231,233,236,238,241],{"class":92,"line":232},10,[90,234,235],{"class":174}," pool_recycle",[90,237,178],{"class":96},[90,239,240],{"class":143},"450",[90,242,229],{"class":100},[90,244,246,249,251],{"class":92,"line":245},11,[90,247,248],{"class":174}," pool_timeout",[90,250,178],{"class":96},[90,252,253],{"class":143},"30\n",[90,255,257],{"class":92,"line":256},12,[90,258,259],{"class":100}," )\n",[90,261,263],{"class":92,"line":262},13,[90,264,127],{"emptyLinePlaceholder":126},[90,266,268],{"class":92,"line":267},14,[90,269,271],{"class":270},"sJ8bj","# Initialize session factory (bind engine after worker_ready signal in production)\n",[90,273,275,278,280],{"class":92,"line":274},15,[90,276,277],{"class":100},"AsyncSessionFactory: Final ",[90,279,178],{"class":96},[90,281,282],{"class":100}," async_sessionmaker(\n",[90,284,286,289,291,294,298,300,303,305,308],{"class":92,"line":285},16,[90,287,288],{"class":174}," bind",[90,290,178],{"class":96},[90,292,293],{"class":100},"setup_db_engine(",[90,295,297],{"class":296},"sZZnC","\"postgresql+asyncpg:\u002F\u002Fuser:pass@localhost\u002Fdb\"",[90,299,201],{"class":100},[90,301,302],{"class":174},"concurrency",[90,304,178],{"class":96},[90,306,307],{"class":143},"8",[90,309,310],{"class":100},"),\n",[90,312,314,317,319],{"class":92,"line":313},17,[90,315,316],{"class":174}," class_",[90,318,178],{"class":96},[90,320,321],{"class":100},"AsyncSession,\n",[90,323,325,328,330],{"class":92,"line":324},18,[90,326,327],{"class":174}," expire_on_commit",[90,329,178],{"class":96},[90,331,332],{"class":143},"False\n",[90,334,336],{"class":92,"line":335},19,[90,337,338],{"class":100},")\n",[61,340,342],{"id":341},"step-2-session-lifecycle-task-execution-pattern","Step 2: Session Lifecycle & Task Execution Pattern",[14,344,345,346,348,349,351,352,355,356,359,360,363,364,367,368,371,372,375,376,379,380,383],{},"Wrap async execution in ",[22,347,40],{}," within sync Celery tasks, or migrate to native async tasks (",[22,350,44],{}," with ",[22,353,354],{},"async def","). Use ",[22,357,358],{},"async with AsyncSession() as session:"," to guarantee automatic rollback on unhandled exceptions. Execute queries using SQLAlchemy 2.0 ",[22,361,362],{},"select()"," syntax. Isolate ",[22,365,366],{},"session.commit()"," in explicit ",[22,369,370],{},"try\u002Fexcept"," blocks and call ",[22,373,374],{},"await session.rollback()"," on failure. Prevent ",[22,377,378],{},"DetachedInstanceError"," by eagerly loading relationships with ",[22,381,382],{},"selectinload()"," before the context manager exits.",[81,385,387],{"className":83,"code":386,"language":85,"meta":86,"style":86},"import asyncio\nfrom celery import Celery\nfrom sqlalchemy import select\nfrom sqlalchemy.orm import selectinload\n\napp = Celery(\"worker\", broker=\"redis:\u002F\u002Flocalhost:6379\u002F0\")\n\n@app.task(bind=True, max_retries=3)\ndef process_user_task(self, user_id: int) -> None:\n async def _execute_db() -> None:\n async with AsyncSessionFactory() as session:\n try:\n stmt = (\n select(User)\n .where(User.id == user_id)\n .options(selectinload(User.profile)) # Prevent lazy-load errors post-commit\n )\n result = await session.execute(stmt)\n user = result.scalar_one()\n user.status = \"active\"\n await session.commit()\n except Exception as exc:\n await session.rollback()\n raise self.retry(exc=exc, countdown=60)\n\n asyncio.run(_execute_db())\n",[22,388,389,396,408,420,432,436,461,465,490,511,529,545,552,562,567,578,586,590,603,613,624,632,647,655,685,690],{"__ignoreMap":86},[90,390,391,393],{"class":92,"line":93},[90,392,104],{"class":96},[90,394,395],{"class":100}," asyncio\n",[90,397,398,400,403,405],{"class":92,"line":110},[90,399,97],{"class":96},[90,401,402],{"class":100}," celery ",[90,404,104],{"class":96},[90,406,407],{"class":100}," Celery\n",[90,409,410,412,415,417],{"class":92,"line":123},[90,411,97],{"class":96},[90,413,414],{"class":100}," sqlalchemy ",[90,416,104],{"class":96},[90,418,419],{"class":100}," select\n",[90,421,422,424,427,429],{"class":92,"line":130},[90,423,97],{"class":96},[90,425,426],{"class":100}," sqlalchemy.orm ",[90,428,104],{"class":96},[90,430,431],{"class":100}," selectinload\n",[90,433,434],{"class":92,"line":156},[90,435,127],{"emptyLinePlaceholder":126},[90,437,438,441,443,446,449,451,454,456,459],{"class":92,"line":165},[90,439,440],{"class":100},"app ",[90,442,178],{"class":96},[90,444,445],{"class":100}," Celery(",[90,447,448],{"class":296},"\"worker\"",[90,450,201],{"class":100},[90,452,453],{"class":174},"broker",[90,455,178],{"class":96},[90,457,458],{"class":296},"\"redis:\u002F\u002Flocalhost:6379\u002F0\"",[90,460,338],{"class":100},[90,462,463],{"class":92,"line":171},[90,464,127],{"emptyLinePlaceholder":126},[90,466,467,469,471,474,476,478,480,483,485,488],{"class":92,"line":184},[90,468,44],{"class":136},[90,470,195],{"class":100},[90,472,473],{"class":174},"bind",[90,475,178],{"class":96},[90,477,226],{"class":143},[90,479,201],{"class":100},[90,481,482],{"class":174},"max_retries",[90,484,178],{"class":96},[90,486,487],{"class":143},"3",[90,489,338],{"class":100},[90,491,492,494,497,500,502,505,508],{"class":92,"line":218},[90,493,133],{"class":96},[90,495,496],{"class":136}," process_user_task",[90,498,499],{"class":100},"(self, user_id: ",[90,501,150],{"class":143},[90,503,504],{"class":100},") -> ",[90,506,507],{"class":143},"None",[90,509,510],{"class":100},":\n",[90,512,513,516,519,522,525,527],{"class":92,"line":232},[90,514,515],{"class":96}," async",[90,517,518],{"class":96}," def",[90,520,521],{"class":136}," _execute_db",[90,523,524],{"class":100},"() -> ",[90,526,507],{"class":143},[90,528,510],{"class":100},[90,530,531,533,536,539,542],{"class":92,"line":245},[90,532,515],{"class":96},[90,534,535],{"class":96}," with",[90,537,538],{"class":100}," AsyncSessionFactory() ",[90,540,541],{"class":96},"as",[90,543,544],{"class":100}," session:\n",[90,546,547,550],{"class":92,"line":256},[90,548,549],{"class":96}," try",[90,551,510],{"class":100},[90,553,554,557,559],{"class":92,"line":262},[90,555,556],{"class":100}," stmt ",[90,558,178],{"class":96},[90,560,561],{"class":100}," (\n",[90,563,564],{"class":92,"line":267},[90,565,566],{"class":100}," select(User)\n",[90,568,569,572,575],{"class":92,"line":274},[90,570,571],{"class":100}," .where(User.id ",[90,573,574],{"class":96},"==",[90,576,577],{"class":100}," user_id)\n",[90,579,580,583],{"class":92,"line":285},[90,581,582],{"class":100}," .options(selectinload(User.profile)) ",[90,584,585],{"class":270},"# Prevent lazy-load errors post-commit\n",[90,587,588],{"class":92,"line":313},[90,589,259],{"class":100},[90,591,592,595,597,600],{"class":92,"line":324},[90,593,594],{"class":100}," result ",[90,596,178],{"class":96},[90,598,599],{"class":96}," await",[90,601,602],{"class":100}," session.execute(stmt)\n",[90,604,605,608,610],{"class":92,"line":335},[90,606,607],{"class":100}," user ",[90,609,178],{"class":96},[90,611,612],{"class":100}," result.scalar_one()\n",[90,614,616,619,621],{"class":92,"line":615},20,[90,617,618],{"class":100}," user.status ",[90,620,178],{"class":96},[90,622,623],{"class":296}," \"active\"\n",[90,625,627,629],{"class":92,"line":626},21,[90,628,599],{"class":96},[90,630,631],{"class":100}," session.commit()\n",[90,633,635,638,641,644],{"class":92,"line":634},22,[90,636,637],{"class":96}," except",[90,639,640],{"class":143}," Exception",[90,642,643],{"class":96}," as",[90,645,646],{"class":100}," exc:\n",[90,648,650,652],{"class":92,"line":649},23,[90,651,599],{"class":96},[90,653,654],{"class":100}," session.rollback()\n",[90,656,658,661,664,667,670,672,675,678,680,683],{"class":92,"line":657},24,[90,659,660],{"class":96}," raise",[90,662,663],{"class":143}," self",[90,665,666],{"class":100},".retry(",[90,668,669],{"class":174},"exc",[90,671,178],{"class":96},[90,673,674],{"class":100},"exc, ",[90,676,677],{"class":174},"countdown",[90,679,178],{"class":96},[90,681,682],{"class":143},"60",[90,684,338],{"class":100},[90,686,688],{"class":92,"line":687},25,[90,689,127],{"emptyLinePlaceholder":126},[90,691,693],{"class":92,"line":692},26,[90,694,695],{"class":100}," asyncio.run(_execute_db())\n",[61,697,699],{"id":698},"step-3-connection-pool-optimization-for-high-throughput-workers","Step 3: Connection Pool Optimization for High-Throughput Workers",[14,701,702],{},"Connection exhaustion is the primary failure mode in async Celery deployments. Tune your pool parameters to match process concurrency and network latency:",[704,705,706,718,730,742,754],"ul",{},[707,708,709,713,714,717],"li",{},[17,710,711],{},[22,712,28],{},": Set exactly to your ",[22,715,716],{},"celery -c"," concurrency value.",[707,719,720,725,726,729],{},[17,721,722],{},[22,723,724],{},"max_overflow",": Allocate a 20% buffer (",[22,727,728],{},"int(concurrency * 0.2)",") for transient spikes.",[707,731,732,737,738,741],{},[17,733,734],{},[22,735,736],{},"pool_recycle",": Configure to ",[22,739,740],{},"300–600"," seconds to preempt PostgreSQL\u002FMySQL idle connection drops.",[707,743,744,749,750,753],{},[17,745,746],{},[22,747,748],{},"pool_timeout",": Set to ",[22,751,752],{},"30"," seconds to fail fast and trigger Celery retries instead of blocking worker threads indefinitely.",[707,755,756,759,760,763],{},[17,757,758],{},"Monitoring",": Query ",[22,761,762],{},"pg_stat_activity"," or equivalent to verify connection reuse versus leak patterns under load.",[61,765,767],{"id":766},"common-pitfalls-error-resolution","Common Pitfalls & Error Resolution",[769,770,771,787],"table",{},[772,773,774],"thead",{},[775,776,777,781,784],"tr",{},[778,779,780],"th",{},"Error \u002F Symptom",[778,782,783],{},"Root Cause",[778,785,786],{},"Production Resolution",[788,789,790,811,848],"tbody",{},[775,791,792,802,805],{},[793,794,795,798,799],"td",{},[22,796,797],{},"GreenletSpawnError"," \u002F ",[22,800,801],{},"Cannot run in an async context",[793,803,804],{},"Sync prefork workers invoking async SQLAlchemy without explicit event loop routing.",[793,806,807,808,810],{},"Wrap all async DB calls in ",[22,809,40],{}," or switch to Celery’s native async task execution mode.",[775,812,813,819,831],{},[793,814,815,818],{},[22,816,817],{},"QueuePool limit reached"," (Connection Exhaustion)",[793,820,821,823,824,826,827,830],{},[22,822,28],{}," + ",[22,825,724],{}," \u003C Celery concurrency, or missing ",[22,828,829],{},"async with"," session context managers.",[793,832,833,834,351,836,839,840,843,844,847],{},"Align ",[22,835,28],{},[22,837,838],{},"--concurrency",", enforce strict ",[22,841,842],{},"async with session:"," blocks, and add ",[22,845,846],{},"pool_timeout=30",".",[775,849,850,855,858],{},[793,851,852,854],{},[22,853,378],{}," on Lazy Load",[793,856,857],{},"Accessing ORM relationships after the async session context closes.",[793,859,860,861,864,865,868,869,872],{},"Use eager loading (",[22,862,863],{},"selectinload","\u002F",[22,866,867],{},"joinedload",") or ",[22,870,871],{},"await session.refresh(instance)"," before exiting the context manager.",[61,874,876],{"id":875},"frequently-asked-questions","Frequently Asked Questions",[14,878,879,882,883,885],{},[17,880,881],{},"Can I use sync Celery workers with SQLAlchemy async sessions?","\nYes. Wrap the async execution block in ",[22,884,40],{}," inside the task. Ensure no synchronous SQLAlchemy calls share the same thread to avoid event loop conflicts.",[14,887,888,891,892,895,896,898,899,901],{},[17,889,890],{},"How do I prevent connection leaks during Celery task retries?","\nAlways use ",[22,893,894],{},"async with async_session() as session:"," and implement explicit ",[22,897,374],{}," on exception. Set ",[22,900,748],{}," and monitor DB-side active connections to verify cleanup after retries.",[14,903,904,907,910,911,914],{},[17,905,906],{},"Does asyncpg or psycopg3 perform better under Celery workloads?",[22,908,909],{},"asyncpg"," generally delivers lower latency for high-concurrency read\u002Fwrite bursts. ",[22,912,913],{},"psycopg3"," offers broader compatibility and native async support. Driver selection directly impacts pool behavior and should match your query profile.",[916,917,918],"style",{},"html pre.shiki code .szBVR, html code.shiki .szBVR{--shiki-default:#D73A49;--shiki-dark:#F97583}html pre.shiki code .sVt8B, html code.shiki .sVt8B{--shiki-default:#24292E;--shiki-dark:#E1E4E8}html pre.shiki code .sScJk, html code.shiki .sScJk{--shiki-default:#6F42C1;--shiki-dark:#B392F0}html pre.shiki code .sj4cs, html code.shiki .sj4cs{--shiki-default:#005CC5;--shiki-dark:#79B8FF}html pre.shiki code .s4XuR, html code.shiki .s4XuR{--shiki-default:#E36209;--shiki-dark:#FFAB70}html pre.shiki code .sJ8bj, html code.shiki .sJ8bj{--shiki-default:#6A737D;--shiki-dark:#6A737D}html pre.shiki code .sZZnC, html code.shiki .sZZnC{--shiki-default:#032F62;--shiki-dark:#9ECBFF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":86,"searchDepth":110,"depth":110,"links":920},[921,922,923,924,925],{"id":63,"depth":110,"text":64},{"id":341,"depth":110,"text":342},{"id":698,"depth":110,"text":699},{"id":766,"depth":110,"text":767},{"id":875,"depth":110,"text":876},"Direct Answer: Core Architecture & Syntax\nTo run SQLAlchemy 2.0 async sessions inside Celery workers safely, you must decouple the event loop from Celery’s synchronous prefork model and align connection pooling with worker concurrency. Initialize create_async_engine() with a pool_size that exactly matches your celery --concurrency flag, defer engine instantiation until the worker_ready signal fires to prevent fork-state corruption, and route all database calls through asyncio.run() inside standard @app.task decorators. Bind your async_sessionmaker to the worker’s isolated event loop to prevent greenlet context collisions. Before scaling worker pools, establish baseline connection behavior by reviewing Async Engines, Dialects, and Connection Pooling to understand how async dialects handle thread-local state and connection lifecycle.","md",{},"\u002Fasync-engines-dialects-and-connection-pooling\u002Fchoosing-between-asyncpg-and-psycopg-async-drivers\u002Fusing-sqlalchemy-async-with-celery-task-workers",{"title":5,"description":926},"async-engines-dialects-and-connection-pooling\u002Fchoosing-between-asyncpg-and-psycopg-async-drivers\u002Fusing-sqlalchemy-async-with-celery-task-workers\u002Findex","iU54D48RwgaILkAnXIINoVqcBVluscM7uZnHGE_rlEY",1778149144399]