[{"data":1,"prerenderedAt":1313},["ShallowReactive",2],{"page-\u002Fadvanced-query-patterns-and-bulk-data-operations\u002Fcommon-table-expressions-ctes-and-recursive-queries\u002F":3},{"id":4,"title":5,"body":6,"description":1306,"extension":1307,"meta":1308,"navigation":126,"path":1309,"seo":1310,"stem":1311,"__hash__":1312},"content\u002Fadvanced-query-patterns-and-bulk-data-operations\u002Fcommon-table-expressions-ctes-and-recursive-queries\u002Findex.md","Common Table Expressions (CTEs) and Recursive Queries in SQLAlchemy 2.0",{"type":7,"value":8,"toc":1296},"minimark",[9,13,22,27,42,54,366,370,377,380,426,434,438,444,660,667,671,674,814,825,829,844,963,966,970,977,990,1077,1080,1084,1221,1225,1244,1257,1270,1292],[10,11,5],"h1",{"id":12},"common-table-expressions-ctes-and-recursive-queries-in-sqlalchemy-20",[14,15,16,17,21],"p",{},"Common Table Expressions (CTEs) have evolved from syntactic sugar into foundational execution primitives for modern relational databases. In SQLAlchemy 2.0, the transition to a unified Core\u002FORM API and native ",[18,19,20],"code",{},"asyncio"," support requires a disciplined approach to CTE construction, execution planning, and async\u002Fawait boundary management. This guide details production-ready patterns for leveraging CTEs in high-throughput, memory-constrained environments.",[23,24,26],"h2",{"id":25},"cte-architecture-in-sqlalchemy-20","CTE Architecture in SQLAlchemy 2.0",[14,28,29,30,33,34,37,38,41],{},"SQLAlchemy 2.0 deprecates the legacy ",[18,31,32],{},"Query.cte()"," construct in favor of the explicit ",[18,35,36],{},"select().cte()"," pattern. This shift aligns with the 2.0 philosophy of explicit, composable SQL generation. A CTE is no longer an implicit subquery but a first-class ",[18,39,40],{},"CTE"," object that participates in the query planner's optimization phase.",[14,43,44,45,47,48,53],{},"When executing long-running CTEs under ",[18,46,20],{},", transaction boundaries must be explicitly scoped. The database driver maintains a cursor state that can block the event loop if not properly yielded. As outlined in ",[49,50,52],"a",{"href":51},"\u002Fadvanced-query-patterns-and-bulk-data-operations\u002F","Advanced Query Patterns and Bulk Data Operations",", execution planning for CTEs involves evaluating whether the planner will inline the expression or materialize it into a temporary workspace. For async workloads, wrapping CTE execution in an explicit transaction context prevents connection pool starvation and ensures deterministic cleanup.",[55,56,61],"pre",{"className":57,"code":58,"language":59,"meta":60,"style":60},"language-python shiki shiki-themes github-light github-dark","from typing import AsyncIterator\nfrom sqlalchemy.ext.asyncio import AsyncSession, create_async_engine\nfrom sqlalchemy import select, text, Column, Integer, String\nfrom sqlalchemy.orm import DeclarativeBase\n\nclass Node(DeclarativeBase):\n __tablename__ = \"nodes\"\n id = Column(Integer, primary_key=True)\n parent_id = Column(Integer, nullable=True)\n payload = Column(String)\n\nasync def execute_async_cte(session: AsyncSession) -> AsyncIterator[tuple[int, str]]:\n # Explicit transaction boundary for async execution\n async with session.begin():\n base_cte = select(Node.id, Node.payload).where(Node.parent_id.is_(None)).cte(name=\"root_nodes\")\n stmt = select(base_cte.c.id, base_cte.c.payload)\n \n # stream_results=True prevents loading the entire result set into memory\n result = await session.execute(stmt.execution_options(stream_results=True))\n async for row in result:\n yield row.id, row.payload\n","python","",[18,62,63,82,95,108,121,128,147,160,185,204,215,220,247,254,266,293,304,310,316,340,357],{"__ignoreMap":60},[64,65,68,72,76,79],"span",{"class":66,"line":67},"line",1,[64,69,71],{"class":70},"szBVR","from",[64,73,75],{"class":74},"sVt8B"," typing ",[64,77,78],{"class":70},"import",[64,80,81],{"class":74}," AsyncIterator\n",[64,83,85,87,90,92],{"class":66,"line":84},2,[64,86,71],{"class":70},[64,88,89],{"class":74}," sqlalchemy.ext.asyncio ",[64,91,78],{"class":70},[64,93,94],{"class":74}," AsyncSession, create_async_engine\n",[64,96,98,100,103,105],{"class":66,"line":97},3,[64,99,71],{"class":70},[64,101,102],{"class":74}," sqlalchemy ",[64,104,78],{"class":70},[64,106,107],{"class":74}," select, text, Column, Integer, String\n",[64,109,111,113,116,118],{"class":66,"line":110},4,[64,112,71],{"class":70},[64,114,115],{"class":74}," sqlalchemy.orm ",[64,117,78],{"class":70},[64,119,120],{"class":74}," DeclarativeBase\n",[64,122,124],{"class":66,"line":123},5,[64,125,127],{"emptyLinePlaceholder":126},true,"\n",[64,129,131,134,138,141,144],{"class":66,"line":130},6,[64,132,133],{"class":70},"class",[64,135,137],{"class":136},"sScJk"," Node",[64,139,140],{"class":74},"(",[64,142,143],{"class":136},"DeclarativeBase",[64,145,146],{"class":74},"):\n",[64,148,150,153,156],{"class":66,"line":149},7,[64,151,152],{"class":74}," __tablename__ ",[64,154,155],{"class":70},"=",[64,157,159],{"class":158},"sZZnC"," \"nodes\"\n",[64,161,163,167,170,173,177,179,182],{"class":66,"line":162},8,[64,164,166],{"class":165},"sj4cs"," id",[64,168,169],{"class":70}," =",[64,171,172],{"class":74}," Column(Integer, ",[64,174,176],{"class":175},"s4XuR","primary_key",[64,178,155],{"class":70},[64,180,181],{"class":165},"True",[64,183,184],{"class":74},")\n",[64,186,188,191,193,195,198,200,202],{"class":66,"line":187},9,[64,189,190],{"class":74}," parent_id ",[64,192,155],{"class":70},[64,194,172],{"class":74},[64,196,197],{"class":175},"nullable",[64,199,155],{"class":70},[64,201,181],{"class":165},[64,203,184],{"class":74},[64,205,207,210,212],{"class":66,"line":206},10,[64,208,209],{"class":74}," payload ",[64,211,155],{"class":70},[64,213,214],{"class":74}," Column(String)\n",[64,216,218],{"class":66,"line":217},11,[64,219,127],{"emptyLinePlaceholder":126},[64,221,223,226,229,232,235,238,241,244],{"class":66,"line":222},12,[64,224,225],{"class":70},"async",[64,227,228],{"class":70}," def",[64,230,231],{"class":136}," execute_async_cte",[64,233,234],{"class":74},"(session: AsyncSession) -> AsyncIterator[tuple[",[64,236,237],{"class":165},"int",[64,239,240],{"class":74},", ",[64,242,243],{"class":165},"str",[64,245,246],{"class":74},"]]:\n",[64,248,250],{"class":66,"line":249},13,[64,251,253],{"class":252},"sJ8bj"," # Explicit transaction boundary for async execution\n",[64,255,257,260,263],{"class":66,"line":256},14,[64,258,259],{"class":70}," async",[64,261,262],{"class":70}," with",[64,264,265],{"class":74}," session.begin():\n",[64,267,269,272,274,277,280,283,286,288,291],{"class":66,"line":268},15,[64,270,271],{"class":74}," base_cte ",[64,273,155],{"class":70},[64,275,276],{"class":74}," select(Node.id, Node.payload).where(Node.parent_id.is_(",[64,278,279],{"class":165},"None",[64,281,282],{"class":74},")).cte(",[64,284,285],{"class":175},"name",[64,287,155],{"class":70},[64,289,290],{"class":158},"\"root_nodes\"",[64,292,184],{"class":74},[64,294,296,299,301],{"class":66,"line":295},16,[64,297,298],{"class":74}," stmt ",[64,300,155],{"class":70},[64,302,303],{"class":74}," select(base_cte.c.id, base_cte.c.payload)\n",[64,305,307],{"class":66,"line":306},17,[64,308,309],{"class":74}," \n",[64,311,313],{"class":66,"line":312},18,[64,314,315],{"class":252}," # stream_results=True prevents loading the entire result set into memory\n",[64,317,319,322,324,327,330,333,335,337],{"class":66,"line":318},19,[64,320,321],{"class":74}," result ",[64,323,155],{"class":70},[64,325,326],{"class":70}," await",[64,328,329],{"class":74}," session.execute(stmt.execution_options(",[64,331,332],{"class":175},"stream_results",[64,334,155],{"class":70},[64,336,181],{"class":165},[64,338,339],{"class":74},"))\n",[64,341,343,345,348,351,354],{"class":66,"line":342},20,[64,344,259],{"class":70},[64,346,347],{"class":70}," for",[64,349,350],{"class":74}," row ",[64,352,353],{"class":70},"in",[64,355,356],{"class":74}," result:\n",[64,358,360,363],{"class":66,"line":359},21,[64,361,362],{"class":70}," yield",[64,364,365],{"class":74}," row.id, row.payload\n",[23,367,369],{"id":368},"non-recursive-cte-construction-optimization","Non-Recursive CTE Construction & Optimization",[14,371,372,373,376],{},"Non-recursive CTEs excel at breaking complex query logic into readable, optimizable stages. In production systems, they frequently replace deeply nested ",[18,374,375],{},"EXISTS"," or correlated subqueries, enabling the query planner to flatten execution paths and apply early filtering.",[14,378,379],{},"Key optimization strategies include:",[381,382,383,399,409],"ul",{},[384,385,386,390,391,394,395,398],"li",{},[387,388,389],"strong",{},"Column Aliasing:"," Explicitly name CTE outputs using ",[18,392,393],{},".label()"," or ",[18,396,397],{},".c"," accessors to avoid planner ambiguity during joins.",[384,400,401,404,405,408],{},[387,402,403],{},"Lateral Joins:"," Combine CTEs with ",[18,406,407],{},"lateral()"," to push row-level computations down to the execution engine.",[384,410,411,414,415,240,418,421,422,425],{},[387,412,413],{},"Subquery Flattening:"," Ensure the CTE does not contain ",[18,416,417],{},"DISTINCT",[18,419,420],{},"ORDER BY",", or ",[18,423,424],{},"LIMIT"," unless strictly necessary, as these force materialization and block predicate pushdown.",[14,427,428,429,433],{},"When integrating CTEs with ORM relationships, prefer explicit joins over implicit loader chains to prevent N+1 query generation. For deeper insights into optimizing multi-entity fetches, refer to ",[49,430,432],{"href":431},"\u002Fadvanced-query-patterns-and-bulk-data-operations\u002Fcomplex-joins-and-relationship-loading-strategies\u002F","Complex Joins and Relationship Loading Strategies",".",[23,435,437],{"id":436},"recursive-query-patterns-for-hierarchical-data","Recursive Query Patterns for Hierarchical Data",[14,439,440,441,433],{},"Recursive CTEs solve graph traversal, organizational charts, and category tree problems natively within the RDBMS. The pattern consists of an anchor member (base case) and a recursive member, combined via ",[18,442,443],{},"union_all()",[55,445,447],{"className":57,"code":446,"language":59,"meta":60,"style":60},"from sqlalchemy import union_all, func\n\ndef build_recursive_hierarchy(session: AsyncSession):\n anchor = select(\n Node.id,\n Node.parent_id,\n func.cast(0, Integer).label(\"depth\"),\n func.cast(Node.id, String).label(\"path\")\n ).where(Node.parent_id.is_(None))\n\n recursive = select(\n Node.id,\n Node.parent_id,\n (anchor.c.depth + 1).label(\"depth\"),\n (anchor.c.path + func.cast(\"->\" + func.cast(Node.id, String), String)).label(\"path\")\n ).join(anchor, Node.parent_id == anchor.c.id)\n\n # Cycle prevention is handled via depth limiting and path tracking\n hierarchy_cte = union_all(anchor, recursive).cte(name=\"hierarchy\", recursive=True)\n \n return select(hierarchy_cte).where(hierarchy_cte.c.depth \u003C 10)\n",[18,448,449,460,464,475,485,490,495,512,522,531,535,544,548,552,570,592,603,607,612,640,644],{"__ignoreMap":60},[64,450,451,453,455,457],{"class":66,"line":67},[64,452,71],{"class":70},[64,454,102],{"class":74},[64,456,78],{"class":70},[64,458,459],{"class":74}," union_all, func\n",[64,461,462],{"class":66,"line":84},[64,463,127],{"emptyLinePlaceholder":126},[64,465,466,469,472],{"class":66,"line":97},[64,467,468],{"class":70},"def",[64,470,471],{"class":136}," build_recursive_hierarchy",[64,473,474],{"class":74},"(session: AsyncSession):\n",[64,476,477,480,482],{"class":66,"line":110},[64,478,479],{"class":74}," anchor ",[64,481,155],{"class":70},[64,483,484],{"class":74}," select(\n",[64,486,487],{"class":66,"line":123},[64,488,489],{"class":74}," Node.id,\n",[64,491,492],{"class":66,"line":130},[64,493,494],{"class":74}," Node.parent_id,\n",[64,496,497,500,503,506,509],{"class":66,"line":149},[64,498,499],{"class":74}," func.cast(",[64,501,502],{"class":165},"0",[64,504,505],{"class":74},", Integer).label(",[64,507,508],{"class":158},"\"depth\"",[64,510,511],{"class":74},"),\n",[64,513,514,517,520],{"class":66,"line":162},[64,515,516],{"class":74}," func.cast(Node.id, String).label(",[64,518,519],{"class":158},"\"path\"",[64,521,184],{"class":74},[64,523,524,527,529],{"class":66,"line":187},[64,525,526],{"class":74}," ).where(Node.parent_id.is_(",[64,528,279],{"class":165},[64,530,339],{"class":74},[64,532,533],{"class":66,"line":206},[64,534,127],{"emptyLinePlaceholder":126},[64,536,537,540,542],{"class":66,"line":217},[64,538,539],{"class":74}," recursive ",[64,541,155],{"class":70},[64,543,484],{"class":74},[64,545,546],{"class":66,"line":222},[64,547,489],{"class":74},[64,549,550],{"class":66,"line":249},[64,551,494],{"class":74},[64,553,554,557,560,563,566,568],{"class":66,"line":256},[64,555,556],{"class":74}," (anchor.c.depth ",[64,558,559],{"class":70},"+",[64,561,562],{"class":165}," 1",[64,564,565],{"class":74},").label(",[64,567,508],{"class":158},[64,569,511],{"class":74},[64,571,572,575,577,579,582,585,588,590],{"class":66,"line":268},[64,573,574],{"class":74}," (anchor.c.path ",[64,576,559],{"class":70},[64,578,499],{"class":74},[64,580,581],{"class":158},"\"->\"",[64,583,584],{"class":70}," +",[64,586,587],{"class":74}," func.cast(Node.id, String), String)).label(",[64,589,519],{"class":158},[64,591,184],{"class":74},[64,593,594,597,600],{"class":66,"line":295},[64,595,596],{"class":74}," ).join(anchor, Node.parent_id ",[64,598,599],{"class":70},"==",[64,601,602],{"class":74}," anchor.c.id)\n",[64,604,605],{"class":66,"line":306},[64,606,127],{"emptyLinePlaceholder":126},[64,608,609],{"class":66,"line":312},[64,610,611],{"class":252}," # Cycle prevention is handled via depth limiting and path tracking\n",[64,613,614,617,619,622,624,626,629,631,634,636,638],{"class":66,"line":318},[64,615,616],{"class":74}," hierarchy_cte ",[64,618,155],{"class":70},[64,620,621],{"class":74}," union_all(anchor, recursive).cte(",[64,623,285],{"class":175},[64,625,155],{"class":70},[64,627,628],{"class":158},"\"hierarchy\"",[64,630,240],{"class":74},[64,632,633],{"class":175},"recursive",[64,635,155],{"class":70},[64,637,181],{"class":165},[64,639,184],{"class":74},[64,641,642],{"class":66,"line":342},[64,643,309],{"class":74},[64,645,646,649,652,655,658],{"class":66,"line":359},[64,647,648],{"class":70}," return",[64,650,651],{"class":74}," select(hierarchy_cte).where(hierarchy_cte.c.depth ",[64,653,654],{"class":70},"\u003C",[64,656,657],{"class":165}," 10",[64,659,184],{"class":74},[14,661,662,663,433],{},"Production recursive queries require strict termination conditions. Without a depth limit or cycle detection array, malformed data triggers infinite recursion, exhausting connection resources. For comprehensive schema design and async traversal strategies, consult ",[49,664,666],{"href":665},"\u002Fadvanced-query-patterns-and-bulk-data-operations\u002Fcommon-table-expressions-ctes-and-recursive-queries\u002Fimplementing-recursive-ctes-for-hierarchical-data-in-sqlalchemy\u002F","Implementing Recursive CTEs for Hierarchical Data in SQLAlchemy",[23,668,670],{"id":669},"analytical-workflows-with-window-functions","Analytical Workflows with Window Functions",[14,672,673],{},"CTEs provide an ideal staging layer for analytical queries. By materializing intermediate aggregations or filtering noise upfront, window functions operate on predictable, indexed datasets.",[55,675,677],{"className":57,"code":676,"language":59,"meta":60,"style":60},"from sqlalchemy import func, over\n\ndef stage_analytical_cte():\n # Stage 1: Filter and aggregate\n base = select(\n Node.id,\n Node.parent_id,\n func.count().over(partition_by=Node.parent_id).label(\"sibling_count\")\n ).where(Node.payload.isnot(None)).cte(name=\"filtered_nodes\")\n\n # Stage 2: Apply ranking on staged output\n stmt = select(\n base.c.id,\n func.rank().over(order_by=base.c.sibling_count.desc()).label(\"rank\")\n )\n return stmt\n",[18,678,679,690,694,704,709,718,722,726,744,762,766,771,779,784,802,807],{"__ignoreMap":60},[64,680,681,683,685,687],{"class":66,"line":67},[64,682,71],{"class":70},[64,684,102],{"class":74},[64,686,78],{"class":70},[64,688,689],{"class":74}," func, over\n",[64,691,692],{"class":66,"line":84},[64,693,127],{"emptyLinePlaceholder":126},[64,695,696,698,701],{"class":66,"line":97},[64,697,468],{"class":70},[64,699,700],{"class":136}," stage_analytical_cte",[64,702,703],{"class":74},"():\n",[64,705,706],{"class":66,"line":110},[64,707,708],{"class":252}," # Stage 1: Filter and aggregate\n",[64,710,711,714,716],{"class":66,"line":123},[64,712,713],{"class":74}," base ",[64,715,155],{"class":70},[64,717,484],{"class":74},[64,719,720],{"class":66,"line":130},[64,721,489],{"class":74},[64,723,724],{"class":66,"line":149},[64,725,494],{"class":74},[64,727,728,731,734,736,739,742],{"class":66,"line":162},[64,729,730],{"class":74}," func.count().over(",[64,732,733],{"class":175},"partition_by",[64,735,155],{"class":70},[64,737,738],{"class":74},"Node.parent_id).label(",[64,740,741],{"class":158},"\"sibling_count\"",[64,743,184],{"class":74},[64,745,746,749,751,753,755,757,760],{"class":66,"line":187},[64,747,748],{"class":74}," ).where(Node.payload.isnot(",[64,750,279],{"class":165},[64,752,282],{"class":74},[64,754,285],{"class":175},[64,756,155],{"class":70},[64,758,759],{"class":158},"\"filtered_nodes\"",[64,761,184],{"class":74},[64,763,764],{"class":66,"line":206},[64,765,127],{"emptyLinePlaceholder":126},[64,767,768],{"class":66,"line":217},[64,769,770],{"class":252}," # Stage 2: Apply ranking on staged output\n",[64,772,773,775,777],{"class":66,"line":222},[64,774,298],{"class":74},[64,776,155],{"class":70},[64,778,484],{"class":74},[64,780,781],{"class":66,"line":249},[64,782,783],{"class":74}," base.c.id,\n",[64,785,786,789,792,794,797,800],{"class":66,"line":256},[64,787,788],{"class":74}," func.rank().over(",[64,790,791],{"class":175},"order_by",[64,793,155],{"class":70},[64,795,796],{"class":74},"base.c.sibling_count.desc()).label(",[64,798,799],{"class":158},"\"rank\"",[64,801,184],{"class":74},[64,803,804],{"class":66,"line":268},[64,805,806],{"class":74}," )\n",[64,808,809,811],{"class":66,"line":295},[64,810,648],{"class":70},[64,812,813],{"class":74}," stmt\n",[14,815,816,817,820,821,433],{},"Optimizing ",[18,818,819],{},"PARTITION BY"," clauses requires aligning partition keys with existing B-tree indexes. When the planner cannot leverage an index for partitioning, it falls back to in-memory sorts, degrading performance linearly with dataset size. Execution plan analysis techniques for these workloads are detailed in ",[49,822,824],{"href":823},"\u002Fadvanced-query-patterns-and-bulk-data-operations\u002Fwindow-functions-and-analytical-queries\u002F","Window Functions and Analytical Queries",[23,826,828],{"id":827},"async-execution-bulk-data-pipelines","Async Execution & Bulk Data Pipelines",[14,830,831,832,835,836,839,840,843],{},"Streaming CTE results through ",[18,833,834],{},"AsyncSession"," requires careful backpressure management. Setting ",[18,837,838],{},"stream_results=True"," instructs the underlying DBAPI (e.g., ",[18,841,842],{},"asyncpg",") to fetch rows in configurable chunks rather than buffering the entire set. This is critical for ETL pipelines where memory constraints dictate throughput.",[55,845,847],{"className":57,"code":846,"language":59,"meta":60,"style":60},"async def stream_to_etl_pipeline(session: AsyncSession, batch_size: int = 1000):\n async with session.begin():\n cte = select(Node.id, Node.payload).cte(name=\"export_batch\")\n stmt = select(cte).execution_options(stream_results=True, yield_per=batch_size)\n \n result = await session.execute(stmt)\n async for chunk in result.partitions(batch_size):\n # Yield to event loop, apply backpressure, or hand off to async queue\n await process_batch(chunk)\n",[18,848,849,870,878,897,922,926,937,951,956],{"__ignoreMap":60},[64,850,851,853,855,858,861,863,865,868],{"class":66,"line":67},[64,852,225],{"class":70},[64,854,228],{"class":70},[64,856,857],{"class":136}," stream_to_etl_pipeline",[64,859,860],{"class":74},"(session: AsyncSession, batch_size: ",[64,862,237],{"class":165},[64,864,169],{"class":70},[64,866,867],{"class":165}," 1000",[64,869,146],{"class":74},[64,871,872,874,876],{"class":66,"line":84},[64,873,259],{"class":70},[64,875,262],{"class":70},[64,877,265],{"class":74},[64,879,880,883,885,888,890,892,895],{"class":66,"line":97},[64,881,882],{"class":74}," cte ",[64,884,155],{"class":70},[64,886,887],{"class":74}," select(Node.id, Node.payload).cte(",[64,889,285],{"class":175},[64,891,155],{"class":70},[64,893,894],{"class":158},"\"export_batch\"",[64,896,184],{"class":74},[64,898,899,901,903,906,908,910,912,914,917,919],{"class":66,"line":110},[64,900,298],{"class":74},[64,902,155],{"class":70},[64,904,905],{"class":74}," select(cte).execution_options(",[64,907,332],{"class":175},[64,909,155],{"class":70},[64,911,181],{"class":165},[64,913,240],{"class":74},[64,915,916],{"class":175},"yield_per",[64,918,155],{"class":70},[64,920,921],{"class":74},"batch_size)\n",[64,923,924],{"class":66,"line":123},[64,925,309],{"class":74},[64,927,928,930,932,934],{"class":66,"line":130},[64,929,321],{"class":74},[64,931,155],{"class":70},[64,933,326],{"class":70},[64,935,936],{"class":74}," session.execute(stmt)\n",[64,938,939,941,943,946,948],{"class":66,"line":149},[64,940,259],{"class":70},[64,942,347],{"class":70},[64,944,945],{"class":74}," chunk ",[64,947,353],{"class":70},[64,949,950],{"class":74}," result.partitions(batch_size):\n",[64,952,953],{"class":66,"line":162},[64,954,955],{"class":252}," # Yield to event loop, apply backpressure, or hand off to async queue\n",[64,957,958,960],{"class":66,"line":187},[64,959,326],{"class":70},[64,961,962],{"class":74}," process_batch(chunk)\n",[14,964,965],{},"When integrating with data engineering frameworks, streaming avoids OOM crashes during DataFrame construction. For production-grade DataFrame streaming and memory mapping, see Using SQLAlchemy 2.0 with Pandas for Data Engineering.",[23,967,969],{"id":968},"materialization-hints-query-planner-control","Materialization Hints & Query Planner Control",[14,971,972,973,976],{},"Modern PostgreSQL and MySQL support explicit CTE materialization hints. By default, PostgreSQL 12+ attempts to inline CTEs (",[18,974,975],{},"NOT MATERIALIZED",") when safe, while older versions or MySQL may force materialization.",[381,978,979,985],{},[384,980,981,984],{},[18,982,983],{},"MATERIALIZED",": Forces the database to evaluate the CTE once and cache the result. Ideal for expensive aggregations referenced multiple times.",[384,986,987,989],{},[18,988,975],{},": Allows predicate pushdown and subquery flattening. Preferred for simple filters or when the CTE is referenced only once.",[55,991,993],{"className":57,"code":992,"language":59,"meta":60,"style":60},"from sqlalchemy import select\n\ndef force_materialized_cte():\n # PostgreSQL-specific hint\n expensive_agg = select(func.sum(Node.id)).cte(name=\"agg\", materialized=True)\n \n # Dialect-agnostic fallback using dialect_options\n # cte = select(...).cte(name=\"agg\", dialect_options={\"postgresql\": {\"materialized\": True}})\n \n return select(expensive_agg)\n",[18,994,995,1006,1010,1019,1024,1052,1056,1061,1066,1070],{"__ignoreMap":60},[64,996,997,999,1001,1003],{"class":66,"line":67},[64,998,71],{"class":70},[64,1000,102],{"class":74},[64,1002,78],{"class":70},[64,1004,1005],{"class":74}," select\n",[64,1007,1008],{"class":66,"line":84},[64,1009,127],{"emptyLinePlaceholder":126},[64,1011,1012,1014,1017],{"class":66,"line":97},[64,1013,468],{"class":70},[64,1015,1016],{"class":136}," force_materialized_cte",[64,1018,703],{"class":74},[64,1020,1021],{"class":66,"line":110},[64,1022,1023],{"class":252}," # PostgreSQL-specific hint\n",[64,1025,1026,1029,1031,1034,1036,1038,1041,1043,1046,1048,1050],{"class":66,"line":123},[64,1027,1028],{"class":74}," expensive_agg ",[64,1030,155],{"class":70},[64,1032,1033],{"class":74}," select(func.sum(Node.id)).cte(",[64,1035,285],{"class":175},[64,1037,155],{"class":70},[64,1039,1040],{"class":158},"\"agg\"",[64,1042,240],{"class":74},[64,1044,1045],{"class":175},"materialized",[64,1047,155],{"class":70},[64,1049,181],{"class":165},[64,1051,184],{"class":74},[64,1053,1054],{"class":66,"line":130},[64,1055,309],{"class":74},[64,1057,1058],{"class":66,"line":149},[64,1059,1060],{"class":252}," # Dialect-agnostic fallback using dialect_options\n",[64,1062,1063],{"class":66,"line":162},[64,1064,1065],{"class":252}," # cte = select(...).cte(name=\"agg\", dialect_options={\"postgresql\": {\"materialized\": True}})\n",[64,1067,1068],{"class":66,"line":187},[64,1069,309],{"class":74},[64,1071,1072,1074],{"class":66,"line":206},[64,1073,648],{"class":70},[64,1075,1076],{"class":74}," select(expensive_agg)\n",[14,1078,1079],{},"Materialization introduces a trade-off: caching reduces redundant computation but consumes temporary disk\u002Fmemory. For persistent result sets that outlive transaction boundaries, consider Creating Materialized Views with SQLAlchemy Core.",[23,1081,1083],{"id":1082},"production-pitfalls-mitigation","Production Pitfalls & Mitigation",[1085,1086,1087,1103],"table",{},[1088,1089,1090],"thead",{},[1091,1092,1093,1097,1100],"tr",{},[1094,1095,1096],"th",{},"Pitfall",[1094,1098,1099],{},"Root Cause",[1094,1101,1102],{},"Mitigation Strategy",[1104,1105,1106,1132,1156,1176,1200],"tbody",{},[1091,1107,1108,1114,1121],{},[1109,1110,1111],"td",{},[387,1112,1113],{},"Infinite recursion in async loops",[1109,1115,1116,1117,1120],{},"Missing ",[18,1118,1119],{},"WHERE"," termination or malformed graph cycles",[1109,1122,1123,1124,1127,1128,1131],{},"Enforce ",[18,1125,1126],{},"depth \u003C N"," filters; implement ",[18,1129,1130],{},"ARRAY"," cycle tracking in recursive member.",[1091,1133,1134,1139,1142],{},[1109,1135,1136],{},[387,1137,1138],{},"Uncontrolled memory allocation",[1109,1140,1141],{},"Unmaterialized CTEs on large datasets with multiple references",[1109,1143,1144,1145,1148,1149,1152,1153,433],{},"Use ",[18,1146,1147],{},"materialized=True"," for multi-reference CTEs; monitor ",[18,1150,1151],{},"temp_buffers"," and ",[18,1154,1155],{},"work_mem",[1091,1157,1158,1166,1169],{},[1109,1159,1160],{},[387,1161,1162,1163,1165],{},"Dialect ",[18,1164,983],{}," incompatibilities",[1109,1167,1168],{},"Assuming uniform hint support across PostgreSQL\u002FMySQL\u002FSQLite",[1109,1170,1171,1172,1175],{},"Abstract hint application via ",[18,1173,1174],{},"dialect_options","; fallback to explicit temp tables if unsupported.",[1091,1177,1178,1183,1193],{},[1109,1179,1180],{},[387,1181,1182],{},"ORM loader conflicts",[1109,1184,1185,1186,1189,1190],{},"Combining CTEs with ",[18,1187,1188],{},"joinedload","\u002F",[18,1191,1192],{},"selectinload",[1109,1194,1195,1196,1199],{},"Use explicit ",[18,1197,1198],{},"select()"," joins; disable eager loaders when querying CTEs directly.",[1091,1201,1202,1207,1210],{},[1109,1203,1204],{},[387,1205,1206],{},"Transaction deadlocks",[1109,1208,1209],{},"Concurrent recursive CTE bulk updates on overlapping rows",[1109,1211,1212,1213,1216,1217,1220],{},"Apply ",[18,1214,1215],{},"SELECT ... FOR UPDATE SKIP LOCKED","; batch updates via ",[18,1218,1219],{},"update().where().values()"," with explicit row ordering.",[23,1222,1224],{"id":1223},"frequently-asked-questions","Frequently Asked Questions",[14,1226,1227,1230,1231,1234,1235,1237,1238,240,1240,1243],{},[387,1228,1229],{},"How does SQLAlchemy 2.0 manage async execution for recursive CTEs?","\nIt utilizes ",[18,1232,1233],{},"AsyncSession.execute()"," with ",[18,1236,838],{}," and async drivers (e.g., ",[18,1239,842],{},[18,1241,1242],{},"aiosqlite",") to enable non-blocking, chunked iteration. This prevents event loop starvation and eliminates out-of-memory errors during deep hierarchy traversal.",[14,1245,1246,1249,1250,1152,1253,1256],{},[387,1247,1248],{},"Can CTEs be safely used in bulk insert\u002Fupdate workflows?","\nYes. By leveraging ",[18,1251,1252],{},"insert().from_select()",[18,1254,1255],{},"update().where()"," patterns, you can perform database-side transformations and set operations before committing. This minimizes network round-trips and leverages ACID guarantees.",[14,1258,1259,1262,1263,1189,1266,1269],{},[387,1260,1261],{},"What is the performance delta between recursive CTEs and Python-level recursion?","\nDatabase-level recursion minimizes network latency, leverages native B-tree indexes, and executes in compiled C code. For deep hierarchies (100+ levels), this typically delivers 10–50x throughput improvements over Python ",[18,1264,1265],{},"while",[18,1267,1268],{},"for"," loops.",[14,1271,1272,1275,1276,1279,1280,1283,1284,1287,1288,1291],{},[387,1273,1274],{},"How do I enforce recursion limits in SQLAlchemy?","\nImplement a ",[18,1277,1278],{},"depth"," integer column in the recursive member, increment it per iteration, and apply a ",[18,1281,1282],{},"WHERE depth \u003C N"," filter. Additionally, configure database-level limits (e.g., PostgreSQL's ",[18,1285,1286],{},"max_stack_depth"," or SQL Server's ",[18,1289,1290],{},"MAXRECURSION",") as a safety net.",[1293,1294,1295],"style",{},"html pre.shiki code .szBVR, html code.shiki .szBVR{--shiki-default:#D73A49;--shiki-dark:#F97583}html pre.shiki code .sVt8B, html code.shiki .sVt8B{--shiki-default:#24292E;--shiki-dark:#E1E4E8}html pre.shiki code .sScJk, html code.shiki .sScJk{--shiki-default:#6F42C1;--shiki-dark:#B392F0}html pre.shiki code .sZZnC, html code.shiki .sZZnC{--shiki-default:#032F62;--shiki-dark:#9ECBFF}html pre.shiki code .sj4cs, html code.shiki .sj4cs{--shiki-default:#005CC5;--shiki-dark:#79B8FF}html pre.shiki code .s4XuR, html code.shiki .s4XuR{--shiki-default:#E36209;--shiki-dark:#FFAB70}html pre.shiki code .sJ8bj, html code.shiki .sJ8bj{--shiki-default:#6A737D;--shiki-dark:#6A737D}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":60,"searchDepth":84,"depth":84,"links":1297},[1298,1299,1300,1301,1302,1303,1304,1305],{"id":25,"depth":84,"text":26},{"id":368,"depth":84,"text":369},{"id":436,"depth":84,"text":437},{"id":669,"depth":84,"text":670},{"id":827,"depth":84,"text":828},{"id":968,"depth":84,"text":969},{"id":1082,"depth":84,"text":1083},{"id":1223,"depth":84,"text":1224},"Common Table Expressions (CTEs) have evolved from syntactic sugar into foundational execution primitives for modern relational databases. In SQLAlchemy 2.0, the transition to a unified Core\u002FORM API and native asyncio support requires a disciplined approach to CTE construction, execution planning, and async\u002Fawait boundary management. This guide details production-ready patterns for leveraging CTEs in high-throughput, memory-constrained environments.","md",{},"\u002Fadvanced-query-patterns-and-bulk-data-operations\u002Fcommon-table-expressions-ctes-and-recursive-queries",{"title":5,"description":1306},"advanced-query-patterns-and-bulk-data-operations\u002Fcommon-table-expressions-ctes-and-recursive-queries\u002Findex","MS_FnEL8kG8qbdSWet4iJqd2v-6nkPR53b-aNdSqm4M",1778149144397]