[{"data":1,"prerenderedAt":1576},["ShallowReactive",2],{"page-\u002Fadvanced-query-patterns-and-bulk-data-operations\u002F":3},{"id":4,"title":5,"body":6,"description":16,"extension":1570,"meta":1571,"navigation":187,"path":1572,"seo":1573,"stem":1574,"__hash__":1575},"content\u002Fadvanced-query-patterns-and-bulk-data-operations\u002Findex.md","Advanced Query Patterns and Bulk Data Operations in SQLAlchemy 2.0",{"type":7,"value":8,"toc":1550},"minimark",[9,13,17,22,27,59,77,81,115,445,449,453,471,488,492,514,589,592,596,600,607,703,711,715,726,972,980,984,988,1007,1187,1195,1199,1214,1343,1347,1472,1476,1500,1518,1528,1546],[10,11,5],"h1",{"id":12},"advanced-query-patterns-and-bulk-data-operations-in-sqlalchemy-20",[14,15,16],"p",{},"Modern Python applications demand database layers that scale predictably under concurrent load, handle complex relational graphs without N+1 degradation, and ingest high-velocity datasets without exhausting memory or connection pools. SQLAlchemy 2.0 delivers a fundamentally restructured execution model, unifying Core and ORM constructs while enforcing explicit async boundaries. This guide details production-ready patterns for advanced querying, analytical SQL generation, and high-throughput bulk operations, with strict adherence to async safety and architectural best practices.",[18,19,21],"h2",{"id":20},"sqlalchemy-20-architecture-async-execution-model","SQLAlchemy 2.0 Architecture & Async Execution Model",[23,24,26],"h3",{"id":25},"core-vs-orm-selectables-in-20","Core vs ORM Selectables in 2.0",[14,28,29,30,34,35,38,39,42,43,46,47,50,51,54,55,58],{},"The legacy ",[31,32,33],"code",{},"session.query()"," API has been deprecated in favor of a unified ",[31,36,37],{},"select()"," construct that bridges SQLAlchemy Core and ORM. In 2.0, all queries originate from ",[31,40,41],{},"sqlalchemy.sql.expression.select()",", which returns a ",[31,44,45],{},"Select"," object that can be executed against either a ",[31,48,49],{},"Connection"," or an ",[31,52,53],{},"AsyncSession",". This architectural shift eliminates the dual-API confusion of 1.x, enforces strict type hinting via ",[31,56,57],{},"Mapped"," annotations, and ensures that query compilation remains dialect-agnostic until execution time.",[14,60,61,62,65,66,69,70,72,73,76],{},"When constructing queries, developers should prefer ",[31,63,64],{},"select(Model)"," over legacy ",[31,67,68],{},"session.query(Model)",". The ORM layer now intercepts ",[31,71,45],{}," objects, automatically resolving mapped columns, relationships, and inheritance hierarchies. This unification simplifies query composition, enables seamless integration with ",[31,74,75],{},"sqlalchemy.ext.asyncio",", and guarantees that analytical constructs compile identically across PostgreSQL, SQLite, and MySQL.",[23,78,80],{"id":79},"asyncsession-lifecycle-connection-pooling","AsyncSession Lifecycle & Connection Pooling",[14,82,83,84,86,87,90,91,94,95,98,99,102,103,106,107,110,111,114],{},"Async execution in SQLAlchemy 2.0 requires explicit lifecycle management. The ",[31,85,53],{}," does not auto-flush or auto-commit; every mutation and retrieval must be awaited. Connection pooling must be configured at the engine level using ",[31,88,89],{},"create_async_engine()",", with ",[31,92,93],{},"pool_size",", ",[31,96,97],{},"max_overflow",", and ",[31,100,101],{},"pool_timeout"," tuned to your application's concurrency profile. Blocking I\u002FO inside an ",[31,104,105],{},"async def"," context will stall the event loop, so drivers like ",[31,108,109],{},"asyncpg"," (PostgreSQL) or ",[31,112,113],{},"aiosqlite"," (SQLite) are mandatory.",[116,117,122],"pre",{"className":118,"code":119,"language":120,"meta":121,"style":121},"language-python shiki shiki-themes github-light github-dark","from typing import Sequence\nfrom sqlalchemy import select, func\nfrom sqlalchemy.ext.asyncio import AsyncSession, create_async_engine\nfrom sqlalchemy.orm import Mapped, mapped_column, DeclarativeBase\n\nclass User(DeclarativeBase):\n __tablename__ = \"users\"\n id: Mapped[int] = mapped_column(primary_key=True)\n email: Mapped[str] = mapped_column(unique=True, index=True)\n status: Mapped[str] = mapped_column(default=\"active\")\n\nasync def fetch_active_users(engine: AsyncEngine, limit: int = 100) -> Sequence[User]:\n stmt = (\n select(User)\n .where(User.status == \"active\")\n .order_by(User.id)\n .limit(limit)\n )\n \n async with AsyncSession(engine) as session:\n result = await session.execute(stmt)\n # scalars() extracts ORM instances from the Result object\n return result.scalars().all()\n","python","",[31,123,124,143,156,169,182,189,208,221,254,287,311,316,342,353,359,373,379,385,391,397,415,429,436],{"__ignoreMap":121},[125,126,129,133,137,140],"span",{"class":127,"line":128},"line",1,[125,130,132],{"class":131},"szBVR","from",[125,134,136],{"class":135},"sVt8B"," typing ",[125,138,139],{"class":131},"import",[125,141,142],{"class":135}," Sequence\n",[125,144,146,148,151,153],{"class":127,"line":145},2,[125,147,132],{"class":131},[125,149,150],{"class":135}," sqlalchemy ",[125,152,139],{"class":131},[125,154,155],{"class":135}," select, func\n",[125,157,159,161,164,166],{"class":127,"line":158},3,[125,160,132],{"class":131},[125,162,163],{"class":135}," sqlalchemy.ext.asyncio ",[125,165,139],{"class":131},[125,167,168],{"class":135}," AsyncSession, create_async_engine\n",[125,170,172,174,177,179],{"class":127,"line":171},4,[125,173,132],{"class":131},[125,175,176],{"class":135}," sqlalchemy.orm ",[125,178,139],{"class":131},[125,180,181],{"class":135}," Mapped, mapped_column, DeclarativeBase\n",[125,183,185],{"class":127,"line":184},5,[125,186,188],{"emptyLinePlaceholder":187},true,"\n",[125,190,192,195,199,202,205],{"class":127,"line":191},6,[125,193,194],{"class":131},"class",[125,196,198],{"class":197},"sScJk"," User",[125,200,201],{"class":135},"(",[125,203,204],{"class":197},"DeclarativeBase",[125,206,207],{"class":135},"):\n",[125,209,211,214,217],{"class":127,"line":210},7,[125,212,213],{"class":135}," __tablename__ ",[125,215,216],{"class":131},"=",[125,218,220],{"class":219},"sZZnC"," \"users\"\n",[125,222,224,228,231,234,237,239,242,246,248,251],{"class":127,"line":223},8,[125,225,227],{"class":226},"sj4cs"," id",[125,229,230],{"class":135},": Mapped[",[125,232,233],{"class":226},"int",[125,235,236],{"class":135},"] ",[125,238,216],{"class":131},[125,240,241],{"class":135}," mapped_column(",[125,243,245],{"class":244},"s4XuR","primary_key",[125,247,216],{"class":131},[125,249,250],{"class":226},"True",[125,252,253],{"class":135},")\n",[125,255,257,260,263,265,267,269,272,274,276,278,281,283,285],{"class":127,"line":256},9,[125,258,259],{"class":135}," email: Mapped[",[125,261,262],{"class":226},"str",[125,264,236],{"class":135},[125,266,216],{"class":131},[125,268,241],{"class":135},[125,270,271],{"class":244},"unique",[125,273,216],{"class":131},[125,275,250],{"class":226},[125,277,94],{"class":135},[125,279,280],{"class":244},"index",[125,282,216],{"class":131},[125,284,250],{"class":226},[125,286,253],{"class":135},[125,288,290,293,295,297,299,301,304,306,309],{"class":127,"line":289},10,[125,291,292],{"class":135}," status: Mapped[",[125,294,262],{"class":226},[125,296,236],{"class":135},[125,298,216],{"class":131},[125,300,241],{"class":135},[125,302,303],{"class":244},"default",[125,305,216],{"class":131},[125,307,308],{"class":219},"\"active\"",[125,310,253],{"class":135},[125,312,314],{"class":127,"line":313},11,[125,315,188],{"emptyLinePlaceholder":187},[125,317,319,322,325,328,331,333,336,339],{"class":127,"line":318},12,[125,320,321],{"class":131},"async",[125,323,324],{"class":131}," def",[125,326,327],{"class":197}," fetch_active_users",[125,329,330],{"class":135},"(engine: AsyncEngine, limit: ",[125,332,233],{"class":226},[125,334,335],{"class":131}," =",[125,337,338],{"class":226}," 100",[125,340,341],{"class":135},") -> Sequence[User]:\n",[125,343,345,348,350],{"class":127,"line":344},13,[125,346,347],{"class":135}," stmt ",[125,349,216],{"class":131},[125,351,352],{"class":135}," (\n",[125,354,356],{"class":127,"line":355},14,[125,357,358],{"class":135}," select(User)\n",[125,360,362,365,368,371],{"class":127,"line":361},15,[125,363,364],{"class":135}," .where(User.status ",[125,366,367],{"class":131},"==",[125,369,370],{"class":219}," \"active\"",[125,372,253],{"class":135},[125,374,376],{"class":127,"line":375},16,[125,377,378],{"class":135}," .order_by(User.id)\n",[125,380,382],{"class":127,"line":381},17,[125,383,384],{"class":135}," .limit(limit)\n",[125,386,388],{"class":127,"line":387},18,[125,389,390],{"class":135}," )\n",[125,392,394],{"class":127,"line":393},19,[125,395,396],{"class":135}," \n",[125,398,400,403,406,409,412],{"class":127,"line":399},20,[125,401,402],{"class":131}," async",[125,404,405],{"class":131}," with",[125,407,408],{"class":135}," AsyncSession(engine) ",[125,410,411],{"class":131},"as",[125,413,414],{"class":135}," session:\n",[125,416,418,421,423,426],{"class":127,"line":417},21,[125,419,420],{"class":135}," result ",[125,422,216],{"class":131},[125,424,425],{"class":131}," await",[125,427,428],{"class":135}," session.execute(stmt)\n",[125,430,432],{"class":127,"line":431},22,[125,433,435],{"class":434},"sJ8bj"," # scalars() extracts ORM instances from the Result object\n",[125,437,439,442],{"class":127,"line":438},23,[125,440,441],{"class":131}," return",[125,443,444],{"class":135}," result.scalars().all()\n",[18,446,448],{"id":447},"advanced-relational-query-patterns","Advanced Relational Query Patterns",[23,450,452],{"id":451},"optimizing-relationship-traversal","Optimizing Relationship Traversal",[14,454,455,456,459,460,463,464,467,468,470],{},"Relationship loading strategies dictate how SQLAlchemy resolves foreign key joins. Indiscriminate use of ",[31,457,458],{},"joinedload"," on one-to-many relationships frequently triggers Cartesian product explosions, exponentially inflating memory consumption and network transfer. Instead, adopt ",[31,461,462],{},"selectinload"," for collections exceeding ~100 rows, which issues a single ",[31,465,466],{},"IN (...)"," query per relationship tier. For predictable, small-cardinality joins, ",[31,469,458],{}," remains optimal due to reduced round trips and database-level join caching.",[14,472,473,474,479,480,483,484,487],{},"When architecting multi-table aggregations, carefully evaluate ",[475,476,478],"a",{"href":477},"\u002Fadvanced-query-patterns-and-bulk-data-operations\u002Fcomplex-joins-and-relationship-loading-strategies\u002F","Complex Joins and Relationship Loading Strategies"," to balance query complexity against execution plan efficiency. Always profile generated SQL using ",[31,481,482],{},"echo=True"," or database-specific ",[31,485,486],{},"EXPLAIN ANALYZE"," before deploying relationship-heavy queries to production.",[23,489,491],{"id":490},"correlated-subqueries-conditional-filtering","Correlated Subqueries & Conditional Filtering",[14,493,494,495,498,499,502,503,506,507,509,510,513],{},"Correlated subqueries and ",[31,496,497],{},"EXISTS"," clauses are essential for conditional filtering without materializing intermediate result sets. In 2.0, these are constructed using ",[31,500,501],{},"select().correlate()"," or ",[31,504,505],{},"select().where(Model.id.in_(...))",". For large datasets, ",[31,508,497],{}," typically outperforms ",[31,511,512],{},"IN"," because the database optimizer can short-circuit evaluation upon finding the first matching row, drastically reducing scan overhead.",[116,515,517],{"className":118,"code":516,"language":120,"meta":121,"style":121},"from sqlalchemy import exists\n\n# Efficient conditional filtering using EXISTS\nsubquery = (\n select(Order.user_id)\n .where(Order.amount > 1000)\n .exists()\n)\n\nstmt = select(User).where(subquery)\n",[31,518,519,530,534,539,548,553,566,571,575,579],{"__ignoreMap":121},[125,520,521,523,525,527],{"class":127,"line":128},[125,522,132],{"class":131},[125,524,150],{"class":135},[125,526,139],{"class":131},[125,528,529],{"class":135}," exists\n",[125,531,532],{"class":127,"line":145},[125,533,188],{"emptyLinePlaceholder":187},[125,535,536],{"class":127,"line":158},[125,537,538],{"class":434},"# Efficient conditional filtering using EXISTS\n",[125,540,541,544,546],{"class":127,"line":171},[125,542,543],{"class":135},"subquery ",[125,545,216],{"class":131},[125,547,352],{"class":135},[125,549,550],{"class":127,"line":184},[125,551,552],{"class":135}," select(Order.user_id)\n",[125,554,555,558,561,564],{"class":127,"line":191},[125,556,557],{"class":135}," .where(Order.amount ",[125,559,560],{"class":131},">",[125,562,563],{"class":226}," 1000",[125,565,253],{"class":135},[125,567,568],{"class":127,"line":210},[125,569,570],{"class":135}," .exists()\n",[125,572,573],{"class":127,"line":223},[125,574,253],{"class":135},[125,576,577],{"class":127,"line":256},[125,578,188],{"emptyLinePlaceholder":187},[125,580,581,584,586],{"class":127,"line":289},[125,582,583],{"class":135},"stmt ",[125,585,216],{"class":131},[125,587,588],{"class":135}," select(User).where(subquery)\n",[14,590,591],{},"Implementing Subqueries and EXISTS Clauses Optimization ensures that conditional filters push down to the storage engine, minimizing Python-side iteration and preventing memory bloat during large dataset scans.",[18,593,595],{"id":594},"analytical-sql-hierarchical-data-processing","Analytical SQL & Hierarchical Data Processing",[23,597,599],{"id":598},"window-functions-for-time-series-ranking","Window Functions for Time-Series & Ranking",[14,601,602,603,606],{},"SQLAlchemy 2.0 fully supports window functions via ",[31,604,605],{},"func.\u003Cfunction>().over()",". These are indispensable for time-series ranking, running totals, and lag\u002Flead calculations. Partitioning and ordering must be explicitly defined to guarantee deterministic results across dialects. While ORM models can map window function results, analytical workloads often benefit from Core-level execution for cleaner query plans and reduced ORM hydration overhead.",[116,608,610],{"className":118,"code":609,"language":120,"meta":121,"style":121},"from sqlalchemy import func\n\n# Ranking users by account creation date within each tenant\nstmt = (\n select(\n User.id,\n User.tenant_id,\n User.created_at,\n func.row_number().over(\n partition_by=User.tenant_id,\n order_by=User.created_at.desc()\n ).label(\"tenant_rank\")\n )\n)\n",[31,611,612,623,627,632,640,645,650,655,660,665,675,685,695,699],{"__ignoreMap":121},[125,613,614,616,618,620],{"class":127,"line":128},[125,615,132],{"class":131},[125,617,150],{"class":135},[125,619,139],{"class":131},[125,621,622],{"class":135}," func\n",[125,624,625],{"class":127,"line":145},[125,626,188],{"emptyLinePlaceholder":187},[125,628,629],{"class":127,"line":158},[125,630,631],{"class":434},"# Ranking users by account creation date within each tenant\n",[125,633,634,636,638],{"class":127,"line":171},[125,635,583],{"class":135},[125,637,216],{"class":131},[125,639,352],{"class":135},[125,641,642],{"class":127,"line":184},[125,643,644],{"class":135}," select(\n",[125,646,647],{"class":127,"line":191},[125,648,649],{"class":135}," User.id,\n",[125,651,652],{"class":127,"line":210},[125,653,654],{"class":135}," User.tenant_id,\n",[125,656,657],{"class":127,"line":223},[125,658,659],{"class":135}," User.created_at,\n",[125,661,662],{"class":127,"line":256},[125,663,664],{"class":135}," func.row_number().over(\n",[125,666,667,670,672],{"class":127,"line":289},[125,668,669],{"class":244}," partition_by",[125,671,216],{"class":131},[125,673,674],{"class":135},"User.tenant_id,\n",[125,676,677,680,682],{"class":127,"line":313},[125,678,679],{"class":244}," order_by",[125,681,216],{"class":131},[125,683,684],{"class":135},"User.created_at.desc()\n",[125,686,687,690,693],{"class":127,"line":318},[125,688,689],{"class":135}," ).label(",[125,691,692],{"class":219},"\"tenant_rank\"",[125,694,253],{"class":135},[125,696,697],{"class":127,"line":344},[125,698,390],{"class":135},[125,700,701],{"class":127,"line":355},[125,702,253],{"class":135},[14,704,705,706,710],{},"When designing dashboards or reporting pipelines, refer to ",[475,707,709],{"href":708},"\u002Fadvanced-query-patterns-and-bulk-data-operations\u002Fwindow-functions-and-analytical-queries\u002F","Window Functions and Analytical Queries"," for dialect-specific compilation notes and performance tuning strategies.",[23,712,714],{"id":713},"recursive-ctes-for-graph-tree-structures","Recursive CTEs for Graph & Tree Structures",[14,716,717,718,721,722,725],{},"Hierarchical data traversal requires recursive Common Table Expressions (CTEs). SQLAlchemy 2.0 implements these via ",[31,719,720],{},"cte(recursive=True)"," combined with ",[31,723,724],{},"union_all()",". The base case initializes the recursion, while the recursive member references the CTE alias to traverse parent-child relationships until termination.",[116,727,729],{"className":118,"code":728,"language":120,"meta":121,"style":121},"from sqlalchemy import Column, Integer, String, select, union_all\n\nclass Category(DeclarativeBase):\n __tablename__ = \"categories\"\n id: Mapped[int] = mapped_column(primary_key=True)\n parent_id: Mapped[int | None] = mapped_column()\n name: Mapped[str] = mapped_column()\n\n# Base case: root categories\nbase = select(Category.id, Category.parent_id, Category.name, Category.id.label(\"depth\")).where(Category.parent_id.is_(None))\n\n# Recursive step\nrecursive = select(\n Category.id, \n Category.parent_id, \n Category.name, \n (Category.id + 1).label(\"depth\")\n).join(Category, Category.parent_id == recursive.c.id)\n\n# Assemble CTE\ntree_cte = base.union_all(recursive).cte(name=\"category_tree\", recursive=True)\n\n# Final query\nstmt = select(tree_cte).order_by(tree_cte.c.depth)\n",[31,730,731,742,746,759,768,790,810,823,827,832,854,858,863,872,877,882,887,905,915,919,924,953,957,962],{"__ignoreMap":121},[125,732,733,735,737,739],{"class":127,"line":128},[125,734,132],{"class":131},[125,736,150],{"class":135},[125,738,139],{"class":131},[125,740,741],{"class":135}," Column, Integer, String, select, union_all\n",[125,743,744],{"class":127,"line":145},[125,745,188],{"emptyLinePlaceholder":187},[125,747,748,750,753,755,757],{"class":127,"line":158},[125,749,194],{"class":131},[125,751,752],{"class":197}," Category",[125,754,201],{"class":135},[125,756,204],{"class":197},[125,758,207],{"class":135},[125,760,761,763,765],{"class":127,"line":171},[125,762,213],{"class":135},[125,764,216],{"class":131},[125,766,767],{"class":219}," \"categories\"\n",[125,769,770,772,774,776,778,780,782,784,786,788],{"class":127,"line":184},[125,771,227],{"class":226},[125,773,230],{"class":135},[125,775,233],{"class":226},[125,777,236],{"class":135},[125,779,216],{"class":131},[125,781,241],{"class":135},[125,783,245],{"class":244},[125,785,216],{"class":131},[125,787,250],{"class":226},[125,789,253],{"class":135},[125,791,792,795,797,800,803,805,807],{"class":127,"line":191},[125,793,794],{"class":135}," parent_id: Mapped[",[125,796,233],{"class":226},[125,798,799],{"class":131}," |",[125,801,802],{"class":226}," None",[125,804,236],{"class":135},[125,806,216],{"class":131},[125,808,809],{"class":135}," mapped_column()\n",[125,811,812,815,817,819,821],{"class":127,"line":210},[125,813,814],{"class":135}," name: Mapped[",[125,816,262],{"class":226},[125,818,236],{"class":135},[125,820,216],{"class":131},[125,822,809],{"class":135},[125,824,825],{"class":127,"line":223},[125,826,188],{"emptyLinePlaceholder":187},[125,828,829],{"class":127,"line":256},[125,830,831],{"class":434},"# Base case: root categories\n",[125,833,834,837,839,842,845,848,851],{"class":127,"line":289},[125,835,836],{"class":135},"base ",[125,838,216],{"class":131},[125,840,841],{"class":135}," select(Category.id, Category.parent_id, Category.name, Category.id.label(",[125,843,844],{"class":219},"\"depth\"",[125,846,847],{"class":135},")).where(Category.parent_id.is_(",[125,849,850],{"class":226},"None",[125,852,853],{"class":135},"))\n",[125,855,856],{"class":127,"line":313},[125,857,188],{"emptyLinePlaceholder":187},[125,859,860],{"class":127,"line":318},[125,861,862],{"class":434},"# Recursive step\n",[125,864,865,868,870],{"class":127,"line":344},[125,866,867],{"class":135},"recursive ",[125,869,216],{"class":131},[125,871,644],{"class":135},[125,873,874],{"class":127,"line":355},[125,875,876],{"class":135}," Category.id, \n",[125,878,879],{"class":127,"line":361},[125,880,881],{"class":135}," Category.parent_id, \n",[125,883,884],{"class":127,"line":375},[125,885,886],{"class":135}," Category.name, \n",[125,888,889,892,895,898,901,903],{"class":127,"line":381},[125,890,891],{"class":135}," (Category.id ",[125,893,894],{"class":131},"+",[125,896,897],{"class":226}," 1",[125,899,900],{"class":135},").label(",[125,902,844],{"class":219},[125,904,253],{"class":135},[125,906,907,910,912],{"class":127,"line":387},[125,908,909],{"class":135},").join(Category, Category.parent_id ",[125,911,367],{"class":131},[125,913,914],{"class":135}," recursive.c.id)\n",[125,916,917],{"class":127,"line":393},[125,918,188],{"emptyLinePlaceholder":187},[125,920,921],{"class":127,"line":399},[125,922,923],{"class":434},"# Assemble CTE\n",[125,925,926,929,931,934,937,939,942,944,947,949,951],{"class":127,"line":417},[125,927,928],{"class":135},"tree_cte ",[125,930,216],{"class":131},[125,932,933],{"class":135}," base.union_all(recursive).cte(",[125,935,936],{"class":244},"name",[125,938,216],{"class":131},[125,940,941],{"class":219},"\"category_tree\"",[125,943,94],{"class":135},[125,945,946],{"class":244},"recursive",[125,948,216],{"class":131},[125,950,250],{"class":226},[125,952,253],{"class":135},[125,954,955],{"class":127,"line":431},[125,956,188],{"emptyLinePlaceholder":187},[125,958,959],{"class":127,"line":438},[125,960,961],{"class":434},"# Final query\n",[125,963,965,967,969],{"class":127,"line":964},24,[125,966,583],{"class":135},[125,968,216],{"class":131},[125,970,971],{"class":135}," select(tree_cte).order_by(tree_cte.c.depth)\n",[14,973,974,975,979],{},"For production graph traversals, consult ",[475,976,978],{"href":977},"\u002Fadvanced-query-patterns-and-bulk-data-operations\u002Fcommon-table-expressions-ctes-and-recursive-queries\u002F","Common Table Expressions (CTEs) and Recursive Queries"," to understand cycle detection, depth limits, and asyncpg\u002Fpsycopg compilation differences.",[18,981,983],{"id":982},"high-throughput-bulk-data-operations","High-Throughput Bulk Data Operations",[23,985,987],{"id":986},"native-bulk-inserts-upserts","Native Bulk Inserts & Upserts",[14,989,990,991,994,995,998,999,1002,1003,1006],{},"ORM-level ",[31,992,993],{},"session.add_all()"," is fundamentally unsuited for high-throughput ingestion. It hydrates objects, tracks state changes, and issues individual ",[31,996,997],{},"INSERT"," statements, creating severe CPU and memory bottlenecks. SQLAlchemy 2.0 mandates Core-level ",[31,1000,1001],{},"insert(Model).values()"," with ",[31,1004,1005],{},"executemany"," optimization. For atomic upserts, leverage dialect-specific conflict resolution clauses.",[116,1008,1010],{"className":118,"code":1009,"language":120,"meta":121,"style":121},"from sqlalchemy import insert\nfrom sqlalchemy.dialects.postgresql import insert as pg_insert\n\nasync def bulk_upsert_users(engine: AsyncEngine, data: list[dict]) -> None:\n stmt = pg_insert(User).values(data)\n upsert_stmt = stmt.on_conflict_do_update(\n index_elements=[\"email\"],\n set_={\"status\": stmt.excluded.status, \"updated_at\": func.now()}\n )\n \n async with AsyncSession(engine) as session:\n await session.execute(\n upsert_stmt,\n execution_options={\"fast_executemany\": True}\n )\n await session.commit()\n",[31,1011,1012,1023,1040,1044,1067,1076,1086,1102,1124,1128,1132,1144,1151,1156,1176,1180],{"__ignoreMap":121},[125,1013,1014,1016,1018,1020],{"class":127,"line":128},[125,1015,132],{"class":131},[125,1017,150],{"class":135},[125,1019,139],{"class":131},[125,1021,1022],{"class":135}," insert\n",[125,1024,1025,1027,1030,1032,1035,1037],{"class":127,"line":145},[125,1026,132],{"class":131},[125,1028,1029],{"class":135}," sqlalchemy.dialects.postgresql ",[125,1031,139],{"class":131},[125,1033,1034],{"class":135}," insert ",[125,1036,411],{"class":131},[125,1038,1039],{"class":135}," pg_insert\n",[125,1041,1042],{"class":127,"line":158},[125,1043,188],{"emptyLinePlaceholder":187},[125,1045,1046,1048,1050,1053,1056,1059,1062,1064],{"class":127,"line":171},[125,1047,321],{"class":131},[125,1049,324],{"class":131},[125,1051,1052],{"class":197}," bulk_upsert_users",[125,1054,1055],{"class":135},"(engine: AsyncEngine, data: list[",[125,1057,1058],{"class":226},"dict",[125,1060,1061],{"class":135},"]) -> ",[125,1063,850],{"class":226},[125,1065,1066],{"class":135},":\n",[125,1068,1069,1071,1073],{"class":127,"line":184},[125,1070,347],{"class":135},[125,1072,216],{"class":131},[125,1074,1075],{"class":135}," pg_insert(User).values(data)\n",[125,1077,1078,1081,1083],{"class":127,"line":191},[125,1079,1080],{"class":135}," upsert_stmt ",[125,1082,216],{"class":131},[125,1084,1085],{"class":135}," stmt.on_conflict_do_update(\n",[125,1087,1088,1091,1093,1096,1099],{"class":127,"line":210},[125,1089,1090],{"class":244}," index_elements",[125,1092,216],{"class":131},[125,1094,1095],{"class":135},"[",[125,1097,1098],{"class":219},"\"email\"",[125,1100,1101],{"class":135},"],\n",[125,1103,1104,1107,1109,1112,1115,1118,1121],{"class":127,"line":223},[125,1105,1106],{"class":244}," set_",[125,1108,216],{"class":131},[125,1110,1111],{"class":135},"{",[125,1113,1114],{"class":219},"\"status\"",[125,1116,1117],{"class":135},": stmt.excluded.status, ",[125,1119,1120],{"class":219},"\"updated_at\"",[125,1122,1123],{"class":135},": func.now()}\n",[125,1125,1126],{"class":127,"line":256},[125,1127,390],{"class":135},[125,1129,1130],{"class":127,"line":289},[125,1131,396],{"class":135},[125,1133,1134,1136,1138,1140,1142],{"class":127,"line":313},[125,1135,402],{"class":131},[125,1137,405],{"class":131},[125,1139,408],{"class":135},[125,1141,411],{"class":131},[125,1143,414],{"class":135},[125,1145,1146,1148],{"class":127,"line":318},[125,1147,425],{"class":131},[125,1149,1150],{"class":135}," session.execute(\n",[125,1152,1153],{"class":127,"line":344},[125,1154,1155],{"class":135}," upsert_stmt,\n",[125,1157,1158,1161,1163,1165,1168,1171,1173],{"class":127,"line":355},[125,1159,1160],{"class":244}," execution_options",[125,1162,216],{"class":131},[125,1164,1111],{"class":135},[125,1166,1167],{"class":219},"\"fast_executemany\"",[125,1169,1170],{"class":135},": ",[125,1172,250],{"class":226},[125,1174,1175],{"class":135},"}\n",[125,1177,1178],{"class":127,"line":361},[125,1179,390],{"class":135},[125,1181,1182,1184],{"class":127,"line":375},[125,1183,425],{"class":131},[125,1185,1186],{"class":135}," session.commit()\n",[14,1188,1189,1190,1194],{},"Architects should review ",[475,1191,1193],{"href":1192},"\u002Fadvanced-query-patterns-and-bulk-data-operations\u002Fhigh-performance-bulk-inserts-and-updates\u002F","High-Performance Bulk Inserts and Updates"," for driver-specific optimizations, transaction batching strategies, and index maintenance during mass writes.",[23,1196,1198],{"id":1197},"async-chunking-memory-management","Async Chunking & Memory Management",[14,1200,1201,1202,1205,1206,1209,1210,1213],{},"Even with native bulk constructs, transmitting millions of rows in a single payload will exhaust connection buffers and trigger ",[31,1203,1204],{},"MemoryError",". Implement generator-based chunking paired with ",[31,1207,1208],{},"yield_per"," to stream results safely. In async contexts, ",[31,1211,1212],{},"session.stream()"," is required to prevent blocking the event loop during large result iteration.",[116,1215,1217],{"className":118,"code":1216,"language":120,"meta":121,"style":121},"from typing import AsyncGenerator\n\nasync def stream_large_dataset(engine: AsyncEngine, batch_size: int = 5000) -> AsyncGenerator[User, None]:\n stmt = select(User).order_by(User.id).execution_options(yield_per=batch_size)\n \n async with AsyncSession(engine) as session:\n async with session.stream(stmt) as result:\n async for partition in result.partitions(batch_size):\n for row in partition:\n yield row\n",[31,1218,1219,1230,1234,1261,1277,1281,1293,1307,1323,1335],{"__ignoreMap":121},[125,1220,1221,1223,1225,1227],{"class":127,"line":128},[125,1222,132],{"class":131},[125,1224,136],{"class":135},[125,1226,139],{"class":131},[125,1228,1229],{"class":135}," AsyncGenerator\n",[125,1231,1232],{"class":127,"line":145},[125,1233,188],{"emptyLinePlaceholder":187},[125,1235,1236,1238,1240,1243,1246,1248,1250,1253,1256,1258],{"class":127,"line":158},[125,1237,321],{"class":131},[125,1239,324],{"class":131},[125,1241,1242],{"class":197}," stream_large_dataset",[125,1244,1245],{"class":135},"(engine: AsyncEngine, batch_size: ",[125,1247,233],{"class":226},[125,1249,335],{"class":131},[125,1251,1252],{"class":226}," 5000",[125,1254,1255],{"class":135},") -> AsyncGenerator[User, ",[125,1257,850],{"class":226},[125,1259,1260],{"class":135},"]:\n",[125,1262,1263,1265,1267,1270,1272,1274],{"class":127,"line":171},[125,1264,347],{"class":135},[125,1266,216],{"class":131},[125,1268,1269],{"class":135}," select(User).order_by(User.id).execution_options(",[125,1271,1208],{"class":244},[125,1273,216],{"class":131},[125,1275,1276],{"class":135},"batch_size)\n",[125,1278,1279],{"class":127,"line":184},[125,1280,396],{"class":135},[125,1282,1283,1285,1287,1289,1291],{"class":127,"line":191},[125,1284,402],{"class":131},[125,1286,405],{"class":131},[125,1288,408],{"class":135},[125,1290,411],{"class":131},[125,1292,414],{"class":135},[125,1294,1295,1297,1299,1302,1304],{"class":127,"line":210},[125,1296,402],{"class":131},[125,1298,405],{"class":131},[125,1300,1301],{"class":135}," session.stream(stmt) ",[125,1303,411],{"class":131},[125,1305,1306],{"class":135}," result:\n",[125,1308,1309,1311,1314,1317,1320],{"class":127,"line":223},[125,1310,402],{"class":131},[125,1312,1313],{"class":131}," for",[125,1315,1316],{"class":135}," partition ",[125,1318,1319],{"class":131},"in",[125,1321,1322],{"class":135}," result.partitions(batch_size):\n",[125,1324,1325,1327,1330,1332],{"class":127,"line":256},[125,1326,1313],{"class":131},[125,1328,1329],{"class":135}," row ",[125,1331,1319],{"class":131},[125,1333,1334],{"class":135}," partition:\n",[125,1336,1337,1340],{"class":127,"line":289},[125,1338,1339],{"class":131}," yield",[125,1341,1342],{"class":135}," row\n",[18,1344,1346],{"id":1345},"production-pitfalls-mitigations","Production Pitfalls & Mitigations",[1348,1349,1350,1366],"table",{},[1351,1352,1353],"thead",{},[1354,1355,1356,1360,1363],"tr",{},[1357,1358,1359],"th",{},"Pitfall",[1357,1361,1362],{},"Impact",[1357,1364,1365],{},"Mitigation",[1367,1368,1369,1393,1415,1435,1450],"tbody",{},[1354,1370,1371,1380,1383],{},[1372,1373,1374,1375,502,1377,1379],"td",{},"Using legacy ",[31,1376,33],{},[31,1378,993],{}," without chunking",[1372,1381,1382],{},"OOM errors, event loop starvation in async contexts",[1372,1384,1385,1386,1388,1389,1392],{},"Migrate to ",[31,1387,37],{}," and ",[31,1390,1391],{},"insert().values()"," with explicit chunking",[1354,1394,1395,1402,1405],{},[1372,1396,1397,1398,1401],{},"Ignoring ",[31,1399,1400],{},"execution_options({\"yield_per\": N})"," during iteration",[1372,1403,1404],{},"Memory bloat, connection pool exhaustion",[1372,1406,1407,1408,1002,1410,1388,1412],{},"Use ",[31,1409,1212],{},[31,1411,1208],{},[31,1413,1414],{},"partitions()",[1354,1416,1417,1423,1426],{},[1372,1418,1419,1420,1422],{},"Applying ",[31,1421,458],{}," indiscriminately on one-to-many relationships",[1372,1424,1425],{},"Cartesian product explosions, query plan degradation",[1372,1427,1428,1429,1431,1432,1434],{},"Default to ",[31,1430,462],{}," for collections; reserve ",[31,1433,458],{}," for small, predictable joins",[1354,1436,1437,1440,1443],{},[1372,1438,1439],{},"Failing to compile CTEs\u002Fwindow functions across DBAPI drivers",[1372,1441,1442],{},"Syntax errors, silent fallback to inefficient scans",[1372,1444,1445,1446,1449],{},"Test against target dialect; use ",[31,1447,1448],{},"sqlalchemy.dialects"," imports for conflict clauses",[1354,1451,1452,1455,1458],{},[1372,1453,1454],{},"Overlooking transaction isolation levels during concurrent bulk writes",[1372,1456,1457],{},"Deadlocks, phantom reads, duplicate key violations",[1372,1459,1460,1461,1464,1465,502,1468,1471],{},"Set ",[31,1462,1463],{},"isolation_level"," at engine creation; use ",[31,1466,1467],{},"SERIALIZABLE",[31,1469,1470],{},"REPEATABLE READ"," for critical upserts",[18,1473,1475],{"id":1474},"frequently-asked-questions","Frequently Asked Questions",[14,1477,1478,1482,1483,1485,1486,1388,1489,1492,1493,1495,1496,1499],{},[1479,1480,1481],"strong",{},"How does SQLAlchemy 2.0 handle async bulk operations differently from 1.x?","\n2.0 enforces ",[31,1484,53],{}," with explicit ",[31,1487,1488],{},"await session.execute()",[31,1490,1491],{},"await session.commit()",", requiring native ",[31,1494,1391],{}," for bulk instead of ORM-level ",[31,1497,1498],{},"add_all()",". This eliminates implicit connection blocking and forces developers to adopt chunked, stream-safe execution patterns.",[14,1501,1502,1511,1512,1514,1515,1517],{},[1479,1503,1504,1505,1507,1508,1510],{},"When should I use ",[31,1506,462],{}," versus ",[31,1509,458],{},"?","\nUse ",[31,1513,462],{}," for large collections to avoid Cartesian product overhead; use ",[31,1516,458],{}," for small, predictable relationships where a single JOIN reduces round trips and leverages database caching.",[14,1519,1520,1523,1524,1527],{},[1479,1521,1522],{},"Can window functions be used directly with the SQLAlchemy ORM?","\nYes, via ",[31,1525,1526],{},"func.row_number().over(partition_by=..., order_by=...)"," mapped to model attributes, though Core constructs often yield cleaner execution plans for analytical workloads.",[14,1529,1530,1533,1534,1537,1538,1541,1542,1545],{},[1479,1531,1532],{},"What is the most efficient way to handle 1M+ row inserts in SQLAlchemy 2.0?","\nChunk data into batches of 10k–50k, use ",[31,1535,1536],{},"session.execute(insert(Model).values(chunk))",", enable ",[31,1539,1540],{},"fast_executemany"," for supported drivers, and apply ",[31,1543,1544],{},"ON CONFLICT"," clauses to avoid duplicate key violations while maintaining atomicity.",[1547,1548,1549],"style",{},"html pre.shiki code .szBVR, html code.shiki .szBVR{--shiki-default:#D73A49;--shiki-dark:#F97583}html pre.shiki code .sVt8B, html code.shiki .sVt8B{--shiki-default:#24292E;--shiki-dark:#E1E4E8}html pre.shiki code .sScJk, html code.shiki .sScJk{--shiki-default:#6F42C1;--shiki-dark:#B392F0}html pre.shiki code .sZZnC, html code.shiki .sZZnC{--shiki-default:#032F62;--shiki-dark:#9ECBFF}html pre.shiki code .sj4cs, html code.shiki .sj4cs{--shiki-default:#005CC5;--shiki-dark:#79B8FF}html pre.shiki code .s4XuR, html code.shiki .s4XuR{--shiki-default:#E36209;--shiki-dark:#FFAB70}html pre.shiki code .sJ8bj, html code.shiki .sJ8bj{--shiki-default:#6A737D;--shiki-dark:#6A737D}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":121,"searchDepth":145,"depth":145,"links":1551},[1552,1556,1560,1564,1568,1569],{"id":20,"depth":145,"text":21,"children":1553},[1554,1555],{"id":25,"depth":158,"text":26},{"id":79,"depth":158,"text":80},{"id":447,"depth":145,"text":448,"children":1557},[1558,1559],{"id":451,"depth":158,"text":452},{"id":490,"depth":158,"text":491},{"id":594,"depth":145,"text":595,"children":1561},[1562,1563],{"id":598,"depth":158,"text":599},{"id":713,"depth":158,"text":714},{"id":982,"depth":145,"text":983,"children":1565},[1566,1567],{"id":986,"depth":158,"text":987},{"id":1197,"depth":158,"text":1198},{"id":1345,"depth":145,"text":1346},{"id":1474,"depth":145,"text":1475},"md",{},"\u002Fadvanced-query-patterns-and-bulk-data-operations",{"title":5,"description":16},"advanced-query-patterns-and-bulk-data-operations\u002Findex","TZY6wZtoNZq6mgFJhjmISsJTsOUdgfXLhaj49bZNoR8",1778149144399]