Merge branch 'main' of https://github.com/apache/airflow into feature/remove-large-in-clause-in-assets

Author: Natanel Rudyuklakir

.github/CODEOWNERS

@@ -149,3 +149,6 @@ Dockerfile.ci @potiuk @ashb @gopidesupavan @amoghrajesh @jscheffl @bugraoz93 @ka
 
 # Shared Libraries
 /shared/ @ashb @amoghrajesh @potiuk
+
+# RMs on release documents
+/dev/README_RELEASE_*.md @potiuk @jscheffl @vincbeck @shahar1 @jedcunningham @bugraoz93

.github/SECURITY.md

@@ -4,16 +4,19 @@
 <!-- START doctoc generated TOC please keep comment here to allow auto update -->
 <!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
 
-- [What should be and should NOT be reported ?](#what-should-be-and-should-not-be-reported-)
-- [How to report the issue ?](#how-to-report-the-issue-)
-- [Is this really a security vulnerability ?](#is-this-really-a-security-vulnerability-)
-- [How do we assess severity of the issue ?](#how-do-we-assess-severity-of-the-issue-)
-- [What happens after you report the issue ?](#what-happens-after-you-report-the-issue-)
-- [Does CVE in Airflow Providers impact Airflow core package ?](#does-cve-in-airflow-providers-impact-airflow-core-package-)
-- [Where do I find more information about Airflow Security ?](#where-do-i-find-more-information-about-airflow-security-)
+- [Apache Airflow Security](#apache-airflow-security)
+  - [What should be and should NOT be reported ?](#what-should-be-and-should-not-be-reported-)
+  - [How to report the issue ?](#how-to-report-the-issue-)
+  - [Is this really a security vulnerability ?](#is-this-really-a-security-vulnerability-)
+  - [How do we assess severity of the issue ?](#how-do-we-assess-severity-of-the-issue-)
+  - [What happens after you report the issue ?](#what-happens-after-you-report-the-issue-)
+  - [Does CVE in Airflow Providers impact Airflow core package ?](#does-cve-in-airflow-providers-impact-airflow-core-package-)
+  - [Where do I find more information about Airflow Security ?](#where-do-i-find-more-information-about-airflow-security-)
 
 <!-- END doctoc generated TOC please keep comment here to allow auto update -->
 
+# Apache Airflow Security
+
 This document contains information on how to report security vulnerabilities in Apache Airflow and
 how security issues reported to the Apache Airflow security team are handled. If you would like
 to learn more, head to the
@@ -30,7 +33,7 @@ e-mail address [security@airflow.apache.org](mailto:security@airflow.apache.org)
 Before sending the report, however, please read the following guidelines first. The guidelines should
 answer the most common questions you might have about reporting vulnerabilities.
 
-### What should be and should NOT be reported ?
+## What should be and should NOT be reported ?
 
 **Only** use the security e-mail address to report undisclosed security vulnerabilities in Apache
 Airflow and to manage the process of fixing such vulnerabilities. We do not accept regular
@@ -45,13 +48,13 @@ with dependencies in Airflow Docker reference image - there is a page that descr
 [Airflow reference Image is fixed at release time](https://airflow.apache.org/docs/docker-stack/index.html#fixing-images-at-release-time) and providing helpful instructions explaining
 how you can build your own image and manage dependencies of Airflow in your own image.
 
-### How to report the issue ?
+## How to report the issue ?
 
 Please send one plain-text email for each vulnerability you are reporting including an explanation
 of how it affects Airflow security. We may ask that you resubmit your report if you send it as an image,
 movie, HTML, or PDF attachment when you could as easily describe it with plain text.
 
-### Is this really a security vulnerability ?
+## Is this really a security vulnerability ?
 
 Before reporting vulnerabilities, please make sure to read and understand the [security model](https://airflow.apache.org/docs/apache-airflow/stable/security/security_model.html)
 of Airflow, because some of the potential security vulnerabilities that are valid for projects that are
@@ -69,7 +72,7 @@ a lot of time on preparing the issue report to follow the guidelines above and w
 save time for yourself and for the Airflow Security team by reading and understanding the security model
 before reporting the issue.
 
-### How do we assess severity of the issue ?
+## How do we assess severity of the issue ?
 
 Severity of the issue is determined based on the criteria described in
 the [Severity Rating blog post](https://security.apache.org/blog/severityrating/) by the Apache Software Foundation Security team.
@@ -79,7 +82,7 @@ do not apply to Airflow, or have a different severity than some generic scoring
 (for example `CVSS`) calculation suggests. So we are not using any generic scoring system.
 
 
-### What happens after you report the issue ?
+## What happens after you report the issue ?
 
 The Airflow Security Team will get back to you after assessing the report. You will usually get
 confirmation that the issue is being worked (or that we quickly assessed it as invalid) within several
@@ -100,7 +103,7 @@ Security issues in Airflow are handled by the Airflow Security Team. Details abo
 Team and how members of it are chosen can be found in the
 [Contributing documentation](https://github.com/apache/airflow/blob/main/contributing-docs/01_roles_in_airflow_project.rst#security-team).
 
-### Does CVE in Airflow Providers impact Airflow core package ?
+## Does CVE in Airflow Providers impact Airflow core package ?
 
 Airflow core package is released separately from provider distributions. While Airflow comes with ``constraints``
 which describe which version of providers have been tested when the version of Airflow was released, the
@@ -110,7 +113,7 @@ not apply to the Airflow core package. There are also Airflow providers released
 Airflow community is not responsible for releasing and announcing security vulnerabilities in them, this
 is handled entirely by the 3rd-parties that release their own providers.
 
-### Where do I find more information about Airflow Security ?
+## Where do I find more information about Airflow Security ?
 
 If you wish to know more about the ASF security process,
 the [ASF Security team's page](https://www.apache.org/security/) describes

.github/instructions/code-review.instructions.md

@@ -0,0 +1,69 @@
+---
+applyTo: "**"
+excludeAgent: "coding-agent"
+---
+
+# Airflow Code Review Instructions
+
+Use these rules when reviewing pull requests to the Apache Airflow repository.
+
+## Architecture Boundaries
+
+- **Scheduler must never run user code.** It only processes serialized Dags. Flag any scheduler-path code that deserializes or executes Dag/task code.
+- **Workers must not access the metadata DB directly.** Task execution communicates with the API server through the Execution API (`/execution` endpoints) only.
+- **Dag Processor and Triggerer run user code in isolated processes.** Code in these components should maintain that isolation.
+- **Providers must not import core internals** like `SUPERVISOR_COMMS` or task-runner plumbing. Providers interact through the public SDK and execution API only.
+
+## Database and Query Correctness
+
+- **N+1 queries**: Flag SQLAlchemy queries that access relationships inside loops without `joinedload()` or `selectinload()`.
+- **`run_id` is only unique per Dag.** Queries that group, partition, or join on `run_id` alone (without `dag_id`) will collide across Dags. Always require `(dag_id, run_id)` together.
+- **Cross-database compatibility**: SQL changes must work on PostgreSQL, MySQL, and SQLite. Flag database-specific features (lateral joins, window functions) without cross-DB handling.
+- **Session discipline**: In `airflow-core`, functions receiving a `session` parameter must not call `session.commit()`. Use keyword-only `session` parameters.
+
+## Code Quality Rules
+
+- No `assert` in production code (stripped in optimized Python).
+- `time.monotonic()` for durations, not `time.time()`.
+- Imports at top of file. Valid exceptions: circular imports, lazy loading for worker isolation, `TYPE_CHECKING` blocks.
+- Guard heavy type-only imports (e.g., `kubernetes.client`) with `TYPE_CHECKING` in multi-process code paths.
+- Unbounded caches are bugs: all `@lru_cache` must have `maxsize`.
+- Resources (files, connections, sessions) must use context managers or `try/finally`.
+
+## Testing Requirements
+
+- New behavior requires tests covering success, failure, and edge cases.
+- Use pytest patterns, not `unittest.TestCase`.
+- Use `spec`/`autospec` when mocking.
+- Use `time_machine` for time-dependent tests.
+- Imports belong at the top of test files, not inside test functions.
+- Issue numbers do not belong in test docstrings.
+
+## API Correctness
+
+- `map_index` must be handled correctly for mapped tasks. Queries without `map_index` filtering may return arbitrary task instances.
+- Execution API changes must follow Cadwyn versioning (CalVer format).
+
+## UI Code (React/TypeScript)
+
+- Avoid `useState + useEffect` to sync derived state. Use nullish coalescing or nullable override patterns instead.
+- Extract shared logic into custom hooks rather than copy-pasting across components.
+
+## AI-Generated Code Signals
+
+Flag these patterns that indicate low-quality AI-generated contributions:
+
+- **Fabricated diffs**: Changes to files or code paths that don't exist in the repository.
+- **Unrelated files included**: Changes to files that have nothing to do with the stated purpose of the PR.
+- **Description doesn't match code**: PR description describes something different from what the code actually does.
+- **No evidence of testing**: Claims of fixes without test evidence, or author admitting they cannot run the test suite.
+- **Over-engineered solutions**: Adding caching layers, complex locking, or benchmark scripts for problems that don't exist or are misunderstood.
+- **Narrating comments**: Comments that restate what the next line does (e.g., `# Add the item to the list` before `list.append(item)`).
+- **Empty PR descriptions**: PRs with just the template filled in and no actual description of the changes.
+
+## Quality Signals to Check
+
+The absence of these signals in a "fix" or "optimization" PR is itself a red flag:
+
+- **Bug fixes need regression tests**: A test that fails without the fix and passes with it.
+- **Existing tests must still pass without modification**: If existing tests need changes to pass, the PR may introduce a behavioral regression.

.github/skills/airflow-translations/locales/ko.md

@@ -0,0 +1,113 @@
+<!-- SPDX-License-Identifier: Apache-2.0
+     https://www.apache.org/licenses/LICENSE-2.0 -->
+
+# Korean (ko)
+
+This document provides locale-specific instructions for translating English Airflow UI strings into Korean. It inherits all global rules from the parent [SKILL.md](../SKILL.md).
+
+## Translation Style
+
+Use wording already established in existing `ko` locale files first. If a term has no established translation yet, prefer natural Korean UI phrasing over literal transliteration.
+
+**English source:**
+
+```json
+"lastDagRun_one": "Last Dag Run",
+"deleteConnection_other": "Delete {{count}} connections"
+```
+
+**Correct** — natural Korean UI wording:
+
+```json
+"lastDagRun_one": "마지막 Dag 실행",
+"deleteConnection_other": "커넥션 {{count}}개 삭제"
+```
+
+**Incorrect** — overly literal or awkward:
+
+```json
+"lastDagRun_one": "마지막 Dag 런",
+"deleteConnection_other": "{{count}} 연결들을 삭제"
+```
+
+## Plural Forms
+
+Korean often uses the same wording for singular and plural. Follow established usage in existing `ko` locale files first. If an existing key pair already distinguishes `_one` and `_other`, keep that distinction. If no established wording exists, use the same translation for both.
+
+**English source:**
+
+```json
+"taskCount_one": "{{count}} Task",
+"taskCount_other": "{{count}} Tasks"
+```
+
+**Correct** — identical for both when no established distinction exists:
+
+```json
+"taskCount_one": "{{count}}개 작업",
+"taskCount_other": "{{count}}개 작업"
+```
+
+## Counters and Spacing
+
+Use counters consistent with existing `ko` locale usage and keep spacing readable:
+
+- Insert a single space between Korean and adjacent English technical terms where needed (`Dag 실행`, `커넥션 ID`).
+- Do not insert a space between numbers/placeholders and counters such as `개` (for example, `{{count}}개`).
+
+```json
+"deleteConnection_other": "커넥션 {{count}}개 삭제",
+"taskCount_one": "{{count}}개 작업",
+"taskCount_other": "{{count}}개 작업",
+"lastDagRun_one": "마지막 Dag 실행",
+"connectionId": "커넥션 ID"
+```
+
+## Particles and Placeholders
+
+Preserve all `{{variable}}` placeholders exactly. Attach Korean particles outside placeholders and reorder phrases only when needed for natural Korean word order.
+
+**English source:**
+
+```json
+"confirmation": "Are you sure you want to delete {{resourceName}}? This action cannot be undone.",
+"description": "{{count}} {{resourceName}} have been successfully deleted. Keys: {{keys}}"
+```
+
+**Correct** — placeholders preserved and particles outside:
+
+```json
+"confirmation": "{{resourceName}}을(를) 삭제하시겠습니까? 이 작업은 되돌릴 수 없습니다.",
+"description": "{{count}}개 {{resourceName}}이(가) 성공적으로 삭제되었습니다. 키: {{keys}}"
+```
+
+**Incorrect** — variable names translated:
+
+```json
+"confirmation": "{{리소스이름}}을(를) 삭제하시겠습니까? 이 작업은 되돌릴 수 없습니다.",
+"description": "{{개수}}개 {{리소스이름}}이(가) 성공적으로 삭제되었습니다. 키: {{키들}}"
+```
+
+## Tone and UI Voice
+
+- Use neutral, slightly formal tone.
+- Keep labels and messages concise for UI.
+- Use polite confirmations in destructive actions:
+  `"{{resourceName}}을(를) 삭제하시겠습니까? 이 작업은 되돌릴 수 없습니다."`
+- Avoid colloquial phrasing.
+
+## Terminology and Casing
+
+- Keep `Dag` casing exactly as `Dag` (never `DAG`).
+- Reuse established Korean role terms from existing `ko` locale files (for example, `스케줄러`, `오퍼레이터`).
+- Prefer established `ko` glossary by key context (for example, `Dag 실행`, `커넥션`, `변수`), and keep stable technical tokens in English: `XCom`, `REST API`, `JSON`, `URL`, `ID`, `UTC`.
+
+## Terminology Reference
+
+The established Korean translations are defined in existing locale files. Before translating, **read the existing ko JSON files** to learn the established terminology:
+
+```
+airflow-core/src/airflow/ui/public/i18n/locales/ko/
+```
+
+Use the translations found in these files as the authoritative glossary. When translating a term, check how it has been translated elsewhere in the locale to maintain consistency. If a term has not been translated yet, refer to the English source in `en/` and apply the rules in this document.

.github/workflows/airflow-distributions-tests.yml

@@ -112,9 +112,10 @@ jobs:
         env:
           DISTRIBUTION_TYPE: "${{ inputs.distribution-cmd-format }}"
           USE_LOCAL_HATCH: "${{ inputs.use-local-venv }}"
+        # Pin on virtualenv is temporary for pypa/hatch#2193
         run: |
           uv tool uninstall hatch || true
-          uv tool install hatch==1.16.4
+          uv tool install hatch==1.16.4 --with 'virtualenv<21'
           breeze release-management "${DISTRIBUTION_TYPE}" --distribution-format wheel
         if: ${{ matrix.python-version == inputs.default-python-version }}
       - name: "Verify wheel packages with twine"

.pre-commit-config.yaml

@@ -39,7 +39,7 @@ repos:
         language: python
         language_version: python311
   - repo: https://github.com/thlorenz/doctoc.git
-    rev: 68f070c98b9a053eabfa7f8899d1f42b9919f98c  # frozen: v2.2.0
+    rev: d7815f1f950f8d5ec933fa4f70208bf316bb13f8  # frozen: v2.3.0
     hooks:
       - id: doctoc
         name: Add TOC for Markdown and RST files
@@ -56,6 +56,7 @@ repos:
         exclude:
             (?x)
             .github/PULL_REQUEST_TEMPLATE\.md$|
+            .github/instructions/|
             .github/skills/
         args:
           - "--maxlevel"
@@ -194,6 +195,7 @@ repos:
         exclude:
           (?x)
           ^scripts/ci/license-templates/|
+          ^\.github/instructions/|
           ^\.github/skills/airflow-translations/SKILL\.md$
       - id: insert-license
         name: Add license for all other files
@@ -625,6 +627,7 @@ repos:
           ^providers/apache/hive/src/airflow/providers/apache/hive/transfers/vertica_to_hive\.py$|
           ^providers/apache/kafka/docs/connections/kafka\.rst$|
           ^providers/apache/spark/docs/decorators/pyspark\.rst$|
+          ^providers/apache/spark/docs/connections/spark-submit.rst$|
           ^providers/apache/spark/src/airflow/providers/apache/spark/decorators/|
           ^providers/apache/spark/src/airflow/providers/apache/spark/hooks/|
           ^providers/apache/spark/src/airflow/providers/apache/spark/operators/|