Fix: metrics fix + queue timeout increase - #499
Open
ividito wants to merge 3 commits into
Open
Conversation
Airflow 3 removed the DagModel.is_active column, so the autoscaling loop in put_airflow_worker_autoscaling_metric_data.py raised AttributeError on its first iteration and the metrics ECS service crash-looped (exit 1 every ~100s). The script scales the worker service directly rather than publishing a metric, so both scale-up and scale-down were dead and workers stayed pinned at desired=1. Observed on both sm2a-dev and sm2a-sit. is_stale is the inverse of the old is_active: a DAG no longer present in its bundle is marked stale. Note: only valid on Airflow 3 - do not merge to a branch still on 2.10.5.
SQS made messages visible again after 30s (the AWS default, which the sm2a module leaves unset on aws_sqs_queue). Any task running longer than that was redelivered, and a second worker attempted to start the same task instance. On Airflow 2 the duplicate was tolerated. On Airflow 3 the Task Execution API rejects the second start with `invalid_state`, the Celery executor reports the task as failed, and the scheduler kills the still-healthy original run. Observed on sm2a-dev: vector-eis-fedsoutput-lfarchive/ingest_vector_task failed at the 30s mark while actively processing, having succeeded on 2.10.5 at 11 minutes. Because predefined_queues is configured, kombu does not create the queue, so this setting must be paired with raising the SQS queue's VisibilityTimeout attribute - that is what actually governs redelivery here.
Matches the value applied to the sm2a-dev SQS queue attribute, and lines up with Celery's own documented default. Long enough for the observed worst case (vector ingest at ~11m) with headroom.
ividito
force-pushed
the
fix/airflow3-autoscaling-is-stale
branch
from
August 14, 2026 19:08
bf181c2 to
8302f1c
Compare
ividito
commented
Aug 14, 2026
| # Must exceed the longest task runtime, or SQS redelivers mid-run and the duplicate | ||
| # delivery fails the task on Airflow 3. The queue's own VisibilityTimeout attribute | ||
| # must match, since predefined_queues means kombu does not create the queue. | ||
| VISIBILITY_TIMEOUT = 1800 # 30m; SQS allows up to 43200 |
Contributor
Author
There was a problem hiding this comment.
NASA-IMPACT/self-managed-apache-airflow#40 is what addresses this through Terraform
botanical
reviewed
Aug 14, 2026
| DagModel.is_active.is_(True), | ||
| # Airflow 3 removed the DagModel.is_active column; is_stale is its | ||
| # inverse (a DAG missing from its bundle is marked stale). | ||
| DagModel.is_stale.is_(False), |
Member
There was a problem hiding this comment.
Airflow 3 removed the DagModel.is_active column
Is this true?
https://airflow.apache.org/docs/apache-airflow/3.0.2/_modules/airflow/models/dag.html#DAG.get_is_active
Contributor
Author
There was a problem hiding this comment.
🤔 Not sure where that method gets used, but it looks like it returns return not self.is_stale. DagModel.is_active is definitely gone.
botanical
approved these changes
Aug 14, 2026
botanical
left a comment
botanical
left a comment
Member
There was a problem hiding this comment.
One question but non-blocking
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix a deprecation in the metrics container. Also increase the default queue timeout - this was impacting backfill tasks, which were taking extra time as a result of migrating in the middle of a fires ingest. There will be an upstream PR to align the SQS settings with this new value - NASA-IMPACT/self-managed-apache-airflow#40
The timeout change has already been applied to dev via the console.