Skip to content

Fix asset watcher triggers deleted on every dag-parsing loop#64625

Closed
vatsrahul1001 wants to merge 1 commit into
mainfrom
fix/asset-watcher-trigger-hash-mismatch
Closed

Fix asset watcher triggers deleted on every dag-parsing loop#64625
vatsrahul1001 wants to merge 1 commit into
mainfrom
fix/asset-watcher-trigger-hash-mismatch

Conversation

@vatsrahul1001

Copy link
Copy Markdown
Contributor

Summary

Fix asset watcher triggers being deleted and recreated on every dag-parsing loop.
encode_trigger wraps kwargs with BaseSerialization.serialize() (e.g., tuple() becomes
{"__type": "tuple", "__var": []}), which is correct for JSON-serializable DAG definitions.
However, add_asset_trigger_references needs raw Python kwargs because Trigger._decrypt_kwargs
returns native Python objects from the DB. This mismatch caused BaseEventTrigger.hash() to
produce different hashes for the same logical trigger, resulting in an infinite delete/recreate
cycle every parsing loop.
Replace encode_trigger with _get_raw_trigger_kwargs that unwraps any BaseSerialization-encoded
values back to native Python objects, ensuring hash consistency between DAG-defined and DB-stored
triggers. encode_trigger remains in use where it belongs — DAG serialization for JSON storage.

  • Yes (please specify the tool below)
    Claude
  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

Comment thread airflow-core/src/airflow/dag_processing/collection.py Outdated
@TJaniF

TJaniF commented Apr 2, 2026

Copy link
Copy Markdown
Contributor

Tested with a local Kafka cluster and works great :)

__init__ py — airflow 2026-04-02 at 2 50 30 PM Airflow 2026-04-02 at 2 50 20 PM

Comment thread airflow-core/src/airflow/dag_processing/collection.py Outdated
@vatsrahul1001 vatsrahul1001 force-pushed the fix/asset-watcher-trigger-hash-mismatch branch from 12d8ca2 to 55960a6 Compare April 2, 2026 13:12
@vatsrahul1001 vatsrahul1001 requested a review from uranusjr April 2, 2026 13:16

@phanikumv phanikumv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add tests !

Comment thread airflow-core/src/airflow/dag_processing/collection.py Outdated
Comment thread airflow-core/src/airflow/dag_processing/collection.py Outdated
Comment thread airflow-core/src/airflow/dag_processing/collection.py

@kaxil kaxil left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left few comments but lgtm directionally but will defer to @uranusjr and @Lee-W for 2nd pass

@vatsrahul1001 vatsrahul1001 force-pushed the fix/asset-watcher-trigger-hash-mismatch branch from 6ac4d7a to 3f10aad Compare April 2, 2026 18:00
@vatsrahul1001 vatsrahul1001 marked this pull request as draft April 2, 2026 20:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants