Skip to content

Fix common.ai example DAGs failing to parse without the sql extra#68497

Merged
kaxil merged 7 commits into
apache:mainfrom
ehab20011:fix/common-ai-example-dags-optional-sqlglot-import
Jun 16, 2026
Merged

Fix common.ai example DAGs failing to parse without the sql extra#68497
kaxil merged 7 commits into
apache:mainfrom
ehab20011:fix/common-ai-example-dags-optional-sqlglot-import

Conversation

@ehab20011

@ehab20011 ehab20011 commented Jun 13, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Fix common.ai example DAGs failing to parse without the [sql] extra

Seven example DAGs in providers/common/ai fail to parse when sqlglot is not installed (i.e. the optional [sql] extra is not present). The DAG processor marks them as import errors, even though sqlglot is intentionally an optional dependency.

Problem

SQLToolset and LLMSQLQueryOperator raise AirflowOptionalProviderFeatureException at import time when sqlglot is missing. The example DAGs import these at the module level, which causes the entire DAG file to fail during parsing. The DAG processor catches this with a generic except BaseException and marks the file as an import error — there is no special handling for AirflowOptionalProviderFeatureException.

Running uv sync (the documented setup) does not install the [sql] extra, so all seven DAGs error on a fresh dev environment.

Fix

Wrap the optional imports in try/except Exception with a None fallback, and guard the DAG definitions that use the optional class with if X is not None:. DAGs in the same file that don't use the optional dependency remain unguarded and always load.

This matches the established pattern already used by two DAGs in the same provider (example_langchain_toolset_bridge.py and example_pydantic_ai_hook.py), as well as other providers in the codebase (e.g. amazon/athena_sql.py, amazon/redshift_sql.py).

Files changed

File DAGs guarded DAGs unguarded
example_agent.py 4 (SQL-dependent) 2 (HookToolset, HITL review)
example_agent_capabilities.py 1 (composed) 2 (Thinking, WebSearch)
example_agent_durable.py 2 (both use SQLToolset) 0
example_agent_skills.py 1 (local skills) 1 (git skills)
example_llm_sql.py 6 (all use LLMSQLQueryOperator) 0
example_llm_survey_analysis.py 2 (both use LLMSQLQueryOperator) 0
example_llm_survey_agentic.py 1 (uses LLMSQLQueryOperator) 0

Before (7 DAG import errors)

Screenshot 2026-06-13 at 1 48 51 AM

After (0 DAG import errors)

Screenshot 2026-06-13 at 1 47 23 AM
Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Anthropic)

Generated-by: Claude Code following the guidelines

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@boring-cyborg

boring-cyborg Bot commented Jun 13, 2026

Copy link
Copy Markdown

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
Here are some useful points:

  • Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example Dag that shows how users should use it.
  • Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
  • Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: dev@airflow.apache.org
    Slack: https://s.apache.org/airflow-slack

@ehab20011 ehab20011 marked this pull request as draft June 13, 2026 05:53
@ehab20011 ehab20011 marked this pull request as ready for review June 13, 2026 05:57
@ehab20011 ehab20011 force-pushed the fix/common-ai-example-dags-optional-sqlglot-import branch from 67d1f55 to be1ca49 Compare June 13, 2026 05:59
kaxil
kaxil requested changes Jun 13, 2026
View reviewed changes

@kaxil kaxil left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The intention of examples Dags is to show how something can be use and easy copy/paste-able and not to be able to run with all
Combination of optionals deps

ehab20011
ehab20011 commented Jun 14, 2026
View reviewed changes

@ehab20011 ehab20011 left a comment •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @kaxil , thanks for the review. I understand you guys wanting to keep examples simple and copy-paste-able.

Just two things I wanted to flag though:

  1. Two example Dags in the same provider already use this exact guard pattern (example_langchain_toolset_bridge.py and example_pydantic_ai_hook.py) both do try/except with None fallback + if X is not None: guards. My PR just follows the same convention. If this pattern isn't the right approach, should those two be updated as well? I would be happy to undo their guards

  2. If the guards aren't the way to go, what would you suggest instead? Running uv sync (the documented dev setup) doesn't install the [sql] extra, so all 7 Dags show up as import errors out of the box. Maybe take a different approach like excluding them from parsing when the extra isn't installed, or adding the extra to the dev dependencies? whatever you think makes sense but it's just something that I noticed when setting up my environment that was a clear error and eye-sore to any developer working on this repository. cc @jroachgolf84

@ehab20011 ehab20011 requested a review from kaxil June 14, 2026 15:05
@kaxil

kaxil commented Jun 16, 2026 •
edited
Loading

Copy link
Copy Markdown
Member
  1. Two example Dags in the same provider already use this exact guard pattern (example_langchain_toolset_bridge.py and example_pydantic_ai_hook.py) both do try/except with None fallback + if X is not None: guards. My PR just follows the same convention. If this pattern isn't the right approach, should those two be updated as well? I would be happy to undo their guards

I don't see a try..except on either of those files:

@kaxil

kaxil commented Jun 16, 2026

Copy link
Copy Markdown
Member

Looks like it was a recent merge: #66161 that loads example dags from all providers.

We can probably do something better with the example dags by raising AirflowOptionalProviderFeatureException or something but that can be a follow-up

@kaxil kaxil merged commit 58480dd into apache:main Jun 16, 2026
81 checks passed
@boring-cyborg

boring-cyborg Bot commented Jun 16, 2026

Copy link
Copy Markdown

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kaxil kaxil kaxil approved these changes
@gopidesupavan gopidesupavan Awaiting requested review from gopidesupavan gopidesupavan is a code owner

Assignees

No one assigned

Labels

area:providers provider:common-ai

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@ehab20011 @kaxil