Skip to content

[v3-3-test] Add Multi-Lang KubernetesExecutor system test (#68709)#69532

Merged
Lee-W merged 2 commits into
apache:v3-3-testfrom
astronomer:backport-f86b03c-to-v3-3-test
Jul 7, 2026
Merged

[v3-3-test] Add Multi-Lang KubernetesExecutor system test (#68709)#69532
Lee-W merged 2 commits into
apache:v3-3-testfrom
astronomer:backport-f86b03c-to-v3-3-test

Conversation

@Lee-W

@Lee-W Lee-W commented Jul 7, 2026

Copy link
Copy Markdown
Member
  • Add Go + Java + Python lang-SDK coordinator system test on KubernetesExecutor

Adds a kind-based system test that runs one Dag (lang_sdk_combined) whose tasks span the Python task-runner and the Go/Java coordinators on the KubernetesExecutor. The golang/java queues route to their coordinators, each carrying a pod_template_file (the worktree-1 feature). Each worker pod's init container stages the queue's artifact (Go binary / Java jar) from a per-language localstack S3 bucket through the DagBundle interface into a shared emptyDir that is the coordinator's executables_root/jars_root. The Go queue runs on the prod image (self-contained static binary); the Java queue runs on the CI image (bundles the JVM the JavaCoordinator needs).

New k8s-test-only bundles: go-sdk/example/k8s_combined and java-sdk/k8s-example. Provisioning is a new breeze command, breeze k8s setup-lang-sdk-test, which builds the artifacts, deploys localstack, uploads artifacts and the Python stub Dag, renders the pod-template image placeholders, and installs the Helm release.

  • Wire up lang-sdk test with breeze k8s command

  • Keep the dags-folder bundle when provisioning the lang-SDK k8s test

The lang-SDK Helm values set config.dag_processor.dag_bundle_config_list to only the localstack S3 bundle, which replaced the default dags-folder bundle. The standard KubernetesExecutor example-Dag tests share the same test run, so dropping dags-folder marked their Dags stale and their dagRun triggers returned
404. Configure the bundles additively via dagProcessor.dagBundleConfigList (the chart-recommended input) so both the example Dags and the lang-SDK stub Dag are served.

  • Move lang-SDK k8s example sources out of go-sdk/java-sdk into kubernetes-tests

The Go and Java coordinator example sources for the KubernetesExecutor lang-SDK system test lived under go-sdk/example/k8s_combined and java-sdk/k8s-example, cluttering the SDK source trees. Move them next to the rest of the test assets under kubernetes-tests/lang_sdk/ (go_example, java_example).

go_example becomes its own Go module resolving the in-repo go-sdk via a replace directive (it no longer sits inside the go-sdk module); java_example stays a standalone Gradle build that resolves the SDK from mavenLocal, now built with java-sdk's gradle wrapper pointed at the moved project. The breeze provisioning command, README, and stub-Dag docs are updated to the new paths. # Conflicts:

dev/breeze/doc/images/output_setup_check-all-params-in-groups.txt # dev/breeze/doc/images/output_setup_regenerate-command-images.txt


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@jason810496 jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just FYI for the further reviewers that we need to wait for #69536 get merged first before merging this one.

jason810496 and others added 2 commits July 7, 2026 21:08
* Add Go + Java + Python lang-SDK coordinator system test on KubernetesExecutor

Adds a kind-based system test that runs one Dag (lang_sdk_combined) whose
tasks span the Python task-runner and the Go/Java coordinators on the
KubernetesExecutor. The golang/java queues route to their coordinators,
each carrying a pod_template_file (the worktree-1 feature). Each worker
pod's init container stages the queue's artifact (Go binary / Java jar)
from a per-language localstack S3 bucket through the DagBundle interface
into a shared emptyDir that is the coordinator's executables_root/jars_root.
The Go queue runs on the prod image (self-contained static binary); the
Java queue runs on the CI image (bundles the JVM the JavaCoordinator needs).

New k8s-test-only bundles: go-sdk/example/k8s_combined and
java-sdk/k8s-example. Provisioning is a new breeze command,
`breeze k8s setup-lang-sdk-test`, which builds the artifacts, deploys
localstack, uploads artifacts and the Python stub Dag, renders the
pod-template image placeholders, and installs the Helm release.

* Wire up lang-sdk test with breeze k8s command

* Keep the dags-folder bundle when provisioning the lang-SDK k8s test

The lang-SDK Helm values set config.dag_processor.dag_bundle_config_list to only
the localstack S3 bundle, which replaced the default dags-folder bundle. The
standard KubernetesExecutor example-Dag tests share the same test run, so
dropping dags-folder marked their Dags stale and their dagRun triggers returned
404. Configure the bundles additively via dagProcessor.dagBundleConfigList (the
chart-recommended input) so both the example Dags and the lang-SDK stub Dag are
served.

* Move lang-SDK k8s example sources out of go-sdk/java-sdk into kubernetes-tests

The Go and Java coordinator example sources for the KubernetesExecutor lang-SDK
system test lived under go-sdk/example/k8s_combined and java-sdk/k8s-example,
cluttering the SDK source trees. Move them next to the rest of the test assets
under kubernetes-tests/lang_sdk/ (go_example, java_example).

go_example becomes its own Go module resolving the in-repo go-sdk via a replace
directive (it no longer sits inside the go-sdk module); java_example stays a
standalone Gradle build that resolves the SDK from mavenLocal, now built with
java-sdk's gradle wrapper pointed at the moved project. The breeze provisioning
command, README, and stub-Dag docs are updated to the new paths.
# Conflicts:
#	dev/breeze/doc/images/output_setup_check-all-params-in-groups.txt
#	dev/breeze/doc/images/output_setup_regenerate-command-images.txt
@jason810496 jason810496 force-pushed the backport-f86b03c-to-v3-3-test branch from a06eaee to 59280d9 Compare July 7, 2026 12:11

@jason810496 jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#69536 was merged and I just rebase the PR. The k8s test should pass this round 🤞

@Lee-W

Lee-W commented Jul 7, 2026

Copy link
Copy Markdown
Member Author

green!

@Lee-W Lee-W merged commit 0cc3c7e into apache:v3-3-test Jul 7, 2026
119 checks passed
@Lee-W Lee-W deleted the backport-f86b03c-to-v3-3-test branch July 7, 2026 13:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants