[v3-3-test] Add Multi-Lang KubernetesExecutor system test (#68709)#69532
Merged
Conversation
jason810496
approved these changes
Jul 7, 2026
c36f6ce to
a06eaee
Compare
1 task
potiuk
approved these changes
Jul 7, 2026
jason810496
reviewed
Jul 7, 2026
Member
There was a problem hiding this comment.
Just FYI for the further reviewers that we need to wait for #69536 get merged first before merging this one.
* Add Go + Java + Python lang-SDK coordinator system test on KubernetesExecutor Adds a kind-based system test that runs one Dag (lang_sdk_combined) whose tasks span the Python task-runner and the Go/Java coordinators on the KubernetesExecutor. The golang/java queues route to their coordinators, each carrying a pod_template_file (the worktree-1 feature). Each worker pod's init container stages the queue's artifact (Go binary / Java jar) from a per-language localstack S3 bucket through the DagBundle interface into a shared emptyDir that is the coordinator's executables_root/jars_root. The Go queue runs on the prod image (self-contained static binary); the Java queue runs on the CI image (bundles the JVM the JavaCoordinator needs). New k8s-test-only bundles: go-sdk/example/k8s_combined and java-sdk/k8s-example. Provisioning is a new breeze command, `breeze k8s setup-lang-sdk-test`, which builds the artifacts, deploys localstack, uploads artifacts and the Python stub Dag, renders the pod-template image placeholders, and installs the Helm release. * Wire up lang-sdk test with breeze k8s command * Keep the dags-folder bundle when provisioning the lang-SDK k8s test The lang-SDK Helm values set config.dag_processor.dag_bundle_config_list to only the localstack S3 bundle, which replaced the default dags-folder bundle. The standard KubernetesExecutor example-Dag tests share the same test run, so dropping dags-folder marked their Dags stale and their dagRun triggers returned 404. Configure the bundles additively via dagProcessor.dagBundleConfigList (the chart-recommended input) so both the example Dags and the lang-SDK stub Dag are served. * Move lang-SDK k8s example sources out of go-sdk/java-sdk into kubernetes-tests The Go and Java coordinator example sources for the KubernetesExecutor lang-SDK system test lived under go-sdk/example/k8s_combined and java-sdk/k8s-example, cluttering the SDK source trees. Move them next to the rest of the test assets under kubernetes-tests/lang_sdk/ (go_example, java_example). go_example becomes its own Go module resolving the in-repo go-sdk via a replace directive (it no longer sits inside the go-sdk module); java_example stays a standalone Gradle build that resolves the SDK from mavenLocal, now built with java-sdk's gradle wrapper pointed at the moved project. The breeze provisioning command, README, and stub-Dag docs are updated to the new paths. # Conflicts: # dev/breeze/doc/images/output_setup_check-all-params-in-groups.txt # dev/breeze/doc/images/output_setup_regenerate-command-images.txt
a06eaee to
59280d9
Compare
jason810496
reviewed
Jul 7, 2026
jason810496
left a comment
Member
There was a problem hiding this comment.
#69536 was merged and I just rebase the PR. The k8s test should pass this round 🤞
Member
Author
|
green! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a kind-based system test that runs one Dag (lang_sdk_combined) whose tasks span the Python task-runner and the Go/Java coordinators on the KubernetesExecutor. The golang/java queues route to their coordinators, each carrying a pod_template_file (the worktree-1 feature). Each worker pod's init container stages the queue's artifact (Go binary / Java jar) from a per-language localstack S3 bucket through the DagBundle interface into a shared emptyDir that is the coordinator's executables_root/jars_root. The Go queue runs on the prod image (self-contained static binary); the Java queue runs on the CI image (bundles the JVM the JavaCoordinator needs).
New k8s-test-only bundles: go-sdk/example/k8s_combined and java-sdk/k8s-example. Provisioning is a new breeze command,
breeze k8s setup-lang-sdk-test, which builds the artifacts, deploys localstack, uploads artifacts and the Python stub Dag, renders the pod-template image placeholders, and installs the Helm release.Wire up lang-sdk test with breeze k8s command
Keep the dags-folder bundle when provisioning the lang-SDK k8s test
The lang-SDK Helm values set config.dag_processor.dag_bundle_config_list to only the localstack S3 bundle, which replaced the default dags-folder bundle. The standard KubernetesExecutor example-Dag tests share the same test run, so dropping dags-folder marked their Dags stale and their dagRun triggers returned
404. Configure the bundles additively via dagProcessor.dagBundleConfigList (the chart-recommended input) so both the example Dags and the lang-SDK stub Dag are served.
The Go and Java coordinator example sources for the KubernetesExecutor lang-SDK system test lived under go-sdk/example/k8s_combined and java-sdk/k8s-example, cluttering the SDK source trees. Move them next to the rest of the test assets under kubernetes-tests/lang_sdk/ (go_example, java_example).
go_example becomes its own Go module resolving the in-repo go-sdk via a replace directive (it no longer sits inside the go-sdk module); java_example stays a standalone Gradle build that resolves the SDK from mavenLocal, now built with java-sdk's gradle wrapper pointed at the moved project. The breeze provisioning command, README, and stub-Dag docs are updated to the new paths. # Conflicts:
dev/breeze/doc/images/output_setup_check-all-params-in-groups.txt # dev/breeze/doc/images/output_setup_regenerate-command-images.txt
Was generative AI tooling used to co-author this PR?
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.