Add Multi-Lang KubernetesExecutor system test#68709
Merged
Merged
Conversation
This was referenced Jun 18, 2026
Merged
6a219ca to
5aeaef4
Compare
…Executor Adds a kind-based system test that runs one Dag (lang_sdk_combined) whose tasks span the Python task-runner and the Go/Java coordinators on the KubernetesExecutor. The golang/java queues route to their coordinators, each carrying a pod_template_file (the worktree-1 feature). Each worker pod's init container stages the queue's artifact (Go binary / Java jar) from a per-language localstack S3 bucket through the DagBundle interface into a shared emptyDir that is the coordinator's executables_root/jars_root. The Go queue runs on the prod image (self-contained static binary); the Java queue runs on the CI image (bundles the JVM the JavaCoordinator needs). New k8s-test-only bundles: go-sdk/example/k8s_combined and java-sdk/k8s-example. Provisioning is a new breeze command, `breeze k8s setup-lang-sdk-test`, which builds the artifacts, deploys localstack, uploads artifacts and the Python stub Dag, renders the pod-template image placeholders, and installs the Helm release.
5aeaef4 to
ec67c69
Compare
The lang-SDK Helm values set config.dag_processor.dag_bundle_config_list to only the localstack S3 bundle, which replaced the default dags-folder bundle. The standard KubernetesExecutor example-Dag tests share the same test run, so dropping dags-folder marked their Dags stale and their dagRun triggers returned 404. Configure the bundles additively via dagProcessor.dagBundleConfigList (the chart-recommended input) so both the example Dags and the lang-SDK stub Dag are served.
…tes-tests The Go and Java coordinator example sources for the KubernetesExecutor lang-SDK system test lived under go-sdk/example/k8s_combined and java-sdk/k8s-example, cluttering the SDK source trees. Move them next to the rest of the test assets under kubernetes-tests/lang_sdk/ (go_example, java_example). go_example becomes its own Go module resolving the in-repo go-sdk via a replace directive (it no longer sits inside the go-sdk module); java_example stays a standalone Gradle build that resolves the SDK from mavenLocal, now built with java-sdk's gradle wrapper pointed at the moved project. The breeze provisioning command, README, and stub-Dag docs are updated to the new paths.
This was referenced Jul 1, 2026
1 task
Lee-W
added a commit
to astronomer/airflow
that referenced
this pull request
Jul 7, 2026
vatsrahul1001
pushed a commit
to astronomer/airflow
that referenced
this pull request
Jul 7, 2026
* Add Go + Java + Python lang-SDK coordinator system test on KubernetesExecutor Adds a kind-based system test that runs one Dag (lang_sdk_combined) whose tasks span the Python task-runner and the Go/Java coordinators on the KubernetesExecutor. The golang/java queues route to their coordinators, each carrying a pod_template_file (the worktree-1 feature). Each worker pod's init container stages the queue's artifact (Go binary / Java jar) from a per-language localstack S3 bucket through the DagBundle interface into a shared emptyDir that is the coordinator's executables_root/jars_root. The Go queue runs on the prod image (self-contained static binary); the Java queue runs on the CI image (bundles the JVM the JavaCoordinator needs). New k8s-test-only bundles: go-sdk/example/k8s_combined and java-sdk/k8s-example. Provisioning is a new breeze command, `breeze k8s setup-lang-sdk-test`, which builds the artifacts, deploys localstack, uploads artifacts and the Python stub Dag, renders the pod-template image placeholders, and installs the Helm release. * Wire up lang-sdk test with breeze k8s command * Keep the dags-folder bundle when provisioning the lang-SDK k8s test The lang-SDK Helm values set config.dag_processor.dag_bundle_config_list to only the localstack S3 bundle, which replaced the default dags-folder bundle. The standard KubernetesExecutor example-Dag tests share the same test run, so dropping dags-folder marked their Dags stale and their dagRun triggers returned 404. Configure the bundles additively via dagProcessor.dagBundleConfigList (the chart-recommended input) so both the example Dags and the lang-SDK stub Dag are served. * Move lang-SDK k8s example sources out of go-sdk/java-sdk into kubernetes-tests The Go and Java coordinator example sources for the KubernetesExecutor lang-SDK system test lived under go-sdk/example/k8s_combined and java-sdk/k8s-example, cluttering the SDK source trees. Move them next to the rest of the test assets under kubernetes-tests/lang_sdk/ (go_example, java_example). go_example becomes its own Go module resolving the in-repo go-sdk via a replace directive (it no longer sits inside the go-sdk module); java_example stays a standalone Gradle build that resolves the SDK from mavenLocal, now built with java-sdk's gradle wrapper pointed at the moved project. The breeze provisioning command, README, and stub-Dag docs are updated to the new paths. # Conflicts: # dev/breeze/doc/images/output_setup_check-all-params-in-groups.txt # dev/breeze/doc/images/output_setup_regenerate-command-images.txt
vatsrahul1001
pushed a commit
to astronomer/airflow
that referenced
this pull request
Jul 7, 2026
jason810496
added a commit
to astronomer/airflow
that referenced
this pull request
Jul 7, 2026
* Add Go + Java + Python lang-SDK coordinator system test on KubernetesExecutor Adds a kind-based system test that runs one Dag (lang_sdk_combined) whose tasks span the Python task-runner and the Go/Java coordinators on the KubernetesExecutor. The golang/java queues route to their coordinators, each carrying a pod_template_file (the worktree-1 feature). Each worker pod's init container stages the queue's artifact (Go binary / Java jar) from a per-language localstack S3 bucket through the DagBundle interface into a shared emptyDir that is the coordinator's executables_root/jars_root. The Go queue runs on the prod image (self-contained static binary); the Java queue runs on the CI image (bundles the JVM the JavaCoordinator needs). New k8s-test-only bundles: go-sdk/example/k8s_combined and java-sdk/k8s-example. Provisioning is a new breeze command, `breeze k8s setup-lang-sdk-test`, which builds the artifacts, deploys localstack, uploads artifacts and the Python stub Dag, renders the pod-template image placeholders, and installs the Helm release. * Wire up lang-sdk test with breeze k8s command * Keep the dags-folder bundle when provisioning the lang-SDK k8s test The lang-SDK Helm values set config.dag_processor.dag_bundle_config_list to only the localstack S3 bundle, which replaced the default dags-folder bundle. The standard KubernetesExecutor example-Dag tests share the same test run, so dropping dags-folder marked their Dags stale and their dagRun triggers returned 404. Configure the bundles additively via dagProcessor.dagBundleConfigList (the chart-recommended input) so both the example Dags and the lang-SDK stub Dag are served. * Move lang-SDK k8s example sources out of go-sdk/java-sdk into kubernetes-tests The Go and Java coordinator example sources for the KubernetesExecutor lang-SDK system test lived under go-sdk/example/k8s_combined and java-sdk/k8s-example, cluttering the SDK source trees. Move them next to the rest of the test assets under kubernetes-tests/lang_sdk/ (go_example, java_example). go_example becomes its own Go module resolving the in-repo go-sdk via a replace directive (it no longer sits inside the go-sdk module); java_example stays a standalone Gradle build that resolves the SDK from mavenLocal, now built with java-sdk's gradle wrapper pointed at the moved project. The breeze provisioning command, README, and stub-Dag docs are updated to the new paths. # Conflicts: # dev/breeze/doc/images/output_setup_check-all-params-in-groups.txt # dev/breeze/doc/images/output_setup_regenerate-command-images.txt
jason810496
pushed a commit
to astronomer/airflow
that referenced
this pull request
Jul 7, 2026
Lee-W
added a commit
that referenced
this pull request
Jul 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The Multi-Lang feature works across executors, the best way to verify whether Multi-Lang works across executors or not is adding k8s system test. We use CeleryExecutor for E2E test in docker-compose, the LocalExecutor is used for normal
breeze start-airflowsetup.How
We haven't settle down the Artifact deployment for Multi-Lang (e.g. How does
*.jars and binaries being sync to worker). We intentionally defer this part to users to figure out the most comfortable way for them, then stabilize as best practice.The current k8s test add a small utility script (
stage_artifacts.py) which leveragesS3DagBundlein theinitContainersection to provision the artifacts in thepodthat KubernetesExecutor created. See more details inkubernetes-tests/lang_sdk/README.md.What
kubernetes-tests/lang_sdk/README.mdregarding how does the test setup.Was generative AI tooling used to co-author this PR?