Speed up lang-SDK k8s test by building Go/Java natively in CI#69411
Merged
potiuk merged 2 commits intoJul 5, 2026
Merged
Conversation
The Multi-Lang KubernetesExecutor system test builds a Go bundle and a Java jar inside throwaway toolchain containers. On a dev host the module, Gradle-distribution and dependency caches persist between runs, but CI runners are ephemeral and nothing cached them, so every scheduled run re-pulled the golang/temurin images and re-downloaded the Gradle distribution and all dependencies — adding roughly ten minutes to each of the six KubernetesExecutor variants that run the test. Build the artifacts with the host toolchain in CI instead, provisioned and cached through actions/setup-go and actions/setup-java, so the toolchain image pulls and cold dependency downloads no longer happen on every run. Local runs keep the containerised build so a dev host still needs neither Go nor a JDK installed.
Follow-ups to the native lang-SDK build: - Move the JDK version (17) into a JAVA_SDK_VERSION breeze constant and surface it as the java-sdk-version selective-checks / build-info output, so the k8s workflow reads it instead of hardcoding the version in YAML. - Restore the Go module/build and Gradle caches with an explicit actions/cache keyed on a "-v1-" salt rather than the setup-* built-in caching, giving an in-repo knob to force-invalidate a poisoned cache (bump the salt) without waiting for a dependency change to rotate it.
jason810496
approved these changes
Jul 5, 2026
jason810496
left a comment
Member
There was a problem hiding this comment.
I like the idea of having "native" host setup for CI to speed up the process! Thanks a lot.
Member
Author
Yeah - while not entirely reproducible, we have the docker run that a) caches nicely for local use b) is reproducible, so we are good with native builds on CI - cache on GitHub is very fast. |
Member
Author
|
Looks good - Back to 13-15 m from 25! |
Contributor
Backport failed to create: v3-3-test. View the failure log Run detailsNote: As of Merging PRs targeted for Airflow 3.X In matter of doubt please ask in #release-management Slack channel.
You can attempt to backport this manually by running: cherry_picker 673183b v3-3-testThis should apply the commit to the v3-3-test branch and leave the commit in conflict state marking After you have resolved the conflicts, you can continue the backport process by running: cherry_picker --continueIf you don't have cherry-picker installed, see the installation guide. |
1 task
jason810496
added a commit
that referenced
this pull request
Jul 8, 2026
…n CI (#69411) (#69580) * Speed up lang-SDK k8s test by building Go/Java natively in CI The Multi-Lang KubernetesExecutor system test builds a Go bundle and a Java jar inside throwaway toolchain containers. On a dev host the module, Gradle-distribution and dependency caches persist between runs, but CI runners are ephemeral and nothing cached them, so every scheduled run re-pulled the golang/temurin images and re-downloaded the Gradle distribution and all dependencies — adding roughly ten minutes to each of the six KubernetesExecutor variants that run the test. Build the artifacts with the host toolchain in CI instead, provisioned and cached through actions/setup-go and actions/setup-java, so the toolchain image pulls and cold dependency downloads no longer happen on every run. Local runs keep the containerised build so a dev host still needs neither Go nor a JDK installed. * Centralize lang-SDK JDK version and add a bustable cache key in CI Follow-ups to the native lang-SDK build: - Move the JDK version (17) into a JAVA_SDK_VERSION breeze constant and surface it as the java-sdk-version selective-checks / build-info output, so the k8s workflow reads it instead of hardcoding the version in YAML. - Restore the Go module/build and Gradle caches with an explicit actions/cache keyed on a "-v1-" salt rather than the setup-* built-in caching, giving an in-repo knob to force-invalidate a poisoned cache (bump the salt) without waiting for a dependency change to rotate it. (cherry picked from commit 673183b) Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Multi-Lang KubernetesExecutor system test (#68709) builds a Go bundle and a Java jar inside throwaway toolchain containers. On a dev host the Go module/build cache, the Gradle distribution and the dependency caches persist between runs, but CI runners are ephemeral and nothing cached them — so every scheduled run re-pulled the
golang/eclipse-temurinimages and re-downloaded the Gradle distribution and all dependencies. That added ~10 min to each of the sixKubernetesExecutor+ standard-naming-off variants (the only ones that run the test), which the daily CI duration-trend alert flagged.This builds the artifacts with the host toolchain in CI instead, provisioned and cached through
actions/setup-goandactions/setup-java(the same pattern the sibling Go SDK tests / Java SDK tests jobs already use). A newLANG_SDK_NATIVE_TOOLCHAIN=trueenv, set only for that variant, makes breeze invoke the hostgo/./gradlewrather thandocker run. Local runs keep the containerised build, so a dev host still needs neither Go nor a JDK installed.Measured on run
28635348211: the parallel provisioning block was ~5m36s, almost entirely toolchain-image pulls + cold Gradle/Maven/Go dependency downloads. Removing the image pulls and warming the caches should cut that to ~1–1.5 min (~4 min/job); the subsequent helm upgrade and the test itself are unaffected.Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 4.8) following the guidelines