Skip to content

java-sdk

Directory actions

More options

Directory actions

More options

Latest commit

 

History

History

java-sdk

README.md

Airflow Java SDK

A JVM SDK for Apache Airflow. You can use any JVM-compatible language to write workflow bundles, and have Airflow consume the result.

The SDK and execution-time logic is implemented in Kotlin. An example is bundled showing how the SDK can be used in Java.

Building the SDK

./gradlew build

Building documentation

./gradlew dokkaGenerate

This uses Dokka to build documentation of the Java SDK. This generates both an HTML representation and Javadoc.

Running the example

The SDK projects must first built and published:

./gradlew publishToMavenLocal -PskipSigning=true

After the build is successful, you should be able to see directories in ~/.m2/repository/org/apache/airflow/.

Now cd example into the example project, and

  • Package the example to ./example/build/bundle

    # We're now in the 'example' directory, so gradlew is in parent.
../gradlew bundle

  • Put the DAG with stub tasks to somewhere Airflow can find.

  • Ensure the java command is available in the same environment the Airflow task worker is in.

  • Configure Airflow to route tasks in the java queue to be run with Java:

    export AIRFLOW__SDK__COORDINATORS='{
  "java": {
    "classpath": "airflow.sdk.coordinators.java.JavaCoordinator",
    "kwargs": {"jars_root": ["/opt/airflow/java-sdk/example/build/bundle"]}
  }
}'
export AIRFLOW__SDK__QUEUE_TO_COORDINATOR='{"java": "java"}'

  • Ensure the Connection and Variable needed by the example DAG are available:

    export AIRFLOW_CONN_TEST_HTTP='{
    "conn_type": "http",
    "login": "user",
    "password": "pass",
    "host": "example.com",
    "port": 1234,
    "extra": {"param1": "val1", "param2": "val2"}
}'
export AIRFLOW_VAR_MY_VARIABLE=123

Publishing

The SDK is published to Maven Central via the ASF Nexus staging repository. The full release process follows the ASF Maven publishing guide.

Any versions published to Maven Central (instead of Snapshots) are considered releases, including alpha, beta, etc (see the ASF Release Policy). Every release therefore requires a PMC vote before it is published; only -SNAPSHOT builds may be published without a vote.

Prerequisites

Bump the version

Edit gradle.properties and set the version for this release:

projectVersion=<VERSION>

Commit the change and push it to the release branch.

Use a Maven-compatible version string, as defined by the Maven version order specification. For example, version 1 beta 1 is 1.0.0-beta1, and the eventual version 1 release is 1.0.0.

NOTE: Editing gradle.properties as above is the standard procedure. You can alternatively override the version for a single command without editing the file, by passing -PprojectVersion=<VERSION> to any Gradle invocation. This is handy for one-off or pre-release builds. Either way, main should stay on a -SNAPSHOT version between releases: a snapshot sorts after <VERSION> and before the GA, so no extra bump is needed after a beta.

Tag the release candidate

Tag the release commit, keeping the RC number in the tag name so a failed vote simply bumps to the next RC (the artifact version itself does not carry the RC suffix). Push the tag before sending the vote so reviewers can check out the exact source being voted on.

git tag -s java-sdk/<VERSION>-rc<N> -m "Java SDK <VERSION> RC <N>"
git push upstream java-sdk/<VERSION>-rc<N>

Verify the POM locally

Before touching any remote repository, publish to your local Maven cache and inspect the generated POM:

rm -rf ~/.m2/repository/org/apache/airflow/  # Start clean.

./gradlew publishToMavenLocal -PskipSigning=true

# The airflow-sdk runtime.
less ~/.m2/repository/org/apache/airflow/airflow-sdk/*/airflow-sdk-*.pom

# The bill of materials of airflow-sdk
less ~/.m2/repository/org/apache/airflow/airflow-sdk-bom/*/*.pom

# The annotation processor for the builder pattern.
less ~/.m2/repository/org/apache/airflow/airflow-sdk-processor/*/airflow-sdk-*.pom

# The Gradle plugin for bundling.
less ~/.m2/repository/org/apache/airflow/airflow-sdk-gradle-plugin/*/airflow-sdk-*.pom

# The Gradle plugin's registration.
less ~/.m2/repository/org/apache/airflow/sdk/org.apache.airflow.sdk.gradle.plugin/*/*.pom

Check that the coordinates, description, license, SCM, and organization fields look correct.

Dry-run against a local repository

To test the full publish flow without touching ASF infrastructure, override the repository URL to a local directory

rm -rf /tmp/local-maven-repo  # Start clean.
./gradlew publish -PmavenUrl=file:///tmp/local-maven-repo -PskipSigning=true
ls /tmp/local-maven-repo/org/apache/airflow/
# This should contain the same components in ~/.m2 as inspected in the previous step.

NOTE: Signing is not required since nothing goes to Maven Central. If you want to test signing, set the GPG private key and passphrase as described in the next section, and remove -PskipSigning=true from the above command.

Publish to ASF Nexus staging

Store the credentials in ~/.gradle/gradle.properties so they are not exposed in your shell history:

mavenUsername=your-asf-nexus-token-username
mavenPassword=your-asf-nexus-token-password
signing.password=your-gpg-key-passphrase

Then stage and close the release. The Gradle Nexus Publish Plugin (applied in the root build.gradle.kts) gathers every module into one staging repository, so there is no per-module fan-out to reconcile.

./gradlew publishToApache closeApacheStagingRepository \
  --no-configuration-cache \
  -P"signing.key=$(gpg --armor --export-secret-keys your-gpg-key-fingerprint)"

NOTE: The signing key is supplied through the command line since it contains newlines, which does not work well in a Gradle properties file.

NOTE: You can also use the following environment variables to provide the credentials instead: ASF_NEXUS_USERNAME, ASF_NEXUS_PASSWORD, SIGNING_KEY, and SIGNING_PASSWORD. This is especially useful on e.g. CI.

NOTE: We enable Gradle's configuration cache globally, but the staging tasks (publishToApache, closeApacheStagingRepository, releaseApacheStagingRepository) talk to the Nexus REST API and are not configuration-cache compatible. Hence the --no-configuration-cache flag on the release commands.

Verify the upload

Under Staging Repositories on the ASF Nexus server, open the closed repository and verify it contains all modules, each with its jar, -sources.jar, -javadoc.jar (where applicable), .pom, and .asc signature.

Check Updated by (should be your ID), Uploaded Date, and Last Modified.

Upload the source package

The closed staging repository from the previous step is the convenience-binary URL you link in the vote.

The signed source package is the artifact the vote is formally on; the Maven artifacts are convenience binaries. The sourceRelease task builds it from the committed java-sdk sources (LICENSE and NOTICE included) and produces its signature and checksum in one step:

# Signing uses your local gpg keyring, so have your key/passphrase ready.
./gradlew sourceRelease -PgitRef=java-sdk/<VERSION>-rc<N>

This writes three files to build/distributions/:

apache-airflow-java-sdk-<VERSION>-src.tar.gz
apache-airflow-java-sdk-<VERSION>-src.tar.gz.asc
apache-airflow-java-sdk-<VERSION>-src.tar.gz.sha512

NOTE: The source archive omits the Gradle wrapper scripts (gradlew, gradlew.bat) and gradle/wrapper/gradle-wrapper.jar since ASF source releases must not contain compiled code (see LEGAL-570), and the scripts are not useful without the jar. The gradle/wrapper/gradle-wrapper.properties file is kept so the archive pins the Gradle version and distribution checksum for verification when the wrapper is regenerated.

Copy the three files into your checkout of the ASF dist dev repo and commit them with Subversion. If you don't already have the repo checked out, replace <dist-dev-checkout> with wherever you want it and <path-to> with this project's location:

# One-time: check out the Airflow dist dev area.
svn checkout https://dist.apache.org/repos/dist/dev/airflow <dist-dev-checkout>

cd <dist-dev-checkout>
mkdir -p java-sdk/<VERSION>-rc<N>
cp <path-to>/java-sdk/build/distributions/apache-airflow-java-sdk-<VERSION>-src.tar.gz* \
   java-sdk/<VERSION>-rc<N>/

svn add --parents java-sdk/<VERSION>-rc<N>
svn commit -m "Add Apache Airflow Java SDK <VERSION>-rc<N> source release candidate"

The commit publishes them under https://dist.apache.org/repos/dist/dev/airflow/java-sdk/<VERSION>-rc<N>/, which is the source-package URL you link in the vote.

Call the vote

Send a [VOTE] email to dev@airflow.apache.org linking the git tag and commit, the source package in dist/dev, the closed Nexus staging repository, and the KEYS file. See "Vote email template" below for the exact fields to fill in, and "Verifying a release" for what to ask reviewers to check.

Vote email template

Subject: [VOTE] Release Apache Airflow Java SDK <VERSION> based on <VERSION>-rc<N>

Hi,

I would like to call a vote to release Apache Airflow Java SDK <VERSION>, based
on release candidate <VERSION>-rc<N>.

Changes since <the last release>: <a one-line description>

The release candidate contains the following Maven artifacts, all under group id org.apache.airflow:

- Main SDK API
    - airflow-sdk
    - airflow-sdk-processor
- Logger helpers
    - airflow-sdk-jpl
    - airflow-sdk-jul
    - airflow-sdk-log4j2
    - airflow-sdk-slf4j
- Gradle plugin and the marker artifact
    - airflow-sdk-gradle-plugin
    - org.apache.airflow.sdk.gradle.plugin
- BOM
    - airflow-sdk-bom

Git information:

- Tag: java-sdk/<VERSION>-rc<N>
- Commit: <full-commit-sha>
- https://github.com/apache/airflow/releases/tag/java-sdk%2F<VERSION>-rc<N>

Source release, signatures and checksums:
https://dist.apache.org/repos/dist/dev/airflow/java-sdk/<VERSION>-rc<N>/

Convenience binaries (staged in the ASF Nexus repository):
https://repository.apache.org/content/repositories/orgapacheairflow-<NNNN>/

KEYS file (public keys used to sign the release):
https://downloads.apache.org/airflow/KEYS

Please review and vote. The vote will remain open for at least 72 hours, until <YYYY-MM-DD HH:MM UTC>, or until the necessary number of binding votes is reached.

[ ] +1 Release this package as Apache Airflow Java SDK <VERSION>
[ ] +0 No opinion
[ ] -1 Do not release, because ...

Only votes from Airflow PMC members are binding, but everyone is welcome and encouraged to test the release and vote.

The verification process can be found in the main repository:
https://github.com/apache/airflow/tree/java-sdk/<VERSION>-rc<N>/java-sdk#verifying-a-release

For more details on ASF release verification, see: https://www.apache.org/info/verification.html

Best,
<your name>

Pre-send checklist:

  • Every staged artifact above resolves in the Nexus staging repository. Cross check against the BOM, not just this list from memory.
  • Changes since rc<N-1> and the vote deadline are filled in.
  • Run grep '<' email.txt on the rendered email and confirm no output — any match means a template placeholder (<N>, <NNNN>, <YYYY-MM-DD ...>, etc.) was left unfilled.

Verifying a release

Anyone on dev@airflow.apache.org can (and should) independently verify a candidate before voting. Below is the checklist a reviewer — or the release manager, before sending the vote — should run against the source package in dist/dev.

  1. Checksum. Confirm the published SHA-512 matches the downloaded tarball:

    sha512sum -c apache-airflow-java-sdk-<VERSION>-src.tar.gz.sha512

  2. Signature. Import the KEYS file and verify the GPG signature:

    curl -O https://downloads.apache.org/airflow/KEYS
gpg --import KEYS
gpg --verify apache-airflow-java-sdk-<VERSION>-src.tar.gz.asc \
    apache-airflow-java-sdk-<VERSION>-src.tar.gz

  3. Diff against the git tag. Extract the tarball and compare it with a clean checkout of the tag it claims to be built from. They should be identical except gradlew, gradlew.bat, and gradle-wrapper.jar. The extracted top-level directory should be apache-airflow-java-sdk-<version> without the -src suffix that only appears in the tarball's own filename:

    tar xzf apache-airflow-java-sdk-<VERSION>-src.tar.gz
git clone --branch java-sdk/<VERSION>-rc<N> \
  https://github.com/apache/airflow.git tag-checkout
diff -rq apache-airflow-java-sdk-<VERSION>/ tag-checkout/java-sdk/ \
  | grep -vE ': (gradlew|gradlew\.bat|gradle-wrapper\.jar)$'

    Any remaining diff output is unexpected and should block the vote.

  4. No binary files. ASF source releases must not contain compiled code. Scan for anything that isn't text:

    find apache-airflow-java-sdk-<VERSION>/ -type f \
  -exec sh -c 'file "$1" | grep -qv text && echo "$1"' _ {} \;

    This should print nothing.

  5. Build from source. Regenerate the Gradle wrapper from a locally installed Gradle (see the Upload the source package section above):

    cd apache-airflow-java-sdk-<VERSION>
gradle wrapper \
    --gradle-version <GRADLE-VERSION> \
    --gradle-distribution-sha256-sum <GRADLE-DISTRIBUTION-SHA>
./gradlew build

    Use the values for <GRADLE-VERSION> and <GRADLE-DISTRIBUTION-SHA> from distributionUrl and distributionSha256Sum in the bundled gradle/wrapper/gradle-wrapper.properties.

  6. Staged-binary smoke test. Resolve the staged Nexus artifacts from a throwaway project to confirm they're actually consumable, following the same pattern as the "Dry-run against a local repository" step: point a repositories {} block at the staging repository URL, declare a dependency on org.apache.airflow:airflow-sdk-bom:<VERSION>, and confirm the transitive artifacts (including airflow-sdk-jpl) resolve and the example bundle builds against them.

After a successful vote

Reply with a [RESULT][VOTE] tally, then:

  1. Release the staging repository so the artifacts sync to Maven Central (a few hours). Nothing is rebuilt or re-signed:

    ./gradlew releaseApacheStagingRepository --no-configuration-cache

    (Or click Release on the repository in the Nexus UI.)

  2. Move the source package from dist/dev to dist/release:

    svn mv https://dist.apache.org/repos/dist/dev/airflow/java-sdk/<VERSION>-rc<N> \
       https://dist.apache.org/repos/dist/release/airflow/java-sdk/<VERSION> \
       -m "Release Apache Airflow Java SDK <VERSION>"

  3. Tag the final version on the same commit that was voted:

    git tag -s java-sdk/<VERSION> <voted-commit-hash> -m "Apache Airflow Java SDK <VERSION>"
git push upstream java-sdk/<VERSION>

    Keep the RC tag for traceability.

  4. Update the download page and wait ~1 hour after promoting so Central has synced. Send the [ANNOUNCE] email.

If the vote fails

Close the vote, drop the staging repository in Nexus, remove the dist/dev candidate, fix the issue, and cut the next RC (...-rc2). The released version stays the same (e.g. <VERSION>); only the RC counter in the tag increments.

Contributing

The user implements a Java application containing task methods annotated (or registered) with the SDK. The application is packaged as a bundle and placed where Airflow can find it.

When the Airflow supervisor identifies that a task should run with Java, it launches the JVM application as a subprocess. The flow is:

  1. JavaCoordinator.execute_task() (Python) scans jars_root, builds the classpath, and spawns java -cp <jars> <MainClass> --comm=<host>:<port> --logs=<host>:<port>.
  2. Server.kt connects to both sockets immediately on startup.
  3. The supervisor sends a StartupDetails MessagePack message; the JVM reads it, looks up the matching task by dag_id + task_id, and calls the user's task method.
  4. During execution the JVM sends requests to the supervisor (GetVariable, GetConnection, GetXCom, SetXCom, etc.) and the supervisor responds. All frames are a 4-byte big-endian length prefix followed by a MessagePack payload.
  5. On completion (or exception) the JVM sends a TaskState message and closes the socket. The JVM process then exits.

Log messages produced by the SDK (not by user code) are forwarded over the --logs socket so the supervisor can append them to Airflow's log store.

The wire protocol is defined in task-sdk/src/airflow/sdk/execution_time/schema/schema.json. execution/Comm.kt implements the framing layer. Adding a new message type requires changes in both schema.json (Python side) and execution/Comm.kt + execution/Client.kt (JVM side).

See Architectural Design Records in the adr directory to learn more.

Repository layout

java-sdk/
├── sdk/          # Core library: public API (org.apache.airflow.sdk) and internal
│                 #   execution layer (org.apache.airflow.sdk.execution)
├── processor/    # Annotation processor that generates *Builder classes
├── plugin/       # Gradle plugin (org.apache.airflow.sdk) — bundle task, manifest
│                 #   attribute injection, and verifyBundleMainClass
├── bom/          # Bill of Materials POM so consumers can import all SDK artifacts
│                 #   at a consistent version
├── slf4j/        # SLF4J logging provider; routes SLF4J calls to the Airflow log store
├── jul/          # java.util.logging handler; routes JUL records to the Airflow log store
├── jpl/          # Java Platform Logging provider (System.Logger, JEP 264); routes JPL
│                 #   calls to the Airflow log store
├── log4j2/       # Log4j 2 appender; routes Log4j 2 events to the Airflow log store
├── example/      # End-to-end example bundle (annotation + interface APIs, Java source)
├── adr/          # Architectural Decision Records for the Java SDK
└── buildSrc/     # Shared Gradle convention plugins (Java version, lint, etc.)

The Python coordinator that launches the JVM subprocess lives outside this directory:

task-sdk/src/airflow/sdk/coordinators/java/   # JavaCoordinator (SubprocessCoordinator subclass)
task-sdk/tests/coordinators/java/             # Python-side unit and integration tests

Testing

# Run all JVM tests
./gradlew test

# Run a specific test class
./gradlew :sdk:test --tests "org.apache.airflow.sdk.execution.CommTest"

For the Python coordinator, use Breeze (never run pytest on the host directly):

breeze testing task-sdk-tests -- task_sdk/coordinators/java

End-to-end tests that exercise a real Airflow environment:

E2E_TEST_MODE=java_sdk uv run --project airflow-e2e-tests pytest \
    tests/airflow_e2e_tests/java_sdk_tests/ -xvs

Coding conventions

  • All SDK and processor source is Kotlin; Java is the public API target, not the implementation language.
  • Keep sdk/src/main/kotlin/ (the public API surface) free of internal implementation details; those belong in the execution/ sub-package.
  • The annotation processor (BuilderProcessor.kt) uses kapt. When adding a new annotation, define it in Builder.kt, handle it in BuilderProcessor.kt, and add a golden-output test in processor/src/test/kotlin/.
  • The Python coordinator subclasses SubprocessCoordinator. Do not reach into the JVM process from Python beyond what _build_execute_task_command provides.
  • Run ./gradlew ktLintCheck spotlessCheck (or ktLintFormat spotlessApply) before submitting — the project enforces Kotlin and Java formatting.
  • All new files need the Apache License header.

Common tasks

Adding a new Client method (e.g. a new Airflow API call):

  1. Regenerate POJO classes from schema.json if the message type is new.
  2. Add Kotlin request/response data classes in execution/Comm.kt or a new file.
  3. Add the public-facing method to Client.kt which delegates to execution/Client.kt for the supervisor wire call.
  4. Write a unit test in sdk/src/test/kotlin/.../ClientTest.kt mocking the socket layer.
  5. Update airflow-core/docs/authoring-and-scheduling/language-sdks/java.rst if the change is user-visible.

Adding a new annotation:

  1. Define the annotation interface in Builder.kt.
  2. Handle it in BuilderProcessor.kt — generate the appropriate code in the *Builder class.
  3. Add a test in BuilderTest.kt with expected generated output.
  4. Update the annotation table in java.rst.

Fixing a framing or protocol bug: focus on execution/Comm.kt and execution/Frame.kt. CommTest.kt covers encode/decode round-trips; add a regression test reproducing the bug before fixing it.

PR checklist

  • Run ./gradlew build test (JVM) and the relevant pytest suite (Python coordinator).
  • Confirm the example bundle still compiles (see "Running the example" above up to the bundling step)
  • If schema.json changed, verify both sides (JVM + Python) handle the new/changed fields.
  • Add or update tests for every changed behaviour.
  • For user-visible changes to task-sdk/, add a newsfragment under airflow-core/newsfragments/.