feat(docker): add attachable Hubble Compose add-on for the 3-node cluster - #3149
feat(docker): add attachable Hubble Compose add-on for the 3-node cluster#3149bitflicker64 wants to merge 5 commits into
Conversation
…ster Add docker-compose-hubble.yml, a Hubble-only add-on that joins the cluster's pre-created external network (HUGEGRAPH_NETWORK, default hugegraph-net) with no depends_on, so attaching, upgrading, or removing Hubble never recreates PD, Store, or Server containers. Attach flow uses an explicit project (-p hugegraph-hubble); the fresh flow brings up cluster plus Hubble in one command with both -f flags. Give the 3-node cluster the Server settings Hubble's PD mode requires: PD registration (HG_SERVER_CLUSTER/USE_PD/REST_URL per replica via a shared env anchor), a required shared auth token secret so tokens validate on every replica, and a required admin password. The Server healthcheck now probes the bound REST URL. Hubble reads the 3x3 topology from hugegraph-hubble-3x3.properties. Document the attach, fresh, and dev-override flows plus migration notes in docker/README.md, update the cluster call sites across the docs, and extend the CI compose contract checks to the cluster file and add-on. Image tags stay on latest until the 1.8.0 release publishes; pin via HUGEGRAPH_VERSION in docker/.env.
…in compose checks Render the combined cluster+Hubble topology with non-default network and version values and assert the overridden network name and all four image tags, so CI fails if any Compose file stops honoring either override. The standalone add-on render keeps asserting the defaults and now strips any runner-level overrides for hermeticity.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #3149 +/- ##
============================================
- Coverage 39.30% 34.74% -4.57%
- Complexity 264 498 +234
============================================
Files 771 782 +11
Lines 65915 67065 +1150
Branches 8754 8951 +197
============================================
- Hits 25910 23302 -2608
- Misses 37241 41163 +3922
+ Partials 2764 2600 -164 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
imbajin
left a comment
There was a problem hiding this comment.
Blocking: yes. Summary: The Compose add-on has a false-green authentication check and operational failover and attachment gaps that can make a healthy-looking deployment unusable. Evidence: exact-head static review across six independent lanes; docker-entrypoint.sh:93-98 requires a 32-byte token, Hubble LiveOperationsCollector uses one pd.server, and the add-on has no data volume.
imbajin
left a comment
There was a problem hiding this comment.
Blocking: yes. Summary: The cluster quickstart currently fails after generating its own credentials, and the Hubble add-on mounts the H2 database outside the path used by Hubble; additional deployment and CI gaps can leave a false-green or state-inconsistent setup. Evidence: exact-head static review across six independent lanes plus Docker Compose render and guard checks; live container startup was unavailable.
| printf "HUGEGRAPH_AUTH_TOKEN_SECRET='%s'\n" "${token_secret}" >> .env | ||
| unset token_secret | ||
| fi | ||
| admin_value="$(sed -nE "s/${pat}HUGEGRAPH_ADMIN_PASSWORD='([^']*)'[[:space:]]*$/\\1/p" .env | tail -n1)" |
There was a problem hiding this comment.
export capture before the value, but both sed commands substitute capture group 1, so HUGEGRAPH_ADMIN_PASSWORD='...' yields an empty value (and an export form yields the word export); lines 216/220 then fail. Use the value capture group and parse or explicitly reject all supported Compose dotenv formats consistently.
There was a problem hiding this comment.
Agreed and also in scope. The optional export group made \1 capture the wrong text. I’ll fix the setup parser to read the value group (and keep parsing as data, not by sourcing .env).
| - "${HUBBLE_PUBLISH_HOST:-127.0.0.1}:8088:8088" | ||
| volumes: | ||
| - ./hugegraph-hubble-3x3.properties:/hubble/conf/hugegraph-hubble.properties:ro | ||
| - hg-hubble-db:/hubble/db |
There was a problem hiding this comment.
/hubble and its datasource is jdbc:h2:file:./db; mounting a volume at /hubble/db leaves the actual /hubble/db.mv.db outside that volume (or makes startup fail because /hubble/db is a directory). Point H2 at a file inside the mounted directory or mount the parent directory, then verify data survives container recreation.
There was a problem hiding this comment.
I’ll fix the persistence boundary so Hubble state survives recreate.
|
|
||
| # ── Shared service defaults ────────────────────────────────────────── | ||
| x-pd-common: &pd-common | ||
| # Pin a release via HUGEGRAPH_VERSION in docker/.env; unset, the image |
There was a problem hiding this comment.
latest with pull_policy: missing; an already-cached older image can ignore PASSWORD and the token secret, while the healthcheck probes public /versions. Pin or fail closed on incompatible images, or add an authenticated readiness smoke check that proves graph requests are rejected without credentials and succeed with them.
There was a problem hiding this comment.
Agree this is a real gap for the auth-required 3-node quickstart. I’d like to keep #3149 scoped to the Hubble add-on and the minimum cluster contract Hubble needs. Changing Server readiness to prove 401/200 and changing default pull policy feels like follow-up cluster hardening rather than the add-on itself. README already documents the manual 401/200 check for incompatible images.
| networks: | ||
| hg-net: | ||
| driver: bridge | ||
| external: true |
There was a problem hiding this comment.
hugegraph-net is external and the file publishes PD/Store REST, gRPC, and Raft ports on all host interfaces; Store has no authentication and PD only checks the internal service username, not the password. Bind these ports to loopback/remove them by default, or require real authentication/TLS and explicit network ACLs before shipping this as the default quickstart.
There was a problem hiding this comment.
Agree on the exposure concern. Default loopback binds / ACL+TLS requirements are a cluster-quickstart security redesign; I’d track that as a follow-up so this PR stays reviewable as a Hubble add-on.
| name: ${HUGEGRAPH_NETWORK:-hugegraph-net} | ||
|
|
||
| volumes: | ||
| hg-hubble-db: |
There was a problem hiding this comment.
hugegraph-hubble, while the combined flow runs under hugegraph-3x3; Compose therefore creates different project-prefixed H2 and upload volumes. Switching flows makes existing state and uploads appear to disappear. Give both volumes stable explicit names (with a multi-cluster override if needed) or document a supported volume migration.
There was a problem hiding this comment.
I’ll give the Hubble DB/upload volumes explicit stable names so attach and combined share the same physical volumes.
| # And a signed-in read must succeed. Compose reads docker/.env by itself, but | ||
| # your shell does not — load it first. Passing the credential through | ||
| # --config keeps it out of argv, where `ps` would expose it to other users. | ||
| set -a; . ./.env; set +a |
There was a problem hiding this comment.
set -a; . ./.env; set +a evaluates command substitutions and other shell syntax before the curl; a value such as HUGEGRAPH_ADMIN_PASSWORD=$(...) runs on the operator host. Parse only the required key with a dotenv-aware parser or avoid sourcing .env directly.
There was a problem hiding this comment.
I’ll replace set -a; . ./.env; set +a with a parse‑only read of the required key, so the verify block does not execute dotenv as a shell.
| hostname: server0 | ||
| environment: | ||
| <<: *server-env | ||
| HG_SERVER_REST_URL: http://server0:8080 |
There was a problem hiding this comment.
http://serverN:8080 with PD while the host publishes ports 8080/8081/8082. Evidence: Server registration uses restserver.url, so a PD-aware client outside the Docker network receives server0/server1/server2, names that only resolve inside hugegraph-net. Provide a separate configurable advertised address or make the external client path use addresses it can resolve.
There was a problem hiding this comment.
Agree for PD-aware clients outside Docker DNS. Hubble on hugegraph-net can use serverN names as-is. Configurable advertised addresses feel like a follow-up to external-client addressing, not required for the add-on path. What do you think?
| .services.store0.image == "hugegraph/store:ci-test-tag" and | ||
| .services.server0.image == "hugegraph/server:ci-test-tag" and | ||
| .services.hubble.image == "hugegraph/hubble:ci-test-tag" and | ||
| .services.server0.environment.HG_SERVER_USE_PD == "true" and |
There was a problem hiding this comment.
server0 for HG_SERVER_USE_PD, HG_SERVER_CLUSTER, HG_SERVER_INIT_STORE_ENABLED, and PASSWORD; server1/server2 only check REST URL and token. A per-replica override can therefore pass CI while breaking discovery or initialization. Apply the required-value assertions to all three Server services.
There was a problem hiding this comment.
Agreed — in scope. This PR introduced the PD/auth Server env for Hubble discovery, so CI should assert the required values on server0/server1/server2, not only server0.
| | select(.value.hostname == $h) | ||
| | .value.hostname + ":" + .value.environment.HG_PD_REST_PORT' \ | ||
| "$rendered")" | ||
| assert_props "cluster=${cluster_name}" "matches HG_SERVER_CLUSTER" |
There was a problem hiding this comment.
assert_props validates cluster, pd.peers, pd.server, and Store targets, but never asserts pd.enabled=true; the current server.direct_url is only meaningful when PD mode is false. Add the explicit mode assertion and a runtime smoke check so Hubble cannot silently fall back to one direct Server.
There was a problem hiding this comment.
I’ll add an explicit assert_props "pd.enabled=true" so the Hubble properties check cannot pass with PD mode disabled.
| HUGEGRAPH_NETWORK=ci-test-net \ | ||
| HUGEGRAPH_VERSION=ci-test-tag \ | ||
| env -u HUBBLE_IMAGE -u HUBBLE_PULL_POLICY -u HUBBLE_PUBLISH_HOST \ | ||
| docker compose --env-file /dev/null -f "$cluster" -f "$addon" \ |
There was a problem hiding this comment.
docker compose config plus jq/grep and never creates the external network, starts the services, probes Hubble /about, verifies PD registration, or checks authenticated graph access. Add a Docker-capable smoke job for the combined and attach flows so entrypoint, DNS, registration, and auth failures cannot remain green.
There was a problem hiding this comment.
Render‑only cannot catch runtime auth/DNS failures. For this PR I’ll keep and strengthen the compose contract assertions (including the in‑scope ones above). A full combined or attached Docker smoke job is useful but adds heavier CI overhead; I’d prefer a follow‑up rather than blocking the add‑on here.
Read docker/.env as data instead of sourcing it as shell, and take the quoted value rather than the optional export capture when validating generated credentials. Point Hubble H2 at a file inside the /hubble/db volume, give attach and combined flows the same explicit volume names, and assert PD/auth env on every Server replica plus pd.enabled=true.
imbajin
left a comment
There was a problem hiding this comment.
Blocking: yes. Summary: Four previously reported deployment blockers remain on this head: incompatible cached images can leave authentication false-green, PD/Store control-plane ports remain exposed without real authentication, PD-registered Server URLs are not resolvable outside Docker DNS, and CI remains render-only. Evidence: exact head 950f645; 21 GitHub checks passed; fresh Docker Compose render, credential guard, topology, and dotenv-parser checks passed locally; six independent review lanes were unavailable because the local review gateway repeatedly failed.
What this PR does
Adds an attachable Hubble deployment for the distributed Compose topology:
docker/docker-compose-hubble.yml(new) — defines only the Hubbleservice. It joins the pre-created cluster network and has no
depends_onon cluster services, so attaching, upgrading, or removing Hubble never
recreates PD, Store, or Server containers.
docker/docker-compose-3pd-3store-3server.yml— adds the ServerPD-registration and auth settings Hubble needs (cluster name, PD mode,
per-replica REST URL, required admin password and shared token secret),
mirroring what fix(docker): integrate Hubble with PD deployment #3143 did for the single-node file.
docker/hugegraph-hubble-3x3.properties(new) — Hubble topologyconfig for the 3-node cluster.
docker/README.md— documents the three flows: attach to a runningcluster, fresh cluster + Hubble in one command, and a local Hubble image
override for development.
.github/workflows/server-ci.yml— extends the existing composecontract checks to the cluster file and the add-on.
commands no longer work with the required-credential guards.
Design rationale (why it is shaped this way)
merges with the canonical cluster file via
-fflags, so the topologyis never maintained in two places. This is the pattern Elastic documents
for attaching Kibana to a running Elasticsearch over a shared Docker
network with versions matched through one variable, and the attach model
TiDB Operator provides via its standalone
TidbDashboardresource.external:networksbefore creating project networks and refuses to adopt unlabeled ones, so
both files declare the shared network as external with a well-known name
(
hugegraph-net, override viaHUGEGRAPH_NETWORK) and setup includes aone-time
docker network create. Verified empirically; the generatedper-project network name cannot support the one-command fresh flow.
-p hugegraph-hubblefor the attach flow: without it theattach project takes the directory name, colliding with other projects
and making documented lifecycle commands cross project boundaries.
latestas the current default: the authenticated PD/Hubbleintegration is not in
1.7.x, so pinning to an existing release wouldship a broken combination. Once the next release publishes, the default
should move to that tag; deployments pin today via
HUGEGRAPH_VERSIONin
docker/.env. (Unpinnedlatestwith no compatibility story is thedocumented failure mode of Dgraph's Ratel — hence the single shared
version variable.)
Breaking change owned by this PR
Existing 3-node deployments: the first
up -dafter this change recreatesall nine containers (network move; named volumes survive), and the graph
APIs begin requiring authentication (401 for previously anonymous
clients). The README carries an upgrade note covering
.envcreation,the orphaned old network, and verifying sign-in before decommissioning
existing access paths.
Validation evidence
(HTTP 200, role SUPERADMIN); Operations view shows 1 logical Server +
3 PD + 3 Store, all UP.
add-on command attached Hubble; all cluster container IDs unchanged.
docker/docker-compose.ymlpath unaffected andhealthy.
Review follow-ups in this update
optional
exportgroup as the password, so a freshly generateddocker/.envfailed the non-empty check immediately.docker/.envas shell../db/hubbleso the file lives inside the/hubble/dbvolume instead of/hubble/db.mv.dboutside it.(
hugegraph-hubble-db,hugegraph-hubble-upload-files).pd.enabled=truein the Hubble properties file.Follow-up work (not in this PR)
Leaving these for later so this PR stays a Hubble add-on:
ignores PASSWORD. Fail closed, or prove 401 without credentials and
200 with them.
Bind to loopback or require real auth/TLS before this is the default
quickstart.
hugegraph-net; PD-aware clients outside that network cannot resolvethem. Advertised addresses are a separate change.
would catch entrypoint, DNS, and auth failures that
config+ jqmiss.