feat(tangle-cli): add python_pipeline authoring DSL#20
Open
Silin144 wants to merge 1 commit into
Open
Conversation
Vendor the Python-first pipeline authoring surface into OSS as `tangle_cli.python_pipeline`, the first phase of moving `compile from-python` out of the internal `tangle-deploy` package. This is a clean additive port: the 17 DSL modules (pipeline/task/subpipeline/ registered/ref/raw + trace/graph/emit IR + cfg/types/errors/ids/ placeholders/task_env/compiler_context) carry no runtime Shopify seam — the only zone-root marker resolution lives in the compiler, not the DSL, and is deferred to a later phase. The package is fully OSS-native: all `tangle_deploy` docstring/reference mentions are genericized, and its public surface is `pipeline, task, registered, ref, raw, subpipeline, TaskEnv, In, Out, Outputs`. External deps are stdlib + PyYAML (already required). Generalize the component_from_func authoring-import strip to recognize BOTH authoring surfaces. `_AUTHORING_IMPORT_MODULE` (a single string) becomes `_AUTHORING_IMPORT_MODULES`, a tuple of `tangle_deploy.python_pipeline` (legacy, re-exports the OSS objects) and `tangle_cli.python_pipeline` (canonical), matched via a new `_is_authoring_module` helper that also covers submodules. Non-authoring `tangle_cli.*` / `tangle_deploy.*` runtime helpers and relative imports are still preserved. The authoring shim (`_ensure_tangle_deploy_ authoring_shim`) is intentionally NOT generalized: faking `tangle_cli.python_pipeline` in sys.modules would shadow the real DSL process-wide and break tracing — the real module is importable in-package so no shim is needed for that path. Tests: new tests/test_python_pipeline_dsl.py locks the public surface, the no-`tangle_deploy`-references guard, snake_to_title_case, the CompileError hierarchy, the In/Out/Outputs markers and the raw() contract; tests/test_component_from_func.py gains parallel `tangle_cli.python_pipeline` strip cases plus unit tests for the `_is_authoring_module` / `_is_authoring_import` predicates. Full suite: 671 passed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Phase 1 of migrating
compile from-pythonout of the internaltangle-deploypackage: vendor the Python-first pipeline authoring DSL into OSS astangle_cli.python_pipeline.This is a clean additive port — no existing behavior changes, no new runtime dependency (stdlib + PyYAML, already required).
The DSL (17 modules)
pipeline,task,subpipeline,registered,ref,rawauthoring entry points; the trace-time IR (trace,graph) lowered to the dehydrated dict shape byemit; pluscfg,types,errors,ids,placeholders,task_env,compiler_context.Public surface:
The DSL carries no runtime Shopify seam — the zone-root marker resolution lives in the compiler (a later phase), not the DSL. All
tangle_deploydocstring/reference mentions are genericized (a guard test enforces zero references remain).Authoring-import strip generalization
component_from_funcstrips authoring imports from baked runtime component source so a component running on a thin image (without the DSL) imports cleanly. Generalized to recognize both authoring surfaces:_AUTHORING_IMPORT_MODULE(single string) →_AUTHORING_IMPORT_MODULEStuple oftangle_deploy.python_pipeline(legacy; will re-export the OSS objects) andtangle_cli.python_pipeline(canonical)._is_authoring_modulehelper also covers submodules (...python_pipeline.types).tangle_cli.*/tangle_deploy.*runtime helpers and relative imports are still preserved.The authoring shim (
_ensure_tangle_deploy_authoring_shim) is intentionally not generalized: fakingtangle_cli.python_pipelineinsys.moduleswould shadow the real DSL process-wide and break tracing. The real module is importable in-package, so no shim is needed for that path.Tests
tests/test_python_pipeline_dsl.py— public import surface +__all__, identity offrom-imported objects, the no-tangle_deploy-references guard,snake_to_title_case, theCompileErrorhierarchy,In/Out/Outputsmarkers, and theraw()contract (str-only, rejects Jinja{%%}/{##}, accepts{{...}}runtime sentinels).tests/test_component_from_func.py— paralleltangle_cli.python_pipelinestrip cases (from-import, dotted@tp.task, aliased, plain import, submodule, multi-name, non-authoring preserved) + unit tests for_is_authoring_module/_is_authoring_import.Full suite: 671 passed.
uv lock --checkandgit diff --checkclean.Follow-ups (not this PR)
PipelineCompiler+ the OSScompilecommand; implement the zone-root marker behind a mutable-global/registry seam.tangle-deployto an identity-preserving re-export of the OSS DSL so every existingfrom tangle_deploy.python_pipeline import ...pipeline keeps working unchanged.🤖 Generated with Claude Code