[dev] C6 - Add aliasing system for accessing deprecated fields in typed configurations - #3211

* centralize config I/O in core.config

* add pytests for core config I/O and generalized_data_converter

* Apply review suggestions (import core_config and os -> pathlib)

* Update tests/pose_estimation_pytorch/modelzoo/test_generalized_data_converter_config.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* modelzoo materialize.py: fix typos corner2move2

* additional tests for core config
- reading breaks for corrupted yaml
- added tests for prefered behavior that is currently not implemented (yaml safeloading and validating config keys)

* auxilliary functions: apply formatting

* add pytorch base config mixin

This mixin provides methods for:
    - Loading configurations from dictionaries or YAML files
    - Validating configuration data against pydantic models
    - Converting configurations to dictionaries
    - Pretty printing configuration data

add imports from utils

* add tests for ConfigMixin

* feat: add structured configuration classes for pose estimation

- Introduced new configuration classes for inference, logging, model, pose, project, runner, and training settings.
- Refactored data loading mechanisms to utilize new configuration structures.
- Moved the multithreading and compilation options in inference configuration to the config module.
- Typed configuration for logging.
- Updated dataset loaders to accept model configurations directly or via file paths.

* Simplify changes (only add typed configs)

* Add typed ProjectConfig as pydantic dataclass
(The fields are kept identical to old multianimal project configs for now)

move ProjectConfig to deeplabcut/core/config

* add pytorch configs as pydantic dataclasses

* copilot suggestions (return transforms, default MultthreadingConfig, correct TypeAdapter usage)

* fix transforms return value at validation

The return value should be the dictionary, not the instantiated transforms

* add comments in yaml using CommentedMap

* Fix skip decorator pytests

* fix missing typing_extensions on python 3.10

* fix None-type handling in update_config_by_dotpath

* Replace config in loaders with new DictConfig

* isolate old config logic in  `make_pose_confg`  and replace with PoseConfig

* update core config loaders in utils.py

* update tests to reflect changes in behaviour

* apply review suggestions: comments, warnings and wrong type

* add deprecated argument sentinel class

* improve `test_read_config_preserves_existing_engine_and_project_path`

* Update `read_config`  and `ConfigMixin`: Add empty field handling for reading from yaml.

* Update ConfigMixin: normalize paths/enums for serialization

* update make_pose_config: replace write_config with PoseConfig.to_yaml

* update ProjectConfig: pathlib Path instead of str

* fix intermediate head_cfg container in build PoseModel

The head_cfg (a part of PoseConfig) was used for dumping modules. This is not supported for typed configs.

The container is now replaced with a plain dict type instead of config.

* fix ProjectConfig list[int] type for corner2move2 field

* fix omegaconf.select (`default` -> keyword argument)

* docs: merge changes from main

* YAML I/O - add centralized, safe ruamel YAML loader and dumper with Path and Enum serialization

* YAML I/O - add yaml representers for DictConfig and ListConfig

* YAML I/O - merge ruamel fix from main (in centralized yaml dumper)

* CFG UPDATE - strict init for PoseConfig and ProjectConfig (forbid extra fields)

* CFG UPDATE - Add TestConfig for pytorch pose_cfg.yaml

* CFG UPDATE - update project config adjust `project_path` when loading from yaml

* CFG UPDATE - Add default individual in ProjectConfig (required in the pipeline)

update ProjectConfig add old fields resnet, croppedtraining

* CFG UPDATE - update PoseConfig NetTypes missing cspnext models

* CFG UPDATE - fix WeightInitialization snapshot path -> None or pathlib.Path

* CFG UPDATE - Add missing (ambiguous) with_identity field

* CFG UPDATE - Fix mismatching bodyparts fields in metadata and proj. config

* CFG UPDATE - add field 'device' to RunnerConfig (used in testscripts only)

* CFG UPDATE - Fix GenSamplingConfig & PoseDatasetParameters: resolve ambiguous bbox_margin

This commit resolves two issues:
- the bbox_margin field that was defined in GenSamplingConfig was always populated with the bbox_margin value from model_cfg['data']['bbox_margin'], which is an implicit way of carrying over config fields. (Potentially breaks when implicit transfer was missing somewhere).
- In most places, GenSamplingConfig is expected to NOT have a field bbox_margin, e.g. when converting to dict it is removed, where in 1 place it is expected (in PoseDataset in dataset.py).

This is now resolved by keeping only the explicit config value in PoseConfig.data.bbox_margin, and adding a ctd_bbox_margin field in PoseDatasetParameters.

* CFG UPDATE - update TrainsettingsConfig: allow WeightInitialization None

* CFG UPDATE - fix make_pose_config initialization from defaults

* TYPE CONVERSION - Add typed -> plain config conversion decorator

* TYPE CONVERSION - convert to plain dict / list where necesary (OmegaConf DictConfig and ListConfig should not be in state dicts)

* TYPE CONVERSION - Add TODO comments: mark unvalidated config updates

* TYPE CONVERSION - update make_pose_config & Loader: dict-updates BEFORE validation

Some config updates currently occur after initial creation and validation of the PoseConfig (e.g. when calling train_network).  This commit makes sure that update_by_dotpath and loader.update_model_cfg are validated afterward

* TYPE CONVERSION - Fix + MARK instance checks for ListConfig and DictConfig

* TYPE CONVERSION - fix unsupported operand type(s) for *: 'int' and 'ListConfig'

* TYPE CONVERSION - Update test_video_set_configuration: assert isinstance(dict) -> assert isinstance(mapping)

* TYPE CONVERSION - update paf_predictor: convert np.int64 to native integer

* NONE HANDLING - fix omegaconf.select default crop_cfg in loaders (not None)

* NONE HANDLING - Add None-safe get methods for configs that are allowed to be missing

* FIX TESTS - fix testscript invalid config: only add detector updates for td models

* docs: merge changes from main

* YAML I/O - Add post_yaml_load_updates hook for ConfigMixin

* VERSIONING - Add config versioning and migration system

* VERSIONING - add tests for versioning and migrationmixin

* VERSIONING - Add sequential step-wise migration chain

* VERSIONING - fix leaking toy migration

* VERSIONING - add duplicate migration guard and eliminate redundant copies

* VERSIONING - adress invalid version numbers

* VERSIONING - move MigrationMixin to versioning

* VERSIONING - increase test coverage for config migration

* VERSIONING - Add extensive logging for config migrations.

* VERSIONING - fix import Self (typing_extensions, for python 3.10)

* VERSIONING - Validate assignment for ProjectConfig and PoseConfig (+ add tests)

* VERSIONING - Update MigrationMixin migrate BEFORE validation (works with validate_assignment=True)

This is necessary to prevent a bug:
The isinstance(data, cls): return data shortcut in MigrationMixin.migrate_then_validate (a model_validator(mode="wrap")). Pydantic re-enters this validator during validate_assignment, passing the current instance as data. The shortcut returned it unchanged, silently discarding every field update. This broke all assignment — not just validation, but even plain cfg.count = 42 was a no-op.

The problem is that model_validator(mode="wrap") wraps the entire validation pipeline, including validate_assignment flow. Migration should only run during construction, not on every field assignment.
The clean solution is to switch from mode="wrap" to mode="before" — it transforms raw input data before validation and doesn't participate in validate_assignment at all.

* FIX TESTS - Update test core config string coercion to path

* TYPED CFG - Update ConfigMixin: Add dict-like functionality for ConfigMixin

* TYPED CFG - update tests for ConfigMixin

* TYPED CFG - Remove OmegaConf DictConfig -> typed ConfigMixin

* CHANGETRACKING - Add ChangeTracking mixin

* CHANGETRACKING - update utils  using the new TrackChanges mixin

* CHANGETRACKING - fix bypass pydantic validation for ChangeTrackingMixin fields

* DOWNST REFACTOR - remove ad-hoc config manipulation in _build_detector

* DOWNST REFACTOR - str concatenation -> pathlib Path

* DOWNST REFACTOR - fix more str concatenations -> pathlib Path

* Update ConfigMixin: add aliasing system (#3211)

* Merge main (version 3.0)

* linting

* FINALIZE INFRASTRUCTURE: Pydantic BaseModel, simplified DLCBaseConfig, field validation

This commit introduces several refactors and simplifactions:

1. Settle with Pydantic BaseModel (no OmegaConf or Pydantic dataclasses, or Mixins)
2. Introduce a separate PoseMetadataConfig, rather than using a ProjectConfig as metadata field on the PoseConfig
3. Add a validation suite for field validators e.g. BodypartPair, UniqueStrList, etc
4. Clean up / update the schemas where necessary (e.g. default values.)
5. Better normalization / serializion for tuples, lists, ndArrays (WeightInitialization)
6. Simplify change tracking on the DLCBaseConfig
7. Simplify versioning migration: only `from_dict`, not on the full class
8. remove duplicate fields and add linting rule
9. add BaseConfig `set_nested` method
10. ProjectConfig: add convenience repair project path

* CLEANUP: Linting and fixes

Apply a set of small fixes and cleanup changes across multiple modules:
- fix Path/string handling in weight_init
- remove many redundant `pass` stubs in abstract classes and placeholders
- adjust loop/range usages (camera calibration, tracklets, DataFrame index)
- improve legacy-argument handling in Loader (DeprecationWarning with stacklevel, error messages) and infer model config path
- reorder and add/remove imports where appropriate
- avoid silently swallowing exceptions after printing (materialize, make_labeled_video)
- use tuple form for startswith check in auxiliary functions
- tweak test random.sample range; and other minor formatting/consistency tweaks. These changes are intended to improve correctness, clarity, and maintainability without altering core behavior.
- Prevent notes on missing fields.
- Fix yaml comments for nested config fields.
- fix validation for None-type read_config_as_dict.
- VersionedConfigs use default factories for PrivateAttrs.
- add missing snapshot_prefix to runner (SA inference requires it).
- fix some faulty default values for modelzoo config templates

* HOOK UP: canonically build structured configs in all main API

1. Refactor PoseConfig: separate modules and add canonical build method.

- Split enums into separate file
- Split PoseMetadata into separate file
- Split PafParamters into separate file
- Add canonical build methods for structured configs: e.g. build from project config

2. Refactor make_pose_config:

- build entire default dict for net_type from the yaml file, before validation.
- deprecate `make_pytorch_test_config`, `make_pytorch_pose_config` and `make_basic_project_config`.

3. Refactor modelzoo - superanimal configs construction via build method

* TESTING: fix tests and add new tests

1. Add tests for centralized config edge cases

Introduce tests/core/config/test_config_breakage.py covering pathological cases for the centralized config model. Tests exercise in-place nested mutation validation (xfail), dirty-state isolation between instances, change-note handling and alias-to-canonical mapping, nested YAML comments (xfail), and normalization/serialization of nested models containing Path and Enum values. Uses ProjectConfig, DLCBaseConfig and DLCVersionedConfig to assert expected dirty-tracking, logging, validation, and YAML output behaviors. add test for cross-field validation with bulk updates

2. fix invalid test imaug (faulty fixture) see DeepLabCut/UnitTestData#4

3. Update testscript_pytorch_multi_animal: ctd_conditions required for ctd shuffle.

The documented primary workflow is: set conditions at shuffle / dataset creation, not defer until inference. This was not reflected in the test, and surfaced with the new structured configs.

This commit makes the test in line with recommended workflow in API docstring; Config docs; BUCTD COLAB notebook; and GUI. -> ctd_conditions are passed at dataset creation time. making the shuffle a fully complete artefact, rather than somthing that requires patching in later API calls.

4. fix testscript multianimal transfer learning pass net_type when creating training dataset

5. add tests:  resume training from existing snapshot

add tests for pose config creation

refactor test_pose_config_creation (test cases for migration to typed)

update xfail for cross fields overrides

remove comment tests

fix test_apis_training

make migrations fixtures to fix registry pollution

use tmp instead of tempfile

tests/core/config: catch expected DeprecationWarnings for aliases

move TEST_DATA_DIR to fixture

use centralized yaml loader instead of pyyaml

update test_modify_train_test_cfg_sets_expected_values

update documentation and tests for write_project_config

update test_pose_config_creation copy dict instead of mutate

update test_pose_config_creation, use centralized yaml loader

add test for ProjectConfig bodypartslist

update test_pose_config_creation separate testing of saving

* CLEANUP: top-level imports and minor fixes
expose get_yaml_loader get_yaml_dumper in core.config

* HOOKUP: expanded use of typed configs in API functions and deprecate old API

- export.py
- deprecate update_config_by_dotpath and update_config
- simplified schema DetectorDataConfig instead of DataConfig

use centralized read_config in trainingsetmanipulation

hook up trainsetmanipulation.py to typed ProjectConfig

update callers of edit_config to use typed updates instead

* CLEANUP: imports and fixes.

Notable:
revert  default `box_score_thresh` in `DetectorModelConfig`
currently a different box_score_thresh is used in inference (0.01) and evaluation (0.6), putting a default value in the config would change behavior.

move deprecation.py from utils to core

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Arash Sal Moslehian <arashsm79@yahoo.com>
Co-authored-by: Cyril Achard <cyril.achard@epfl.ch>