(deeplabcut) C:\Users\samador>python -m deeplabcut
Loading DLC 3.0.0rc13...
Starting GUI...
QPixmap::scaled: Pixmap is a null pixmap
Loading DLC 3.0.0rc13...
Shiboken::Conversions::_pythonToCppCopy: Cannot copy-convert 00007FF9B5436740 (NoneType) to C++.
Shiboken::Conversions::_pythonToCppCopy: Cannot copy-convert 00007FF9B5436740 (NoneType) to C++.
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 183/183 [00:02<00:00, 70.21it/s]
WARNING: DirectWrite: CreateFontFaceFromHDC() failed (Indicates an error in an input file such as a font file.) for QFontDef(Family="MS Sans Serif", pointsize=12, pixelsize=16, styleHint=5, weight=400, stretch=100, hintingPreference=0) LOGFONT("MS Sans Serif", lfWidth=0, lfHeight=-16) dpi=96
WARNING:vispy:DirectWrite: CreateFontFaceFromHDC() failed (Indicates an error in an input file such as a font file.) for QFontDef(Family="MS Sans Serif", pointsize=12, pixelsize=16, styleHint=5, weight=400, stretch=100, hintingPreference=0) LOGFONT("MS Sans Serif", lfWidth=0, lfHeight=-16) dpi=96
INFO:console:Creating images with labels by SAK.
INFO:console:If all the labels are ok, then use the function 'create_training_dataset' to create the training dataset!
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 173/173 [00:00<00:00, 3637.13it/s]
INFO:console:Utilizing the following graph:
INFO:console:
INFO:console:[[0, 1], [0, 2], [0, 3], [0, 4], [0, 5], [0, 6], [0, 7], [0, 8], [0, 9], [0, 10], [1, 2], [1, 3], [1, 4], [1, 5], [1, 6], [1, 7], [1, 8], [1, 9], [1, 10], [2, 3], [2, 4], [2, 5], [2, 6], [2, 7], [2, 8], [2, 9], [2, 10], [3, 4], [3, 5], [3, 6], [3, 7], [3, 8], [3, 9], [3, 10], [4, 5], [4, 6], [4, 7], [4, 8], [4, 9], [4, 10], [5, 6], [5, 7], [5, 8], [5, 9], [5, 10], [6, 7], [6, 8], [6, 9], [6, 10], [7, 8], [7, 9], [7, 10], [8, 9], [8, 10], [9, 10]]
INFO:console:Creating training data for: Shuffle:
INFO:console:
INFO:console:1
INFO:console:
INFO:console:TrainFraction:
INFO:console:
INFO:console:0.95
INFO:console:The training dataset is successfully created. Use the function 'train_network' to start training. Happy training!
INFO:console:Training dataset successfully created.
Training with configuration:
data:
bbox_margin: 20
colormode: RGB
inference:
normalize_images: True
train:
affine:
p: 0.5
rotation: 30
scaling: [0.5, 1.25]
translation: 0
crop_sampling:
width: 448
height: 448
max_shift: 0.1
method: hybrid
gaussian_noise: 12.75
motion_blur: True
normalize_images: True
device: auto
inference:
multithreading:
enabled: True
queue_length: 4
timeout: 30.0
compile:
enabled: False
backend: inductor
autocast:
enabled: False
metadata:
project_path: C:\Users\samador\Documents\DEEPLABCUT\Sand_Summer_2025-SAK-2026-03-28-SAK-2026-03-28
pose_config_path: C:\Users\samador\Documents\DEEPLABCUT\Sand_Summer_2025-SAK-2026-03-28-SAK-2026-03-28\dlc-models-pytorch\iteration-0\Sand_Summer_2025-SAK-2026-03-28Mar28-trainset95shuffle1\train\pytorch_config.yaml
bodyparts: ['COM', 'head', 'caudal_end', 'L1', 'L2', 'L3', 'L4', 'R4', 'R3', 'R2', 'R1']
unique_bodyparts: []
individuals: ['spider_1']
with_identity: False
method: bu
model:
backbone:
type: ResNet
model_name: resnet50_gn
output_stride: 16
freeze_bn_stats: False
freeze_bn_weights: False
backbone_output_channels: 2048
heads:
bodypart:
type: DLCRNetHead
predictor:
type: PartAffinityFieldPredictor
num_animals: 1
num_multibodyparts: 11
num_uniquebodyparts: 0
nms_radius: 5
sigma: 1.0
locref_stdev: 7.2801
min_affinity: 0.05
graph: [[0, 1], [0, 2], [0, 3], [0, 4], [0, 5], [0, 6], [0, 7], [0, 8], [0, 9], [0, 10], [1, 2], [1, 3], [1, 4], [1, 5], [1, 6], [1, 7], [1, 8], [1, 9], [1, 10], [2, 3], [2, 4], [2, 5], [2, 6], [2, 7], [2, 8], [2, 9], [2, 10], [3, 4], [3, 5], [3, 6], [3, 7], [3, 8], [3, 9], [3, 10], [4, 5], [4, 6], [4, 7], [4, 8], [4, 9], [4, 10], [5, 6], [5, 7], [5, 8], [5, 9], [5, 10], [6, 7], [6, 8], [6, 9], [6, 10], [7, 8], [7, 9], [7, 10], [8, 9], [8, 10], [9, 10]]
edges_to_keep: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54]
apply_sigmoid: True
clip_scores: False
target_generator:
type: SequentialGenerator
generators: [{'type': 'HeatmapPlateauGenerator', 'num_heatmaps': 11, 'pos_dist_thresh': 17, 'heatmap_mode': 'KEYPOINT', 'gradient_masking': False, 'generate_locref': True, 'locref_std': 7.2801}, {'type': 'PartAffinityFieldGenerator', 'graph': [[0, 1], [0, 2], [0, 3], [0, 4], [0, 5], [0, 6], [0, 7], [0, 8], [0, 9], [0, 10], [1, 2], [1, 3], [1, 4], [1, 5], [1, 6], [1, 7], [1, 8], [1, 9], [1, 10], [2, 3], [2, 4], [2, 5], [2, 6], [2, 7], [2, 8], [2, 9], [2, 10], [3, 4], [3, 5], [3, 6], [3, 7], [3, 8], [3, 9], [3, 10], [4, 5], [4, 6], [4, 7], [4, 8], [4, 9], [4, 10], [5, 6], [5, 7], [5, 8], [5, 9], [5, 10], [6, 7], [6, 8], [6, 9], [6, 10], [7, 8], [7, 9], [7, 10], [8, 9], [8, 10], [9, 10]], 'width': 20}]
criterion:
heatmap:
type: WeightedBCECriterion
weight: 1.0
locref:
type: WeightedHuberCriterion
weight: 0.05
paf:
type: WeightedHuberCriterion
weight: 0.1
heatmap_config:
channels: [2048, 11]
kernel_size: [3]
strides: [2]
locref_config:
channels: [2048, 22]
kernel_size: [3]
strides: [2]
paf_config:
channels: [2048, 110]
kernel_size: [3]
strides: [2]
num_stages: 5
net_type: resnet_50
runner:
type: PoseTrainingRunner
gpus: None
key_metric: test.mAP
key_metric_asc: True
eval_interval: 10
optimizer:
type: AdamW
params:
lr: 0.0005
scheduler:
type: LRListScheduler
params:
lr_list: [[0.0001], [1e-05]]
milestones: [90, 120]
snapshots:
max_snapshots: 5
save_epochs: 50
save_optimizer_state: False
train_settings:
batch_size: 8
dataloader_workers: 0
dataloader_pin_memory: False
display_iters: 1000
epochs: 200
seed: 42
Loading pretrained weights from Hugging Face hub (timm/resnet50_gn.a1h_in1k)
HTTP Request: HEAD https://huggingface.co/timm/resnet50_gn.a1h_in1k/resolve/main/model.safetensors "HTTP/1.1 302 Found"
[timm/resnet50_gn.a1h_in1k] Safe alternative available for 'pytorch_model.bin' (as 'model.safetensors'). Loading weights using safetensors.
Data Transforms:
Training: Compose([
Affine(always_apply=False, p=0.5, interpolation=1, mask_interpolation=0, cval=0, mode=0, scale={'x': (0.5, 1.25), 'y': (0.5, 1.25)}, translate_percent=None, translate_px={'x': (0, 0), 'y': (0, 0)}, rotate=(-30, 30), fit_output=False, shear={'x': (0.0, 0.0), 'y': (0.0, 0.0)}, cval_mask=0, keep_ratio=True, rotate_method='largest_box'),
PadIfNeeded(always_apply=True, p=1.0, min_height=448, min_width=448, pad_height_divisor=None, pad_width_divisor=None, position=PositionType.CENTER, border_mode=0, value=None, mask_value=None),
KeypointAwareCrop(always_apply=True, p=1.0, width=448, height=448, max_shift=0.1, crop_sampling='hybrid'),
MotionBlur(always_apply=False, p=0.5, blur_limit=(3, 7), allow_shifted=True),
GaussNoise(always_apply=False, p=0.5, var_limit=(0, 162.5625), per_channel=True, mean=0),
Normalize(always_apply=False, p=1.0, mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225], max_pixel_value=255.0),
], p=1.0, bbox_params={'format': 'coco', 'label_fields': ['bbox_labels'], 'min_area': 0.0, 'min_visibility': 0.0, 'min_width': 0.0, 'min_height': 0.0, 'check_each_transform': True}, keypoint_params={'format': 'xy', 'label_fields': ['class_labels'], 'remove_invisible': False, 'angle_in_degrees': True, 'check_each_transform': True}, additional_targets={}, is_check_shapes=True)
Validation: Compose([
Normalize(always_apply=False, p=1.0, mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225], max_pixel_value=255.0),
], p=1.0, bbox_params={'format': 'coco', 'label_fields': ['bbox_labels'], 'min_area': 0.0, 'min_visibility': 0.0, 'min_width': 0.0, 'min_height': 0.0, 'check_each_transform': True}, keypoint_params={'format': 'xy', 'label_fields': ['class_labels'], 'remove_invisible': False, 'angle_in_degrees': True, 'check_each_transform': True}, additional_targets={}, is_check_shapes=True)
Using 173 images and 10 for testing
Starting pose model training...
--------------------------------------------------
Epoch 1/200 (lr=0.0005), train loss 0.04817, GPU: 4408.0/4095.5 MiB
Epoch 2/200 (lr=0.0005), train loss 0.01571, GPU: 4408.0/4095.5 MiB
Epoch 3/200 (lr=0.0005), train loss 0.01445, GPU: 4408.0/4095.5 MiB
Epoch 4/200 (lr=0.0005), train loss 0.01272, GPU: 4408.0/4095.5 MiB
Epoch 5/200 (lr=0.0005), train loss 0.01025, GPU: 4408.0/4095.5 MiB
Epoch 6/200 (lr=0.0005), train loss 0.00834, GPU: 4408.0/4095.5 MiB
Epoch 7/200 (lr=0.0005), train loss 0.00679, GPU: 4408.0/4095.5 MiB
Epoch 8/200 (lr=0.0005), train loss 0.00553, GPU: 4408.0/4095.5 MiB
Epoch 9/200 (lr=0.0005), train loss 0.00497, GPU: 4408.0/4095.5 MiB
Training for epoch 10 done, starting evaluation
Epoch 10/200 (lr=0.0005), train loss 0.00416, valid loss 0.04771, GPU: 4408.0/4095.5 MiB
Model performance:
metrics/test.rmse: 110.52
metrics/test.rmse_pcutoff: nan
metrics/test.mAP: 0.00
metrics/test.mAR: 0.00
Epoch 11/200 (lr=0.0005), train loss 0.00368, GPU: 4408.0/4095.5 MiB
Epoch 12/200 (lr=0.0005), train loss 0.00360, GPU: 4408.0/4095.5 MiB
Epoch 13/200 (lr=0.0005), train loss 0.00324, GPU: 4408.0/4095.5 MiB
Epoch 14/200 (lr=0.0005), train loss 0.00310, GPU: 4408.0/4095.5 MiB
Epoch 15/200 (lr=0.0005), train loss 0.00274, GPU: 4408.0/4095.5 MiB
Epoch 16/200 (lr=0.0005), train loss 0.00296, GPU: 4408.0/4095.5 MiB
Epoch 17/200 (lr=0.0005), train loss 0.00273, GPU: 4408.0/4095.5 MiB
Epoch 18/200 (lr=0.0005), train loss 0.00265, GPU: 4408.0/4095.5 MiB
Epoch 19/200 (lr=0.0005), train loss 0.00242, GPU: 4408.0/4095.5 MiB
Training for epoch 20 done, starting evaluation
Epoch 20/200 (lr=0.0005), train loss 0.00216, valid loss 0.03662, GPU: 4408.0/4095.5 MiB
Model performance:
metrics/test.rmse: 65.49
metrics/test.rmse_pcutoff: 5.12
metrics/test.mAP: 14.52
metrics/test.mAR: 18.00
Epoch 21/200 (lr=0.0005), train loss 0.00214, GPU: 4408.0/4095.5 MiB
Epoch 22/200 (lr=0.0005), train loss 0.00226, GPU: 4408.0/4095.5 MiB
Epoch 23/200 (lr=0.0005), train loss 0.00198, GPU: 4408.0/4095.5 MiB
Epoch 24/200 (lr=0.0005), train loss 0.00180, GPU: 4408.0/4095.5 MiB
Epoch 25/200 (lr=0.0005), train loss 0.00180, GPU: 4408.0/4095.5 MiB
Epoch 26/200 (lr=0.0005), train loss 0.00182, GPU: 4408.0/4095.5 MiB
Epoch 27/200 (lr=0.0005), train loss 0.00162, GPU: 4408.0/4095.5 MiB
Epoch 28/200 (lr=0.0005), train loss 0.00164, GPU: 4408.0/4095.5 MiB
Epoch 29/200 (lr=0.0005), train loss 0.00158, GPU: 4408.0/4095.5 MiB
Training for epoch 30 done, starting evaluation
Epoch 30/200 (lr=0.0005), train loss 0.00149, valid loss 0.03653, GPU: 4408.0/4095.5 MiB
Model performance:
metrics/test.rmse: 52.33
metrics/test.rmse_pcutoff: 5.91
metrics/test.mAP: 23.05
metrics/test.mAR: 26.00
Epoch 31/200 (lr=0.0005), train loss 0.00167, GPU: 4408.0/4095.5 MiB
Epoch 32/200 (lr=0.0005), train loss 0.00144, GPU: 4408.0/4095.5 MiB
Epoch 33/200 (lr=0.0005), train loss 0.00155, GPU: 4408.0/4095.5 MiB
Epoch 34/200 (lr=0.0005), train loss 0.00137, GPU: 4408.0/4095.5 MiB
Epoch 35/200 (lr=0.0005), train loss 0.00129, GPU: 4408.0/4095.5 MiB
Epoch 36/200 (lr=0.0005), train loss 0.00139, GPU: 4408.0/4095.5 MiB
Epoch 37/200 (lr=0.0005), train loss 0.00135, GPU: 4408.0/4095.5 MiB
Epoch 38/200 (lr=0.0005), train loss 0.00136, GPU: 4408.0/4095.5 MiB
Epoch 39/200 (lr=0.0005), train loss 0.00129, GPU: 4408.0/4095.5 MiB
Training for epoch 40 done, starting evaluation
Epoch 40/200 (lr=0.0005), train loss 0.00131, valid loss 0.03071, GPU: 4408.0/4095.5 MiB
Model performance:
metrics/test.rmse: 19.98
metrics/test.rmse_pcutoff: 3.98
metrics/test.mAP: 66.85
metrics/test.mAR: 68.00
Epoch 41/200 (lr=0.0005), train loss 0.00114, GPU: 4408.0/4095.5 MiB
Epoch 42/200 (lr=0.0005), train loss 0.00121, GPU: 4408.0/4095.5 MiB
Epoch 43/200 (lr=0.0005), train loss 0.00116, GPU: 4408.0/4095.5 MiB
Epoch 44/200 (lr=0.0005), train loss 0.00120, GPU: 4408.0/4095.5 MiB
Epoch 45/200 (lr=0.0005), train loss 0.00118, GPU: 4408.0/4095.5 MiB
Epoch 46/200 (lr=0.0005), train loss 0.00115, GPU: 4408.0/4095.5 MiB
Epoch 47/200 (lr=0.0005), train loss 0.00106, GPU: 4408.0/4095.5 MiB
Epoch 48/200 (lr=0.0005), train loss 0.00106, GPU: 4408.0/4095.5 MiB
Epoch 49/200 (lr=0.0005), train loss 0.00102, GPU: 4408.0/4095.5 MiB
Training for epoch 50 done, starting evaluation
Epoch 50/200 (lr=0.0005), train loss 0.00105, valid loss 0.02666, GPU: 4408.0/4095.5 MiB
Model performance:
metrics/test.rmse: 13.30
metrics/test.rmse_pcutoff: 3.52
metrics/test.mAP: 79.78
metrics/test.mAR: 83.00
Epoch 51/200 (lr=0.0005), train loss 0.00110, GPU: 4408.0/4095.5 MiB
Epoch 52/200 (lr=0.0005), train loss 0.00108, GPU: 4408.0/4095.5 MiB
Epoch 53/200 (lr=0.0005), train loss 0.00109, GPU: 4408.0/4095.5 MiB
Epoch 54/200 (lr=0.0005), train loss 0.00110, GPU: 4408.0/4095.5 MiB
Epoch 55/200 (lr=0.0005), train loss 0.00105, GPU: 4408.0/4095.5 MiB
Epoch 56/200 (lr=0.0005), train loss 0.00095, GPU: 4408.0/4095.5 MiB
Epoch 57/200 (lr=0.0005), train loss 0.00106, GPU: 4408.0/4095.5 MiB
Epoch 58/200 (lr=0.0005), train loss 0.00114, GPU: 4408.0/4095.5 MiB
Epoch 59/200 (lr=0.0005), train loss 0.00101, GPU: 4408.0/4095.5 MiB
Training for epoch 60 done, starting evaluation
Epoch 60/200 (lr=0.0005), train loss 0.00097, valid loss 0.02824, GPU: 4408.0/4095.5 MiB
Model performance:
metrics/test.rmse: 20.04
metrics/test.rmse_pcutoff: 4.18
metrics/test.mAP: 71.17
metrics/test.mAR: 75.00
Epoch 61/200 (lr=0.0005), train loss 0.00096, GPU: 4408.0/4095.5 MiB
Epoch 62/200 (lr=0.0005), train loss 0.00093, GPU: 4408.0/4095.5 MiB
Epoch 63/200 (lr=0.0005), train loss 0.00099, GPU: 4408.0/4095.5 MiB
Epoch 64/200 (lr=0.0005), train loss 0.00098, GPU: 4408.0/4095.5 MiB
Epoch 65/200 (lr=0.0005), train loss 0.00100, GPU: 4408.0/4095.5 MiB
Epoch 66/200 (lr=0.0005), train loss 0.00092, GPU: 4408.0/4095.5 MiB
Epoch 67/200 (lr=0.0005), train loss 0.00091, GPU: 4408.0/4095.5 MiB
Epoch 68/200 (lr=0.0005), train loss 0.00098, GPU: 4408.0/4095.5 MiB
Epoch 69/200 (lr=0.0005), train loss 0.00099, GPU: 4408.0/4095.5 MiB
Training for epoch 70 done, starting evaluation
Epoch 70/200 (lr=0.0005), train loss 0.00090, valid loss 0.02239, GPU: 4408.0/4095.5 MiB
Model performance:
metrics/test.rmse: 4.27
metrics/test.rmse_pcutoff: 3.93
metrics/test.mAP: 98.61
metrics/test.mAR: 99.00
Epoch 71/200 (lr=0.0005), train loss 0.00093, GPU: 4408.0/4095.5 MiB
Epoch 72/200 (lr=0.0005), train loss 0.00087, GPU: 4408.0/4095.5 MiB
Epoch 73/200 (lr=0.0005), train loss 0.00087, GPU: 4408.0/4095.5 MiB
Epoch 74/200 (lr=0.0005), train loss 0.00082, GPU: 4408.0/4095.5 MiB
Epoch 75/200 (lr=0.0005), train loss 0.00086, GPU: 4408.0/4095.5 MiB
Epoch 76/200 (lr=0.0005), train loss 0.00089, GPU: 4408.0/4095.5 MiB
Epoch 77/200 (lr=0.0005), train loss 0.00086, GPU: 4408.0/4095.5 MiB
Epoch 78/200 (lr=0.0005), train loss 0.00104, GPU: 4408.0/4095.5 MiB
Epoch 79/200 (lr=0.0005), train loss 0.00111, GPU: 4408.0/4095.5 MiB
Training for epoch 80 done, starting evaluation
Epoch 80/200 (lr=0.0005), train loss 0.00095, valid loss 0.01714, GPU: 4408.0/4095.5 MiB
Model performance:
metrics/test.rmse: 4.45
metrics/test.rmse_pcutoff: 4.34
metrics/test.mAP: 98.61
metrics/test.mAR: 99.00
Epoch 81/200 (lr=0.0005), train loss 0.00088, GPU: 4408.0/4095.5 MiB
Epoch 82/200 (lr=0.0005), train loss 0.00087, GPU: 4408.0/4095.5 MiB
Epoch 83/200 (lr=0.0005), train loss 0.00089, GPU: 4408.0/4095.5 MiB
Epoch 84/200 (lr=0.0005), train loss 0.00083, GPU: 4408.0/4095.5 MiB
Epoch 85/200 (lr=0.0005), train loss 0.00077, GPU: 4408.0/4095.5 MiB
Epoch 86/200 (lr=0.0005), train loss 0.00085, GPU: 4408.0/4095.5 MiB
Epoch 87/200 (lr=0.0005), train loss 0.00085, GPU: 4408.0/4095.5 MiB
Epoch 88/200 (lr=0.0005), train loss 0.00084, GPU: 4408.0/4095.5 MiB
Epoch 89/200 (lr=0.0005), train loss 0.00085, GPU: 4408.0/4095.5 MiB
Training for epoch 90 done, starting evaluation
Epoch 90/200 (lr=0.0001), train loss 0.00081, valid loss 0.01781, GPU: 4408.0/4095.5 MiB
Model performance:
metrics/test.rmse: 4.53
metrics/test.rmse_pcutoff: 4.19
metrics/test.mAP: 97.03
metrics/test.mAR: 98.00
Epoch 91/200 (lr=0.0001), train loss 0.00071, GPU: 4408.0/4095.5 MiB
Epoch 92/200 (lr=0.0001), train loss 0.00066, GPU: 4408.0/4095.5 MiB
Epoch 93/200 (lr=0.0001), train loss 0.00063, GPU: 4408.0/4095.5 MiB
Epoch 94/200 (lr=0.0001), train loss 0.00064, GPU: 4408.0/4095.5 MiB
Epoch 95/200 (lr=0.0001), train loss 0.00067, GPU: 4408.0/4095.5 MiB
Epoch 96/200 (lr=0.0001), train loss 0.00062, GPU: 4408.0/4095.5 MiB
Epoch 97/200 (lr=0.0001), train loss 0.00058, GPU: 4408.0/4095.5 MiB
Epoch 98/200 (lr=0.0001), train loss 0.00060, GPU: 4408.0/4095.5 MiB
Epoch 99/200 (lr=0.0001), train loss 0.00061, GPU: 4408.0/4095.5 MiB
Training for epoch 100 done, starting evaluation
Epoch 100/200 (lr=0.0001), train loss 0.00062, valid loss 0.02061, GPU: 4408.0/4095.5 MiB
Model performance:
metrics/test.rmse: 4.20
metrics/test.rmse_pcutoff: 3.85
metrics/test.mAP: 98.42
metrics/test.mAR: 99.00
Epoch 101/200 (lr=0.0001), train loss 0.00061, GPU: 4408.0/4095.5 MiB
Epoch 102/200 (lr=0.0001), train loss 0.00060, GPU: 4408.0/4095.5 MiB
Epoch 103/200 (lr=0.0001), train loss 0.00063, GPU: 4408.0/4095.5 MiB
Epoch 104/200 (lr=0.0001), train loss 0.00061, GPU: 4408.0/4095.5 MiB
Epoch 105/200 (lr=0.0001), train loss 0.00061, GPU: 4408.0/4095.5 MiB
Epoch 106/200 (lr=0.0001), train loss 0.00062, GPU: 4408.0/4095.5 MiB
Epoch 107/200 (lr=0.0001), train loss 0.00061, GPU: 4408.0/4095.5 MiB
Epoch 108/200 (lr=0.0001), train loss 0.00057, GPU: 4408.0/4095.5 MiB
Epoch 109/200 (lr=0.0001), train loss 0.00057, GPU: 4408.0/4095.5 MiB
Training for epoch 110 done, starting evaluation
Epoch 110/200 (lr=0.0001), train loss 0.00062, valid loss 0.02231, GPU: 4408.0/4095.5 MiB
Model performance:
metrics/test.rmse: 4.49
metrics/test.rmse_pcutoff: 4.04
metrics/test.mAP: 96.63
metrics/test.mAR: 98.00
Epoch 111/200 (lr=0.0001), train loss 0.00060, GPU: 4408.0/4095.5 MiB
Epoch 112/200 (lr=0.0001), train loss 0.00060, GPU: 4408.0/4095.5 MiB
Epoch 113/200 (lr=0.0001), train loss 0.00058, GPU: 4408.0/4095.5 MiB
Epoch 114/200 (lr=0.0001), train loss 0.00060, GPU: 4408.0/4095.5 MiB
Epoch 115/200 (lr=0.0001), train loss 0.00058, GPU: 4408.0/4095.5 MiB
Epoch 116/200 (lr=0.0001), train loss 0.00060, GPU: 4408.0/4095.5 MiB
Epoch 117/200 (lr=0.0001), train loss 0.00051, GPU: 4408.0/4095.5 MiB
Epoch 118/200 (lr=0.0001), train loss 0.00058, GPU: 4408.0/4095.5 MiB
Epoch 119/200 (lr=0.0001), train loss 0.00061, GPU: 4408.0/4095.5 MiB
Training for epoch 120 done, starting evaluation
Epoch 120/200 (lr=1e-05), train loss 0.00059, valid loss 0.02172, GPU: 4408.0/4095.5 MiB
Model performance:
metrics/test.rmse: 4.36
metrics/test.rmse_pcutoff: 3.89
metrics/test.mAP: 96.42
metrics/test.mAR: 98.00
Epoch 121/200 (lr=1e-05), train loss 0.00056, GPU: 4408.0/4095.5 MiB
Epoch 122/200 (lr=1e-05), train loss 0.00052, GPU: 4408.0/4095.5 MiB
Epoch 123/200 (lr=1e-05), train loss 0.00054, GPU: 4408.0/4095.5 MiB
Epoch 124/200 (lr=1e-05), train loss 0.00056, GPU: 4408.0/4095.5 MiB
Epoch 125/200 (lr=1e-05), train loss 0.00053, GPU: 4408.0/4095.5 MiB
Epoch 126/200 (lr=1e-05), train loss 0.00053, GPU: 4408.0/4095.5 MiB
Epoch 127/200 (lr=1e-05), train loss 0.00055, GPU: 4408.0/4095.5 MiB
Epoch 128/200 (lr=1e-05), train loss 0.00052, GPU: 4408.0/4095.5 MiB
Epoch 129/200 (lr=1e-05), train loss 0.00051, GPU: 4408.0/4095.5 MiB
Training for epoch 130 done, starting evaluation
Epoch 130/200 (lr=1e-05), train loss 0.00052, valid loss 0.02294, GPU: 4408.0/4095.5 MiB
Model performance:
metrics/test.rmse: 4.38
metrics/test.rmse_pcutoff: 3.89
metrics/test.mAP: 96.42
metrics/test.mAR: 98.00
Epoch 131/200 (lr=1e-05), train loss 0.00056, GPU: 4408.0/4095.5 MiB
Epoch 132/200 (lr=1e-05), train loss 0.00053, GPU: 4408.0/4095.5 MiB
Epoch 133/200 (lr=1e-05), train loss 0.00055, GPU: 4408.0/4095.5 MiB
Epoch 134/200 (lr=1e-05), train loss 0.00051, GPU: 4408.0/4095.5 MiB
Epoch 135/200 (lr=1e-05), train loss 0.00055, GPU: 4408.0/4095.5 MiB
Epoch 136/200 (lr=1e-05), train loss 0.00055, GPU: 4408.0/4095.5 MiB
Epoch 137/200 (lr=1e-05), train loss 0.00052, GPU: 4408.0/4095.5 MiB
Epoch 138/200 (lr=1e-05), train loss 0.00054, GPU: 4408.0/4095.5 MiB
Epoch 139/200 (lr=1e-05), train loss 0.00057, GPU: 4408.0/4095.5 MiB
Training for epoch 140 done, starting evaluation
Epoch 140/200 (lr=1e-05), train loss 0.00053, valid loss 0.02296, GPU: 4408.0/4095.5 MiB
Model performance:
metrics/test.rmse: 5.51
metrics/test.rmse_pcutoff: 3.80
metrics/test.mAP: 94.06
metrics/test.mAR: 96.00
Epoch 141/200 (lr=1e-05), train loss 0.00059, GPU: 4408.0/4095.5 MiB
Epoch 142/200 (lr=1e-05), train loss 0.00052, GPU: 4408.0/4095.5 MiB
Epoch 143/200 (lr=1e-05), train loss 0.00054, GPU: 4408.0/4095.5 MiB
Epoch 144/200 (lr=1e-05), train loss 0.00051, GPU: 4408.0/4095.5 MiB
Epoch 145/200 (lr=1e-05), train loss 0.00051, GPU: 4408.0/4095.5 MiB
Epoch 146/200 (lr=1e-05), train loss 0.00050, GPU: 4408.0/4095.5 MiB
Epoch 147/200 (lr=1e-05), train loss 0.00054, GPU: 4408.0/4095.5 MiB
Epoch 148/200 (lr=1e-05), train loss 0.00051, GPU: 4408.0/4095.5 MiB
Epoch 149/200 (lr=1e-05), train loss 0.00052, GPU: 4408.0/4095.5 MiB
Training for epoch 150 done, starting evaluation
Epoch 150/200 (lr=1e-05), train loss 0.00053, valid loss 0.02303, GPU: 4408.0/4095.5 MiB
Model performance:
metrics/test.rmse: 4.33
metrics/test.rmse_pcutoff: 3.93
metrics/test.mAP: 96.42
metrics/test.mAR: 98.00
Epoch 151/200 (lr=1e-05), train loss 0.00052, GPU: 4408.0/4095.5 MiB
Epoch 152/200 (lr=1e-05), train loss 0.00052, GPU: 4408.0/4095.5 MiB
Epoch 153/200 (lr=1e-05), train loss 0.00054, GPU: 4408.0/4095.5 MiB
Epoch 154/200 (lr=1e-05), train loss 0.00053, GPU: 4408.0/4095.5 MiB
Epoch 155/200 (lr=1e-05), train loss 0.00053, GPU: 4408.0/4095.5 MiB
Epoch 156/200 (lr=1e-05), train loss 0.00053, GPU: 4408.0/4095.5 MiB
Epoch 157/200 (lr=1e-05), train loss 0.00052, GPU: 4408.0/4095.5 MiB
Epoch 158/200 (lr=1e-05), train loss 0.00058, GPU: 4408.0/4095.5 MiB
Epoch 159/200 (lr=1e-05), train loss 0.00049, GPU: 4408.0/4095.5 MiB
Training for epoch 160 done, starting evaluation
Epoch 160/200 (lr=1e-05), train loss 0.00054, valid loss 0.02343, GPU: 4408.0/4095.5 MiB
Model performance:
metrics/test.rmse: 4.39
metrics/test.rmse_pcutoff: 4.02
metrics/test.mAP: 96.42
metrics/test.mAR: 98.00
Epoch 161/200 (lr=1e-05), train loss 0.00055, GPU: 4408.0/4095.5 MiB
Epoch 162/200 (lr=1e-05), train loss 0.00053, GPU: 4408.0/4095.5 MiB
Epoch 163/200 (lr=1e-05), train loss 0.00054, GPU: 4408.0/4095.5 MiB
Epoch 164/200 (lr=1e-05), train loss 0.00052, GPU: 4408.0/4095.5 MiB
Epoch 165/200 (lr=1e-05), train loss 0.00054, GPU: 4408.0/4095.5 MiB
Epoch 166/200 (lr=1e-05), train loss 0.00057, GPU: 4408.0/4095.5 MiB
Epoch 167/200 (lr=1e-05), train loss 0.00055, GPU: 4408.0/4095.5 MiB
Epoch 168/200 (lr=1e-05), train loss 0.00053, GPU: 4408.0/4095.5 MiB
Epoch 169/200 (lr=1e-05), train loss 0.00051, GPU: 4408.0/4095.5 MiB
Training for epoch 170 done, starting evaluation
Epoch 170/200 (lr=1e-05), train loss 0.00054, valid loss 0.02411, GPU: 4408.0/4095.5 MiB
Model performance:
metrics/test.rmse: 5.53
metrics/test.rmse_pcutoff: 3.84
metrics/test.mAP: 94.06
metrics/test.mAR: 96.00
Epoch 171/200 (lr=1e-05), train loss 0.00051, GPU: 4408.0/4095.5 MiB
Epoch 172/200 (lr=1e-05), train loss 0.00051, GPU: 4408.0/4095.5 MiB
Epoch 173/200 (lr=1e-05), train loss 0.00051, GPU: 4408.0/4095.5 MiB
Epoch 174/200 (lr=1e-05), train loss 0.00049, GPU: 4408.0/4095.5 MiB
Epoch 175/200 (lr=1e-05), train loss 0.00051, GPU: 4408.0/4095.5 MiB
Epoch 176/200 (lr=1e-05), train loss 0.00053, GPU: 4408.0/4095.5 MiB
Epoch 177/200 (lr=1e-05), train loss 0.00051, GPU: 4408.0/4095.5 MiB
Epoch 178/200 (lr=1e-05), train loss 0.00055, GPU: 4408.0/4095.5 MiB
Epoch 179/200 (lr=1e-05), train loss 0.00052, GPU: 4408.0/4095.5 MiB
Training for epoch 180 done, starting evaluation
Epoch 180/200 (lr=1e-05), train loss 0.00052, valid loss 0.02420, GPU: 4408.0/4095.5 MiB
Model performance:
metrics/test.rmse: 4.41
metrics/test.rmse_pcutoff: 3.87
metrics/test.mAP: 96.42
metrics/test.mAR: 98.00
Epoch 181/200 (lr=1e-05), train loss 0.00050, GPU: 4408.0/4095.5 MiB
Epoch 182/200 (lr=1e-05), train loss 0.00050, GPU: 4408.0/4095.5 MiB
Epoch 183/200 (lr=1e-05), train loss 0.00052, GPU: 4408.0/4095.5 MiB
Epoch 184/200 (lr=1e-05), train loss 0.00049, GPU: 4408.0/4095.5 MiB
Epoch 185/200 (lr=1e-05), train loss 0.00057, GPU: 4408.0/4095.5 MiB
Epoch 186/200 (lr=1e-05), train loss 0.00051, GPU: 4408.0/4095.5 MiB
Epoch 187/200 (lr=1e-05), train loss 0.00052, GPU: 4408.0/4095.5 MiB
Epoch 188/200 (lr=1e-05), train loss 0.00051, GPU: 4408.0/4095.5 MiB
Epoch 189/200 (lr=1e-05), train loss 0.00057, GPU: 4408.0/4095.5 MiB
Training for epoch 190 done, starting evaluation
Epoch 190/200 (lr=1e-05), train loss 0.00052, valid loss 0.02488, GPU: 4408.0/4095.5 MiB
Model performance:
metrics/test.rmse: 5.58
metrics/test.rmse_pcutoff: 3.88
metrics/test.mAP: 94.06
metrics/test.mAR: 96.00
Epoch 191/200 (lr=1e-05), train loss 0.00056, GPU: 4408.0/4095.5 MiB
Epoch 192/200 (lr=1e-05), train loss 0.00050, GPU: 4408.0/4095.5 MiB
Epoch 193/200 (lr=1e-05), train loss 0.00054, GPU: 4408.0/4095.5 MiB
Epoch 194/200 (lr=1e-05), train loss 0.00054, GPU: 4408.0/4095.5 MiB
Epoch 195/200 (lr=1e-05), train loss 0.00055, GPU: 4408.0/4095.5 MiB
Epoch 196/200 (lr=1e-05), train loss 0.00052, GPU: 4408.0/4095.5 MiB
Epoch 197/200 (lr=1e-05), train loss 0.00051, GPU: 4408.0/4095.5 MiB
Epoch 198/200 (lr=1e-05), train loss 0.00050, GPU: 4408.0/4095.5 MiB
Epoch 199/200 (lr=1e-05), train loss 0.00055, GPU: 4408.0/4095.5 MiB
Training for epoch 200 done, starting evaluation
Epoch 200/200 (lr=1e-05), train loss 0.00047, valid loss 0.02439, GPU: 4408.0/4095.5 MiB
Model performance:
metrics/test.rmse: 5.54
metrics/test.rmse_pcutoff: 3.87
metrics/test.mAP: 94.06
metrics/test.mAR: 96.00
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 173/173 [00:03<00:00, 54.99it/s]
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:00<00:00, 18.20it/s]
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 173/173 [00:02<00:00, 69.68it/s]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:00<00:00, 116.75it/s]
WARNING: QThread::wait: Thread tried to wait on itself
WARNING: QThread::wait: Thread tried to wait on itself
Is there an existing issue for this?
Operating System
Windows 11
DeepLabCut version
v3.0.0rc13
What engine are you using?
pytorch
DeepLabCut mode
multi animal
Device type
11th Gen Intel Core i7-11800H @2.30 GHz, 32. GB RAM
NVIDIA RTX A2000 Laptop GPU
Bug description 🐛
Using the DLC GUI, I created a new project and imported a previously tracked video + its labeled-data. The new config.yaml file has the exact same names for the bodyparts, individual animal, and scorer as for the previous project in which I tracked the video. To check this, I can see the labels in napari just fine.
Using default settings for the next steps, DLC creates a training dataset, trains a network, and evaluates it without errors. However, it does not create a .h5 file when I try to run Analyze Videos.
The error message at the bottom of the GUI says: "No .h5 files were created during video analysis. Please check your code and ensure that the video inference and output generation are correct" The machine labels .h5 is indeed now being created in the labeled-data folder.
I have previously been able to train and run an entire session including generating the videos without problems. Therefore, my DLC installation should be fine.
However, that project somehow got corrupted and started having this issue. We could not figure out how to fix it, so we had to start over using the labeled frames (we hope!)
It seems like it should be possible to start again using the previously labeled frames and video, but something is going wrong. Please help us avoid having to relabel 11 bodyparts in > 180 frames again!
Thanks,
Suzanne
Steps To Reproduce
config.yaml file
relevant log output
Code of Conduct