Skip to content

Project trains, evaluates but does not create .h5 file or analyze videos #3260

Description

@amadorkane

Is there an existing issue for this?

  • I have searched the existing issues

Operating System

Windows 11

DeepLabCut version

v3.0.0rc13

What engine are you using?

pytorch

DeepLabCut mode

multi animal

Device type

11th Gen Intel Core i7-11800H @2.30 GHz, 32. GB RAM
NVIDIA RTX A2000 Laptop GPU

Bug description 🐛

Using the DLC GUI, I created a new project and imported a previously tracked video + its labeled-data. The new config.yaml file has the exact same names for the bodyparts, individual animal, and scorer as for the previous project in which I tracked the video. To check this, I can see the labels in napari just fine.

Using default settings for the next steps, DLC creates a training dataset, trains a network, and evaluates it without errors. However, it does not create a .h5 file when I try to run Analyze Videos.

The error message at the bottom of the GUI says: "No .h5 files were created during video analysis. Please check your code and ensure that the video inference and output generation are correct" The machine labels .h5 is indeed now being created in the labeled-data folder.

I have previously been able to train and run an entire session including generating the videos without problems. Therefore, my DLC installation should be fine.

However, that project somehow got corrupted and started having this issue. We could not figure out how to fix it, so we had to start over using the labeled frames (we hope!)

It seems like it should be possible to start again using the previously labeled frames and video, but something is going wrong. Please help us avoid having to relabel 11 bodyparts in > 180 frames again!
Thanks,
Suzanne

Steps To Reproduce

  1. I ran the DLC GUI after checking the video and project folders, and all other information in the config.yaml file (see below) was correct
  2. Loaded the project
  3. Checked the labels using napari
  4. Created a training dataset
  5. Trained the network
  6. Evaluated it
  7. Ran Analyze Videos
config.yaml file
# Project definitions (do not edit)
Task: Sand_Summer_2025-SAK-2026-03-28
scorer: SAK
date: Mar28
multianimalproject: true
identity: false


# Project path (change when moving around)
project_path: 
  C:\Users\samador\Documents\DEEPLABCUT\Sand_Summer_2025-SAK-2026-03-28-SAK-2026-03-28


# Default DeepLabCut engine to use for shuffle creation (either pytorch or tensorflow)
engine: pytorch


# Annotation data set configuration (and individual video cropping parameters)
video_sets:
  ? C:\Users\samador\Documents\DEEPLABCUT\Sand_Summer_2025-SAK-2026-03-28-SAK-2026-03-28\videos\vid_2025-06-18_14-51-10_croprot.avi
  : crop: 0, 204, 0, 240
individuals:
- spider_1
uniquebodyparts: []
multianimalbodyparts:
- COM
- head
- caudal_end
- L1
- L2
- L3
- L4
- R4
- R3
- R2
- R1
bodyparts: MULTI!


# Fraction of video to start/stop when extracting frames for labeling/refinement
start: 0
stop: 1
numframes2pick: 20


# Plotting configuration
skeleton:
- - bodypart1
  - bodypart2
- - bodypart2
  - bodypart3
- - bodypart1
  - bodypart3
skeleton_color: black
pcutoff: 0.6
dotsize: 12
alphavalue: 0.7
colormap: rainbow


# Training,Evaluation and Analysis configuration
TrainingFraction:
- 0.95
iteration: 0
default_net_type: resnet_50
default_augmenter: albumentations
default_track_method: ellipse
snapshotindex: -1
detector_snapshotindex: -1
batch_size: 8


# Cropping Parameters (for analysis and outlier frame detection)
cropping: false
#if cropping is true for analysis, then set the values here:
x1: 0
x2: 640
y1: 277
y2: 624


# Refinement configuration (parameters from annotation dataset configuration also relevant in this stage)
corner2move2:
- 50
- 50
move2corner: true


# Conversion tables to fine-tune SuperAnimal weights
SuperAnimalConversionTables:
detector_batch_size: 1
relevant log output
(deeplabcut) C:\Users\samador>python -m deeplabcut
Loading DLC 3.0.0rc13...
Starting GUI...
QPixmap::scaled: Pixmap is a null pixmap
Loading DLC 3.0.0rc13...
Shiboken::Conversions::_pythonToCppCopy: Cannot copy-convert 00007FF9B5436740 (NoneType) to C++.
Shiboken::Conversions::_pythonToCppCopy: Cannot copy-convert 00007FF9B5436740 (NoneType) to C++.
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 183/183 [00:02<00:00, 70.21it/s]
WARNING: DirectWrite: CreateFontFaceFromHDC() failed (Indicates an error in an input file such as a font file.) for QFontDef(Family="MS Sans Serif", pointsize=12, pixelsize=16, styleHint=5, weight=400, stretch=100, hintingPreference=0) LOGFONT("MS Sans Serif", lfWidth=0, lfHeight=-16) dpi=96
WARNING:vispy:DirectWrite: CreateFontFaceFromHDC() failed (Indicates an error in an input file such as a font file.) for QFontDef(Family="MS Sans Serif", pointsize=12, pixelsize=16, styleHint=5, weight=400, stretch=100, hintingPreference=0) LOGFONT("MS Sans Serif", lfWidth=0, lfHeight=-16) dpi=96
INFO:console:Creating images with labels by SAK.
INFO:console:If all the labels are ok, then use the function 'create_training_dataset' to create the training dataset!
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 173/173 [00:00<00:00, 3637.13it/s]
INFO:console:Utilizing the following graph:
INFO:console:
INFO:console:[[0, 1], [0, 2], [0, 3], [0, 4], [0, 5], [0, 6], [0, 7], [0, 8], [0, 9], [0, 10], [1, 2], [1, 3], [1, 4], [1, 5], [1, 6], [1, 7], [1, 8], [1, 9], [1, 10], [2, 3], [2, 4], [2, 5], [2, 6], [2, 7], [2, 8], [2, 9], [2, 10], [3, 4], [3, 5], [3, 6], [3, 7], [3, 8], [3, 9], [3, 10], [4, 5], [4, 6], [4, 7], [4, 8], [4, 9], [4, 10], [5, 6], [5, 7], [5, 8], [5, 9], [5, 10], [6, 7], [6, 8], [6, 9], [6, 10], [7, 8], [7, 9], [7, 10], [8, 9], [8, 10], [9, 10]]
INFO:console:Creating training data for: Shuffle:
INFO:console:
INFO:console:1
INFO:console:
INFO:console:TrainFraction:
INFO:console:
INFO:console:0.95
INFO:console:The training dataset is successfully created. Use the function 'train_network' to start training. Happy training!
INFO:console:Training dataset successfully created.
Training with configuration:
data:
  bbox_margin: 20
  colormode: RGB
  inference:
    normalize_images: True
  train:
    affine:
      p: 0.5
      rotation: 30
      scaling: [0.5, 1.25]
      translation: 0
    crop_sampling:
      width: 448
      height: 448
      max_shift: 0.1
      method: hybrid
    gaussian_noise: 12.75
    motion_blur: True
    normalize_images: True
device: auto
inference:
  multithreading:
    enabled: True
    queue_length: 4
    timeout: 30.0
  compile:
    enabled: False
    backend: inductor
  autocast:
    enabled: False
metadata:
  project_path: C:\Users\samador\Documents\DEEPLABCUT\Sand_Summer_2025-SAK-2026-03-28-SAK-2026-03-28
  pose_config_path: C:\Users\samador\Documents\DEEPLABCUT\Sand_Summer_2025-SAK-2026-03-28-SAK-2026-03-28\dlc-models-pytorch\iteration-0\Sand_Summer_2025-SAK-2026-03-28Mar28-trainset95shuffle1\train\pytorch_config.yaml
  bodyparts: ['COM', 'head', 'caudal_end', 'L1', 'L2', 'L3', 'L4', 'R4', 'R3', 'R2', 'R1']
  unique_bodyparts: []
  individuals: ['spider_1']
  with_identity: False
method: bu
model:
  backbone:
    type: ResNet
    model_name: resnet50_gn
    output_stride: 16
    freeze_bn_stats: False
    freeze_bn_weights: False
  backbone_output_channels: 2048
  heads:
    bodypart:
      type: DLCRNetHead
      predictor:
        type: PartAffinityFieldPredictor
        num_animals: 1
        num_multibodyparts: 11
        num_uniquebodyparts: 0
        nms_radius: 5
        sigma: 1.0
        locref_stdev: 7.2801
        min_affinity: 0.05
        graph: [[0, 1], [0, 2], [0, 3], [0, 4], [0, 5], [0, 6], [0, 7], [0, 8], [0, 9], [0, 10], [1, 2], [1, 3], [1, 4], [1, 5], [1, 6], [1, 7], [1, 8], [1, 9], [1, 10], [2, 3], [2, 4], [2, 5], [2, 6], [2, 7], [2, 8], [2, 9], [2, 10], [3, 4], [3, 5], [3, 6], [3, 7], [3, 8], [3, 9], [3, 10], [4, 5], [4, 6], [4, 7], [4, 8], [4, 9], [4, 10], [5, 6], [5, 7], [5, 8], [5, 9], [5, 10], [6, 7], [6, 8], [6, 9], [6, 10], [7, 8], [7, 9], [7, 10], [8, 9], [8, 10], [9, 10]]
        edges_to_keep: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54]
        apply_sigmoid: True
        clip_scores: False
      target_generator:
        type: SequentialGenerator
        generators: [{'type': 'HeatmapPlateauGenerator', 'num_heatmaps': 11, 'pos_dist_thresh': 17, 'heatmap_mode': 'KEYPOINT', 'gradient_masking': False, 'generate_locref': True, 'locref_std': 7.2801}, {'type': 'PartAffinityFieldGenerator', 'graph': [[0, 1], [0, 2], [0, 3], [0, 4], [0, 5], [0, 6], [0, 7], [0, 8], [0, 9], [0, 10], [1, 2], [1, 3], [1, 4], [1, 5], [1, 6], [1, 7], [1, 8], [1, 9], [1, 10], [2, 3], [2, 4], [2, 5], [2, 6], [2, 7], [2, 8], [2, 9], [2, 10], [3, 4], [3, 5], [3, 6], [3, 7], [3, 8], [3, 9], [3, 10], [4, 5], [4, 6], [4, 7], [4, 8], [4, 9], [4, 10], [5, 6], [5, 7], [5, 8], [5, 9], [5, 10], [6, 7], [6, 8], [6, 9], [6, 10], [7, 8], [7, 9], [7, 10], [8, 9], [8, 10], [9, 10]], 'width': 20}]
      criterion:
        heatmap:
          type: WeightedBCECriterion
          weight: 1.0
        locref:
          type: WeightedHuberCriterion
          weight: 0.05
        paf:
          type: WeightedHuberCriterion
          weight: 0.1
      heatmap_config:
        channels: [2048, 11]
        kernel_size: [3]
        strides: [2]
      locref_config:
        channels: [2048, 22]
        kernel_size: [3]
        strides: [2]
      paf_config:
        channels: [2048, 110]
        kernel_size: [3]
        strides: [2]
      num_stages: 5
net_type: resnet_50
runner:
  type: PoseTrainingRunner
  gpus: None
  key_metric: test.mAP
  key_metric_asc: True
  eval_interval: 10
  optimizer:
    type: AdamW
    params:
      lr: 0.0005
  scheduler:
    type: LRListScheduler
    params:
      lr_list: [[0.0001], [1e-05]]
      milestones: [90, 120]
  snapshots:
    max_snapshots: 5
    save_epochs: 50
    save_optimizer_state: False
train_settings:
  batch_size: 8
  dataloader_workers: 0
  dataloader_pin_memory: False
  display_iters: 1000
  epochs: 200
  seed: 42
Loading pretrained weights from Hugging Face hub (timm/resnet50_gn.a1h_in1k)
HTTP Request: HEAD https://huggingface.co/timm/resnet50_gn.a1h_in1k/resolve/main/model.safetensors "HTTP/1.1 302 Found"
[timm/resnet50_gn.a1h_in1k] Safe alternative available for 'pytorch_model.bin' (as 'model.safetensors'). Loading weights using safetensors.
Data Transforms:
  Training:   Compose([
  Affine(always_apply=False, p=0.5, interpolation=1, mask_interpolation=0, cval=0, mode=0, scale={'x': (0.5, 1.25), 'y': (0.5, 1.25)}, translate_percent=None, translate_px={'x': (0, 0), 'y': (0, 0)}, rotate=(-30, 30), fit_output=False, shear={'x': (0.0, 0.0), 'y': (0.0, 0.0)}, cval_mask=0, keep_ratio=True, rotate_method='largest_box'),
  PadIfNeeded(always_apply=True, p=1.0, min_height=448, min_width=448, pad_height_divisor=None, pad_width_divisor=None, position=PositionType.CENTER, border_mode=0, value=None, mask_value=None),
  KeypointAwareCrop(always_apply=True, p=1.0, width=448, height=448, max_shift=0.1, crop_sampling='hybrid'),
  MotionBlur(always_apply=False, p=0.5, blur_limit=(3, 7), allow_shifted=True),
  GaussNoise(always_apply=False, p=0.5, var_limit=(0, 162.5625), per_channel=True, mean=0),
  Normalize(always_apply=False, p=1.0, mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225], max_pixel_value=255.0),
], p=1.0, bbox_params={'format': 'coco', 'label_fields': ['bbox_labels'], 'min_area': 0.0, 'min_visibility': 0.0, 'min_width': 0.0, 'min_height': 0.0, 'check_each_transform': True}, keypoint_params={'format': 'xy', 'label_fields': ['class_labels'], 'remove_invisible': False, 'angle_in_degrees': True, 'check_each_transform': True}, additional_targets={}, is_check_shapes=True)
  Validation: Compose([
  Normalize(always_apply=False, p=1.0, mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225], max_pixel_value=255.0),
], p=1.0, bbox_params={'format': 'coco', 'label_fields': ['bbox_labels'], 'min_area': 0.0, 'min_visibility': 0.0, 'min_width': 0.0, 'min_height': 0.0, 'check_each_transform': True}, keypoint_params={'format': 'xy', 'label_fields': ['class_labels'], 'remove_invisible': False, 'angle_in_degrees': True, 'check_each_transform': True}, additional_targets={}, is_check_shapes=True)
Using 173 images and 10 for testing

Starting pose model training...
--------------------------------------------------
Epoch 1/200 (lr=0.0005), train loss 0.04817, GPU: 4408.0/4095.5 MiB
Epoch 2/200 (lr=0.0005), train loss 0.01571, GPU: 4408.0/4095.5 MiB
Epoch 3/200 (lr=0.0005), train loss 0.01445, GPU: 4408.0/4095.5 MiB
Epoch 4/200 (lr=0.0005), train loss 0.01272, GPU: 4408.0/4095.5 MiB
Epoch 5/200 (lr=0.0005), train loss 0.01025, GPU: 4408.0/4095.5 MiB
Epoch 6/200 (lr=0.0005), train loss 0.00834, GPU: 4408.0/4095.5 MiB
Epoch 7/200 (lr=0.0005), train loss 0.00679, GPU: 4408.0/4095.5 MiB
Epoch 8/200 (lr=0.0005), train loss 0.00553, GPU: 4408.0/4095.5 MiB
Epoch 9/200 (lr=0.0005), train loss 0.00497, GPU: 4408.0/4095.5 MiB
Training for epoch 10 done, starting evaluation
Epoch 10/200 (lr=0.0005), train loss 0.00416, valid loss 0.04771, GPU: 4408.0/4095.5 MiB
Model performance:
  metrics/test.rmse:         110.52
  metrics/test.rmse_pcutoff:    nan
  metrics/test.mAP:            0.00
  metrics/test.mAR:            0.00
Epoch 11/200 (lr=0.0005), train loss 0.00368, GPU: 4408.0/4095.5 MiB
Epoch 12/200 (lr=0.0005), train loss 0.00360, GPU: 4408.0/4095.5 MiB
Epoch 13/200 (lr=0.0005), train loss 0.00324, GPU: 4408.0/4095.5 MiB
Epoch 14/200 (lr=0.0005), train loss 0.00310, GPU: 4408.0/4095.5 MiB
Epoch 15/200 (lr=0.0005), train loss 0.00274, GPU: 4408.0/4095.5 MiB
Epoch 16/200 (lr=0.0005), train loss 0.00296, GPU: 4408.0/4095.5 MiB
Epoch 17/200 (lr=0.0005), train loss 0.00273, GPU: 4408.0/4095.5 MiB
Epoch 18/200 (lr=0.0005), train loss 0.00265, GPU: 4408.0/4095.5 MiB
Epoch 19/200 (lr=0.0005), train loss 0.00242, GPU: 4408.0/4095.5 MiB
Training for epoch 20 done, starting evaluation
Epoch 20/200 (lr=0.0005), train loss 0.00216, valid loss 0.03662, GPU: 4408.0/4095.5 MiB
Model performance:
  metrics/test.rmse:          65.49
  metrics/test.rmse_pcutoff:   5.12
  metrics/test.mAP:           14.52
  metrics/test.mAR:           18.00
Epoch 21/200 (lr=0.0005), train loss 0.00214, GPU: 4408.0/4095.5 MiB
Epoch 22/200 (lr=0.0005), train loss 0.00226, GPU: 4408.0/4095.5 MiB
Epoch 23/200 (lr=0.0005), train loss 0.00198, GPU: 4408.0/4095.5 MiB
Epoch 24/200 (lr=0.0005), train loss 0.00180, GPU: 4408.0/4095.5 MiB
Epoch 25/200 (lr=0.0005), train loss 0.00180, GPU: 4408.0/4095.5 MiB
Epoch 26/200 (lr=0.0005), train loss 0.00182, GPU: 4408.0/4095.5 MiB
Epoch 27/200 (lr=0.0005), train loss 0.00162, GPU: 4408.0/4095.5 MiB
Epoch 28/200 (lr=0.0005), train loss 0.00164, GPU: 4408.0/4095.5 MiB
Epoch 29/200 (lr=0.0005), train loss 0.00158, GPU: 4408.0/4095.5 MiB
Training for epoch 30 done, starting evaluation
Epoch 30/200 (lr=0.0005), train loss 0.00149, valid loss 0.03653, GPU: 4408.0/4095.5 MiB
Model performance:
  metrics/test.rmse:          52.33
  metrics/test.rmse_pcutoff:   5.91
  metrics/test.mAP:           23.05
  metrics/test.mAR:           26.00
Epoch 31/200 (lr=0.0005), train loss 0.00167, GPU: 4408.0/4095.5 MiB
Epoch 32/200 (lr=0.0005), train loss 0.00144, GPU: 4408.0/4095.5 MiB
Epoch 33/200 (lr=0.0005), train loss 0.00155, GPU: 4408.0/4095.5 MiB
Epoch 34/200 (lr=0.0005), train loss 0.00137, GPU: 4408.0/4095.5 MiB
Epoch 35/200 (lr=0.0005), train loss 0.00129, GPU: 4408.0/4095.5 MiB
Epoch 36/200 (lr=0.0005), train loss 0.00139, GPU: 4408.0/4095.5 MiB
Epoch 37/200 (lr=0.0005), train loss 0.00135, GPU: 4408.0/4095.5 MiB
Epoch 38/200 (lr=0.0005), train loss 0.00136, GPU: 4408.0/4095.5 MiB
Epoch 39/200 (lr=0.0005), train loss 0.00129, GPU: 4408.0/4095.5 MiB
Training for epoch 40 done, starting evaluation
Epoch 40/200 (lr=0.0005), train loss 0.00131, valid loss 0.03071, GPU: 4408.0/4095.5 MiB
Model performance:
  metrics/test.rmse:          19.98
  metrics/test.rmse_pcutoff:   3.98
  metrics/test.mAP:           66.85
  metrics/test.mAR:           68.00
Epoch 41/200 (lr=0.0005), train loss 0.00114, GPU: 4408.0/4095.5 MiB
Epoch 42/200 (lr=0.0005), train loss 0.00121, GPU: 4408.0/4095.5 MiB
Epoch 43/200 (lr=0.0005), train loss 0.00116, GPU: 4408.0/4095.5 MiB
Epoch 44/200 (lr=0.0005), train loss 0.00120, GPU: 4408.0/4095.5 MiB
Epoch 45/200 (lr=0.0005), train loss 0.00118, GPU: 4408.0/4095.5 MiB
Epoch 46/200 (lr=0.0005), train loss 0.00115, GPU: 4408.0/4095.5 MiB
Epoch 47/200 (lr=0.0005), train loss 0.00106, GPU: 4408.0/4095.5 MiB
Epoch 48/200 (lr=0.0005), train loss 0.00106, GPU: 4408.0/4095.5 MiB
Epoch 49/200 (lr=0.0005), train loss 0.00102, GPU: 4408.0/4095.5 MiB
Training for epoch 50 done, starting evaluation
Epoch 50/200 (lr=0.0005), train loss 0.00105, valid loss 0.02666, GPU: 4408.0/4095.5 MiB
Model performance:
  metrics/test.rmse:          13.30
  metrics/test.rmse_pcutoff:   3.52
  metrics/test.mAP:           79.78
  metrics/test.mAR:           83.00
Epoch 51/200 (lr=0.0005), train loss 0.00110, GPU: 4408.0/4095.5 MiB
Epoch 52/200 (lr=0.0005), train loss 0.00108, GPU: 4408.0/4095.5 MiB
Epoch 53/200 (lr=0.0005), train loss 0.00109, GPU: 4408.0/4095.5 MiB
Epoch 54/200 (lr=0.0005), train loss 0.00110, GPU: 4408.0/4095.5 MiB
Epoch 55/200 (lr=0.0005), train loss 0.00105, GPU: 4408.0/4095.5 MiB
Epoch 56/200 (lr=0.0005), train loss 0.00095, GPU: 4408.0/4095.5 MiB
Epoch 57/200 (lr=0.0005), train loss 0.00106, GPU: 4408.0/4095.5 MiB
Epoch 58/200 (lr=0.0005), train loss 0.00114, GPU: 4408.0/4095.5 MiB
Epoch 59/200 (lr=0.0005), train loss 0.00101, GPU: 4408.0/4095.5 MiB
Training for epoch 60 done, starting evaluation
Epoch 60/200 (lr=0.0005), train loss 0.00097, valid loss 0.02824, GPU: 4408.0/4095.5 MiB
Model performance:
  metrics/test.rmse:          20.04
  metrics/test.rmse_pcutoff:   4.18
  metrics/test.mAP:           71.17
  metrics/test.mAR:           75.00
Epoch 61/200 (lr=0.0005), train loss 0.00096, GPU: 4408.0/4095.5 MiB
Epoch 62/200 (lr=0.0005), train loss 0.00093, GPU: 4408.0/4095.5 MiB
Epoch 63/200 (lr=0.0005), train loss 0.00099, GPU: 4408.0/4095.5 MiB
Epoch 64/200 (lr=0.0005), train loss 0.00098, GPU: 4408.0/4095.5 MiB
Epoch 65/200 (lr=0.0005), train loss 0.00100, GPU: 4408.0/4095.5 MiB
Epoch 66/200 (lr=0.0005), train loss 0.00092, GPU: 4408.0/4095.5 MiB
Epoch 67/200 (lr=0.0005), train loss 0.00091, GPU: 4408.0/4095.5 MiB
Epoch 68/200 (lr=0.0005), train loss 0.00098, GPU: 4408.0/4095.5 MiB
Epoch 69/200 (lr=0.0005), train loss 0.00099, GPU: 4408.0/4095.5 MiB
Training for epoch 70 done, starting evaluation
Epoch 70/200 (lr=0.0005), train loss 0.00090, valid loss 0.02239, GPU: 4408.0/4095.5 MiB
Model performance:
  metrics/test.rmse:           4.27
  metrics/test.rmse_pcutoff:   3.93
  metrics/test.mAP:           98.61
  metrics/test.mAR:           99.00
Epoch 71/200 (lr=0.0005), train loss 0.00093, GPU: 4408.0/4095.5 MiB
Epoch 72/200 (lr=0.0005), train loss 0.00087, GPU: 4408.0/4095.5 MiB
Epoch 73/200 (lr=0.0005), train loss 0.00087, GPU: 4408.0/4095.5 MiB
Epoch 74/200 (lr=0.0005), train loss 0.00082, GPU: 4408.0/4095.5 MiB
Epoch 75/200 (lr=0.0005), train loss 0.00086, GPU: 4408.0/4095.5 MiB
Epoch 76/200 (lr=0.0005), train loss 0.00089, GPU: 4408.0/4095.5 MiB
Epoch 77/200 (lr=0.0005), train loss 0.00086, GPU: 4408.0/4095.5 MiB
Epoch 78/200 (lr=0.0005), train loss 0.00104, GPU: 4408.0/4095.5 MiB
Epoch 79/200 (lr=0.0005), train loss 0.00111, GPU: 4408.0/4095.5 MiB
Training for epoch 80 done, starting evaluation
Epoch 80/200 (lr=0.0005), train loss 0.00095, valid loss 0.01714, GPU: 4408.0/4095.5 MiB
Model performance:
  metrics/test.rmse:           4.45
  metrics/test.rmse_pcutoff:   4.34
  metrics/test.mAP:           98.61
  metrics/test.mAR:           99.00
Epoch 81/200 (lr=0.0005), train loss 0.00088, GPU: 4408.0/4095.5 MiB
Epoch 82/200 (lr=0.0005), train loss 0.00087, GPU: 4408.0/4095.5 MiB
Epoch 83/200 (lr=0.0005), train loss 0.00089, GPU: 4408.0/4095.5 MiB
Epoch 84/200 (lr=0.0005), train loss 0.00083, GPU: 4408.0/4095.5 MiB
Epoch 85/200 (lr=0.0005), train loss 0.00077, GPU: 4408.0/4095.5 MiB
Epoch 86/200 (lr=0.0005), train loss 0.00085, GPU: 4408.0/4095.5 MiB
Epoch 87/200 (lr=0.0005), train loss 0.00085, GPU: 4408.0/4095.5 MiB
Epoch 88/200 (lr=0.0005), train loss 0.00084, GPU: 4408.0/4095.5 MiB
Epoch 89/200 (lr=0.0005), train loss 0.00085, GPU: 4408.0/4095.5 MiB
Training for epoch 90 done, starting evaluation
Epoch 90/200 (lr=0.0001), train loss 0.00081, valid loss 0.01781, GPU: 4408.0/4095.5 MiB
Model performance:
  metrics/test.rmse:           4.53
  metrics/test.rmse_pcutoff:   4.19
  metrics/test.mAP:           97.03
  metrics/test.mAR:           98.00
Epoch 91/200 (lr=0.0001), train loss 0.00071, GPU: 4408.0/4095.5 MiB
Epoch 92/200 (lr=0.0001), train loss 0.00066, GPU: 4408.0/4095.5 MiB
Epoch 93/200 (lr=0.0001), train loss 0.00063, GPU: 4408.0/4095.5 MiB
Epoch 94/200 (lr=0.0001), train loss 0.00064, GPU: 4408.0/4095.5 MiB
Epoch 95/200 (lr=0.0001), train loss 0.00067, GPU: 4408.0/4095.5 MiB
Epoch 96/200 (lr=0.0001), train loss 0.00062, GPU: 4408.0/4095.5 MiB
Epoch 97/200 (lr=0.0001), train loss 0.00058, GPU: 4408.0/4095.5 MiB
Epoch 98/200 (lr=0.0001), train loss 0.00060, GPU: 4408.0/4095.5 MiB
Epoch 99/200 (lr=0.0001), train loss 0.00061, GPU: 4408.0/4095.5 MiB
Training for epoch 100 done, starting evaluation
Epoch 100/200 (lr=0.0001), train loss 0.00062, valid loss 0.02061, GPU: 4408.0/4095.5 MiB
Model performance:
  metrics/test.rmse:           4.20
  metrics/test.rmse_pcutoff:   3.85
  metrics/test.mAP:           98.42
  metrics/test.mAR:           99.00
Epoch 101/200 (lr=0.0001), train loss 0.00061, GPU: 4408.0/4095.5 MiB
Epoch 102/200 (lr=0.0001), train loss 0.00060, GPU: 4408.0/4095.5 MiB
Epoch 103/200 (lr=0.0001), train loss 0.00063, GPU: 4408.0/4095.5 MiB
Epoch 104/200 (lr=0.0001), train loss 0.00061, GPU: 4408.0/4095.5 MiB
Epoch 105/200 (lr=0.0001), train loss 0.00061, GPU: 4408.0/4095.5 MiB
Epoch 106/200 (lr=0.0001), train loss 0.00062, GPU: 4408.0/4095.5 MiB
Epoch 107/200 (lr=0.0001), train loss 0.00061, GPU: 4408.0/4095.5 MiB
Epoch 108/200 (lr=0.0001), train loss 0.00057, GPU: 4408.0/4095.5 MiB
Epoch 109/200 (lr=0.0001), train loss 0.00057, GPU: 4408.0/4095.5 MiB
Training for epoch 110 done, starting evaluation
Epoch 110/200 (lr=0.0001), train loss 0.00062, valid loss 0.02231, GPU: 4408.0/4095.5 MiB
Model performance:
  metrics/test.rmse:           4.49
  metrics/test.rmse_pcutoff:   4.04
  metrics/test.mAP:           96.63
  metrics/test.mAR:           98.00
Epoch 111/200 (lr=0.0001), train loss 0.00060, GPU: 4408.0/4095.5 MiB
Epoch 112/200 (lr=0.0001), train loss 0.00060, GPU: 4408.0/4095.5 MiB
Epoch 113/200 (lr=0.0001), train loss 0.00058, GPU: 4408.0/4095.5 MiB
Epoch 114/200 (lr=0.0001), train loss 0.00060, GPU: 4408.0/4095.5 MiB
Epoch 115/200 (lr=0.0001), train loss 0.00058, GPU: 4408.0/4095.5 MiB
Epoch 116/200 (lr=0.0001), train loss 0.00060, GPU: 4408.0/4095.5 MiB
Epoch 117/200 (lr=0.0001), train loss 0.00051, GPU: 4408.0/4095.5 MiB
Epoch 118/200 (lr=0.0001), train loss 0.00058, GPU: 4408.0/4095.5 MiB
Epoch 119/200 (lr=0.0001), train loss 0.00061, GPU: 4408.0/4095.5 MiB
Training for epoch 120 done, starting evaluation
Epoch 120/200 (lr=1e-05), train loss 0.00059, valid loss 0.02172, GPU: 4408.0/4095.5 MiB
Model performance:
  metrics/test.rmse:           4.36
  metrics/test.rmse_pcutoff:   3.89
  metrics/test.mAP:           96.42
  metrics/test.mAR:           98.00
Epoch 121/200 (lr=1e-05), train loss 0.00056, GPU: 4408.0/4095.5 MiB
Epoch 122/200 (lr=1e-05), train loss 0.00052, GPU: 4408.0/4095.5 MiB
Epoch 123/200 (lr=1e-05), train loss 0.00054, GPU: 4408.0/4095.5 MiB
Epoch 124/200 (lr=1e-05), train loss 0.00056, GPU: 4408.0/4095.5 MiB
Epoch 125/200 (lr=1e-05), train loss 0.00053, GPU: 4408.0/4095.5 MiB
Epoch 126/200 (lr=1e-05), train loss 0.00053, GPU: 4408.0/4095.5 MiB
Epoch 127/200 (lr=1e-05), train loss 0.00055, GPU: 4408.0/4095.5 MiB
Epoch 128/200 (lr=1e-05), train loss 0.00052, GPU: 4408.0/4095.5 MiB
Epoch 129/200 (lr=1e-05), train loss 0.00051, GPU: 4408.0/4095.5 MiB
Training for epoch 130 done, starting evaluation
Epoch 130/200 (lr=1e-05), train loss 0.00052, valid loss 0.02294, GPU: 4408.0/4095.5 MiB
Model performance:
  metrics/test.rmse:           4.38
  metrics/test.rmse_pcutoff:   3.89
  metrics/test.mAP:           96.42
  metrics/test.mAR:           98.00
Epoch 131/200 (lr=1e-05), train loss 0.00056, GPU: 4408.0/4095.5 MiB
Epoch 132/200 (lr=1e-05), train loss 0.00053, GPU: 4408.0/4095.5 MiB
Epoch 133/200 (lr=1e-05), train loss 0.00055, GPU: 4408.0/4095.5 MiB
Epoch 134/200 (lr=1e-05), train loss 0.00051, GPU: 4408.0/4095.5 MiB
Epoch 135/200 (lr=1e-05), train loss 0.00055, GPU: 4408.0/4095.5 MiB
Epoch 136/200 (lr=1e-05), train loss 0.00055, GPU: 4408.0/4095.5 MiB
Epoch 137/200 (lr=1e-05), train loss 0.00052, GPU: 4408.0/4095.5 MiB
Epoch 138/200 (lr=1e-05), train loss 0.00054, GPU: 4408.0/4095.5 MiB
Epoch 139/200 (lr=1e-05), train loss 0.00057, GPU: 4408.0/4095.5 MiB
Training for epoch 140 done, starting evaluation
Epoch 140/200 (lr=1e-05), train loss 0.00053, valid loss 0.02296, GPU: 4408.0/4095.5 MiB
Model performance:
  metrics/test.rmse:           5.51
  metrics/test.rmse_pcutoff:   3.80
  metrics/test.mAP:           94.06
  metrics/test.mAR:           96.00
Epoch 141/200 (lr=1e-05), train loss 0.00059, GPU: 4408.0/4095.5 MiB
Epoch 142/200 (lr=1e-05), train loss 0.00052, GPU: 4408.0/4095.5 MiB
Epoch 143/200 (lr=1e-05), train loss 0.00054, GPU: 4408.0/4095.5 MiB
Epoch 144/200 (lr=1e-05), train loss 0.00051, GPU: 4408.0/4095.5 MiB
Epoch 145/200 (lr=1e-05), train loss 0.00051, GPU: 4408.0/4095.5 MiB
Epoch 146/200 (lr=1e-05), train loss 0.00050, GPU: 4408.0/4095.5 MiB
Epoch 147/200 (lr=1e-05), train loss 0.00054, GPU: 4408.0/4095.5 MiB
Epoch 148/200 (lr=1e-05), train loss 0.00051, GPU: 4408.0/4095.5 MiB
Epoch 149/200 (lr=1e-05), train loss 0.00052, GPU: 4408.0/4095.5 MiB
Training for epoch 150 done, starting evaluation
Epoch 150/200 (lr=1e-05), train loss 0.00053, valid loss 0.02303, GPU: 4408.0/4095.5 MiB
Model performance:
  metrics/test.rmse:           4.33
  metrics/test.rmse_pcutoff:   3.93
  metrics/test.mAP:           96.42
  metrics/test.mAR:           98.00
Epoch 151/200 (lr=1e-05), train loss 0.00052, GPU: 4408.0/4095.5 MiB
Epoch 152/200 (lr=1e-05), train loss 0.00052, GPU: 4408.0/4095.5 MiB
Epoch 153/200 (lr=1e-05), train loss 0.00054, GPU: 4408.0/4095.5 MiB
Epoch 154/200 (lr=1e-05), train loss 0.00053, GPU: 4408.0/4095.5 MiB
Epoch 155/200 (lr=1e-05), train loss 0.00053, GPU: 4408.0/4095.5 MiB
Epoch 156/200 (lr=1e-05), train loss 0.00053, GPU: 4408.0/4095.5 MiB
Epoch 157/200 (lr=1e-05), train loss 0.00052, GPU: 4408.0/4095.5 MiB
Epoch 158/200 (lr=1e-05), train loss 0.00058, GPU: 4408.0/4095.5 MiB
Epoch 159/200 (lr=1e-05), train loss 0.00049, GPU: 4408.0/4095.5 MiB
Training for epoch 160 done, starting evaluation
Epoch 160/200 (lr=1e-05), train loss 0.00054, valid loss 0.02343, GPU: 4408.0/4095.5 MiB
Model performance:
  metrics/test.rmse:           4.39
  metrics/test.rmse_pcutoff:   4.02
  metrics/test.mAP:           96.42
  metrics/test.mAR:           98.00
Epoch 161/200 (lr=1e-05), train loss 0.00055, GPU: 4408.0/4095.5 MiB
Epoch 162/200 (lr=1e-05), train loss 0.00053, GPU: 4408.0/4095.5 MiB
Epoch 163/200 (lr=1e-05), train loss 0.00054, GPU: 4408.0/4095.5 MiB
Epoch 164/200 (lr=1e-05), train loss 0.00052, GPU: 4408.0/4095.5 MiB
Epoch 165/200 (lr=1e-05), train loss 0.00054, GPU: 4408.0/4095.5 MiB
Epoch 166/200 (lr=1e-05), train loss 0.00057, GPU: 4408.0/4095.5 MiB
Epoch 167/200 (lr=1e-05), train loss 0.00055, GPU: 4408.0/4095.5 MiB
Epoch 168/200 (lr=1e-05), train loss 0.00053, GPU: 4408.0/4095.5 MiB
Epoch 169/200 (lr=1e-05), train loss 0.00051, GPU: 4408.0/4095.5 MiB
Training for epoch 170 done, starting evaluation
Epoch 170/200 (lr=1e-05), train loss 0.00054, valid loss 0.02411, GPU: 4408.0/4095.5 MiB
Model performance:
  metrics/test.rmse:           5.53
  metrics/test.rmse_pcutoff:   3.84
  metrics/test.mAP:           94.06
  metrics/test.mAR:           96.00
Epoch 171/200 (lr=1e-05), train loss 0.00051, GPU: 4408.0/4095.5 MiB
Epoch 172/200 (lr=1e-05), train loss 0.00051, GPU: 4408.0/4095.5 MiB
Epoch 173/200 (lr=1e-05), train loss 0.00051, GPU: 4408.0/4095.5 MiB
Epoch 174/200 (lr=1e-05), train loss 0.00049, GPU: 4408.0/4095.5 MiB
Epoch 175/200 (lr=1e-05), train loss 0.00051, GPU: 4408.0/4095.5 MiB
Epoch 176/200 (lr=1e-05), train loss 0.00053, GPU: 4408.0/4095.5 MiB
Epoch 177/200 (lr=1e-05), train loss 0.00051, GPU: 4408.0/4095.5 MiB
Epoch 178/200 (lr=1e-05), train loss 0.00055, GPU: 4408.0/4095.5 MiB
Epoch 179/200 (lr=1e-05), train loss 0.00052, GPU: 4408.0/4095.5 MiB
Training for epoch 180 done, starting evaluation
Epoch 180/200 (lr=1e-05), train loss 0.00052, valid loss 0.02420, GPU: 4408.0/4095.5 MiB
Model performance:
  metrics/test.rmse:           4.41
  metrics/test.rmse_pcutoff:   3.87
  metrics/test.mAP:           96.42
  metrics/test.mAR:           98.00
Epoch 181/200 (lr=1e-05), train loss 0.00050, GPU: 4408.0/4095.5 MiB
Epoch 182/200 (lr=1e-05), train loss 0.00050, GPU: 4408.0/4095.5 MiB
Epoch 183/200 (lr=1e-05), train loss 0.00052, GPU: 4408.0/4095.5 MiB
Epoch 184/200 (lr=1e-05), train loss 0.00049, GPU: 4408.0/4095.5 MiB
Epoch 185/200 (lr=1e-05), train loss 0.00057, GPU: 4408.0/4095.5 MiB
Epoch 186/200 (lr=1e-05), train loss 0.00051, GPU: 4408.0/4095.5 MiB
Epoch 187/200 (lr=1e-05), train loss 0.00052, GPU: 4408.0/4095.5 MiB
Epoch 188/200 (lr=1e-05), train loss 0.00051, GPU: 4408.0/4095.5 MiB
Epoch 189/200 (lr=1e-05), train loss 0.00057, GPU: 4408.0/4095.5 MiB
Training for epoch 190 done, starting evaluation
Epoch 190/200 (lr=1e-05), train loss 0.00052, valid loss 0.02488, GPU: 4408.0/4095.5 MiB
Model performance:
  metrics/test.rmse:           5.58
  metrics/test.rmse_pcutoff:   3.88
  metrics/test.mAP:           94.06
  metrics/test.mAR:           96.00
Epoch 191/200 (lr=1e-05), train loss 0.00056, GPU: 4408.0/4095.5 MiB
Epoch 192/200 (lr=1e-05), train loss 0.00050, GPU: 4408.0/4095.5 MiB
Epoch 193/200 (lr=1e-05), train loss 0.00054, GPU: 4408.0/4095.5 MiB
Epoch 194/200 (lr=1e-05), train loss 0.00054, GPU: 4408.0/4095.5 MiB
Epoch 195/200 (lr=1e-05), train loss 0.00055, GPU: 4408.0/4095.5 MiB
Epoch 196/200 (lr=1e-05), train loss 0.00052, GPU: 4408.0/4095.5 MiB
Epoch 197/200 (lr=1e-05), train loss 0.00051, GPU: 4408.0/4095.5 MiB
Epoch 198/200 (lr=1e-05), train loss 0.00050, GPU: 4408.0/4095.5 MiB
Epoch 199/200 (lr=1e-05), train loss 0.00055, GPU: 4408.0/4095.5 MiB
Training for epoch 200 done, starting evaluation
Epoch 200/200 (lr=1e-05), train loss 0.00047, valid loss 0.02439, GPU: 4408.0/4095.5 MiB
Model performance:
  metrics/test.rmse:           5.54
  metrics/test.rmse_pcutoff:   3.87
  metrics/test.mAP:           94.06
  metrics/test.mAR:           96.00
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 173/173 [00:03<00:00, 54.99it/s]
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:00<00:00, 18.20it/s]
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 173/173 [00:02<00:00, 69.68it/s]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:00<00:00, 116.75it/s]
WARNING: QThread::wait: Thread tried to wait on itself
WARNING: QThread::wait: Thread tried to wait on itself

Code of Conduct

Metadata

Metadata

Assignees

Labels

GUIissues relating to GUI

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions