loss is nan #6

fengluodb · 2022-09-06T06:29:57Z

I train the net from scratch as the redeme said. However, after 5 epoch, the loss is always nan, acc is 0, iou is 0.

Lr: 9.611e-03 | Update: nan mean,nan std | Epoch: [4][990/1047] | Time 1.768 (1.854) | Data 0.618 (0.706) | Loss nan (nan) | acc 0.000 (0.000) | IoU 0.000 (0.000) | [1 day, 18:24:23]
Lr: 9.610e-03 | Update: nan mean,nan std | Epoch: [4][1000/1047] | Time 1.738 (1.854) | Data 0.638 (0.706) | Loss nan (nan) | acc 0.000 (0.000) | IoU 0.000 (0.000) | [1 day, 18:23:44]
Lr: 9.609e-03 | Update: nan mean,nan std | Epoch: [4][1010/1047] | Time 1.907 (1.854) | Data 0.668 (0.706) | Loss nan (nan) | acc 0.000 (0.000) | IoU 0.000 (0.000) | [1 day, 18:23:12]
Lr: 9.608e-03 | Update: nan mean,nan std | Epoch: [4][1020/1047] | Time 1.808 (1.854) | Data 0.648 (0.706) | Loss nan (nan) | acc 0.000 (0.000) | IoU 0.000 (0.000) | [1 day, 18:22:35]
Lr: 9.608e-03 | Update: nan mean,nan std | Epoch: [4][1030/1047] | Time 1.855 (1.853) | Data 0.633 (0.705) | Loss nan (nan) | acc 0.000 (0.000) | IoU 0.000 (0.000) | [1 day, 18:21:58]
Lr: 9.607e-03 | Update: nan mean,nan std | Epoch: [4][1040/1047] | Time 1.763 (1.853) | Data 0.627 (0.705) | Loss nan (nan) | acc 0.000 (0.000) | IoU 0.000 (0.000) | [1 day, 18:21:18]

The text was updated successfully, but these errors were encountered:

MaxChanger · 2022-09-06T07:58:23Z

Hi, could you provide more details, such as, the running command, and which config file you used

fengluodb · 2022-09-06T08:03:49Z

My running command is python train.py -d DATAROOT -ac ./train_yaml/mos_coarse_stage.yml. As for config file, I just change the batch size to 8, epoch to 50 and workers to 0.

MaxChanger · 2022-09-06T08:06:41Z

Did you use KITTI-Road data? How many GPUs?
On 4*3090, this training code was verified multiple times before release.

fengluodb · 2022-09-06T08:08:58Z

I don't use KITTI-Road data. I commnet them. I just use SemanticKitti dataset. I use 4*3090

split: # sequence numbers
  train:
    - 0  # 4541 scans
    - 1  # 1101
    - 2  # 4661
    - 3  # 801
    - 4  # 271
    - 5  # 2761
    - 6  # 1101
    - 7  # 1105
    - 9  # 1591
    - 10 # 1201
    # - 30 # 297  add new kitti-road seq
    # - 31 # 188  add new kitti-road seq
    # - 32 # 430  add new kitti-road seq
    # - 33 # 430  add new kitti-road seq
    # - 34 # 390  add new kitti-road seq
    # - 40 # 1170 add new kitti-road seq
  valid:
    - 8  # 4071
    # - 35 # 78   add new kitti-road seq
    # - 36 # 420  add new kitti-road seq
    # - 37 # 936  add new kitti-road seq
    # - 38 # 339  add new kitti-road seq
    # - 39 # 279  add new kitti-road seq
    # - 41 # 837  add new kitti-road seq
  test:
    - 11
    - 12
    - 13
    - 14
    - 15
    - 16
    - 17
    - 18
    - 19
    - 20
    - 21

MaxChanger · 2022-09-06T08:12:05Z

There are some output log I just ran, I will use your config later to train and see if I can reproduce your problem. You can check the residual image by the way.

output log

export CUDA_VISIBLE_DEVICES=0,5,6,7
python train.py -d ~/Repo/MotionSeg3D/data/ -ac ./train_yaml/mos_coarse_stage.yml -l log/ours_motionseg3d

 Opening arch config file ./train_yaml/mos_coarse_stage.yml
 Opening arch config file config/labels/semantic-kitti-mos.yaml
Channel of range image input =  5
Number of residual images input =  8
----------
INTERFACE:
  dataset: /home1/xxxxxx/Repo/MotionSeg3D/data/
  arch_cfg: ./train_yaml/mos_coarse_stage.yml
  data_cfg: config/labels/semantic-kitti-mos.yaml
  Total of Trainable Parameters: 13.61M
  log: log/ours_motionseg3d/logs/2022-9-06-15:55
  pretrained: None
----------

 No pretrained directory found.
Copying files to log/ours_motionseg3d/logs/2022-9-06-15:55 for further reference.
Sequences folder exists! Using sequences from /home/xxxxxxx/Repo/MotionSeg3D/data/sequences
parsing seq 00
parsing seq 01
parsing seq 02
parsing seq 03
parsing seq 04
parsing seq 05
parsing seq 06
parsing seq 07
parsing seq 09
parsing seq 10
parsing seq 30
parsing seq 31
parsing seq 32
parsing seq 33
parsing seq 34
parsing seq 40
 There are 22035 frames in total.
  Drop residual_images_1 in seq00: 4541 -> 1558
  Drop residual_images_2 in seq00: 4541 -> 1558
  Drop residual_images_3 in seq00: 4541 -> 1558
  Drop residual_images_4 in seq00: 4541 -> 1558
  Drop residual_images_5 in seq00: 4541 -> 1558
  Drop residual_images_6 in seq00: 4541 -> 1558
  Drop residual_images_7 in seq00: 4541 -> 1558
  Drop residual_images_8 in seq00: 4541 -> 1558
Seq 00 drop 2983: 4541 -> 1558
  Drop residual_images_1 in seq01: 1101 -> 899
  Drop residual_images_2 in seq01: 1101 -> 899
  Drop residual_images_3 in seq01: 1101 -> 899
  Drop residual_images_4 in seq01: 1101 -> 899
  Drop residual_images_5 in seq01: 1101 -> 899
  Drop residual_images_6 in seq01: 1101 -> 899
  Drop residual_images_7 in seq01: 1101 -> 899
  Drop residual_images_8 in seq01: 1101 -> 899
Seq 01 drop 202: 1101 -> 899
  Drop residual_images_1 in seq02: 4661 -> 1237
  Drop residual_images_2 in seq02: 4661 -> 1237
  Drop residual_images_3 in seq02: 4661 -> 1237
  Drop residual_images_4 in seq02: 4661 -> 1237
  Drop residual_images_5 in seq02: 4661 -> 1237
  Drop residual_images_6 in seq02: 4661 -> 1237
  Drop residual_images_7 in seq02: 4661 -> 1237
  Drop residual_images_8 in seq02: 4661 -> 1237
Seq 02 drop 3424: 4661 -> 1237
  Drop residual_images_1 in seq03: 801 -> 341
  Drop residual_images_2 in seq03: 801 -> 341
  Drop residual_images_3 in seq03: 801 -> 341
  Drop residual_images_4 in seq03: 801 -> 341
  Drop residual_images_5 in seq03: 801 -> 341
  Drop residual_images_6 in seq03: 801 -> 341
  Drop residual_images_7 in seq03: 801 -> 341
  Drop residual_images_8 in seq03: 801 -> 341
Seq 03 drop 460: 801 -> 341
  Drop residual_images_1 in seq04: 271 -> 271
  Drop residual_images_2 in seq04: 271 -> 271
  Drop residual_images_3 in seq04: 271 -> 271
  Drop residual_images_4 in seq04: 271 -> 271
  Drop residual_images_5 in seq04: 271 -> 271
  Drop residual_images_6 in seq04: 271 -> 271
  Drop residual_images_7 in seq04: 271 -> 271
  Drop residual_images_8 in seq04: 271 -> 271
Seq 04 drop 0: 271 -> 271
  Drop residual_images_1 in seq05: 2761 -> 1423
  Drop residual_images_2 in seq05: 2761 -> 1423
  Drop residual_images_3 in seq05: 2761 -> 1423
  Drop residual_images_4 in seq05: 2761 -> 1423
  Drop residual_images_5 in seq05: 2761 -> 1423
  Drop residual_images_6 in seq05: 2761 -> 1423
  Drop residual_images_7 in seq05: 2761 -> 1423
  Drop residual_images_8 in seq05: 2761 -> 1423
Seq 05 drop 1338: 2761 -> 1423
  Drop residual_images_1 in seq06: 1101 -> 428
  Drop residual_images_2 in seq06: 1101 -> 428
  Drop residual_images_3 in seq06: 1101 -> 428
  Drop residual_images_4 in seq06: 1101 -> 428
  Drop residual_images_5 in seq06: 1101 -> 428
  Drop residual_images_6 in seq06: 1101 -> 428
  Drop residual_images_7 in seq06: 1101 -> 428
  Drop residual_images_8 in seq06: 1101 -> 428
Seq 06 drop 673: 1101 -> 428
  Drop residual_images_1 in seq07: 1101 -> 871
  Drop residual_images_2 in seq07: 1101 -> 871
  Drop residual_images_3 in seq07: 1101 -> 871
  Drop residual_images_4 in seq07: 1101 -> 871
  Drop residual_images_5 in seq07: 1101 -> 871
  Drop residual_images_6 in seq07: 1101 -> 871
  Drop residual_images_7 in seq07: 1101 -> 871
  Drop residual_images_8 in seq07: 1101 -> 871
Seq 07 drop 230: 1101 -> 871
  Drop residual_images_1 in seq09: 1591 -> 827
  Drop residual_images_2 in seq09: 1591 -> 827
  Drop residual_images_3 in seq09: 1591 -> 827
  Drop residual_images_4 in seq09: 1591 -> 827
  Drop residual_images_5 in seq09: 1591 -> 827
  Drop residual_images_6 in seq09: 1591 -> 827
  Drop residual_images_7 in seq09: 1591 -> 827
  Drop residual_images_8 in seq09: 1591 -> 827
Seq 09 drop 764: 1591 -> 827
  Drop residual_images_1 in seq10: 1201 -> 524
  Drop residual_images_2 in seq10: 1201 -> 524
  Drop residual_images_3 in seq10: 1201 -> 524
  Drop residual_images_4 in seq10: 1201 -> 524
  Drop residual_images_5 in seq10: 1201 -> 524
  Drop residual_images_6 in seq10: 1201 -> 524
  Drop residual_images_7 in seq10: 1201 -> 524
  Drop residual_images_8 in seq10: 1201 -> 524
Seq 10 drop 677: 1201 -> 524
  Drop residual_images_1 in seq30: 297 -> 297
  Drop residual_images_2 in seq30: 297 -> 297
  Drop residual_images_3 in seq30: 297 -> 297
  Drop residual_images_4 in seq30: 297 -> 297
  Drop residual_images_5 in seq30: 297 -> 297
  Drop residual_images_6 in seq30: 297 -> 297
  Drop residual_images_7 in seq30: 297 -> 297
  Drop residual_images_8 in seq30: 297 -> 297
Seq 30 drop 0: 297 -> 297
  Drop residual_images_1 in seq31: 188 -> 188
  Drop residual_images_2 in seq31: 188 -> 188
  Drop residual_images_3 in seq31: 188 -> 188
  Drop residual_images_4 in seq31: 188 -> 188
  Drop residual_images_5 in seq31: 188 -> 188
  Drop residual_images_6 in seq31: 188 -> 188
  Drop residual_images_7 in seq31: 188 -> 188
  Drop residual_images_8 in seq31: 188 -> 188
Seq 31 drop 0: 188 -> 188
  Drop residual_images_1 in seq32: 430 -> 430
  Drop residual_images_2 in seq32: 430 -> 430
  Drop residual_images_3 in seq32: 430 -> 430
  Drop residual_images_4 in seq32: 430 -> 430
  Drop residual_images_5 in seq32: 430 -> 430
  Drop residual_images_6 in seq32: 430 -> 430
  Drop residual_images_7 in seq32: 430 -> 430
  Drop residual_images_8 in seq32: 430 -> 430
Seq 32 drop 0: 430 -> 430
  Drop residual_images_1 in seq33: 430 -> 430
  Drop residual_images_2 in seq33: 430 -> 430
  Drop residual_images_3 in seq33: 430 -> 430
  Drop residual_images_4 in seq33: 430 -> 430
  Drop residual_images_5 in seq33: 430 -> 430
  Drop residual_images_6 in seq33: 430 -> 430
  Drop residual_images_7 in seq33: 430 -> 430
  Drop residual_images_8 in seq33: 430 -> 430
Seq 33 drop 0: 430 -> 430
  Drop residual_images_1 in seq34: 390 -> 390
  Drop residual_images_2 in seq34: 390 -> 390
  Drop residual_images_3 in seq34: 390 -> 390
  Drop residual_images_4 in seq34: 390 -> 390
  Drop residual_images_5 in seq34: 390 -> 390
  Drop residual_images_6 in seq34: 390 -> 390
  Drop residual_images_7 in seq34: 390 -> 390
  Drop residual_images_8 in seq34: 390 -> 390
Seq 34 drop 0: 390 -> 390
  Drop residual_images_1 in seq40: 1170 -> 1170
  Drop residual_images_2 in seq40: 1170 -> 1170
  Drop residual_images_3 in seq40: 1170 -> 1170
  Drop residual_images_4 in seq40: 1170 -> 1170
  Drop residual_images_5 in seq40: 1170 -> 1170
  Drop residual_images_6 in seq40: 1170 -> 1170
  Drop residual_images_7 in seq40: 1170 -> 1170
  Drop residual_images_8 in seq40: 1170 -> 1170
Seq 40 drop 0: 1170 -> 1170
 Remove 10751 frames.
 New use 11284 frames.
 Using 11284 scans from sequences [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 30, 31, 32, 33, 34, 40]
Sequences folder exists! Using sequences from /home/xxxxxx/Repo/MotionSeg3D/data/sequences
parsing seq 08
parsing seq 35
parsing seq 36
parsing seq 37
parsing seq 38
parsing seq 39
parsing seq 41
 There are 6960 frames in total.
 Using 6960 scans from sequences [8, 35, 36, 37, 38, 39, 41]
Loss weights from content:  tensor([  0.0000,   1.0014, 296.4371])
Channel of range image input =  5
Number of residual images input =  8
Training in device:  cuda
Let's use 4 GPUs!

Ignoring class  0  in IoU evaluation
[IOU EVAL] IGNORE:  tensor([0])
[IOU EVAL] INCLUDE:  tensor([1, 2])

Lr: 1.773e-05 | Update: 1.579e-01 mean,3.646e-01 std | Epoch: [0][0/564] | Time 48.405 (48.405) | Data 19.875 (19.875) | Loss 1.9651 (1.9651) | acc 0.417 (0.417) | IoU 0.210 (0.210) | [66 days, 20:33:02]
Lr: 1.950e-04 | Update: 1.278e-02 mean,3.152e-02 std | Epoch: [0][10/564] | Time 2.075 (6.279) | Data 0.095 (1.888) | Loss 1.9103 (1.9772) | acc 0.460 (0.431) | IoU 0.234 (0.218) | [7 days, 23:53:33]
Lr: 3.723e-04 | Update: 6.040e-03 mean,1.477e-02 std | Epoch: [0][20/564] | Time 1.953 (4.204) | Data 0.111 (1.035) | Loss 1.7721 (1.9081) | acc 0.561 (0.469) | IoU 0.284 (0.237) | [5 days, 3:06:25]
Lr: 5.496e-04 | Update: 2.098e-03 mean,5.005e-03 std | Epoch: [0][30/564] | Time 1.870 (3.486) | Data 0.107 (0.733) | Loss 1.6438 (1.8074) | acc 0.514 (0.497) | IoU 0.259 (0.252) | [4 days, 3:06:18]
Lr: 7.270e-04 | Update: 3.818e-03 mean,9.948e-03 std | Epoch: [0][40/564] | Time 1.828 (3.141) | Data 0.086 (0.577) | Loss 1.4263 (1.7247) | acc 0.547 (0.511) | IoU 0.279 (0.259) | [3 days, 15:19:15]
Lr: 9.043e-04 | Update: 1.696e-03 mean,4.019e-03 std | Epoch: [0][50/564] | Time 2.065 (2.932) | Data 0.089 (0.481) | Loss 1.2419 (1.6415) | acc 0.618 (0.525) | IoU 0.314 (0.267) | [3 days, 8:10:42]
Lr: 1.082e-03 | Update: 8.822e-04 mean,1.949e-03 std | Epoch: [0][60/564] | Time 2.052 (2.884) | Data 0.076 (0.502) | Loss 1.1353 (1.5668) | acc 0.694 (0.547) | IoU 0.353 (0.278) | [3 days, 7:30:13]
Lr: 1.259e-03 | Update: 1.966e-03 mean,4.621e-03 std | Epoch: [0][70/564] | Time 2.164 (2.868) | Data 0.086 (0.523) | Loss 1.0365 (1.5015) | acc 0.745 (0.573) | IoU 0.384 (0.292) | [3 days, 7:36:59]
Lr: 1.436e-03 | Update: 1.700e-03 mean,4.092e-03 std | Epoch: [0][80/564] | Time 2.164 (2.819) | Data 0.085 (0.499) | Loss 1.0809 (1.4473) | acc 0.748 (0.596) | IoU 0.387 (0.304) | [3 days, 5:54:46]
Lr: 1.613e-03 | Update: 1.201e-03 mean,2.953e-03 std | Epoch: [0][90/564] | Time 2.282 (2.759) | Data 0.085 (0.455) | Loss 0.9169 (1.3977) | acc 0.831 (0.618) | IoU 0.428 (0.316) | [3 days, 3:27:07]
Lr: 1.791e-03 | Update: 2.153e-03 mean,5.089e-03 std | Epoch: [0][100/564] | Time 2.216 (2.731) | Data 0.082 (0.446) | Loss 0.9267 (1.3535) | acc 0.840 (0.639) | IoU 0.432 (0.327) | [3 days, 2:33:50]
Lr: 1.968e-03 | Update: 2.456e-03 mean,6.732e-03 std | Epoch: [0][110/564] | Time 2.460 (2.710) | Data 0.102 (0.435) | Loss 1.1077 (1.3181) | acc 0.741 (0.656) | IoU 0.382 (0.336) | [3 days, 1:49:05]
Lr: 2.145e-03 | Update: 1.373e-03 mean,2.921e-03 std | Epoch: [0][120/564] | Time 2.064 (2.686) | Data 0.073 (0.425) | Loss 0.9094 (1.2877) | acc 0.752 (0.664) | IoU 0.407 (0.341) | [3 days, 0:58:56]
Lr: 2.323e-03 | Update: 1.989e-03 mean,4.530e-03 std | Epoch: [0][130/564] | Time 2.143 (2.649) | Data 0.086 (0.400) | Loss 0.8917 (1.2582) | acc 0.909 (0.676) | IoU 0.470 (0.347) | [2 days, 23:32:21]
Lr: 2.500e-03 | Update: 9.505e-04 mean,2.062e-03 std | Epoch: [0][140/564] | Time 2.309 (2.649) | Data 0.079 (0.405) | Loss 0.8591 (1.2347) | acc 0.807 (0.687) | IoU 0.411 (0.353) | [2 days, 23:39:18]
Lr: 2.677e-03 | Update: 6.679e-04 mean,1.608e-03 std | Epoch: [0][150/564] | Time 2.249 (2.649) | Data 0.108 (0.408) | Loss 0.8893 (1.2146) | acc 0.722 (0.696) | IoU 0.378 (0.359) | [2 days, 23:41:58]
Lr: 2.855e-03 | Update: 1.439e-03 mean,3.152e-03 std | Epoch: [0][160/564] | Time 2.214 (2.660) | Data 0.095 (0.427) | Loss 0.8394 (1.1945) | acc 0.858 (0.702) | IoU 0.446 (0.362) | [3 days, 0:24:30]
Lr: 3.032e-03 | Update: 1.132e-03 mean,2.688e-03 std | Epoch: [0][170/564] | Time 2.255 (2.637) | Data 0.091 (0.409) | Loss 0.8871 (1.1771) | acc 0.822 (0.710) | IoU 0.433 (0.366) | [2 days, 23:25:10]
Lr: 3.209e-03 | Update: 1.026e-03 mean,2.266e-03 std | Epoch: [0][180/564] | Time 2.114 (2.635) | Data 0.080 (0.410) | Loss 0.8078 (1.1587) | acc 0.881 (0.720) | IoU 0.453 (0.372) | [2 days, 23:24:15]
Lr: 3.387e-03 | Update: 1.523e-03 mean,3.854e-03 std | Epoch: [0][190/564] | Time 2.088 (2.619) | Data 0.081 (0.396) | Loss 0.8563 (1.1420) | acc 0.843 (0.728) | IoU 0.435 (0.377) | [2 days, 22:41:19]
Lr: 3.564e-03 | Update: 1.542e-03 mean,3.290e-03 std | Epoch: [0][200/564] | Time 2.272 (2.608) | Data 0.084 (0.385) | Loss 0.9149 (1.1314) | acc 0.755 (0.736) | IoU 0.385 (0.381) | [2 days, 22:10:41]
Lr: 3.741e-03 | Update: 6.770e-04 mean,1.423e-03 std | Epoch: [0][210/564] | Time 2.240 (2.596) | Data 0.086 (0.381) | Loss 0.8294 (1.1216) | acc 0.857 (0.738) | IoU 0.440 (0.382) | [2 days, 21:46:33]
Lr: 3.918e-03 | Update: 1.067e-03 mean,2.263e-03 std | Epoch: [0][220/564] | Time 2.149 (2.599) | Data 0.095 (0.388) | Loss 0.8311 (1.1085) | acc 0.846 (0.746) | IoU 0.431 (0.387) | [2 days, 22:00:12]
Lr: 4.096e-03 | Update: 1.733e-03 mean,3.930e-03 std | Epoch: [0][230/564] | Time 2.183 (2.592) | Data 0.088 (0.384) | Loss 0.8637 (1.0968) | acc 0.864 (0.751) | IoU 0.448 (0.389) | [2 days, 21:44:29]

fengluodb · 2022-09-06T08:19:07Z

My output log has some difference with you. Maybe the problem is that.

Let's use 4 GPUs!
/home/db/miniconda3/envs/mos3d/lib/python3.7/site-packages/torch/optim/lr_scheduler.py:136: UserWarning: Detected call of `lr_scheduler.step()` before `optimizer.step()`. In PyTorch 1.1.0 and later, you should call them in the opposite order: `optimizer.step()` before `lr_scheduler.step()`.  Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate
  "https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate", UserWarning)
Ignoring class  0  in IoU evaluation
[IOU EVAL] IGNORE:  tensor([0])
[IOU EVAL] INCLUDE:  tensor([1, 2])
/home/db/miniconda3/envs/mos3d/lib/python3.7/site-packages/torch/nn/parallel/_functions.py:64: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.
  warnings.warn('Was asked to gather along dimension 0, but all '

MaxChanger · 2022-09-06T08:23:17Z

Yeah, I also have this warning, but I don't think it matters. I deleted these lines for cleanliness.
I have run up your Settings, convenient to tell me how long you run? If you don't remember, you can see it in the Tensorboard

fengluodb · 2022-09-06T08:25:18Z

I just run 3 hours. But the loss is always Nan.

MaxChanger · 2022-09-06T08:32:11Z

I just run 3 hours. But the loss is always Nan.

Thanks, I need to confirm whether the loss of the first 4 epochs is correct?
Everything seems to be working fine for me so far, it may take up to three hours.

fengluodb · 2022-09-06T08:38:31Z

Maybe I find the problem. I debug find that the model output is fulll with NaN.

fengluodb · 2022-09-06T08:43:32Z

I just find the in_vol is full with 0. Maybe the problem is raised, because the data read have error.

          [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ...,  0.0000e+00,
            0.0000e+00,  0.0000e+00],
          [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ...,  0.0000e+00,
            0.0000e+00,  0.0000e+00],
          [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ...,  0.0000e+00,
            0.0000e+00,  0.0000e+00]]]])

MaxChanger · 2022-09-06T08:45:14Z

Do you have check the visualization of residual images? Some results show be like as follows, right are 3 figures.

fengluodb · 2022-09-06T08:49:15Z

My don't generate the visualization of residual images in server. But I generate the visualization of residual images in my coumupter, it's normal.

fengluodb · 2022-09-06T09:21:38Z

I check the value of in_vol. I don't konw whether the value is normal, but they are so small

MaxChanger · 2022-09-06T12:51:15Z

Hi, waiting for a long time, I trained 4 epochs here, and no nan appeared when I reached the fifth epoch. I will continue to train to see if your issue will appear. The output log as follow

output log

Sequences folder exists! Using sequences from /home1/xxxx/Repo/MotionSeg3D/data/sequences
parsing seq 08
 There are 4071 frames in total.
 Using 4071 scans from sequences [8]
Loss weights from content:  tensor([  0.0000,   1.0210, 296.4371])
Channel of range image input =  5
Number of residual images input =  8
  "https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate", UserWarning)
Ignoring class  0  in IoU evaluation
[IOU EVAL] IGNORE:  tensor([0])
[IOU EVAL] INCLUDE:  tensor([1, 2])
/home1/xxxx/anaconda3/envs/mos3d/lib/python3.7/site-packages/torch/nn/parallel/_functions.py:64: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.
  warnings.warn('Was asked to gather along dimension 0, but all '
Lr: 9.551e-06 | Update: 1.579e-01 mean,3.646e-01 std | Epoch: [0][0/1047] | Time 12.647 (12.647) | Data 1.301 (1.301) | Loss 1.9132 (1.9132) | acc 0.503 (0.503) | IoU 0.261 (0.261) | [25 days, 8:28:35]
Lr: 1.051e-04 | Update: 1.625e-02 mean,4.200e-02 std | Epoch: [0][10/1047] | Time 2.750 (3.430) | Data 1.667 (1.417) | Loss 1.9071 (1.9316) | acc 0.531 (0.517) | IoU 0.270 (0.261) | [8 days, 19:25:03]
Lr: 2.006e-04 | Update: 5.366e-03 mean,1.278e-02 std | Epoch: [0][20/1047] | Time 2.015 (2.849) | Data 0.944 (1.286) | Loss 1.7890 (1.8875) | acc 0.624 (0.545) | IoU 0.317 (0.275) | [7 days, 12:22:01]
Lr: 2.961e-04 | Update: 3.678e-03 mean,8.984e-03 std | Epoch: [0][30/1047] | Time 2.373 (2.719) | Data 1.314 (1.341) | Loss 1.5860 (1.8333) | acc 0.703 (0.579) | IoU 0.355 (0.293) | [7 days, 9:02:51]
Lr: 3.916e-04 | Update: 2.311e-03 mean,5.385e-03 std | Epoch: [0][40/1047] | Time 2.581 (2.629) | Data 1.406 (1.310) | Loss 1.5813 (1.7658) | acc 0.687 (0.604) | IoU 0.345 (0.306) | [7 days, 3:49:10]
Lr: 4.871e-04 | Update: 1.333e-03 mean,3.043e-03 std | Epoch: [0][50/1047] | Time 2.354 (2.588) | Data 1.326 (1.304) | Loss 1.3176 (1.7056) | acc 0.734 (0.629) | IoU 0.376 (0.319) | [7 days, 1:44:31]
Lr: 5.826e-04 | Update: 1.689e-03 mean,4.035e-03 std | Epoch: [0][60/1047] | Time 2.331 (2.568) | Data 1.300 (1.320) | Loss 1.2942 (1.6491) | acc 0.697 (0.650) | IoU 0.355 (0.331) | [7 days, 1:32:38]
Lr: 6.781e-04 | Update: 1.268e-03 mean,2.949e-03 std | Epoch: [0][70/1047] | Time 2.278 (2.559) | Data 1.205 (1.325) | Loss 1.1285 (1.5918) | acc 0.876 (0.672) | IoU 0.481 (0.343) | [7 days, 1:23:16]
Lr: 7.736e-04 | Update: 8.686e-04 mean,1.962e-03 std | Epoch: [0][80/1047] | Time 2.927 (2.552) | Data 1.840 (1.339) | Loss 1.0479 (1.5344) | acc 0.840 (0.690) | IoU 0.424 (0.353) | [7 days, 1:39:24]
Lr: 8.691e-04 | Update: 1.198e-03 mean,2.806e-03 std | Epoch: [0][90/1047] | Time 2.352 (2.552) | Data 1.300 (1.354) | Loss 1.0332 (1.4817) | acc 0.794 (0.711) | IoU 0.406 (0.364) | [7 days, 2:18:49]
Lr: 9.647e-04 | Update: 8.290e-04 mean,1.899e-03 std | Epoch: [0][100/1047] | Time 2.344 (2.549) | Data 1.133 (1.360) | Loss 0.9701 (1.4306) | acc 0.860 (0.722) | IoU 0.454 (0.370) | [7 days, 2:24:19]
Lr: 1.060e-03 | Update: 9.992e-04 mean,2.213e-03 std | Epoch: [0][110/1047] | Time 2.461 (2.543) | Data 1.489 (1.360) | Loss 0.9070 (1.3859) | acc 0.852 (0.736) | IoU 0.447 (0.378) | [7 days, 2:09:17]
Lr: 1.156e-03 | Update: 7.116e-04 mean,1.664e-03 std | Epoch: [0][120/1047] | Time 2.225 (2.530) | Data 1.039 (1.343) | Loss 0.8492 (1.3452) | acc 0.894 (0.747) | IoU 0.485 (0.384) | [7 days, 0:48:36]
Lr: 1.251e-03 | Update: 1.089e-03 mean,2.731e-03 std | Epoch: [0][130/1047] | Time 2.437 (2.525) | Data 1.333 (1.339) | Loss 0.7845 (1.3100) | acc 0.885 (0.757) | IoU 0.465 (0.390) | [7 days, 0:23:48]
Lr: 1.347e-03 | Update: 1.280e-03 mean,3.044e-03 std | Epoch: [0][140/1047] | Time 2.684 (2.526) | Data 1.554 (1.344) | Loss 0.6662 (1.2746) | acc 0.941 (0.770) | IoU 0.488 (0.396) | [7 days, 0:40:42]
Lr: 1.442e-03 | Update: 5.753e-04 mean,1.273e-03 std | Epoch: [0][150/1047] | Time 2.673 (2.528) | Data 1.515 (1.351) | Loss 0.7347 (1.2447) | acc 0.894 (0.779) | IoU 0.473 (0.401) | [7 days, 1:03:58]
Lr: 1.538e-03 | Update: 1.711e-03 mean,4.238e-03 std | Epoch: [0][160/1047] | Time 2.182 (2.525) | Data 1.159 (1.353) | Loss 0.7408 (1.2163) | acc 0.929 (0.785) | IoU 0.474 (0.405) | [7 days, 1:00:21]
Lr: 1.633e-03 | Update: 1.667e-03 mean,3.765e-03 std | Epoch: [0][170/1047] | Time 2.383 (2.527) | Data 1.331 (1.360) | Loss 0.8254 (1.1950) | acc 0.862 (0.791) | IoU 0.434 (0.408) | [7 days, 1:21:42]
Lr: 1.729e-03 | Update: 6.654e-04 mean,1.433e-03 std | Epoch: [0][180/1047] | Time 2.767 (2.523) | Data 1.534 (1.360) | Loss 0.8085 (1.1774) | acc 0.845 (0.798) | IoU 0.458 (0.412) | [7 days, 1:12:08]
Lr: 1.824e-03 | Update: 8.009e-04 mean,1.794e-03 std | Epoch: [0][190/1047] | Time 2.275 (2.519) | Data 1.056 (1.355) | Loss 0.6827 (1.1555) | acc 0.996 (0.803) | IoU 0.526 (0.416) | [7 days, 0:45:59]
Lr: 1.920e-03 | Update: 2.597e-03 mean,5.816e-03 std | Epoch: [0][200/1047] | Time 2.424 (2.513) | Data 1.224 (1.349) | Loss 1.2700 (1.1395) | acc 0.925 (0.808) | IoU 0.523 (0.419) | [7 days, 0:16:33]
Lr: 2.015e-03 | Update: 5.836e-04 mean,1.235e-03 std | Epoch: [0][210/1047] | Time 2.691 (2.511) | Data 1.645 (1.352) | Loss 0.7207 (1.1247) | acc 0.873 (0.812) | IoU 0.444 (0.421) | [7 days, 0:17:34]
Lr: 2.111e-03 | Update: 1.918e-03 mean,4.756e-03 std | Epoch: [0][220/1047] | Time 2.726 (2.512) | Data 1.570 (1.354) | Loss 0.7615 (1.1113) | acc 0.874 (0.816) | IoU 0.446 (0.423) | [7 days, 0:25:49]
Lr: 2.206e-03 | Update: 1.322e-03 mean,3.124e-03 std | Epoch: [0][230/1047] | Time 2.476 (2.508) | Data 1.243 (1.352) | Loss 0.7561 (1.1010) | acc 0.929 (0.821) | IoU 0.472 (0.426) | [7 days, 0:10:27]
Lr: 2.302e-03 | Update: 1.535e-03 mean,3.500e-03 std | Epoch: [0][240/1047] | Time 2.670 (2.505) | Data 1.289 (1.346) | Loss 0.8113 (1.0902) | acc 0.894 (0.823) | IoU 0.452 (0.428) | [6 days, 23:44:34]
Lr: 2.397e-03 | Update: 2.856e-03 mean,6.554e-03 std | Epoch: [0][250/1047] | Time 2.240 (2.499) | Data 1.212 (1.343) | Loss 1.0410 (1.0808) | acc 0.815 (0.827) | IoU 0.424 (0.430) | [6 days, 23:21:14]
Lr: 2.493e-03 | Update: 7.094e-04 mean,1.485e-03 std | Epoch: [0][260/1047] | Time 2.647 (2.502) | Data 1.476 (1.346) | Loss 0.7701 (1.0721) | acc 0.975 (0.830) | IoU 0.518 (0.432) | [6 days, 23:37:21]
Lr: 2.588e-03 | Update: 1.118e-03 mean,2.737e-03 std | Epoch: [0][270/1047] | Time 2.134 (2.497) | Data 1.070 (1.339) | Loss 0.7353 (1.0603) | acc 0.916 (0.833) | IoU 0.470 (0.434) | [6 days, 23:04:37]
Lr: 2.684e-03 | Update: 1.065e-03 mean,2.424e-03 std | Epoch: [0][280/1047] | Time 2.310 (2.495) | Data 1.222 (1.338) | Loss 0.6982 (1.0513) | acc 0.946 (0.836) | IoU 0.523 (0.436) | [6 days, 22:54:31]
Lr: 2.779e-03 | Update: 8.265e-04 mean,1.761e-03 std | Epoch: [0][290/1047] | Time 2.697 (2.495) | Data 1.620 (1.341) | Loss 0.7314 (1.0423) | acc 0.903 (0.839) | IoU 0.470 (0.438) | [6 days, 23:01:24]
Lr: 2.875e-03 | Update: 2.241e-03 mean,4.692e-03 std | Epoch: [0][300/1047] | Time 2.080 (2.486) | Data 1.049 (1.334) | Loss 1.1129 (1.0350) | acc 0.967 (0.841) | IoU 0.503 (0.439) | [6 days, 22:19:54]
Lr: 2.970e-03 | Update: 7.071e-04 mean,1.441e-03 std | Epoch: [0][310/1047] | Time 2.816 (2.489) | Data 1.703 (1.336) | Loss 0.7379 (1.0282) | acc 0.902 (0.844) | IoU 0.458 (0.440) | [6 days, 22:32:46]
Lr: 3.066e-03 | Update: 1.630e-03 mean,3.690e-03 std | Epoch: [0][320/1047] | Time 2.144 (2.492) | Data 1.018 (1.339) | Loss 0.8597 (1.0210) | acc 0.826 (0.846) | IoU 0.423 (0.442) | [6 days, 22:44:51]
Lr: 3.161e-03 | Update: 1.472e-03 mean,3.088e-03 std | Epoch: [0][330/1047] | Time 2.465 (2.494) | Data 1.317 (1.341) | Loss 0.8644 (1.0154) | acc 0.924 (0.848) | IoU 0.526 (0.443) | [6 days, 22:58:11]
Lr: 3.257e-03 | Update: 1.005e-03 mean,2.851e-03 std | Epoch: [0][340/1047] | Time 2.831 (2.500) | Data 1.651 (1.347) | Loss 0.6173 (1.0065) | acc 0.937 (0.850) | IoU 0.498 (0.444) | [6 days, 23:29:32]
Lr: 3.352e-03 | Update: 7.584e-04 mean,1.554e-03 std | Epoch: [0][350/1047] | Time 2.417 (2.501) | Data 1.270 (1.347) | Loss 0.6959 (0.9999) | acc 0.904 (0.852) | IoU 0.473 (0.445) | [6 days, 23:28:05]
Lr: 3.448e-03 | Update: 9.897e-04 mean,2.158e-03 std | Epoch: [0][360/1047] | Time 2.639 (2.504) | Data 1.551 (1.350) | Loss 0.8042 (0.9960) | acc 0.865 (0.854) | IoU 0.439 (0.447) | [6 days, 23:45:33]
Lr: 3.543e-03 | Update: 1.748e-03 mean,4.505e-03 std | Epoch: [0][370/1047] | Time 1.926 (2.503) | Data 0.887 (1.350) | Loss 0.7487 (0.9888) | acc 0.927 (0.855) | IoU 0.474 (0.447) | [6 days, 23:40:38]
Lr: 3.639e-03 | Update: 1.938e-03 mean,4.059e-03 std | Epoch: [0][380/1047] | Time 2.537 (2.505) | Data 1.311 (1.351) | Loss 0.8351 (0.9838) | acc 0.879 (0.857) | IoU 0.479 (0.448) | [6 days, 23:48:32]
Lr: 3.734e-03 | Update: 5.555e-03 mean,1.380e-02 std | Epoch: [0][390/1047] | Time 2.743 (2.508) | Data 1.656 (1.354) | Loss 1.1202 (0.9799) | acc 0.916 (0.858) | IoU 0.486 (0.449) | [7 days, 0:03:23]
Lr: 3.830e-03 | Update: 1.649e-03 mean,4.768e-03 std | Epoch: [0][400/1047] | Time 2.657 (2.511) | Data 1.373 (1.358) | Loss 0.6852 (0.9746) | acc 0.950 (0.859) | IoU 0.483 (0.450) | [7 days, 0:22:25]
Lr: 3.926e-03 | Update: 3.202e-03 mean,8.860e-03 std | Epoch: [0][410/1047] | Time 2.643 (2.516) | Data 1.527 (1.363) | Loss 0.8996 (0.9747) | acc 0.859 (0.860) | IoU 0.432 (0.451) | [7 days, 0:46:14]
Lr: 4.021e-03 | Update: 9.669e-04 mean,1.989e-03 std | Epoch: [0][420/1047] | Time 2.615 (2.520) | Data 1.337 (1.366) | Loss 0.8768 (0.9727) | acc 0.883 (0.861) | IoU 0.463 (0.451) | [7 days, 1:05:07]
Lr: 4.117e-03 | Update: 1.969e-03 mean,4.417e-03 std | Epoch: [0][430/1047] | Time 2.724 (2.524) | Data 1.584 (1.370) | Loss 0.7839 (0.9700) | acc 0.960 (0.862) | IoU 0.589 (0.452) | [7 days, 1:22:42]
Lr: 4.212e-03 | Update: 1.953e-03 mean,3.675e-03 std | Epoch: [0][440/1047] | Time 2.528 (2.528) | Data 1.344 (1.373) | Loss 1.2602 (0.9667) | acc 0.918 (0.864) | IoU 0.470 (0.453) | [7 days, 1:40:26]
Lr: 4.308e-03 | Update: 2.655e-03 mean,1.010e-02 std | Epoch: [0][450/1047] | Time 2.430 (2.530) | Data 1.303 (1.375) | Loss 0.8391 (0.9637) | acc 0.996 (0.864) | IoU 0.515 (0.453) | [7 days, 1:53:47]
Lr: 4.403e-03 | Update: 1.135e-03 mean,2.366e-03 std | Epoch: [0][460/1047] | Time 2.772 (2.532) | Data 1.563 (1.377) | Loss 0.8759 (0.9599) | acc 0.935 (0.865) | IoU 0.483 (0.454) | [7 days, 2:01:57]
Lr: 4.499e-03 | Update: 1.648e-03 mean,3.519e-03 std | Epoch: [0][470/1047] | Time 2.034 (2.533) | Data 0.878 (1.377) | Loss 0.7658 (0.9558) | acc 0.925 (0.866) | IoU 0.474 (0.455) | [7 days, 2:02:54]
Lr: 4.594e-03 | Update: 5.714e-04 mean,1.138e-03 std | Epoch: [0][480/1047] | Time 2.334 (2.535) | Data 1.311 (1.379) | Loss 0.6352 (0.9527) | acc 0.880 (0.867) | IoU 0.445 (0.455) | [7 days, 2:15:22]
Lr: 4.690e-03 | Update: 5.623e-04 mean,1.094e-03 std | Epoch: [0][490/1047] | Time 2.745 (2.536) | Data 1.571 (1.380) | Loss 0.7400 (0.9498) | acc 0.855 (0.868) | IoU 0.433 (0.456) | [7 days, 2:17:59]
Lr: 4.785e-03 | Update: 9.261e-04 mean,1.800e-03 std | Epoch: [0][500/1047] | Time 2.464 (2.545) | Data 1.305 (1.389) | Loss 0.7770 (0.9467) | acc 0.914 (0.869) | IoU 0.479 (0.457) | [7 days, 3:05:10]
Lr: 4.881e-03 | Update: 1.127e-03 mean,2.135e-03 std | Epoch: [0][510/1047] | Time 2.559 (2.544) | Data 1.477 (1.389) | Loss 0.7733 (0.9435) | acc 0.918 (0.870) | IoU 0.468 (0.457) | [7 days, 3:02:18]
Lr: 4.976e-03 | Update: 9.463e-04 mean,2.094e-03 std | Epoch: [0][520/1047] | Time 2.665 (2.544) | Data 1.597 (1.391) | Loss 0.7514 (0.9412) | acc 0.934 (0.871) | IoU 0.483 (0.458) | [7 days, 3:05:02]
Lr: 5.072e-03 | Update: 6.239e-04 mean,1.230e-03 std | Epoch: [0][530/1047] | Time 2.713 (2.544) | Data 1.510 (1.392) | Loss 0.6412 (0.9358) | acc 0.979 (0.873) | IoU 0.578 (0.460) | [7 days, 3:08:33]
Lr: 5.167e-03 | Update: 1.812e-03 mean,3.781e-03 std | Epoch: [0][540/1047] | Time 2.457 (2.543) | Data 1.411 (1.392) | Loss 0.7798 (0.9315) | acc 0.908 (0.875) | IoU 0.478 (0.461) | [7 days, 3:04:31]
Lr: 5.263e-03 | Update: 6.219e-04 mean,1.292e-03 std | Epoch: [0][550/1047] | Time 2.201 (2.541) | Data 0.919 (1.392) | Loss 0.5778 (0.9270) | acc 0.964 (0.875) | IoU 0.585 (0.462) | [7 days, 2:59:49]
Lr: 5.358e-03 | Update: 2.284e-03 mean,4.718e-03 std | Epoch: [0][560/1047] | Time 2.921 (2.544) | Data 1.785 (1.395) | Loss 1.2335 (0.9233) | acc 0.982 (0.877) | IoU 0.594 (0.463) | [7 days, 3:12:30]
Lr: 5.454e-03 | Update: 1.555e-03 mean,3.969e-03 std | Epoch: [0][570/1047] | Time 2.355 (2.543) | Data 1.223 (1.395) | Loss 0.7468 (0.9196) | acc 0.943 (0.877) | IoU 0.520 (0.463) | [7 days, 3:09:46]
Lr: 5.549e-03 | Update: 4.145e-04 mean,7.546e-04 std | Epoch: [0][580/1047] | Time 2.626 (2.542) | Data 1.511 (1.395) | Loss 0.6135 (0.9162) | acc 0.936 (0.879) | IoU 0.483 (0.464) | [7 days, 3:08:01]
Lr: 5.645e-03 | Update: 6.645e-04 mean,1.345e-03 std | Epoch: [0][590/1047] | Time 2.785 (2.539) | Data 1.661 (1.394) | Loss 0.8057 (0.9163) | acc 0.814 (0.878) | IoU 0.410 (0.464) | [7 days, 2:57:30]
Lr: 5.740e-03 | Update: 8.223e-04 mean,1.517e-03 std | Epoch: [0][600/1047] | Time 2.347 (2.539) | Data 1.221 (1.395) | Loss 0.7623 (0.9141) | acc 0.912 (0.879) | IoU 0.462 (0.464) | [7 days, 2:59:36]
Lr: 5.836e-03 | Update: 4.141e-04 mean,8.162e-04 std | Epoch: [0][610/1047] | Time 2.408 (2.539) | Data 1.341 (1.396) | Loss 0.6397 (0.9103) | acc 0.971 (0.880) | IoU 0.544 (0.466) | [7 days, 3:00:07]
Lr: 5.931e-03 | Update: 6.788e-04 mean,1.265e-03 std | Epoch: [0][620/1047] | Time 2.557 (2.538) | Data 1.481 (1.397) | Loss 0.6688 (0.9083) | acc 0.956 (0.881) | IoU 0.501 (0.466) | [7 days, 2:58:59]
Lr: 6.027e-03 | Update: 1.531e-03 mean,3.549e-03 std | Epoch: [0][630/1047] | Time 2.335 (2.537) | Data 1.278 (1.396) | Loss 0.7941 (0.9060) | acc 0.889 (0.882) | IoU 0.493 (0.467) | [7 days, 2:54:15]
Lr: 6.122e-03 | Update: 1.401e-03 mean,2.835e-03 std | Epoch: [0][640/1047] | Time 2.279 (2.537) | Data 1.138 (1.396) | Loss 0.7996 (0.9050) | acc 0.990 (0.882) | IoU 0.612 (0.467) | [7 days, 2:52:54]
Lr: 6.218e-03 | Update: 3.342e-04 mean,6.851e-04 std | Epoch: [0][650/1047] | Time 2.338 (2.536) | Data 1.291 (1.396) | Loss 0.6704 (0.9028) | acc 0.919 (0.883) | IoU 0.470 (0.468) | [7 days, 2:48:42]
Lr: 6.313e-03 | Update: 9.230e-04 mean,2.055e-03 std | Epoch: [0][660/1047] | Time 1.989 (2.534) | Data 0.914 (1.394) | Loss 0.6238 (0.8998) | acc 0.998 (0.884) | IoU 0.695 (0.469) | [7 days, 2:38:58]
Lr: 6.409e-03 | Update: 1.361e-03 mean,2.987e-03 std | Epoch: [0][670/1047] | Time 2.269 (2.533) | Data 1.312 (1.394) | Loss 0.7666 (0.8969) | acc 0.934 (0.885) | IoU 0.493 (0.470) | [7 days, 2:35:30]
Lr: 6.504e-03 | Update: 6.876e-04 mean,1.506e-03 std | Epoch: [0][680/1047] | Time 2.780 (2.532) | Data 1.701 (1.395) | Loss 0.6020 (0.8936) | acc 0.913 (0.886) | IoU 0.475 (0.470) | [7 days, 2:35:49]
Lr: 6.600e-03 | Update: 8.734e-04 mean,1.623e-03 std | Epoch: [0][690/1047] | Time 2.647 (2.531) | Data 1.510 (1.394) | Loss 0.6495 (0.8917) | acc 0.905 (0.887) | IoU 0.471 (0.470) | [7 days, 2:29:06]
Lr: 6.695e-03 | Update: 6.435e-04 mean,1.537e-03 std | Epoch: [0][700/1047] | Time 2.386 (2.529) | Data 1.299 (1.393) | Loss 0.5610 (0.8883) | acc 0.997 (0.888) | IoU 0.693 (0.471) | [7 days, 2:20:02]
Lr: 6.791e-03 | Update: 1.886e-03 mean,3.829e-03 std | Epoch: [0][710/1047] | Time 2.497 (2.528) | Data 1.427 (1.392) | Loss 0.7742 (0.8858) | acc 0.932 (0.888) | IoU 0.472 (0.472) | [7 days, 2:14:30]
Lr: 6.886e-03 | Update: 9.669e-04 mean,1.753e-03 std | Epoch: [0][720/1047] | Time 2.476 (2.525) | Data 1.410 (1.389) | Loss 0.7748 (0.8839) | acc 0.929 (0.889) | IoU 0.483 (0.473) | [7 days, 1:58:02]
Lr: 6.982e-03 | Update: 1.690e-03 mean,3.547e-03 std | Epoch: [0][730/1047] | Time 2.569 (2.526) | Data 1.297 (1.389) | Loss 1.1382 (0.8822) | acc 0.987 (0.890) | IoU 0.630 (0.473) | [7 days, 1:59:14]
Lr: 7.077e-03 | Update: 1.390e-03 mean,2.881e-03 std | Epoch: [0][740/1047] | Time 2.743 (2.524) | Data 1.659 (1.386) | Loss 0.8406 (0.8806) | acc 0.899 (0.890) | IoU 0.481 (0.473) | [7 days, 1:46:09]
Lr: 7.173e-03 | Update: 1.045e-03 mean,1.934e-03 std | Epoch: [0][750/1047] | Time 2.384 (2.524) | Data 1.194 (1.387) | Loss 0.7398 (0.8788) | acc 0.990 (0.891) | IoU 0.666 (0.474) | [7 days, 1:48:34]
Lr: 7.268e-03 | Update: 5.514e-04 mean,9.696e-04 std | Epoch: [0][760/1047] | Time 2.420 (2.523) | Data 1.256 (1.385) | Loss 0.7007 (0.8769) | acc 0.897 (0.891) | IoU 0.464 (0.474) | [7 days, 1:41:37]
Lr: 7.364e-03 | Update: 6.106e-04 mean,1.214e-03 std | Epoch: [0][770/1047] | Time 2.536 (2.523) | Data 1.447 (1.385) | Loss 0.6077 (0.8740) | acc 0.992 (0.892) | IoU 0.665 (0.475) | [7 days, 1:39:00]
Lr: 7.459e-03 | Update: 9.396e-04 mean,1.622e-03 std | Epoch: [0][780/1047] | Time 2.343 (2.523) | Data 1.278 (1.385) | Loss 0.8258 (0.8722) | acc 0.965 (0.893) | IoU 0.494 (0.476) | [7 days, 1:38:59]
Lr: 7.555e-03 | Update: 4.524e-04 mean,9.995e-04 std | Epoch: [0][790/1047] | Time 2.314 (2.523) | Data 1.089 (1.386) | Loss 0.7351 (0.8713) | acc 0.838 (0.893) | IoU 0.444 (0.476) | [7 days, 1:40:14]
Lr: 7.650e-03 | Update: 5.381e-04 mean,9.517e-04 std | Epoch: [0][800/1047] | Time 2.540 (2.522) | Data 1.354 (1.384) | Loss 0.7208 (0.8704) | acc 0.887 (0.893) | IoU 0.455 (0.476) | [7 days, 1:31:44]
Lr: 7.746e-03 | Update: 4.082e-04 mean,7.186e-04 std | Epoch: [0][810/1047] | Time 2.039 (2.521) | Data 0.807 (1.383) | Loss 0.6611 (0.8690) | acc 0.960 (0.894) | IoU 0.506 (0.476) | [7 days, 1:25:55]
Lr: 7.841e-03 | Update: 4.009e-04 mean,6.378e-04 std | Epoch: [0][820/1047] | Time 2.429 (2.520) | Data 1.299 (1.381) | Loss 0.6614 (0.8671) | acc 0.954 (0.894) | IoU 0.530 (0.477) | [7 days, 1:18:01]
Lr: 7.937e-03 | Update: 6.184e-04 mean,1.092e-03 std | Epoch: [0][830/1047] | Time 2.495 (2.519) | Data 1.510 (1.380) | Loss 0.8044 (0.8662) | acc 0.907 (0.895) | IoU 0.494 (0.477) | [7 days, 1:10:18]
Lr: 8.032e-03 | Update: 3.739e-04 mean,6.361e-04 std | Epoch: [0][840/1047] | Time 2.118 (2.517) | Data 0.999 (1.378) | Loss 0.6879 (0.8641) | acc 0.984 (0.895) | IoU 0.549 (0.477) | [7 days, 1:00:42]
Lr: 8.128e-03 | Update: 2.605e-04 mean,6.422e-04 std | Epoch: [0][850/1047] | Time 2.493 (2.517) | Data 1.276 (1.378) | Loss 0.5741 (0.8615) | acc 0.950 (0.896) | IoU 0.482 (0.478) | [7 days, 0:58:47]
Lr: 8.223e-03 | Update: 3.486e-04 mean,6.885e-04 std | Epoch: [0][860/1047] | Time 2.359 (2.515) | Data 1.047 (1.376) | Loss 0.6499 (0.8610) | acc 0.859 (0.897) | IoU 0.457 (0.478) | [7 days, 0:49:11]
Lr: 8.319e-03 | Update: 2.398e-03 mean,4.698e-03 std | Epoch: [0][870/1047] | Time 2.359 (2.514) | Data 1.248 (1.375) | Loss 0.8596 (0.8603) | acc 0.912 (0.897) | IoU 0.466 (0.478) | [7 days, 0:44:18]
Lr: 8.415e-03 | Update: 6.602e-04 mean,1.329e-03 std | Epoch: [0][880/1047] | Time 2.502 (2.514) | Data 1.309 (1.374) | Loss 0.7988 (0.8591) | acc 0.874 (0.897) | IoU 0.446 (0.478) | [7 days, 0:41:43]
Lr: 8.510e-03 | Update: 5.196e-04 mean,9.264e-04 std | Epoch: [0][890/1047] | Time 2.280 (2.514) | Data 1.226 (1.374) | Loss 0.5362 (0.8571) | acc 0.994 (0.898) | IoU 0.662 (0.479) | [7 days, 0:38:04]
Lr: 8.606e-03 | Update: 1.692e-03 mean,3.439e-03 std | Epoch: [0][900/1047] | Time 2.888 (2.515) | Data 1.602 (1.375) | Loss 0.6705 (0.8547) | acc 0.977 (0.898) | IoU 0.624 (0.480) | [7 days, 0:42:42]
Lr: 8.701e-03 | Update: 4.740e-04 mean,8.510e-04 std | Epoch: [0][910/1047] | Time 2.303 (2.515) | Data 1.167 (1.374) | Loss 0.6010 (0.8526) | acc 0.939 (0.899) | IoU 0.495 (0.480) | [7 days, 0:41:24]
Lr: 8.797e-03 | Update: 9.422e-04 mean,1.727e-03 std | Epoch: [0][920/1047] | Time 2.289 (2.513) | Data 1.006 (1.372) | Loss 0.7965 (0.8508) | acc 0.989 (0.899) | IoU 0.670 (0.481) | [7 days, 0:31:07]
Lr: 8.892e-03 | Update: 3.877e-04 mean,6.446e-04 std | Epoch: [0][930/1047] | Time 2.231 (2.513) | Data 1.077 (1.372) | Loss 0.5667 (0.8487) | acc 0.927 (0.900) | IoU 0.472 (0.481) | [7 days, 0:28:01]
Lr: 8.988e-03 | Update: 5.974e-04 mean,1.021e-03 std | Epoch: [0][940/1047] | Time 2.318 (2.511) | Data 1.166 (1.370) | Loss 0.6684 (0.8479) | acc 0.932 (0.900) | IoU 0.522 (0.481) | [7 days, 0:20:08]
Lr: 9.083e-03 | Update: 4.019e-04 mean,7.437e-04 std | Epoch: [0][950/1047] | Time 2.653 (2.511) | Data 1.550 (1.370) | Loss 0.5914 (0.8464) | acc 0.956 (0.900) | IoU 0.493 (0.482) | [7 days, 0:16:33]
Lr: 9.179e-03 | Update: 3.633e-04 mean,7.251e-04 std | Epoch: [0][960/1047] | Time 2.518 (2.512) | Data 1.453 (1.371) | Loss 0.5754 (0.8452) | acc 0.962 (0.901) | IoU 0.572 (0.482) | [7 days, 0:22:49]
Lr: 9.274e-03 | Update: 8.452e-04 mean,1.432e-03 std | Epoch: [0][970/1047] | Time 2.385 (2.511) | Data 1.347 (1.370) | Loss 0.9197 (0.8446) | acc 0.896 (0.901) | IoU 0.490 (0.482) | [7 days, 0:15:09]
Lr: 9.370e-03 | Update: 3.664e-04 mean,8.862e-04 std | Epoch: [0][980/1047] | Time 2.176 (2.511) | Data 1.030 (1.370) | Loss 0.5625 (0.8433) | acc 0.940 (0.901) | IoU 0.477 (0.482) | [7 days, 0:15:59]
Lr: 9.465e-03 | Update: 6.226e-04 mean,1.149e-03 std | Epoch: [0][990/1047] | Time 2.714 (2.511) | Data 1.617 (1.371) | Loss 0.8802 (0.8416) | acc 0.986 (0.902) | IoU 0.526 (0.483) | [7 days, 0:17:00]
Lr: 9.561e-03 | Update: 1.485e-03 mean,2.518e-03 std | Epoch: [0][1000/1047] | Time 2.563 (2.510) | Data 1.485 (1.370) | Loss 0.9931 (0.8409) | acc 0.893 (0.902) | IoU 0.475 (0.483) | [7 days, 0:11:47]
Lr: 9.656e-03 | Update: 2.691e-03 mean,4.982e-03 std | Epoch: [0][1010/1047] | Time 2.781 (2.511) | Data 1.665 (1.371) | Loss 0.9067 (0.8402) | acc 0.969 (0.903) | IoU 0.597 (0.483) | [7 days, 0:15:58]
Lr: 9.752e-03 | Update: 4.912e-04 mean,9.211e-04 std | Epoch: [0][1020/1047] | Time 2.317 (2.510) | Data 1.186 (1.370) | Loss 0.6171 (0.8390) | acc 0.962 (0.903) | IoU 0.539 (0.483) | [7 days, 0:09:40]
Lr: 9.847e-03 | Update: 1.077e-03 mean,1.846e-03 std | Epoch: [0][1030/1047] | Time 2.844 (2.509) | Data 1.730 (1.369) | Loss 0.9557 (0.8378) | acc 0.966 (0.903) | IoU 0.604 (0.484) | [7 days, 0:05:22]
Lr: 9.943e-03 | Update: 4.561e-04 mean,7.759e-04 std | Epoch: [0][1040/1047] | Time 2.756 (2.508) | Data 1.754 (1.368) | Loss 0.6414 (0.8366) | acc 0.930 (0.904) | IoU 0.486 (0.484) | [6 days, 23:57:31]
Best mean iou in training set so far, save model!
Validation:: 100%|████████████████████████████| 508/508 [20:14<00:00,  2.39s/it]
********************************************************************************
Validation set:
Time avg per batch 2.390
Loss avg 0.6212
Jaccard avg 0.3498
WCE avg 0.2714
Acc avg 0.915623
IoU avg 0.475279
IoU class 0 [unlabeled] = 0.000000
IoU class 1 [static] = 0.915363
IoU class 2 [moving] = 0.035195
********************************************************************************
Best mean iou in validation so far, save model!
********************************************************************************
Lr: 1.000e-02 | Update: 6.661e-04 mean,1.239e-03 std | Epoch: [1][0/1047] | Time 2.168 (2.507) | Data 1.165 (1.368) | Loss 0.6463 (0.6463) | acc 0.953 (0.953) | IoU 0.519 (0.519) | [9 days, 2:09:17]
Lr: 9.999e-03 | Update: 8.192e-04 mean,1.428e-03 std | Epoch: [1][10/1047] | Time 2.537 (2.504) | Data 1.236 (1.365) | Loss 0.7102 (0.6836) | acc 0.994 (0.965) | IoU 0.586 (0.528) | [9 days, 1:53:08]
Lr: 9.998e-03 | Update: 5.564e-04 mean,1.015e-03 std | Epoch: [1][20/1047] | Time 2.308 (2.500) | Data 1.216 (1.362) | Loss 0.6141 (0.6716) | acc 0.919 (0.950) | IoU 0.474 (0.529) | [9 days, 1:35:13]
Lr: 9.997e-03 | Update: 5.368e-04 mean,9.257e-04 std | Epoch: [1][30/1047] | Time 1.674 (2.497) | Data 0.665 (1.358) | Loss 0.5842 (0.6826) | acc 0.948 (0.959) | IoU 0.509 (0.547) | [9 days, 1:16:34]
Lr: 9.996e-03 | Update: 4.074e-04 mean,7.694e-04 std | Epoch: [1][40/1047] | Time 2.087 (2.494) | Data 0.930 (1.355) | Loss 0.6891 (0.6953) | acc 0.918 (0.939) | IoU 0.466 (0.527) | [9 days, 1:00:28]
Lr: 9.995e-03 | Update: 4.502e-04 mean,7.399e-04 std | Epoch: [1][50/1047] | Time 2.152 (2.491) | Data 1.097 (1.352) | Loss 0.6182 (0.6930) | acc 0.918 (0.943) | IoU 0.511 (0.526) | [9 days, 0:42:53]
Lr: 9.994e-03 | Update: 4.795e-04 mean,8.169e-04 std | Epoch: [1][60/1047] | Time 2.257 (2.488) | Data 1.172 (1.349) | Loss 0.6281 (0.6976) | acc 0.950 (0.938) | IoU 0.517 (0.519) | [9 days, 0:26:57]
Lr: 9.993e-03 | Update: 1.535e-03 mean,2.619e-03 std | Epoch: [1][70/1047] | Time 2.086 (2.485) | Data 0.973 (1.346) | Loss 0.8987 (0.6945) | acc 0.962 (0.944) | IoU 0.518 (0.526) | [9 days, 0:11:40]
Lr: 9.992e-03 | Update: 5.964e-04 mean,1.015e-03 std | Epoch: [1][80/1047] | Time 2.117 (2.481) | Data 1.142 (1.342) | Loss 0.7664 (0.6973) | acc 0.868 (0.941) | IoU 0.440 (0.523) | [8 days, 23:51:03]
Lr: 9.991e-03 | Update: 1.051e-03 mean,1.839e-03 std | Epoch: [1][90/1047] | Time 2.017 (2.479) | Data 0.772 (1.340) | Loss 0.7672 (0.6988) | acc 0.980 (0.942) | IoU 0.500 (0.521) | [8 days, 23:37:44]
Lr: 9.990e-03 | Update: 5.267e-04 mean,1.023e-03 std | Epoch: [1][100/1047] | Time 2.408 (2.476) | Data 1.182 (1.336) | Loss 0.7222 (0.7019) | acc 0.883 (0.938) | IoU 0.446 (0.518) | [8 days, 23:19:53]
Lr: 9.989e-03 | Update: 1.351e-03 mean,2.296e-03 std | Epoch: [1][110/1047] | Time 1.848 (2.472) | Data 0.760 (1.333) | Loss 0.8049 (0.7058) | acc 0.874 (0.937) | IoU 0.448 (0.517) | [8 days, 23:03:02]
Lr: 9.988e-03 | Update: 6.096e-04 mean,9.729e-04 std | Epoch: [1][120/1047] | Time 2.193 (2.469) | Data 1.010 (1.331) | Loss 0.6868 (0.7071) | acc 0.987 (0.934) | IoU 0.526 (0.514) | [8 days, 22:48:03]
Lr: 9.988e-03 | Update: 5.171e-04 mean,8.854e-04 std | Epoch: [1][130/1047] | Time 1.867 (2.467) | Data 0.753 (1.329) | Loss 0.6957 (0.7080) | acc 0.918 (0.935) | IoU 0.484 (0.516) | [8 days, 22:37:02]
Lr: 9.987e-03 | Update: 9.760e-04 mean,1.632e-03 std | Epoch: [1][140/1047] | Time 2.196 (2.465) | Data 0.879 (1.326) | Loss 0.6056 (0.7064) | acc 0.986 (0.935) | IoU 0.596 (0.516) | [8 days, 22:21:36]
Lr: 9.986e-03 | Update: 1.078e-03 mean,1.816e-03 std | Epoch: [1][150/1047] | Time 2.169 (2.462) | Data 1.053 (1.323) | Loss 0.8640 (0.7111) | acc 0.886 (0.935) | IoU 0.462 (0.515) | [8 days, 22:04:58]
Lr: 9.985e-03 | Update: 4.061e-04 mean,7.449e-04 std | Epoch: [1][160/1047] | Time 2.066 (2.459) | Data 0.758 (1.320) | Loss 0.4711 (0.7139) | acc 0.975 (0.935) | IoU 0.552 (0.515) | [8 days, 21:50:14]
Lr: 9.984e-03 | Update: 2.174e-04 mean,3.537e-04 std | Epoch: [1][170/1047] | Time 2.183 (2.456) | Data 1.117 (1.317) | Loss 0.6116 (0.7098) | acc 0.945 (0.935) | IoU 0.511 (0.515) | [8 days, 21:35:01]
Lr: 9.983e-03 | Update: 4.760e-04 mean,8.167e-04 std | Epoch: [1][180/1047] | Time 1.959 (2.454) | Data 0.705 (1.315) | Loss 0.5687 (0.7084) | acc 0.897 (0.937) | IoU 0.456 (0.516) | [8 days, 21:22:36]
Lr: 9.982e-03 | Update: 4.058e-04 mean,6.828e-04 std | Epoch: [1][190/1047] | Time 2.086 (2.451) | Data 0.977 (1.312) | Loss 0.5245 (0.7038) | acc 0.983 (0.936) | IoU 0.582 (0.515) | [8 days, 21:09:08]
Lr: 9.981e-03 | Update: 1.755e-03 mean,2.859e-03 std | Epoch: [1][200/1047] | Time 2.374 (2.449) | Data 1.250 (1.310) | Loss 0.9470 (0.7034) | acc 0.955 (0.939) | IoU 0.540 (0.519) | [8 days, 20:56:31]
Lr: 9.980e-03 | Update: 3.865e-04 mean,6.273e-04 std | Epoch: [1][210/1047] | Time 2.101 (2.446) | Data 1.063 (1.307) | Loss 0.7084 (0.7033) | acc 0.896 (0.937) | IoU 0.461 (0.517) | [8 days, 20:40:50]
Lr: 9.979e-03 | Update: 3.937e-04 mean,6.325e-04 std | Epoch: [1][220/1047] | Time 1.996 (2.444) | Data 0.778 (1.304) | Loss 0.5857 (0.7019) | acc 0.925 (0.937) | IoU 0.486 (0.517) | [8 days, 20:26:23]
Lr: 9.978e-03 | Update: 4.642e-04 mean,7.834e-04 std | Epoch: [1][230/1047] | Time 2.565 (2.441) | Data 1.431 (1.302) | Loss 0.6222 (0.7042) | acc 0.921 (0.936) | IoU 0.492 (0.517) | [8 days, 20:12:58]
Lr: 9.977e-03 | Update: 3.713e-04 mean,5.843e-04 std | Epoch: [1][240/1047] | Time 2.192 (2.439) | Data 1.070 (1.300) | Loss 0.6053 (0.7037) | acc 0.965 (0.937) | IoU 0.596 (0.518) | [8 days, 20:03:05]
Lr: 9.976e-03 | Update: 5.921e-04 mean,1.003e-03 std | Epoch: [1][250/1047] | Time 2.056 (2.437) | Data 0.889 (1.298) | Loss 0.6812 (0.7016) | acc 0.970 (0.937) | IoU 0.508 (0.517) | [8 days, 19:50:19]
Lr: 9.975e-03 | Update: 4.818e-04 mean,7.490e-04 std | Epoch: [1][260/1047] | Time 2.159 (2.435) | Data 1.138 (1.296) | Loss 0.6329 (0.6988) | acc 0.941 (0.939) | IoU 0.496 (0.521) | [8 days, 19:38:18]
Lr: 9.974e-03 | Update: 3.753e-04 mean,6.455e-04 std | Epoch: [1][270/1047] | Time 1.823 (2.432) | Data 0.696 (1.294) | Loss 0.5991 (0.6982) | acc 0.930 (0.938) | IoU 0.485 (0.519) | [8 days, 19:26:01]
Lr: 9.973e-03 | Update: 1.136e-03 mean,1.893e-03 std | Epoch: [1][280/1047] | Time 2.080 (2.431) | Data 0.907 (1.292) | Loss 1.0192 (0.6973) | acc 0.977 (0.939) | IoU 0.516 (0.520) | [8 days, 19:15:28]
Lr: 9.972e-03 | Update: 3.413e-04 mean,6.132e-04 std | Epoch: [1][290/1047] | Time 2.416 (2.429) | Data 1.319 (1.290) | Loss 0.6255 (0.6958) | acc 0.898 (0.938) | IoU 0.461 (0.519) | [8 days, 19:07:09]
Lr: 9.971e-03 | Update: 1.021e-03 mean,1.905e-03 std | Epoch: [1][300/1047] | Time 2.338 (2.427) | Data 1.260 (1.289) | Loss 0.7921 (0.6950) | acc 0.985 (0.940) | IoU 0.699 (0.521) | [8 days, 18:58:11]
Lr: 9.970e-03 | Update: 6.826e-04 mean,1.193e-03 std | Epoch: [1][310/1047] | Time 1.839 (2.425) | Data 0.680 (1.286) | Loss 1.0124 (0.6938) | acc 0.942 (0.940) | IoU 0.493 (0.521) | [8 days, 18:46:04]
Lr: 9.969e-03 | Update: 1.355e-03 mean,2.464e-03 std | Epoch: [1][320/1047] | Time 1.968 (2.424) | Data 0.805 (1.285) | Loss 0.7558 (0.6932) | acc 0.971 (0.941) | IoU 0.517 (0.521) | [8 days, 18:37:01]
Lr: 9.968e-03 | Update: 4.943e-04 mean,8.049e-04 std | Epoch: [1][330/1047] | Time 2.157 (2.423) | Data 0.962 (1.284) | Loss 0.7105 (0.6919) | acc 0.948 (0.941) | IoU 0.484 (0.522) | [8 days, 18:34:01]
Lr: 9.967e-03 | Update: 1.473e-03 mean,2.464e-03 std | Epoch: [1][340/1047] | Time 1.742 (2.421) | Data 0.696 (1.282) | Loss 0.8216 (0.6930) | acc 0.979 (0.942) | IoU 0.516 (0.523) | [8 days, 18:20:15]
Lr: 9.966e-03 | Update: 6.841e-04 mean,1.155e-03 std | Epoch: [1][350/1047] | Time 2.185 (2.419) | Data 1.062 (1.280) | Loss 0.6928 (0.6915) | acc 0.963 (0.942) | IoU 0.553 (0.523) | [8 days, 18:11:22]
Lr: 9.966e-03 | Update: 5.011e-04 mean,8.797e-04 std | Epoch: [1][360/1047] | Time 2.545 (2.418) | Data 1.501 (1.279) | Loss 0.5821 (0.6919) | acc 0.910 (0.942) | IoU 0.475 (0.523) | [8 days, 18:05:31]
Lr: 9.965e-03 | Update: 5.356e-04 mean,8.798e-04 std | Epoch: [1][370/1047] | Time 2.060 (2.417) | Data 0.963 (1.278) | Loss 0.6512 (0.6917) | acc 0.976 (0.943) | IoU 0.534 (0.524) | [8 days, 17:57:37]
Lr: 9.964e-03 | Update: 2.823e-04 mean,4.791e-04 std | Epoch: [1][380/1047] | Time 2.174 (2.415) | Data 1.043 (1.276) | Loss 0.4583 (0.6894) | acc 0.980 (0.943) | IoU 0.534 (0.525) | [8 days, 17:46:36]
Lr: 9.963e-03 | Update: 7.853e-04 mean,1.388e-03 std | Epoch: [1][390/1047] | Time 2.378 (2.413) | Data 1.347 (1.274) | Loss 0.8234 (0.6899) | acc 0.966 (0.944) | IoU 0.518 (0.525) | [8 days, 17:39:03]
Lr: 9.962e-03 | Update: 2.684e-04 mean,5.790e-04 std | Epoch: [1][400/1047] | Time 2.146 (2.412) | Data 1.112 (1.273) | Loss 0.5726 (0.6898) | acc 0.917 (0.944) | IoU 0.472 (0.524) | [8 days, 17:33:20]
Lr: 9.961e-03 | Update: 9.585e-04 mean,1.651e-03 std | Epoch: [1][410/1047] | Time 2.229 (2.411) | Data 1.097 (1.272) | Loss 0.8193 (0.6888) | acc 0.975 (0.944) | IoU 0.499 (0.525) | [8 days, 17:26:15]
Lr: 9.960e-03 | Update: 3.723e-04 mean,5.733e-04 std | Epoch: [1][420/1047] | Time 2.000 (2.409) | Data 0.797 (1.270) | Loss 0.4709 (0.6871) | acc 0.951 (0.945) | IoU 0.499 (0.525) | [8 days, 17:16:09]
Lr: 9.959e-03 | Update: 3.546e-04 mean,5.208e-04 std | Epoch: [1][430/1047] | Time 2.412 (2.408) | Data 1.284 (1.268) | Loss 0.6429 (0.6852) | acc 0.951 (0.945) | IoU 0.494 (0.526) | [8 days, 17:06:25]
Lr: 9.958e-03 | Update: 3.640e-04 mean,6.258e-04 std | Epoch: [1][440/1047] | Time 2.057 (2.406) | Data 0.798 (1.266) | Loss 0.4489 (0.6826) | acc 0.985 (0.946) | IoU 0.530 (0.527) | [8 days, 16:56:03]
Lr: 9.957e-03 | Update: 3.100e-04 mean,6.062e-04 std | Epoch: [1][450/1047] | Time 2.189 (2.404) | Data 0.971 (1.265) | Loss 0.5231 (0.6815) | acc 0.932 (0.946) | IoU 0.496 (0.528) | [8 days, 16:47:22]
Lr: 9.956e-03 | Update: 4.023e-04 mean,6.638e-04 std | Epoch: [1][460/1047] | Time 2.409 (2.403) | Data 1.315 (1.264) | Loss 0.5911 (0.6803) | acc 0.981 (0.947) | IoU 0.533 (0.527) | [8 days, 16:40:25]
Lr: 9.955e-03 | Update: 5.970e-04 mean,9.486e-04 std | Epoch: [1][470/1047] | Time 2.459 (2.402) | Data 1.304 (1.263) | Loss 0.6564 (0.6805) | acc 0.889 (0.947) | IoU 0.466 (0.528) | [8 days, 16:35:06]
Lr: 9.954e-03 | Update: 2.072e-03 mean,3.423e-03 std | Epoch: [1][480/1047] | Time 1.821 (2.401) | Data 0.621 (1.261) | Loss 1.5904 (0.6820) | acc 0.963 (0.946) | IoU 0.500 (0.527) | [8 days, 16:27:55]
Lr: 9.953e-03 | Update: 4.798e-04 mean,8.193e-04 std | Epoch: [1][490/1047] | Time 2.044 (2.400) | Data 0.780 (1.260) | Loss 0.7137 (0.6807) | acc 0.931 (0.947) | IoU 0.472 (0.528) | [8 days, 16:20:43]
Lr: 9.952e-03 | Update: 2.930e-04 mean,5.397e-04 std | Epoch: [1][500/1047] | Time 2.394 (2.399) | Data 1.378 (1.259) | Loss 0.6003 (0.6803) | acc 0.924 (0.947) | IoU 0.482 (0.527) | [8 days, 16:13:10]
Lr: 9.951e-03 | Update: 3.052e-04 mean,4.483e-04 std | Epoch: [1][510/1047] | Time 2.396 (2.398) | Data 1.281 (1.257) | Loss 0.5956 (0.6796) | acc 0.948 (0.947) | IoU 0.540 (0.528) | [8 days, 16:07:03]
Lr: 9.950e-03 | Update: 4.942e-04 mean,7.485e-04 std | Epoch: [1][520/1047] | Time 2.615 (2.397) | Data 1.472 (1.257) | Loss 0.6124 (0.6783) | acc 0.944 (0.947) | IoU 0.495 (0.528) | [8 days, 16:01:37]
Lr: 9.949e-03 | Update: 3.981e-04 mean,7.767e-04 std | Epoch: [1][530/1047] | Time 2.632 (2.396) | Data 1.504 (1.256) | Loss 0.6537 (0.6784) | acc 0.902 (0.946) | IoU 0.456 (0.527) | [8 days, 15:57:36]
Lr: 9.948e-03 | Update: 2.191e-04 mean,4.208e-04 std | Epoch: [1][540/1047] | Time 2.425 (2.395) | Data 1.303 (1.255) | Loss 0.6230 (0.6782) | acc 0.946 (0.947) | IoU 0.492 (0.528) | [8 days, 15:53:05]
Lr: 9.947e-03 | Update: 7.328e-04 mean,1.205e-03 std | Epoch: [1][550/1047] | Time 2.639 (2.394) | Data 1.469 (1.254) | Loss 0.7139 (0.6774) | acc 0.986 (0.948) | IoU 0.640 (0.529) | [8 days, 15:48:04]
Lr: 9.946e-03 | Update: 3.760e-04 mean,6.023e-04 std | Epoch: [1][560/1047] | Time 2.434 (2.393) | Data 1.283 (1.253) | Loss 0.6200 (0.6767) | acc 0.952 (0.947) | IoU 0.544 (0.529) | [8 days, 15:41:44]
Lr: 9.945e-03 | Update: 6.692e-04 mean,1.064e-03 std | Epoch: [1][570/1047] | Time 1.645 (2.392) | Data 0.582 (1.252) | Loss 0.6151 (0.6751) | acc 0.983 (0.948) | IoU 0.537 (0.529) | [8 days, 15:33:56]
Lr: 9.944e-03 | Update: 2.154e-04 mean,5.035e-04 std | Epoch: [1][580/1047] | Time 2.158 (2.392) | Data 0.937 (1.251) | Loss 0.6403 (0.6747) | acc 0.911 (0.948) | IoU 0.466 (0.530) | [8 days, 15:31:12]
Lr: 9.944e-03 | Update: 3.014e-04 mean,5.295e-04 std | Epoch: [1][590/1047] | Time 2.300 (2.391) | Data 1.152 (1.250) | Loss 0.5594 (0.6748) | acc 0.959 (0.948) | IoU 0.503 (0.529) | [8 days, 15:24:51]
Lr: 9.943e-03 | Update: 2.931e-04 mean,5.556e-04 std | Epoch: [1][600/1047] | Time 1.959 (2.390) | Data 0.773 (1.250) | Loss 0.6065 (0.6746) | acc 0.919 (0.948) | IoU 0.477 (0.530) | [8 days, 15:22:01]
Lr: 9.942e-03 | Update: 6.888e-04 mean,1.263e-03 std | Epoch: [1][610/1047] | Time 2.587 (2.390) | Data 1.412 (1.250) | Loss 0.6928 (0.6750) | acc 0.916 (0.947) | IoU 0.518 (0.530) | [8 days, 15:21:51]
Lr: 9.941e-03 | Update: 3.302e-04 mean,6.462e-04 std | Epoch: [1][620/1047] | Time 2.878 (2.390) | Data 1.626 (1.250) | Loss 0.4457 (0.6750) | acc 0.990 (0.947) | IoU 0.632 (0.530) | [8 days, 15:21:40]
Lr: 9.940e-03 | Update: 1.751e-04 mean,3.219e-04 std | Epoch: [1][630/1047] | Time 2.397 (2.391) | Data 1.218 (1.250) | Loss 0.5590 (0.6745) | acc 0.972 (0.948) | IoU 0.569 (0.530) | [8 days, 15:23:39]
Lr: 9.939e-03 | Update: 3.210e-04 mean,5.631e-04 std | Epoch: [1][640/1047] | Time 2.340 (2.392) | Data 1.048 (1.251) | Loss 0.4458 (0.6731) | acc 0.996 (0.948) | IoU 0.758 (0.531) | [8 days, 15:26:02]
Lr: 9.938e-03 | Update: 2.840e-03 mean,5.020e-03 std | Epoch: [1][650/1047] | Time 2.509 (2.391) | Data 1.218 (1.250) | Loss 0.9586 (0.6723) | acc 0.963 (0.949) | IoU 0.512 (0.532) | [8 days, 15:23:44]
Lr: 9.937e-03 | Update: 3.434e-04 mean,5.058e-04 std | Epoch: [1][660/1047] | Time 2.044 (2.392) | Data 0.920 (1.250) | Loss 0.6074 (0.6722) | acc 0.963 (0.949) | IoU 0.508 (0.531) | [8 days, 15:23:01]
Lr: 9.936e-03 | Update: 5.295e-04 mean,8.214e-04 std | Epoch: [1][670/1047] | Time 2.259 (2.391) | Data 1.134 (1.250) | Loss 0.6103 (0.6721) | acc 0.897 (0.948) | IoU 0.464 (0.532) | [8 days, 15:21:09]
Lr: 9.935e-03 | Update: 6.809e-04 mean,1.180e-03 std | Epoch: [1][680/1047] | Time 2.494 (2.391) | Data 1.282 (1.249) | Loss 0.7588 (0.6714) | acc 0.992 (0.949) | IoU 0.599 (0.532) | [8 days, 15:19:04]
Lr: 9.934e-03 | Update: 6.857e-04 mean,9.862e-04 std | Epoch: [1][690/1047] | Time 2.619 (2.392) | Data 1.453 (1.249) | Loss 0.5607 (0.6702) | acc 0.973 (0.949) | IoU 0.549 (0.533) | [8 days, 15:21:04]
Lr: 9.933e-03 | Update: 2.673e-04 mean,4.328e-04 std | Epoch: [1][700/1047] | Time 2.987 (2.393) | Data 1.657 (1.250) | Loss 0.6079 (0.6700) | acc 0.902 (0.949) | IoU 0.471 (0.533) | [8 days, 15:23:19]
Lr: 9.932e-03 | Update: 2.070e-04 mean,4.226e-04 std | Epoch: [1][710/1047] | Time 2.385 (2.394) | Data 1.256 (1.250) | Loss 0.5902 (0.6699) | acc 0.937 (0.949) | IoU 0.533 (0.532) | [8 days, 15:26:07]
Lr: 9.931e-03 | Update: 2.500e-04 mean,5.642e-04 std | Epoch: [1][720/1047] | Time 2.429 (2.394) | Data 1.313 (1.250) | Loss 0.5640 (0.6697) | acc 0.941 (0.949) | IoU 0.492 (0.533) | [8 days, 15:27:15]
Lr: 9.930e-03 | Update: 3.767e-04 mean,6.110e-04 std | Epoch: [1][730/1047] | Time 2.654 (2.395) | Data 1.478 (1.250) | Loss 0.6344 (0.6687) | acc 0.967 (0.949) | IoU 0.520 (0.532) | [8 days, 15:28:02]
Lr: 9.929e-03 | Update: 2.113e-04 mean,3.341e-04 std | Epoch: [1][740/1047] | Time 2.606 (2.395) | Data 1.340 (1.251) | Loss 0.5183 (0.6682) | acc 0.958 (0.949) | IoU 0.532 (0.533) | [8 days, 15:29:11]
Lr: 9.928e-03 | Update: 2.922e-04 mean,5.332e-04 std | Epoch: [1][750/1047] | Time 2.534 (2.396) | Data 1.329 (1.251) | Loss 0.4648 (0.6664) | acc 0.996 (0.949) | IoU 0.792 (0.534) | [8 days, 15:30:49]
Lr: 9.927e-03 | Update: 2.829e-04 mean,4.424e-04 std | Epoch: [1][760/1047] | Time 2.477 (2.397) | Data 1.220 (1.251) | Loss 0.5655 (0.6659) | acc 0.960 (0.950) | IoU 0.509 (0.534) | [8 days, 15:32:32]
Lr: 9.926e-03 | Update: 4.624e-04 mean,8.842e-04 std | Epoch: [1][770/1047] | Time 1.778 (2.395) | Data 0.758 (1.250) | Loss 0.5747 (0.6652) | acc 0.939 (0.950) | IoU 0.500 (0.535) | [8 days, 15:24:31]
Lr: 9.925e-03 | Update: 2.997e-04 mean,5.012e-04 std | Epoch: [1][780/1047] | Time 2.063 (2.392) | Data 0.963 (1.248) | Loss 0.6152 (0.6648) | acc 0.947 (0.950) | IoU 0.499 (0.534) | [8 days, 15:12:44]
Lr: 9.924e-03 | Update: 1.016e-03 mean,1.675e-03 std | Epoch: [1][790/1047] | Time 4.231 (2.391) | Data 2.844 (1.247) | Loss 0.8184 (0.6659) | acc 0.978 (0.950) | IoU 0.516 (0.534) | [8 days, 15:06:46]
Lr: 9.924e-03 | Update: 3.118e-04 mean,5.419e-04 std | Epoch: [1][800/1047] | Time 2.369 (2.391) | Data 1.156 (1.246) | Loss 0.6951 (0.6652) | acc 0.929 (0.950) | IoU 0.471 (0.535) | [8 days, 15:03:51]
Lr: 9.923e-03 | Update: 2.489e-04 mean,3.719e-04 std | Epoch: [1][810/1047] | Time 2.428 (2.392) | Data 1.377 (1.247) | Loss 0.4693 (0.6644) | acc 0.998 (0.951) | IoU 0.647 (0.535) | [8 days, 15:06:38]
Lr: 9.922e-03 | Update: 1.925e-04 mean,3.923e-04 std | Epoch: [1][820/1047] | Time 2.162 (2.392) | Data 0.846 (1.247) | Loss 0.4429 (0.6633) | acc 0.957 (0.951) | IoU 0.485 (0.536) | [8 days, 15:05:53]
Lr: 9.921e-03 | Update: 5.991e-04 mean,1.028e-03 std | Epoch: [1][830/1047] | Time 2.254 (2.391) | Data 1.084 (1.246) | Loss 0.5472 (0.6630) | acc 0.991 (0.951) | IoU 0.696 (0.536) | [8 days, 15:00:29]
Lr: 9.920e-03 | Update: 2.466e-04 mean,3.989e-04 std | Epoch: [1][840/1047] | Time 2.424 (2.391) | Data 1.172 (1.247) | Loss 0.5757 (0.6625) | acc 0.976 (0.952) | IoU 0.574 (0.536) | [8 days, 15:01:39]
Lr: 9.919e-03 | Update: 8.646e-04 mean,1.422e-03 std | Epoch: [1][850/1047] | Time 2.241 (2.390) | Data 1.162 (1.246) | Loss 0.7730 (0.6620) | acc 0.969 (0.952) | IoU 0.491 (0.537) | [8 days, 14:56:38]
Lr: 9.918e-03 | Update: 3.103e-04 mean,4.995e-04 std | Epoch: [1][860/1047] | Time 2.292 (2.389) | Data 1.230 (1.245) | Loss 0.5566 (0.6613) | acc 0.975 (0.952) | IoU 0.501 (0.536) | [8 days, 14:53:03]
Lr: 9.917e-03 | Update: 2.776e-04 mean,4.240e-04 std | Epoch: [1][870/1047] | Time 2.551 (2.390) | Data 1.410 (1.246) | Loss 0.5053 (0.6600) | acc 0.977 (0.952) | IoU 0.537 (0.537) | [8 days, 14:54:03]
Lr: 9.916e-03 | Update: 3.043e-04 mean,5.510e-04 std | Epoch: [1][880/1047] | Time 2.383 (2.390) | Data 1.224 (1.245) | Loss 0.4781 (0.6597) | acc 0.926 (0.952) | IoU 0.470 (0.537) | [8 days, 14:53:42]
Lr: 9.915e-03 | Update: 1.268e-03 mean,2.531e-03 std | Epoch: [1][890/1047] | Time 2.389 (2.389) | Data 1.161 (1.245) | Loss 0.6767 (0.6588) | acc 0.974 (0.953) | IoU 0.541 (0.538) | [8 days, 14:50:57]
Lr: 9.914e-03 | Update: 5.882e-04 mean,9.479e-04 std | Epoch: [1][900/1047] | Time 2.677 (2.389) | Data 1.715 (1.245) | Loss 0.6088 (0.6583) | acc 0.950 (0.953) | IoU 0.567 (0.538) | [8 days, 14:48:49]
Lr: 9.913e-03 | Update: 4.117e-04 mean,6.550e-04 std | Epoch: [1][910/1047] | Time 2.072 (2.388) | Data 0.987 (1.244) | Loss 0.5004 (0.6572) | acc 0.991 (0.953) | IoU 0.754 (0.539) | [8 days, 14:44:11]
Lr: 9.912e-03 | Update: 3.112e-04 mean,5.620e-04 std | Epoch: [1][920/1047] | Time 2.193 (2.389) | Data 1.020 (1.244) | Loss 0.6126 (0.6564) | acc 0.931 (0.953) | IoU 0.470 (0.539) | [8 days, 14:45:04]
Lr: 9.911e-03 | Update: 4.146e-04 mean,7.106e-04 std | Epoch: [1][930/1047] | Time 1.780 (2.388) | Data 0.691 (1.244) | Loss 0.4506 (0.6553) | acc 0.995 (0.953) | IoU 0.678 (0.540) | [8 days, 14:41:14]
Lr: 9.910e-03 | Update: 2.579e-03 mean,5.018e-03 std | Epoch: [1][940/1047] | Time 2.324 (2.389) | Data 1.242 (1.244) | Loss 0.8745 (0.6554) | acc 0.957 (0.953) | IoU 0.544 (0.540) | [8 days, 14:43:06]
Lr: 9.909e-03 | Update: 6.806e-04 mean,1.216e-03 std | Epoch: [1][950/1047] | Time 2.280 (2.388) | Data 0.999 (1.244) | Loss 0.8988 (0.6552) | acc 0.919 (0.953) | IoU 0.462 (0.540) | [8 days, 14:41:44]
Lr: 9.908e-03 | Update: 7.226e-04 mean,1.334e-03 std | Epoch: [1][960/1047] | Time 2.104 (2.388) | Data 0.879 (1.243) | Loss 0.6019 (0.6544) | acc 0.993 (0.953) | IoU 0.707 (0.541) | [8 days, 14:38:04]
Lr: 9.907e-03 | Update: 3.287e-04 mean,6.334e-04 std | Epoch: [1][970/1047] | Time 2.614 (2.387) | Data 1.493 (1.243) | Loss 0.4688 (0.6547) | acc 0.917 (0.953) | IoU 0.487 (0.541) | [8 days, 14:33:45]
Lr: 9.906e-03 | Update: 2.489e-04 mean,3.915e-04 std | Epoch: [1][980/1047] | Time 2.547 (2.388) | Data 1.314 (1.243) | Loss 0.5174 (0.6544) | acc 0.973 (0.953) | IoU 0.510 (0.540) | [8 days, 14:35:13]
Lr: 9.905e-03 | Update: 3.514e-04 mean,5.433e-04 std | Epoch: [1][990/1047] | Time 2.426 (2.387) | Data 1.357 (1.243) | Loss 0.5753 (0.6535) | acc 0.983 (0.953) | IoU 0.648 (0.542) | [8 days, 14:32:21]
Lr: 9.904e-03 | Update: 3.851e-04 mean,5.714e-04 std | Epoch: [1][1000/1047] | Time 2.385 (2.387) | Data 1.182 (1.243) | Loss 0.5885 (0.6532) | acc 0.940 (0.953) | IoU 0.529 (0.542) | [8 days, 14:33:28]
Lr: 9.904e-03 | Update: 4.517e-04 mean,6.910e-04 std | Epoch: [1][1010/1047] | Time 2.574 (2.387) | Data 1.573 (1.243) | Loss 0.6277 (0.6538) | acc 0.970 (0.953) | IoU 0.492 (0.542) | [8 days, 14:31:41]
Lr: 9.903e-03 | Update: 5.795e-04 mean,9.510e-04 std | Epoch: [1][1020/1047] | Time 2.402 (2.387) | Data 1.325 (1.243) | Loss 0.6783 (0.6528) | acc 0.991 (0.954) | IoU 0.729 (0.543) | [8 days, 14:29:07]
Lr: 9.902e-03 | Update: 4.181e-04 mean,6.529e-04 std | Epoch: [1][1030/1047] | Time 1.948 (2.386) | Data 0.809 (1.242) | Loss 0.5133 (0.6520) | acc 0.968 (0.954) | IoU 0.520 (0.543) | [8 days, 14:27:35]
Lr: 9.901e-03 | Update: 1.173e-03 mean,1.992e-03 std | Epoch: [1][1040/1047] | Time 2.097 (2.386) | Data 0.999 (1.242) | Loss 0.7131 (0.6514) | acc 0.994 (0.954) | IoU 0.670 (0.543) | [8 days, 14:24:02]
Best mean iou in training set so far, save model!
Validation:: 100%|████████████████████████████| 508/508 [17:01<00:00,  2.01s/it]
********************************************************************************
Validation set:
Time avg per batch 2.200
Loss avg 13.7639
Jaccard avg 0.9974
WCE avg 12.7665
Acc avg 0.003885
IoU avg 0.001944
IoU class 0 [unlabeled] = 0.000000
IoU class 1 [static] = 0.000284
IoU class 2 [moving] = 0.003604
********************************************************************************
Lr: 9.900e-03 | Update: 7.302e-04 mean,1.437e-03 std | Epoch: [2][0/1047] | Time 2.114 (2.386) | Data 0.840 (1.241) | Loss 0.6113 (0.6113) | acc 0.971 (0.971) | IoU 0.516 (0.516) | [8 days, 10:03:40]
Lr: 9.899e-03 | Update: 4.767e-04 mean,9.448e-04 std | Epoch: [2][10/1047] | Time 2.334 (2.384) | Data 1.097 (1.240) | Loss 0.5328 (0.6021) | acc 0.985 (0.975) | IoU 0.601 (0.560) | [8 days, 9:53:55]
Lr: 9.898e-03 | Update: 5.232e-04 mean,1.126e-03 std | Epoch: [2][20/1047] | Time 2.163 (2.382) | Data 1.139 (1.238) | Loss 0.5637 (0.6008) | acc 0.932 (0.960) | IoU 0.535 (0.548) | [8 days, 9:44:48]
Lr: 9.897e-03 | Update: 5.666e-04 mean,9.479e-04 std | Epoch: [2][30/1047] | Time 2.250 (2.380) | Data 1.212 (1.236) | Loss 0.5411 (0.6173) | acc 0.971 (0.956) | IoU 0.560 (0.545) | [8 days, 9:34:07]
Lr: 9.896e-03 | Update: 7.623e-04 mean,1.784e-03 std | Epoch: [2][40/1047] | Time 1.895 (2.378) | Data 0.752 (1.234) | Loss 0.5960 (0.6318) | acc 0.979 (0.963) | IoU 0.559 (0.553) | [8 days, 9:23:48]
Lr: 9.895e-03 | Update: 3.129e-04 mean,4.859e-04 std | Epoch: [2][50/1047] | Time 2.102 (2.377) | Data 0.955 (1.233) | Loss 0.4739 (0.6183) | acc 0.987 (0.962) | IoU 0.648 (0.549) | [8 days, 9:15:03]
Lr: 9.894e-03 | Update: 5.574e-04 mean,8.774e-04 std | Epoch: [2][60/1047] | Time 2.254 (2.375) | Data 1.055 (1.231) | Loss 0.5306 (0.6185) | acc 0.984 (0.966) | IoU 0.678 (0.560) | [8 days, 9:06:24]
Lr: 9.893e-03 | Update: 4.641e-04 mean,7.335e-04 std | Epoch: [2][70/1047] | Time 2.183 (2.373) | Data 1.067 (1.229) | Loss 0.6492 (0.6177) | acc 0.931 (0.963) | IoU 0.596 (0.556) | [8 days, 8:56:57]
Lr: 9.892e-03 | Update: 2.106e-04 mean,3.928e-04 std | Epoch: [2][80/1047] | Time 1.809 (2.371) | Data 0.645 (1.228) | Loss 0.4385 (0.6127) | acc 0.993 (0.962) | IoU 0.601 (0.554) | [8 days, 8:46:36]
Lr: 9.891e-03 | Update: 2.046e-04 mean,3.084e-04 std | Epoch: [2][90/1047] | Time 2.277 (2.370) | Data 0.998 (1.226) | Loss 0.5058 (0.6035) | acc 0.982 (0.965) | IoU 0.556 (0.562) | [8 days, 8:37:30]
Lr: 9.891e-03 | Update: 4.168e-04 mean,6.710e-04 std | Epoch: [2][100/1047] | Time 2.075 (2.368) | Data 0.835 (1.224) | Loss 0.5121 (0.6043) | acc 0.971 (0.965) | IoU 0.582 (0.562) | [8 days, 8:27:01]
Lr: 9.890e-03 | Update: 5.656e-04 mean,8.294e-04 std | Epoch: [2][110/1047] | Time 2.321 (2.366) | Data 1.251 (1.222) | Loss 0.6429 (0.6054) | acc 0.959 (0.965) | IoU 0.540 (0.561) | [8 days, 8:17:39]
Lr: 9.889e-03 | Update: 2.557e-04 mean,4.434e-04 std | Epoch: [2][120/1047] | Time 1.979 (2.364) | Data 0.962 (1.221) | Loss 0.5127 (0.6060) | acc 0.982 (0.966) | IoU 0.514 (0.564) | [8 days, 8:08:24]
Lr: 9.888e-03 | Update: 5.815e-04 mean,8.653e-04 std | Epoch: [2][130/1047] | Time 1.921 (2.363) | Data 0.725 (1.220) | Loss 0.7251 (0.6102) | acc 0.903 (0.964) | IoU 0.464 (0.561) | [8 days, 8:01:37]
Lr: 9.887e-03 | Update: 5.680e-04 mean,9.375e-04 std | Epoch: [2][140/1047] | Time 2.054 (2.362) | Data 0.746 (1.218) | Loss 0.6710 (0.6148) | acc 0.976 (0.963) | IoU 0.525 (0.558) | [8 days, 7:52:48]
Lr: 9.886e-03 | Update: 6.988e-04 mean,1.017e-03 std | Epoch: [2][150/1047] | Time 2.080 (2.360) | Data 0.922 (1.217) | Loss 0.6536 (0.6088) | acc 0.988 (0.964) | IoU 0.663 (0.563) | [8 days, 7:45:00]
Lr: 9.885e-03 | Update: 3.271e-04 mean,5.020e-04 std | Epoch: [2][160/1047] | Time 2.041 (2.358) | Data 0.738 (1.215) | Loss 0.5155 (0.6096) | acc 0.981 (0.965) | IoU 0.703 (0.562) | [8 days, 7:35:34]
Lr: 9.884e-03 | Update: 7.991e-04 mean,1.390e-03 std | Epoch: [2][170/1047] | Time 2.480 (2.357) | Data 1.378 (1.214) | Loss 0.6676 (0.6082) | acc 0.949 (0.965) | IoU 0.568 (0.563) | [8 days, 7:28:47]
Lr: 9.883e-03 | Update: 4.451e-04 mean,6.543e-04 std | Epoch: [2][180/1047] | Time 2.085 (2.355) | Data 0.974 (1.212) | Loss 0.5417 (0.6082) | acc 0.968 (0.964) | IoU 0.531 (0.561) | [8 days, 7:19:18]
Lr: 9.882e-03 | Update: 3.184e-04 mean,4.742e-04 std | Epoch: [2][190/1047] | Time 1.863 (2.354) | Data 0.772 (1.210) | Loss 0.5784 (0.6070) | acc 0.980 (0.965) | IoU 0.572 (0.563) | [8 days, 7:10:31]
Lr: 9.881e-03 | Update: 1.657e-04 mean,2.698e-04 std | Epoch: [2][200/1047] | Time 2.047 (2.353) | Data 0.741 (1.209) | Loss 0.4553 (0.6043) | acc 0.990 (0.966) | IoU 0.582 (0.564) | [8 days, 7:03:13]
Lr: 9.880e-03 | Update: 3.405e-04 mean,5.319e-04 std | Epoch: [2][210/1047] | Time 1.973 (2.351) | Data 0.672 (1.207) | Loss 0.5519 (0.6018) | acc 0.984 (0.967) | IoU 0.625 (0.566) | [8 days, 6:54:58]
Lr: 9.879e-03 | Update: 2.167e-04 mean,3.565e-04 std | Epoch: [2][220/1047] | Time 2.050 (2.350) | Data 0.965 (1.206) | Loss 0.4210 (0.5997) | acc 0.989 (0.967) | IoU 0.647 (0.565) | [8 days, 6:47:49]
Lr: 9.878e-03 | Update: 5.496e-04 mean,9.348e-04 std | Epoch: [2][230/1047] | Time 1.717 (2.349) | Data 0.687 (1.205) | Loss 0.6956 (0.5991) | acc 0.986 (0.968) | IoU 0.620 (0.567) | [8 days, 6:39:58]
Lr: 9.877e-03 | Update: 2.273e-04 mean,4.180e-04 std | Epoch: [2][240/1047] | Time 1.715 (2.347) | Data 0.641 (1.203) | Loss 0.5140 (0.5967) | acc 0.981 (0.968) | IoU 0.550 (0.569) | [8 days, 6:31:23]
Lr: 9.876e-03 | Update: 1.175e-03 mean,2.382e-03 std | Epoch: [2][250/1047] | Time 1.765 (2.346) | Data 0.678 (1.202) | Loss 0.6459 (0.5963) | acc 0.979 (0.969) | IoU 0.558 (0.571) | [8 days, 6:23:56]
Lr: 9.875e-03 | Update: 3.305e-04 mean,5.586e-04 std | Epoch: [2][260/1047] | Time 1.866 (2.344) | Data 0.776 (1.200) | Loss 0.5167 (0.5981) | acc 0.951 (0.969) | IoU 0.527 (0.570) | [8 days, 6:15:34]
Lr: 9.874e-03 | Update: 2.648e-04 mean,4.097e-04 std | Epoch: [2][270/1047] | Time 1.917 (2.343) | Data 0.779 (1.199) | Loss 0.5131 (0.5970) | acc 0.993 (0.969) | IoU 0.606 (0.573) | [8 days, 6:07:42]
Lr: 9.873e-03 | Update: 2.461e-04 mean,4.216e-04 std | Epoch: [2][280/1047] | Time 1.959 (2.342) | Data 0.782 (1.198) | Loss 0.5798 (0.5967) | acc 0.997 (0.970) | IoU 0.731 (0.573) | [8 days, 6:01:14]
Lr: 9.872e-03 | Update: 2.617e-04 mean,3.898e-04 std | Epoch: [2][290/1047] | Time 2.092 (2.341) | Data 0.778 (1.197) | Loss 0.5591 (0.5968) | acc 0.948 (0.970) | IoU 0.502 (0.575) | [8 days, 5:54:43]
Lr: 9.872e-03 | Update: 3.706e-04 mean,5.668e-04 std | Epoch: [2][300/1047] | Time 2.102 (2.339) | Data 0.917 (1.195) | Loss 0.5084 (0.5955) | acc 0.993 (0.970) | IoU 0.688 (0.575) | [8 days, 5:48:00]
Lr: 9.871e-03 | Update: 3.789e-04 mean,5.565e-04 std | Epoch: [2][310/1047] | Time 2.032 (2.338) | Data 0.881 (1.194) | Loss 0.5947 (0.5963) | acc 0.942 (0.970) | IoU 0.496 (0.574) | [8 days, 5:39:52]
Lr: 9.870e-03 | Update: 2.561e-04 mean,3.936e-04 std | Epoch: [2][320/1047] | Time 2.260 (2.337) | Data 1.019 (1.193) | Loss 0.4854 (0.5959) | acc 0.991 (0.970) | IoU 0.644 (0.574) | [8 days, 5:33:11]
Lr: 9.869e-03 | Update: 2.404e-04 mean,4.036e-04 std | Epoch: [2][330/1047] | Time 2.169 (2.335) | Data 1.114 (1.192) | Loss 0.5057 (0.5949) | acc 0.994 (0.970) | IoU 0.693 (0.575) | [8 days, 5:26:15]
Lr: 9.868e-03 | Update: 2.016e-04 mean,3.798e-04 std | Epoch: [2][340/1047] | Time 1.724 (2.334) | Data 0.708 (1.190) | Loss 0.6499 (0.5949) | acc 0.921 (0.970) | IoU 0.466 (0.574) | [8 days, 5:19:08]
Lr: 9.867e-03 | Update: 9.583e-04 mean,1.438e-03 std | Epoch: [2][350/1047] | Time 2.173 (2.333) | Data 1.060 (1.189) | Loss 0.7757 (0.5943) | acc 0.981 (0.969) | IoU 0.522 (0.573) | [8 days, 5:11:33]
Lr: 9.866e-03 | Update: 1.977e-04 mean,3.676e-04 std | Epoch: [2][360/1047] | Time 2.054 (2.331) | Data 0.972 (1.188) | Loss 0.3646 (0.5924) | acc 0.956 (0.970) | IoU 0.527 (0.575) | [8 days, 5:04:12]
Lr: 9.865e-03 | Update: 2.715e-04 mean,3.884e-04 std | Epoch: [2][370/1047] | Time 2.345 (2.330) | Data 1.212 (1.187) | Loss 0.4868 (0.5915) | acc 0.982 (0.969) | IoU 0.656 (0.574) | [8 days, 4:57:15]
Lr: 9.864e-03 | Update: 4.739e-04 mean,7.169e-04 std | Epoch: [2][380/1047] | Time 2.016 (2.329) | Data 0.887 (1.185) | Loss 0.5739 (0.5921) | acc 0.967 (0.969) | IoU 0.604 (0.574) | [8 days, 4:50:33]
Lr: 9.863e-03 | Update: 5.131e-04 mean,8.309e-04 std | Epoch: [2][390/1047] | Time 1.825 (2.328) | Data 0.696 (1.184) | Loss 0.5815 (0.5920) | acc 0.934 (0.969) | IoU 0.550 (0.573) | [8 days, 4:44:35]
Lr: 9.862e-03 | Update: 2.829e-04 mean,4.466e-04 std | Epoch: [2][400/1047] | Time 1.803 (2.327) | Data 0.628 (1.183) | Loss 0.4179 (0.5906) | acc 0.989 (0.969) | IoU 0.595 (0.573) | [8 days, 4:37:18]
Lr: 9.861e-03 | Update: 2.994e-04 mean,4.201e-04 std | Epoch: [2][410/1047] | Time 2.131 (2.326) | Data 1.128 (1.182) | Loss 0.5244 (0.5898) | acc 0.937 (0.969) | IoU 0.533 (0.574) | [8 days, 4:31:28]
Lr: 9.860e-03 | Update: 2.803e-04 mean,4.588e-04 std | Epoch: [2][420/1047] | Time 2.024 (2.325) | Data 1.004 (1.180) | Loss 0.5438 (0.5897) | acc 0.972 (0.968) | IoU 0.515 (0.572) | [8 days, 4:24:22]
Lr: 9.859e-03 | Update: 2.510e-03 mean,4.268e-03 std | Epoch: [2][430/1047] | Time 1.820 (2.324) | Data 0.753 (1.180) | Loss 1.1294 (0.5892) | acc 0.987 (0.969) | IoU 0.717 (0.574) | [8 days, 4:19:05]
Lr: 9.858e-03 | Update: 7.258e-04 mean,1.380e-03 std | Epoch: [2][440/1047] | Time 2.096 (2.323) | Data 0.848 (1.179) | Loss 0.5499 (0.5894) | acc 0.916 (0.968) | IoU 0.494 (0.573) | [8 days, 4:13:37]
Lr: 9.857e-03 | Update: 2.975e-04 mean,4.934e-04 std | Epoch: [2][450/1047] | Time 2.310 (2.322) | Data 1.242 (1.178) | Loss 0.4865 (0.5903) | acc 0.962 (0.968) | IoU 0.583 (0.572) | [8 days, 4:07:23]
Lr: 9.856e-03 | Update: 2.298e-04 mean,3.500e-04 std | Epoch: [2][460/1047] | Time 2.071 (2.321) | Data 0.782 (1.177) | Loss 0.4810 (0.5894) | acc 0.994 (0.968) | IoU 0.597 (0.572) | [8 days, 4:03:41]
Lr: 9.855e-03 | Update: 1.884e-04 mean,3.490e-04 std | Epoch: [2][470/1047] | Time 2.526 (2.320) | Data 1.246 (1.176) | Loss 0.3772 (0.5887) | acc 0.989 (0.968) | IoU 0.601 (0.573) | [8 days, 3:58:27]
Lr: 9.854e-03 | Update: 3.068e-04 mean,4.535e-04 std | Epoch: [2][480/1047] | Time 2.101 (2.319) | Data 0.984 (1.175) | Loss 0.5765 (0.5897) | acc 0.972 (0.969) | IoU 0.531 (0.573) | [8 days, 3:51:23]
Lr: 9.854e-03 | Update: 3.816e-04 mean,6.505e-04 std | Epoch: [2][490/1047] | Time 2.304 (2.318) | Data 1.129 (1.174) | Loss 0.5067 (0.5894) | acc 0.991 (0.969) | IoU 0.649 (0.574) | [8 days, 3:45:37]
Lr: 9.853e-03 | Update: 3.958e-04 mean,6.682e-04 std | Epoch: [2][500/1047] | Time 2.118 (2.317) | Data 1.090 (1.173) | Loss 0.5750 (0.5874) | acc 0.983 (0.969) | IoU 0.534 (0.574) | [8 days, 3:39:17]
Lr: 9.852e-03 | Update: 2.595e-04 mean,3.979e-04 std | Epoch: [2][510/1047] | Time 2.189 (2.316) | Data 1.125 (1.171) | Loss 0.4795 (0.5867) | acc 0.985 (0.970) | IoU 0.568 (0.575) | [8 days, 3:32:08]
Lr: 9.851e-03 | Update: 3.953e-04 mean,6.530e-04 std | Epoch: [2][520/1047] | Time 1.966 (2.314) | Data 0.892 (1.170) | Loss 0.6137 (0.5870) | acc 0.958 (0.970) | IoU 0.484 (0.574) | [8 days, 3:25:23]
Lr: 9.850e-03 | Update: 2.604e-04 mean,4.414e-04 std | Epoch: [2][530/1047] | Time 2.246 (2.313) | Data 1.045 (1.169) | Loss 0.3437 (0.5856) | acc 0.990 (0.970) | IoU 0.561 (0.575) | [8 days, 3:18:40]
Lr: 9.849e-03 | Update: 2.235e-04 mean,3.578e-04 std | Epoch: [2][540/1047] | Time 2.011 (2.312) | Data 0.911 (1.168) | Loss 0.4731 (0.5859) | acc 0.991 (0.970) | IoU 0.605 (0.576) | [8 days, 3:12:51]
Lr: 9.848e-03 | Update: 2.988e-04 mean,4.733e-04 std | Epoch: [2][550/1047] | Time 2.140 (2.311) | Data 0.841 (1.167) | Loss 0.4493 (0.5846) | acc 0.981 (0.971) | IoU 0.642 (0.576) | [8 days, 3:07:02]
Lr: 9.847e-03 | Update: 3.306e-04 mean,5.251e-04 std | Epoch: [2][560/1047] | Time 1.960 (2.310) | Data 0.866 (1.166) | Loss 0.5785 (0.5845) | acc 0.995 (0.971) | IoU 0.716 (0.577) | [8 days, 3:01:15]
Lr: 9.846e-03 | Update: 5.788e-04 mean,8.871e-04 std | Epoch: [2][570/1047] | Time 1.821 (2.309) | Data 0.694 (1.165) | Loss 0.6331 (0.5846) | acc 0.945 (0.971) | IoU 0.490 (0.577) | [8 days, 2:56:11]
Lr: 9.845e-03 | Update: 3.449e-04 mean,5.058e-04 std | Epoch: [2][580/1047] | Time 1.804 (2.308) | Data 0.709 (1.164) | Loss 0.5373 (0.5855) | acc 0.954 (0.971) | IoU 0.531 (0.576) | [8 days, 2:49:40]
Lr: 9.844e-03 | Update: 3.421e-04 mean,5.160e-04 std | Epoch: [2][590/1047] | Time 2.301 (2.307) | Data 1.034 (1.163) | Loss 0.5497 (0.5848) | acc 0.990 (0.971) | IoU 0.544 (0.577) | [8 days, 2:44:47]
Lr: 9.843e-03 | Update: 2.671e-04 mean,4.005e-04 std | Epoch: [2][600/1047] | Time 2.286 (2.306) | Data 1.154 (1.162) | Loss 0.4290 (0.5843) | acc 0.997 (0.971) | IoU 0.804 (0.578) | [8 days, 2:39:05]
Lr: 9.842e-03 | Update: 3.421e-04 mean,5.043e-04 std | Epoch: [2][610/1047] | Time 1.880 (2.305) | Data 0.799 (1.161) | Loss 0.4707 (0.5841) | acc 0.985 (0.971) | IoU 0.604 (0.579) | [8 days, 2:32:45]
Lr: 9.841e-03 | Update: 2.452e-04 mean,3.642e-04 std | Epoch: [2][620/1047] | Time 2.108 (2.305) | Data 0.862 (1.160) | Loss 0.5597 (0.5854) | acc 0.944 (0.971) | IoU 0.532 (0.578) | [8 days, 2:28:28]
Lr: 9.840e-03 | Update: 1.868e-04 mean,3.252e-04 std | Epoch: [2][630/1047] | Time 2.243 (2.304) | Data 1.159 (1.159) | Loss 0.4841 (0.5849) | acc 0.991 (0.971) | IoU 0.633 (0.578) | [8 days, 2:23:31]
Lr: 9.839e-03 | Update: 2.588e-04 mean,3.762e-04 std | Epoch: [2][640/1047] | Time 2.077 (2.303) | Data 0.939 (1.158) | Loss 0.3959 (0.5842) | acc 0.987 (0.971) | IoU 0.541 (0.578) | [8 days, 2:16:52]
Lr: 9.838e-03 | Update: 4.803e-04 mean,7.683e-04 std | Epoch: [2][650/1047] | Time 2.206 (2.302) | Data 1.059 (1.157) | Loss 0.5278 (0.5834) | acc 0.995 (0.971) | IoU 0.690 (0.579) | [8 days, 2:13:07]
Lr: 9.837e-03 | Update: 4.534e-04 mean,6.769e-04 std | Epoch: [2][660/1047] | Time 2.340 (2.301) | Data 1.174 (1.156) | Loss 0.5687 (0.5834) | acc 0.981 (0.972) | IoU 0.533 (0.579) | [8 days, 2:06:51]
Lr: 9.837e-03 | Update: 2.673e-04 mean,3.992e-04 std | Epoch: [2][670/1047] | Time 1.842 (2.300) | Data 0.822 (1.155) | Loss 0.4827 (0.5829) | acc 0.978 (0.972) | IoU 0.603 (0.579) | [8 days, 2:01:09]
Lr: 9.836e-03 | Update: 2.553e-04 mean,4.748e-04 std | Epoch: [2][680/1047] | Time 1.980 (2.299) | Data 0.711 (1.154) | Loss 0.5558 (0.5828) | acc 0.995 (0.972) | IoU 0.639 (0.579) | [8 days, 1:55:25]
Lr: 9.835e-03 | Update: 4.682e-04 mean,6.814e-04 std | Epoch: [2][690/1047] | Time 2.079 (2.298) | Data 0.958 (1.153) | Loss 0.7038 (0.5828) | acc 0.977 (0.972) | IoU 0.585 (0.580) | [8 days, 1:49:57]
Lr: 9.834e-03 | Update: 1.682e-04 mean,3.055e-04 std | Epoch: [2][700/1047] | Time 2.138 (2.297) | Data 0.952 (1.152) | Loss 0.5143 (0.5823) | acc 0.947 (0.972) | IoU 0.501 (0.580) | [8 days, 1:44:40]
Lr: 9.833e-03 | Update: 5.005e-04 mean,7.660e-04 std | Epoch: [2][710/1047] | Time 2.285 (2.296) | Data 1.130 (1.151) | Loss 0.6014 (0.5819) | acc 0.990 (0.972) | IoU 0.663 (0.579) | [8 days, 1:39:18]
Lr: 9.832e-03 | Update: 1.932e-04 mean,3.568e-04 std | Epoch: [2][720/1047] | Time 2.147 (2.296) | Data 0.802 (1.151) | Loss 0.5058 (0.5816) | acc 0.959 (0.972) | IoU 0.509 (0.579) | [8 days, 1:35:47]
Lr: 9.831e-03 | Update: 1.939e-04 mean,3.215e-04 std | Epoch: [2][730/1047] | Time 2.266 (2.295) | Data 1.131 (1.150) | Loss 0.4759 (0.5811) | acc 0.990 (0.972) | IoU 0.663 (0.579) | [8 days, 1:31:04]
Lr: 9.830e-03 | Update: 6.998e-04 mean,1.041e-03 std | Epoch: [2][740/1047] | Time 2.190 (2.294) | Data 1.141 (1.149) | Loss 0.6195 (0.5804) | acc 0.978 (0.972) | IoU 0.573 (0.580) | [8 days, 1:25:00]
Lr: 9.829e-03 | Update: 2.000e-03 mean,3.336e-03 std | Epoch: [2][750/1047] | Time 1.867 (2.293) | Data 0.691 (1.148) | Loss 0.6657 (0.5800) | acc 0.968 (0.972) | IoU 0.551 (0.580) | [8 days, 1:20:42]
Lr: 9.828e-03 | Update: 1.729e-04 mean,2.763e-04 std | Epoch: [2][760/1047] | Time 2.127 (2.292) | Data 0.764 (1.147) | Loss 0.4223 (0.5788) | acc 0.990 (0.972) | IoU 0.718 (0.580) | [8 days, 1:15:31]
Lr: 9.827e-03 | Update: 2.343e-04 mean,3.843e-04 std | Epoch: [2][770/1047] | Time 1.986 (2.292) | Data 0.724 (1.146) | Loss 0.4848 (0.5780) | acc 0.964 (0.972) | IoU 0.545 (0.581) | [8 days, 1:12:10]
Lr: 9.826e-03 | Update: 7.675e-04 mean,1.161e-03 std | Epoch: [2][780/1047] | Time 2.142 (2.291) | Data 1.066 (1.146) | Loss 0.8327 (0.5785) | acc 0.970 (0.972) | IoU 0.566 (0.581) | [8 days, 1:07:20]
Lr: 9.825e-03 | Update: 2.095e-04 mean,3.881e-04 std | Epoch: [2][790/1047] | Time 2.006 (2.290) | Data 0.892 (1.145) | Loss 0.4056 (0.5782) | acc 0.983 (0.972) | IoU 0.501 (0.581) | [8 days, 1:02:30]
Lr: 9.824e-03 | Update: 2.582e-04 mean,3.942e-04 std | Epoch: [2][800/1047] | Time 2.012 (2.289) | Data 0.868 (1.144) | Loss 0.6374 (0.5786) | acc 0.978 (0.973) | IoU 0.569 (0.581) | [8 days, 0:56:07]
Lr: 9.823e-03 | Update: 2.704e-04 mean,4.045e-04 std | Epoch: [2][810/1047] | Time 2.126 (2.288) | Data 1.060 (1.143) | Loss 0.5159 (0.5781) | acc 0.992 (0.973) | IoU 0.671 (0.581) | [8 days, 0:50:51]
Lr: 9.822e-03 | Update: 1.890e-04 mean,3.500e-04 std | Epoch: [2][820/1047] | Time 2.007 (2.287) | Data 0.904 (1.142) | Loss 0.3577 (0.5771) | acc 0.996 (0.973) | IoU 0.685 (0.582) | [8 days, 0:45:48]
Lr: 9.821e-03 | Update: 4.574e-04 mean,7.285e-04 std | Epoch: [2][830/1047] | Time 2.085 (2.287) | Data 0.931 (1.141) | Loss 0.6718 (0.5769) | acc 0.981 (0.973) | IoU 0.552 (0.583) | [8 days, 0:43:04]
Lr: 9.820e-03 | Update: 1.195e-03 mean,1.839e-03 std | Epoch: [2][840/1047] | Time 2.348 (2.286) | Data 1.034 (1.141) | Loss 0.6892 (0.5769) | acc 0.952 (0.973) | IoU 0.511 (0.583) | [8 days, 0:38:16]
Lr: 9.820e-03 | Update: 4.959e-04 mean,7.981e-04 std | Epoch: [2][850/1047] | Time 2.053 (2.285) | Data 0.911 (1.140) | Loss 0.3600 (0.5760) | acc 0.995 (0.973) | IoU 0.773 (0.584) | [8 days, 0:32:57]
Lr: 9.819e-03 | Update: 1.981e-04 mean,3.109e-04 std | Epoch: [2][860/1047] | Time 2.218 (2.284) | Data 0.854 (1.139) | Loss 0.4770 (0.5759) | acc 0.955 (0.973) | IoU 0.548 (0.584) | [8 days, 0:28:39]
Lr: 9.818e-03 | Update: 2.137e-04 mean,3.143e-04 std | Epoch: [2][870/1047] | Time 1.887 (2.283) | Data 0.699 (1.138) | Loss 0.5650 (0.5761) | acc 0.972 (0.973) | IoU 0.516 (0.584) | [8 days, 0:22:14]
Lr: 9.817e-03 | Update: 7.041e-04 mean,1.160e-03 std | Epoch: [2][880/1047] | Time 2.011 (2.282) | Data 0.777 (1.137) | Loss 0.5781 (0.5758) | acc 0.995 (0.973) | IoU 0.760 (0.585) | [8 days, 0:17:02]
Lr: 9.816e-03 | Update: 2.724e-04 mean,4.208e-04 std | Epoch: [2][890/1047] | Time 1.926 (2.282) | Data 0.740 (1.136) | Loss 0.4449 (0.5754) | acc 0.936 (0.973) | IoU 0.500 (0.584) | [8 days, 0:12:03]
Lr: 9.815e-03 | Update: 2.555e-04 mean,3.567e-04 std | Epoch: [2][900/1047] | Time 2.101 (2.281) | Data 0.826 (1.135) | Loss 0.4801 (0.5753) | acc 0.981 (0.973) | IoU 0.517 (0.584) | [8 days, 0:08:22]
Lr: 9.814e-03 | Update: 3.533e-04 mean,5.482e-04 std | Epoch: [2][910/1047] | Time 2.003 (2.280) | Data 0.798 (1.135) | Loss 0.4913 (0.5751) | acc 0.978 (0.973) | IoU 0.522 (0.584) | [8 days, 0:03:48]
Lr: 9.813e-03 | Update: 2.049e-04 mean,3.171e-04 std | Epoch: [2][920/1047] | Time 2.152 (2.279) | Data 0.960 (1.134) | Loss 0.4821 (0.5745) | acc 0.981 (0.973) | IoU 0.560 (0.584) | [7 days, 23:59:00]
Lr: 9.812e-03 | Update: 3.398e-04 mean,5.524e-04 std | Epoch: [2][930/1047] | Time 1.876 (2.279) | Data 0.684 (1.133) | Loss 0.5895 (0.5753) | acc 0.989 (0.973) | IoU 0.704 (0.584) | [7 days, 23:56:18]
Lr: 9.811e-03 | Update: 3.204e-04 mean,4.326e-04 std | Epoch: [2][940/1047] | Time 1.984 (2.278) | Data 0.832 (1.133) | Loss 0.6829 (0.5759) | acc 0.934 (0.973) | IoU 0.532 (0.584) | [7 days, 23:52:02]
Lr: 9.810e-03 | Update: 1.923e-04 mean,3.159e-04 std | Epoch: [2][950/1047] | Time 2.104 (2.277) | Data 1.019 (1.132) | Loss 0.4712 (0.5759) | acc 0.979 (0.973) | IoU 0.612 (0.583) | [7 days, 23:47:27]
Lr: 9.809e-03 | Update: 6.764e-04 mean,1.047e-03 std | Epoch: [2][960/1047] | Time 1.912 (2.277) | Data 0.830 (1.131) | Loss 0.5642 (0.5756) | acc 0.995 (0.973) | IoU 0.720 (0.584) | [7 days, 23:43:36]
Lr: 9.808e-03 | Update: 4.035e-04 mean,7.429e-04 std | Epoch: [2][970/1047] | Time 1.846 (2.276) | Data 0.827 (1.130) | Loss 0.5801 (0.5761) | acc 0.978 (0.973) | IoU 0.572 (0.584) | [7 days, 23:38:54]
Lr: 9.807e-03 | Update: 5.201e-04 mean,7.222e-04 std | Epoch: [2][980/1047] | Time 2.553 (2.276) | Data 1.508 (1.130) | Loss 0.6024 (0.5757) | acc 0.989 (0.973) | IoU 0.661 (0.584) | [7 days, 23:37:14]
Lr: 9.806e-03 | Update: 2.817e-04 mean,4.138e-04 std | Epoch: [2][990/1047] | Time 2.015 (2.275) | Data 0.985 (1.129) | Loss 0.5154 (0.5757) | acc 0.979 (0.973) | IoU 0.677 (0.585) | [7 days, 23:32:42]
Lr: 9.805e-03 | Update: 4.057e-04 mean,6.047e-04 std | Epoch: [2][1000/1047] | Time 2.191 (2.275) | Data 1.129 (1.129) | Loss 0.5675 (0.5755) | acc 0.985 (0.973) | IoU 0.627 (0.585) | [7 days, 23:31:00]
Lr: 9.804e-03 | Update: 4.980e-04 mean,7.848e-04 std | Epoch: [2][1010/1047] | Time 1.978 (2.274) | Data 0.826 (1.129) | Loss 0.5484 (0.5747) | acc 0.991 (0.974) | IoU 0.703 (0.586) | [7 days, 23:27:30]
Lr: 9.804e-03 | Update: 2.737e-04 mean,4.208e-04 std | Epoch: [2][1020/1047] | Time 2.041 (2.274) | Data 0.812 (1.128) | Loss 0.4948 (0.5750) | acc 0.973 (0.974) | IoU 0.567 (0.586) | [7 days, 23:23:59]
Lr: 9.803e-03 | Update: 2.185e-04 mean,3.759e-04 std | Epoch: [2][1030/1047] | Time 2.482 (2.273) | Data 1.372 (1.127) | Loss 0.5026 (0.5750) | acc 0.985 (0.974) | IoU 0.530 (0.586) | [7 days, 23:20:07]
Lr: 9.802e-03 | Update: 1.719e-04 mean,3.503e-04 std | Epoch: [2][1040/1047] | Time 2.230 (2.273) | Data 1.120 (1.127) | Loss 0.4875 (0.5744) | acc 0.983 (0.974) | IoU 0.527 (0.586) | [7 days, 23:18:48]
Best mean iou in training set so far, save model!
Validation:: 100%|████████████████████████████| 508/508 [13:08<00:00,  1.55s/it]
********************************************************************************
Validation set:
Time avg per batch 1.984
Loss avg 0.4537
Jaccard avg 0.2476
WCE avg 0.2061
Acc avg 0.994580
IoU avg 0.652938
IoU class 0 [unlabeled] = 0.000000
IoU class 1 [static] = 0.994566
IoU class 2 [moving] = 0.311309
********************************************************************************
Best mean iou in validation so far, save model!
********************************************************************************
Lr: 9.801e-03 | Update: 3.175e-04 mean,4.770e-04 std | Epoch: [3][0/1047] | Time 1.666 (2.272) | Data 0.544 (1.127) | Loss 0.3843 (0.3843) | acc 0.995 (0.995) | IoU 0.725 (0.725) | [7 days, 18:28:05]
Lr: 9.800e-03 | Update: 3.091e-04 mean,4.233e-04 std | Epoch: [3][10/1047] | Time 1.863 (2.272) | Data 0.723 (1.126) | Loss 0.4977 (0.4958) | acc 0.987 (0.991) | IoU 0.544 (0.665) | [7 days, 18:23:54]
Lr: 9.798e-03 | Update: 3.157e-04 mean,4.726e-04 std | Epoch: [3][30/1047] | Time 1.804 (2.270) | Data 0.613 (1.124) | Loss 0.4201 (0.5200) | acc 0.994 (0.989) | IoU 0.778 (0.641) | [7 days, 18:14:42]
Lr: 9.797e-03 | Update: 7.679e-04 mean,1.100e-03 std | Epoch: [3][40/1047] | Time 1.854 (2.269) | Data 0.695 (1.124) | Loss 0.6597 (0.5282) | acc 0.977 (0.987) | IoU 0.589 (0.629) | [7 days, 18:10:02]
Lr: 9.796e-03 | Update: 3.945e-04 mean,5.962e-04 std | Epoch: [3][50/1047] | Time 1.778 (2.268) | Data 0.685 (1.123) | Loss 0.5584 (0.5322) | acc 0.960 (0.984) | IoU 0.496 (0.615) | [7 days, 18:05:09]
Lr: 9.795e-03 | Update: 9.331e-04 mean,1.443e-03 std | Epoch: [3][60/1047] | Time 1.775 (2.267) | Data 0.717 (1.122) | Loss 0.7657 (0.5414) | acc 0.961 (0.982) | IoU 0.493 (0.609) | [7 days, 18:00:15]
Lr: 9.794e-03 | Update: 2.003e-04 mean,2.806e-04 std | Epoch: [3][70/1047] | Time 1.875 (2.266) | Data 0.862 (1.121) | Loss 0.4307 (0.5382) | acc 0.994 (0.982) | IoU 0.723 (0.609) | [7 days, 17:54:49]
Lr: 9.793e-03 | Update: 3.548e-04 mean,5.141e-04 std | Epoch: [3][80/1047] | Time 2.127 (2.266) | Data 0.939 (1.120) | Loss 0.5594 (0.5382) | acc 0.981 (0.983) | IoU 0.616 (0.614) | [7 days, 17:50:14]
Lr: 9.792e-03 | Update: 2.590e-04 mean,3.451e-04 std | Epoch: [3][100/1047] | Time 2.143 (2.264) | Data 0.828 (1.119) | Loss 0.3809 (0.5330) | acc 0.997 (0.984) | IoU 0.828 (0.616) | [7 days, 17:41:18]
Lr: 9.791e-03 | Update: 1.925e-04 mean,2.884e-04 std | Epoch: [3][110/1047] | Time 1.884 (2.263) | Data 0.782 (1.118) | Loss 0.5354 (0.5357) | acc 0.969 (0.983) | IoU 0.509 (0.612) | [7 days, 17:36:02]
Lr: 9.790e-03 | Update: 3.232e-04 mean,5.650e-04 std | Epoch: [3][120/1047] | Time 1.816 (2.262) | Data 0.662 (1.117) | Loss 0.4771 (0.5326) | acc 0.996 (0.984) | IoU 0.788 (0.615) | [7 days, 17:31:08]
Lr: 9.789e-03 | Update: 7.322e-04 mean,1.389e-03 std | Epoch: [3][130/1047] | Time 1.783 (2.261) | Data 0.701 (1.116) | Loss 0.5049 (0.5335) | acc 0.976 (0.983) | IoU 0.589 (0.615) | [7 days, 17:26:23]
Lr: 9.788e-03 | Update: 9.494e-04 mean,1.488e-03 std | Epoch: [3][140/1047] | Time 2.318 (2.261) | Data 1.056 (1.115) | Loss 0.5711 (0.5322) | acc 0.990 (0.984) | IoU 0.704 (0.613) | [7 days, 17:21:15]
Lr: 9.787e-03 | Update: 3.352e-04 mean,5.493e-04 std | Epoch: [3][150/1047] | Time 1.776 (2.260) | Data 0.673 (1.114) | Loss 0.5472 (0.5329) | acc 0.950 (0.983) | IoU 0.499 (0.614) | [7 days, 17:15:54]
Lr: 9.786e-03 | Update: 7.534e-04 mean,1.057e-03 std | Epoch: [3][160/1047] | Time 1.821 (2.259) | Data 0.682 (1.114) | Loss 0.5959 (0.5331) | acc 0.987 (0.983) | IoU 0.693 (0.612) | [7 days, 17:11:36]
Lr: 9.785e-03 | Update: 3.515e-04 mean,5.620e-04 std | Epoch: [3][170/1047] | Time 1.956 (2.258) | Data 0.893 (1.113) | Loss 0.4188 (0.5349) | acc 0.984 (0.982) | IoU 0.665 (0.610) | [7 days, 17:06:27]
Lr: 9.784e-03 | Update: 6.282e-04 mean,1.156e-03 std | Epoch: [3][180/1047] | Time 1.911 (2.257) | Data 0.867 (1.112) | Loss 0.8165 (0.5365) | acc 0.990 (0.982) | IoU 0.631 (0.608) | [7 days, 17:01:47]
Lr: 9.783e-03 | Update: 3.352e-04 mean,5.033e-04 std | Epoch: [3][190/1047] | Time 1.741 (2.256) | Data 0.606 (1.111) | Loss 0.5209 (0.5384) | acc 0.964 (0.982) | IoU 0.573 (0.608) | [7 days, 16:57:08]
Lr: 9.782e-03 | Update: 2.818e-04 mean,4.257e-04 std | Epoch: [3][200/1047] | Time 1.835 (2.256) | Data 0.730 (1.111) | Loss 0.5229 (0.5401) | acc 0.981 (0.981) | IoU 0.535 (0.605) | [7 days, 16:53:44]
Lr: 9.781e-03 | Update: 1.979e-04 mean,2.973e-04 std | Epoch: [3][210/1047] | Time 1.794 (2.255) | Data 0.651 (1.110) | Loss 0.5238 (0.5386) | acc 0.981 (0.981) | IoU 0.520 (0.606) | [7 days, 16:48:33]
Lr: 9.780e-03 | Update: 1.628e-04 mean,2.660e-04 std | Epoch: [3][220/1047] | Time 2.023 (2.254) | Data 0.892 (1.109) | Loss 0.3079 (0.5353) | acc 0.997 (0.982) | IoU 0.689 (0.607) | [7 days, 16:44:21]
Lr: 9.779e-03 | Update: 8.521e-04 mean,1.318e-03 std | Epoch: [3][230/1047] | Time 1.856 (2.253) | Data 0.708 (1.108) | Loss 0.7230 (0.5341) | acc 0.982 (0.982) | IoU 0.574 (0.609) | [7 days, 16:38:26]
Lr: 9.778e-03 | Update: 5.677e-04 mean,8.038e-04 std | Epoch: [3][240/1047] | Time 2.309 (2.252) | Data 1.202 (1.108) | Loss 0.5910 (0.5333) | acc 0.985 (0.982) | IoU 0.621 (0.611) | [7 days, 16:35:19]
Lr: 9.778e-03 | Update: 3.074e-04 mean,4.231e-04 std | Epoch: [3][250/1047] | Time 1.796 (2.252) | Data 0.639 (1.107) | Loss 0.5500 (0.5318) | acc 0.969 (0.982) | IoU 0.588 (0.610) | [7 days, 16:30:54]
Lr: 9.777e-03 | Update: 3.090e-04 mean,4.234e-04 std | Epoch: [3][260/1047] | Time 2.156 (2.251) | Data 1.042 (1.107) | Loss 0.5961 (0.5320) | acc 0.973 (0.981) | IoU 0.504 (0.608) | [7 days, 16:27:56]
Lr: 9.776e-03 | Update: 2.415e-04 mean,3.474e-04 std | Epoch: [3][270/1047] | Time 2.250 (2.250) | Data 1.052 (1.106) | Loss 0.5005 (0.5331) | acc 0.989 (0.981) | IoU 0.684 (0.608) | [7 days, 16:23:41]
Lr: 9.775e-03 | Update: 5.617e-04 mean,8.913e-04 std | Epoch: [3][280/1047] | Time 1.913 (2.250) | Data 0.841 (1.105) | Loss 0.6020 (0.5329) | acc 0.992 (0.981) | IoU 0.740 (0.609) | [7 days, 16:19:21]
Lr: 9.774e-03 | Update: 3.998e-04 mean,5.547e-04 std | Epoch: [3][290/1047] | Time 1.715 (2.249) | Data 0.655 (1.104) | Loss 0.6074 (0.5345) | acc 0.956 (0.981) | IoU 0.512 (0.610) | [7 days, 16:14:43]
Lr: 9.773e-03 | Update: 3.318e-04 mean,5.683e-04 std | Epoch: [3][300/1047] | Time 1.927 (2.248) | Data 0.751 (1.104) | Loss 0.4406 (0.5354) | acc 0.975 (0.981) | IoU 0.587 (0.608) | [7 days, 16:11:13]
Lr: 9.772e-03 | Update: 3.803e-04 mean,9.867e-04 std | Epoch: [3][310/1047] | Time 1.737 (2.247) | Data 0.582 (1.103) | Loss 0.4780 (0.5356) | acc 0.991 (0.981) | IoU 0.588 (0.608) | [7 days, 16:05:37]
Lr: 9.771e-03 | Update: 3.619e-04 mean,5.252e-04 std | Epoch: [3][320/1047] | Time 2.161 (2.247) | Data 0.906 (1.102) | Loss 0.5141 (0.5364) | acc 0.990 (0.981) | IoU 0.679 (0.609) | [7 days, 16:02:07]
Lr: 9.770e-03 | Update: 3.299e-04 mean,7.432e-04 std | Epoch: [3][330/1047] | Time 1.866 (2.246) | Data 0.734 (1.101) | Loss 0.4687 (0.5345) | acc 0.971 (0.981) | IoU 0.514 (0.611) | [7 days, 15:57:04]
Lr: 9.769e-03 | Update: 4.687e-04 mean,9.404e-04 std | Epoch: [3][340/1047] | Time 2.241 (2.245) | Data 1.145 (1.101) | Loss 0.4288 (0.5356) | acc 0.992 (0.981) | IoU 0.747 (0.611) | [7 days, 15:54:18]
Lr: 9.768e-03 | Update: 6.603e-04 mean,2.736e-03 std | Epoch: [3][350/1047] | Time 2.052 (2.244) | Data 0.989 (1.100) | Loss 0.5898 (0.5351) | acc 0.987 (0.981) | IoU 0.676 (0.612) | [7 days, 15:49:15]
Lr: 9.767e-03 | Update: 7.283e-04 mean,1.082e-03 std | Epoch: [3][360/1047] | Time 2.110 (2.244) | Data 0.858 (1.099) | Loss 0.6633 (0.5355) | acc 0.984 (0.981) | IoU 0.591 (0.612) | [7 days, 15:44:26]
Lr: 9.766e-03 | Update: 1.931e-04 mean,4.504e-04 std | Epoch: [3][370/1047] | Time 2.161 (2.243) | Data 1.173 (1.099) | Loss 0.4189 (0.5359) | acc 0.996 (0.982) | IoU 0.815 (0.613) | [7 days, 15:40:04]
Lr: 9.765e-03 | Update: 5.083e-04 mean,1.235e-03 std | Epoch: [3][380/1047] | Time 2.063 (2.242) | Data 0.982 (1.098) | Loss 0.4221 (0.5360) | acc 0.987 (0.982) | IoU 0.573 (0.614) | [7 days, 15:35:48]
Lr: 9.764e-03 | Update: 2.561e-04 mean,3.896e-04 std | Epoch: [3][390/1047] | Time 2.124 (2.241) | Data 1.130 (1.097) | Loss 0.4602 (0.5351) | acc 0.997 (0.982) | IoU 0.655 (0.614) | [7 days, 15:31:28]
Lr: 9.763e-03 | Update: 3.602e-04 mean,6.213e-04 std | Epoch: [3][400/1047] | Time 2.087 (2.241) | Data 1.078 (1.096) | Loss 0.5326 (0.5361) | acc 0.981 (0.982) | IoU 0.522 (0.615) | [7 days, 15:26:21]
Lr: 9.763e-03 | Update: 3.721e-04 mean,5.358e-04 std | Epoch: [3][410/1047] | Time 2.276 (2.240) | Data 0.987 (1.095) | Loss 0.5046 (0.5358) | acc 0.982 (0.982) | IoU 0.531 (0.614) | [7 days, 15:21:35]
Lr: 9.762e-03 | Update: 5.129e-04 mean,8.834e-04 std | Epoch: [3][420/1047] | Time 2.260 (2.239) | Data 1.173 (1.095) | Loss 0.4826 (0.5347) | acc 0.994 (0.982) | IoU 0.632 (0.615) | [7 days, 15:17:01]
Lr: 9.761e-03 | Update: 2.568e-04 mean,4.813e-04 std | Epoch: [3][430/1047] | Time 2.041 (2.238) | Data 0.867 (1.094) | Loss 0.4584 (0.5342) | acc 0.979 (0.982) | IoU 0.700 (0.617) | [7 days, 15:12:50]
Lr: 9.760e-03 | Update: 4.874e-04 mean,7.271e-04 std | Epoch: [3][440/1047] | Time 1.773 (2.237) | Data 0.668 (1.093) | Loss 0.4474 (0.5345) | acc 0.991 (0.982) | IoU 0.672 (0.615) | [7 days, 15:08:19]
Lr: 9.759e-03 | Update: 3.244e-04 mean,5.188e-04 std | Epoch: [3][450/1047] | Time 1.915 (2.237) | Data 0.791 (1.092) | Loss 0.5497 (0.5347) | acc 0.975 (0.982) | IoU 0.571 (0.615) | [7 days, 15:03:56]
Lr: 9.758e-03 | Update: 7.291e-04 mean,1.159e-03 std | Epoch: [3][460/1047] | Time 2.524 (2.236) | Data 1.547 (1.092) | Loss 0.5897 (0.5343) | acc 0.983 (0.982) | IoU 0.616 (0.615) | [7 days, 15:00:28]
Lr: 9.757e-03 | Update: 4.333e-04 mean,8.282e-04 std | Epoch: [3][470/1047] | Time 1.954 (2.235) | Data 0.754 (1.091) | Loss 0.5732 (0.5342) | acc 0.986 (0.982) | IoU 0.551 (0.615) | [7 days, 14:56:34]
Lr: 9.756e-03 | Update: 4.693e-04 mean,1.393e-03 std | Epoch: [3][480/1047] | Time 1.858 (2.235) | Data 0.800 (1.090) | Loss 0.4676 (0.5342) | acc 0.994 (0.982) | IoU 0.833 (0.615) | [7 days, 14:51:54]
Lr: 9.755e-03 | Update: 3.936e-04 mean,8.751e-04 std | Epoch: [3][490/1047] | Time 2.396 (2.234) | Data 1.216 (1.090) | Loss 0.4018 (0.5335) | acc 0.985 (0.982) | IoU 0.574 (0.615) | [7 days, 14:47:57]
Lr: 9.754e-03 | Update: 5.359e-04 mean,8.874e-04 std | Epoch: [3][500/1047] | Time 1.717 (2.233) | Data 0.607 (1.089) | Loss 0.5866 (0.5331) | acc 0.975 (0.982) | IoU 0.560 (0.616) | [7 days, 14:44:10]
Lr: 9.753e-03 | Update: 2.256e-04 mean,3.775e-04 std | Epoch: [3][510/1047] | Time 1.711 (2.233) | Data 0.608 (1.088) | Loss 0.4466 (0.5340) | acc 0.981 (0.982) | IoU 0.527 (0.615) | [7 days, 14:40:20]
Lr: 9.752e-03 | Update: 1.816e-04 mean,2.683e-04 std | Epoch: [3][520/1047] | Time 1.952 (2.232) | Data 0.770 (1.088) | Loss 0.4479 (0.5348) | acc 0.987 (0.982) | IoU 0.683 (0.616) | [7 days, 14:36:42]
Lr: 9.751e-03 | Update: 7.500e-04 mean,1.266e-03 std | Epoch: [3][530/1047] | Time 2.344 (2.232) | Data 1.277 (1.088) | Loss 0.6598 (0.5348) | acc 0.991 (0.982) | IoU 0.571 (0.615) | [7 days, 14:34:43]
Lr: 9.750e-03 | Update: 2.552e-04 mean,3.595e-04 std | Epoch: [3][540/1047] | Time 2.127 (2.231) | Data 1.121 (1.087) | Loss 0.5989 (0.5359) | acc 0.976 (0.982) | IoU 0.518 (0.615) | [7 days, 14:30:39]
Lr: 9.749e-03 | Update: 2.337e-04 mean,3.763e-04 std | Epoch: [3][550/1047] | Time 2.349 (2.230) | Data 1.310 (1.086) | Loss 0.5642 (0.5374) | acc 0.927 (0.982) | IoU 0.480 (0.614) | [7 days, 14:27:00]
Lr: 9.748e-03 | Update: 4.409e-04 mean,7.191e-04 std | Epoch: [3][560/1047] | Time 1.867 (2.230) | Data 0.759 (1.086) | Loss 0.5449 (0.5370) | acc 0.996 (0.982) | IoU 0.855 (0.615) | [7 days, 14:23:05]
Lr: 9.748e-03 | Update: 2.289e-04 mean,4.062e-04 std | Epoch: [3][570/1047] | Time 1.997 (2.229) | Data 0.926 (1.085) | Loss 0.5158 (0.5367) | acc 0.970 (0.982) | IoU 0.518 (0.614) | [7 days, 14:19:25]
Lr: 9.747e-03 | Update: 2.256e-04 mean,3.451e-04 std | Epoch: [3][580/1047] | Time 2.248 (2.229) | Data 1.145 (1.085) | Loss 0.4289 (0.5367) | acc 0.985 (0.982) | IoU 0.585 (0.615) | [7 days, 14:16:02]
Lr: 9.746e-03 | Update: 1.823e-04 mean,2.620e-04 std | Epoch: [3][590/1047] | Time 1.990 (2.228) | Data 0.978 (1.084) | Loss 0.3642 (0.5365) | acc 0.988 (0.982) | IoU 0.691 (0.615) | [7 days, 14:12:17]
Lr: 9.745e-03 | Update: 6.050e-04 mean,1.368e-03 std | Epoch: [3][600/1047] | Time 2.304 (2.227) | Data 1.185 (1.083) | Loss 0.4770 (0.5355) | acc 0.996 (0.982) | IoU 0.796 (0.616) | [7 days, 14:08:25]
Lr: 9.744e-03 | Update: 2.787e-04 mean,3.950e-04 std | Epoch: [3][610/1047] | Time 2.122 (2.227) | Data 1.062 (1.083) | Loss 0.6003 (0.5348) | acc 0.959 (0.982) | IoU 0.494 (0.617) | [7 days, 14:05:19]
Lr: 9.743e-03 | Update: 3.591e-04 mean,4.963e-04 std | Epoch: [3][620/1047] | Time 2.092 (2.226) | Data 0.956 (1.082) | Loss 0.3653 (0.5346) | acc 0.995 (0.982) | IoU 0.816 (0.618) | [7 days, 14:01:41]
Lr: 9.742e-03 | Update: 8.768e-04 mean,1.375e-03 std | Epoch: [3][630/1047] | Time 1.836 (2.226) | Data 0.688 (1.082) | Loss 0.6331 (0.5335) | acc 0.989 (0.982) | IoU 0.728 (0.619) | [7 days, 13:57:42]
Lr: 9.741e-03 | Update: 2.968e-04 mean,9.457e-04 std | Epoch: [3][640/1047] | Time 1.677 (2.225) | Data 0.598 (1.081) | Loss 0.4676 (0.5323) | acc 0.979 (0.982) | IoU 0.605 (0.619) | [7 days, 13:53:28]
Lr: 9.740e-03 | Update: 4.550e-04 mean,1.266e-03 std | Epoch: [3][650/1047] | Time 1.945 (2.224) | Data 0.897 (1.080) | Loss 0.4449 (0.5318) | acc 0.987 (0.982) | IoU 0.674 (0.619) | [7 days, 13:49:45]
Lr: 9.739e-03 | Update: 2.182e-04 mean,3.868e-04 std | Epoch: [3][660/1047] | Time 2.102 (2.224) | Data 0.938 (1.080) | Loss 0.3899 (0.5316) | acc 0.989 (0.983) | IoU 0.629 (0.620) | [7 days, 13:46:14]
Lr: 9.738e-03 | Update: 2.081e-04 mean,4.281e-04 std | Epoch: [3][670/1047] | Time 2.080 (2.223) | Data 0.970 (1.079) | Loss 0.4258 (0.5310) | acc 0.990 (0.983) | IoU 0.637 (0.619) | [7 days, 13:43:03]
Lr: 9.737e-03 | Update: 3.575e-04 mean,1.509e-03 std | Epoch: [3][680/1047] | Time 2.216 (2.223) | Data 0.988 (1.079) | Loss 0.4740 (0.5310) | acc 0.994 (0.983) | IoU 0.705 (0.620) | [7 days, 13:40:16]
Lr: 9.736e-03 | Update: 2.871e-04 mean,1.015e-03 std | Epoch: [3][690/1047] | Time 1.992 (2.222) | Data 0.920 (1.078) | Loss 0.5494 (0.5318) | acc 0.941 (0.982) | IoU 0.495 (0.618) | [7 days, 13:37:02]
Lr: 9.735e-03 | Update: 5.273e-04 mean,1.215e-03 std | Epoch: [3][700/1047] | Time 1.799 (2.221) | Data 0.692 (1.077) | Loss 0.5683 (0.5316) | acc 0.992 (0.982) | IoU 0.624 (0.618) | [7 days, 13:33:05]
Lr: 9.734e-03 | Update: 3.204e-04 mean,1.439e-03 std | Epoch: [3][710/1047] | Time 2.083 (2.221) | Data 0.792 (1.077) | Loss 0.5562 (0.5312) | acc 0.984 (0.982) | IoU 0.615 (0.618) | [7 days, 13:29:04]
Lr: 9.733e-03 | Update: 7.754e-04 mean,2.320e-03 std | Epoch: [3][720/1047] | Time 1.930 (2.220) | Data 0.702 (1.076) | Loss 0.6014 (0.5306) | acc 0.994 (0.982) | IoU 0.602 (0.620) | [7 days, 13:25:24]
Lr: 9.733e-03 | Update: 5.115e-04 mean,9.507e-04 std | Epoch: [3][730/1047] | Time 2.300 (2.220) | Data 1.136 (1.076) | Loss 0.6103 (0.5304) | acc 0.975 (0.982) | IoU 0.595 (0.620) | [7 days, 13:23:16]
Lr: 9.732e-03 | Update: 9.304e-04 mean,1.738e-03 std | Epoch: [3][740/1047] | Time 2.122 (2.219) | Data 0.854 (1.075) | Loss 0.6873 (0.5297) | acc 0.989 (0.982) | IoU 0.693 (0.620) | [7 days, 13:19:53]
Lr: 9.731e-03 | Update: 7.912e-04 mean,6.322e-03 std | Epoch: [3][750/1047] | Time 1.996 (2.219) | Data 0.906 (1.075) | Loss 0.5796 (0.5309) | acc 0.969 (0.982) | IoU 0.554 (0.620) | [7 days, 13:16:52]
Lr: 9.730e-03 | Update: 2.519e-04 mean,1.160e-03 std | Epoch: [3][760/1047] | Time 2.102 (2.218) | Data 0.924 (1.074) | Loss 0.4265 (0.5305) | acc 0.993 (0.982) | IoU 0.596 (0.619) | [7 days, 13:13:38]
Lr: 9.729e-03 | Update: 6.046e-04 mean,9.555e-04 std | Epoch: [3][770/1047] | Time 1.850 (2.218) | Data 0.730 (1.074) | Loss 0.6389 (0.5311) | acc 0.987 (0.982) | IoU 0.742 (0.620) | [7 days, 13:10:35]
Lr: 9.728e-03 | Update: 2.827e-04 mean,4.301e-04 std | Epoch: [3][780/1047] | Time 1.989 (2.217) | Data 0.787 (1.073) | Loss 0.3655 (0.5304) | acc 0.985 (0.982) | IoU 0.675 (0.620) | [7 days, 13:07:23]
Lr: 9.727e-03 | Update: 1.173e-03 mean,2.412e-03 std | Epoch: [3][790/1047] | Time 2.157 (2.217) | Data 1.019 (1.073) | Loss 0.7535 (0.5317) | acc 0.983 (0.982) | IoU 0.707 (0.619) | [7 days, 13:05:36]
Lr: 9.726e-03 | Update: 2.200e-04 mean,3.800e-04 std | Epoch: [3][800/1047] | Time 2.052 (2.217) | Data 0.894 (1.073) | Loss 0.4345 (0.5317) | acc 0.982 (0.982) | IoU 0.576 (0.619) | [7 days, 13:03:11]
Lr: 9.725e-03 | Update: 1.825e-04 mean,2.921e-04 std | Epoch: [3][810/1047] | Time 2.101 (2.216) | Data 0.771 (1.072) | Loss 0.4201 (0.5320) | acc 0.993 (0.982) | IoU 0.563 (0.619) | [7 days, 13:00:39]
Lr: 9.724e-03 | Update: 4.435e-04 mean,2.692e-03 std | Epoch: [3][820/1047] | Time 2.213 (2.216) | Data 0.948 (1.072) | Loss 0.4940 (0.5317) | acc 0.989 (0.982) | IoU 0.682 (0.619) | [7 days, 12:58:24]
Lr: 9.723e-03 | Update: 3.909e-04 mean,2.772e-03 std | Epoch: [3][830/1047] | Time 2.385 (2.216) | Data 1.232 (1.072) | Loss 0.4965 (0.5314) | acc 0.992 (0.982) | IoU 0.680 (0.619) | [7 days, 12:57:01]
Lr: 9.722e-03 | Update: 3.212e-04 mean,5.312e-04 std | Epoch: [3][840/1047] | Time 1.993 (2.215) | Data 0.931 (1.071) | Loss 0.5639 (0.5316) | acc 0.984 (0.983) | IoU 0.687 (0.619) | [7 days, 12:54:29]
Lr: 9.721e-03 | Update: 2.252e-04 mean,4.337e-04 std | Epoch: [3][850/1047] | Time 2.363 (2.215) | Data 1.131 (1.071) | Loss 0.4774 (0.5318) | acc 0.981 (0.982) | IoU 0.532 (0.619) | [7 days, 12:53:25]
Lr: 9.720e-03 | Update: 2.507e-04 mean,8.882e-04 std | Epoch: [3][860/1047] | Time 1.795 (2.215) | Data 0.677 (1.071) | Loss 0.4698 (0.5317) | acc 0.997 (0.983) | IoU 0.812 (0.619) | [7 days, 12:51:26]
Lr: 9.719e-03 | Update: 2.353e-04 mean,9.835e-04 std | Epoch: [3][870/1047] | Time 2.282 (2.215) | Data 1.131 (1.071) | Loss 0.3869 (0.5306) | acc 0.998 (0.983) | IoU 0.785 (0.621) | [7 days, 12:50:10]
Lr: 9.719e-03 | Update: 4.326e-04 mean,8.740e-04 std | Epoch: [3][880/1047] | Time 2.050 (2.215) | Data 1.050 (1.070) | Loss 0.6500 (0.5311) | acc 0.978 (0.983) | IoU 0.545 (0.621) | [7 days, 12:47:23]
Lr: 9.718e-03 | Update: 3.512e-04 mean,1.912e-03 std | Epoch: [3][890/1047] | Time 1.875 (2.214) | Data 0.852 (1.070) | Loss 0.4652 (0.5307) | acc 0.971 (0.983) | IoU 0.579 (0.620) | [7 days, 12:45:04]
Lr: 9.717e-03 | Update: 7.114e-04 mean,6.770e-03 std | Epoch: [3][900/1047] | Time 1.968 (2.214) | Data 0.887 (1.070) | Loss 0.4206 (0.5305) | acc 0.996 (0.983) | IoU 0.734 (0.620) | [7 days, 12:43:43]
Lr: 9.716e-03 | Update: 1.189e-03 mean,3.030e-03 std | Epoch: [3][910/1047] | Time 1.769 (2.214) | Data 0.627 (1.069) | Loss 0.7551 (0.5300) | acc 0.977 (0.983) | IoU 0.514 (0.621) | [7 days, 12:41:44]
Lr: 9.715e-03 | Update: 4.493e-04 mean,2.384e-03 std | Epoch: [3][920/1047] | Time 2.362 (2.214) | Data 1.309 (1.070) | Loss 0.4035 (0.5296) | acc 0.996 (0.983) | IoU 0.739 (0.621) | [7 days, 12:41:21]
Lr: 9.714e-03 | Update: 2.313e-04 mean,3.613e-04 std | Epoch: [3][930/1047] | Time 2.139 (2.214) | Data 0.897 (1.069) | Loss 0.4852 (0.5294) | acc 0.991 (0.983) | IoU 0.561 (0.622) | [7 days, 12:40:21]
Lr: 9.713e-03 | Update: 9.208e-04 mean,3.023e-03 std | Epoch: [3][940/1047] | Time 2.195 (2.214) | Data 1.010 (1.069) | Loss 0.6028 (0.5296) | acc 0.974 (0.983) | IoU 0.577 (0.622) | [7 days, 12:38:24]
Lr: 9.712e-03 | Update: 3.297e-04 mean,1.346e-03 std | Epoch: [3][950/1047] | Time 2.100 (2.214) | Data 0.996 (1.069) | Loss 0.4020 (0.5296) | acc 0.998 (0.983) | IoU 0.882 (0.622) | [7 days, 12:37:53]
Lr: 9.711e-03 | Update: 2.827e-04 mean,4.881e-04 std | Epoch: [3][960/1047] | Time 2.139 (2.213) | Data 0.959 (1.069) | Loss 0.5431 (0.5303) | acc 0.951 (0.983) | IoU 0.483 (0.622) | [7 days, 12:36:19]
Lr: 9.710e-03 | Update: 1.970e-03 mean,6.581e-03 std | Epoch: [3][970/1047] | Time 1.956 (2.213) | Data 0.717 (1.069) | Loss 0.7916 (0.5303) | acc 0.987 (0.983) | IoU 0.613 (0.621) | [7 days, 12:34:48]
Lr: 9.709e-03 | Update: 8.624e-04 mean,6.192e-03 std | Epoch: [3][980/1047] | Time 2.314 (2.213) | Data 1.017 (1.068) | Loss 0.6677 (0.5300) | acc 0.980 (0.983) | IoU 0.526 (0.621) | [7 days, 12:33:30]
Lr: 9.708e-03 | Update: 1.925e-02 mean,3.685e-01 std | Epoch: [3][990/1047] | Time 1.961 (2.213) | Data 0.804 (1.068) | Loss 0.5366 (0.5307) | acc 0.979 (0.983) | IoU 0.540 (0.621) | [7 days, 12:32:52]
Lr: 9.707e-03 | Update: 3.198e-04 mean,4.814e-04 std | Epoch: [3][1000/1047] | Time 2.086 (2.213) | Data 0.959 (1.068) | Loss 0.5425 (0.5306) | acc 0.986 (0.983) | IoU 0.529 (0.621) | [7 days, 12:30:55]
Lr: 9.706e-03 | Update: 1.181e-04 mean,2.420e-04 std | Epoch: [3][1010/1047] | Time 2.259 (2.213) | Data 1.052 (1.068) | Loss 0.3732 (0.5305) | acc 0.996 (0.983) | IoU 0.816 (0.621) | [7 days, 12:30:44]
Lr: 9.706e-03 | Update: 3.094e-04 mean,6.237e-04 std | Epoch: [3][1020/1047] | Time 2.112 (2.213) | Data 0.957 (1.068) | Loss 0.6100 (0.5306) | acc 0.979 (0.983) | IoU 0.506 (0.622) | [7 days, 12:30:41]
Lr: 9.705e-03 | Update: 5.776e-04 mean,8.948e-04 std | Epoch: [3][1030/1047] | Time 2.194 (2.213) | Data 1.019 (1.068) | Loss 0.5599 (0.5303) | acc 0.988 (0.983) | IoU 0.658 (0.622) | [7 days, 12:29:26]
Lr: 9.704e-03 | Update: 4.823e-04 mean,6.858e-04 std | Epoch: [3][1040/1047] | Time 2.284 (2.213) | Data 1.100 (1.068) | Loss 0.5914 (0.5295) | acc 0.990 (0.983) | IoU 0.663 (0.622) | [7 days, 12:29:38]
Best mean iou in training set so far, save model!
Validation:: 100%|████████████████████████████| 508/508 [12:38<00:00,  1.49s/it]
********************************************************************************
Validation set:
Time avg per batch 1.862
Loss avg 0.4427
Jaccard avg 0.2673
WCE avg 0.1753
Acc avg 0.982607
IoU avg 0.564428
IoU class 0 [unlabeled] = 0.000000
IoU class 1 [static] = 0.982555
IoU class 2 [moving] = 0.146301
********************************************************************************
Lr: 9.703e-03 | Update: 3.961e-04 mean,1.863e-03 std | Epoch: [4][0/1047] | Time 1.768 (2.213) | Data 0.599 (1.068) | Loss 0.6463 (0.6463) | acc 0.954 (0.954) | IoU 0.480 (0.480) | [7 days, 9:40:30]
Lr: 9.702e-03 | Update: 2.115e-04 mean,4.134e-04 std | Epoch: [4][10/1047] | Time 2.184 (2.213) | Data 0.942 (1.068) | Loss 0.5510 (0.5529) | acc 0.960 (0.953) | IoU 0.534 (0.533) | [7 days, 9:38:37]
Lr: 9.701e-03 | Update: 3.978e-04 mean,1.165e-03 std | Epoch: [4][20/1047] | Time 2.197 (2.212) | Data 1.082 (1.068) | Loss 0.5317 (0.5511) | acc 0.990 (0.965) | IoU 0.580 (0.564) | [7 days, 9:36:13]
Lr: 9.700e-03 | Update: 2.977e-04 mean,1.577e-03 std | Epoch: [4][30/1047] | Time 2.253 (2.212) | Data 1.132 (1.067) | Loss 0.4496 (0.5401) | acc 0.994 (0.974) | IoU 0.700 (0.598) | [7 days, 9:33:40]
Lr: 9.699e-03 | Update: 3.292e-04 mean,1.373e-03 std | Epoch: [4][40/1047] | Time 1.962 (2.211) | Data 0.883 (1.067) | Loss 0.5206 (0.5256) | acc 0.980 (0.976) | IoU 0.523 (0.598) | [7 days, 9:30:45]
Lr: 9.698e-03 | Update: 3.630e-04 mean,9.636e-04 std | Epoch: [4][50/1047] | Time 2.033 (2.211) | Data 0.970 (1.066) | Loss 0.4992 (0.5168) | acc 0.976 (0.979) | IoU 0.540 (0.622) | [7 days, 9:27:53]
Lr: 9.697e-03 | Update: 4.473e-04 mean,3.720e-03 std | Epoch: [4][60/1047] | Time 2.243 (2.211) | Data 1.093 (1.066) | Loss 0.4191 (0.5128) | acc 0.994 (0.980) | IoU 0.628 (0.624) | [7 days, 9:25:20]
Lr: 9.696e-03 | Update: 4.709e-04 mean,1.119e-03 std | Epoch: [4][70/1047] | Time 2.006 (2.210) | Data 0.914 (1.065) | Loss 0.5927 (0.5263) | acc 0.982 (0.981) | IoU 0.710 (0.623) | [7 days, 9:22:51]
Lr: 9.696e-03 | Update: 3.359e-04 mean,1.837e-03 std | Epoch: [4][80/1047] | Time 1.948 (2.210) | Data 0.887 (1.065) | Loss 0.4118 (0.5311) | acc 0.953 (0.976) | IoU 0.591 (0.610) | [7 days, 9:20:20]
Lr: 9.695e-03 | Update: 5.346e-04 mean,1.387e-03 std | Epoch: [4][90/1047] | Time 2.096 (2.210) | Data 0.775 (1.065) | Loss 0.5551 (0.5350) | acc 0.989 (0.975) | IoU 0.782 (0.607) | [7 days, 9:18:38]
Lr: 9.694e-03 | Update: 5.087e-04 mean,1.008e-03 std | Epoch: [4][100/1047] | Time 1.946 (2.209) | Data 0.841 (1.064) | Loss 0.4723 (0.5271) | acc 0.989 (0.976) | IoU 0.718 (0.613) | [7 days, 9:15:52]
Lr: 9.693e-03 | Update: 4.740e-04 mean,3.148e-03 std | Epoch: [4][110/1047] | Time 2.304 (2.209) | Data 1.154 (1.064) | Loss 0.3969 (0.5217) | acc 0.987 (0.977) | IoU 0.633 (0.616) | [7 days, 9:14:00]
Lr: 9.692e-03 | Update: 2.007e-03 mean,2.321e-02 std | Epoch: [4][120/1047] | Time 1.852 (2.209) | Data 0.619 (1.064) | Loss 0.5922 (0.5229) | acc 0.998 (0.978) | IoU 0.696 (0.616) | [7 days, 9:12:13]
Lr: 9.691e-03 | Update: 4.207e-04 mean,8.630e-04 std | Epoch: [4][130/1047] | Time 2.030 (2.209) | Data 0.772 (1.064) | Loss 0.5038 (0.5181) | acc 0.992 (0.979) | IoU 0.600 (0.618) | [7 days, 9:10:33]
Lr: 9.690e-03 | Update: 5.247e-04 mean,4.613e-03 std | Epoch: [4][140/1047] | Time 2.175 (2.208) | Data 0.891 (1.063) | Loss 0.4335 (0.5156) | acc 0.982 (0.980) | IoU 0.523 (0.624) | [7 days, 9:07:49]
Lr: 9.689e-03 | Update: 5.526e-04 mean,5.104e-03 std | Epoch: [4][150/1047] | Time 2.350 (2.208) | Data 1.231 (1.063) | Loss 0.4549 (0.5106) | acc 0.977 (0.980) | IoU 0.654 (0.630) | [7 days, 9:05:54]
Lr: 9.688e-03 | Update: 9.248e-04 mean,8.237e-03 std | Epoch: [4][160/1047] | Time 2.162 (2.208) | Data 0.843 (1.063) | Loss 0.5884 (0.5144) | acc 0.983 (0.980) | IoU 0.564 (0.626) | [7 days, 9:03:46]
Lr: 9.687e-03 | Update: 5.169e-03 mean,7.377e-02 std | Epoch: [4][170/1047] | Time 2.177 (2.207) | Data 0.960 (1.062) | Loss 0.8142 (0.5163) | acc 0.992 (0.981) | IoU 0.798 (0.628) | [7 days, 9:00:59]
Lr: 9.686e-03 | Update: 1.825e-03 mean,2.508e-02 std | Epoch: [4][180/1047] | Time 2.222 (2.207) | Data 1.084 (1.062) | Loss 0.6210 (0.5152) | acc 0.973 (0.981) | IoU 0.505 (0.630) | [7 days, 8:58:31]
Lr: 9.685e-03 | Update: 8.755e-04 mean,4.471e-03 std | Epoch: [4][190/1047] | Time 2.362 (2.206) | Data 1.216 (1.061) | Loss 0.6294 (0.5144) | acc 0.992 (0.982) | IoU 0.792 (0.635) | [7 days, 8:56:01]
Lr: 9.684e-03 | Update: 3.301e-04 mean,9.818e-04 std | Epoch: [4][200/1047] | Time 2.049 (2.206) | Data 0.730 (1.061) | Loss 0.4828 (0.5118) | acc 0.988 (0.982) | IoU 0.585 (0.635) | [7 days, 8:53:44]
Lr: 9.683e-03 | Update: 7.042e-04 mean,5.814e-03 std | Epoch: [4][210/1047] | Time 2.224 (2.206) | Data 1.109 (1.061) | Loss 0.5619 (0.5133) | acc 0.962 (0.982) | IoU 0.521 (0.634) | [7 days, 8:51:28]
Lr: 9.683e-03 | Update: 1.014e-03 mean,1.564e-02 std | Epoch: [4][220/1047] | Time 2.218 (2.205) | Data 1.023 (1.060) | Loss 0.4523 (0.5120) | acc 0.990 (0.982) | IoU 0.678 (0.636) | [7 days, 8:49:09]
Lr: 9.682e-03 | Update: 2.823e-04 mean,1.214e-03 std | Epoch: [4][230/1047] | Time 2.065 (2.205) | Data 0.900 (1.060) | Loss 0.4262 (0.5116) | acc 0.986 (0.982) | IoU 0.604 (0.636) | [7 days, 8:46:36]
Lr: 9.681e-03 | Update: 3.110e-04 mean,4.882e-04 std | Epoch: [4][240/1047] | Time 2.241 (2.205) | Data 1.197 (1.060) | Loss 0.4328 (0.5093) | acc 0.990 (0.983) | IoU 0.531 (0.637) | [7 days, 8:44:37]
Lr: 9.680e-03 | Update: 2.261e-04 mean,3.471e-04 std | Epoch: [4][250/1047] | Time 1.834 (2.204) | Data 0.623 (1.059) | Loss 0.4283 (0.5079) | acc 0.988 (0.983) | IoU 0.614 (0.637) | [7 days, 8:42:09]
Lr: 9.679e-03 | Update: 5.991e-04 mean,1.485e-03 std | Epoch: [4][260/1047] | Time 1.783 (2.204) | Data 0.690 (1.059) | Loss 0.4824 (0.5089) | acc 0.993 (0.983) | IoU 0.841 (0.636) | [7 days, 8:39:42]
Lr: 9.678e-03 | Update: 6.473e-04 mean,2.559e-03 std | Epoch: [4][270/1047] | Time 2.176 (2.204) | Data 0.947 (1.059) | Loss 0.7600 (0.5095) | acc 0.982 (0.983) | IoU 0.689 (0.637) | [7 days, 8:38:06]
Lr: 9.677e-03 | Update: 3.938e-04 mean,8.063e-04 std | Epoch: [4][280/1047] | Time 1.918 (2.203) | Data 0.760 (1.058) | Loss 0.4227 (0.5091) | acc 0.997 (0.983) | IoU 0.831 (0.638) | [7 days, 8:34:42]
Lr: 9.676e-03 | Update: 5.323e-04 mean,6.093e-03 std | Epoch: [4][290/1047] | Time 2.085 (2.203) | Data 0.803 (1.058) | Loss 0.4241 (0.5089) | acc 0.984 (0.983) | IoU 0.671 (0.639) | [7 days, 8:32:51]
Lr: 9.675e-03 | Update: 4.684e-04 mean,2.547e-03 std | Epoch: [4][300/1047] | Time 2.233 (2.203) | Data 1.015 (1.057) | Loss 0.4789 (0.5080) | acc 0.996 (0.984) | IoU 0.613 (0.638) | [7 days, 8:30:53]
Lr: 9.674e-03 | Update: 9.827e-04 mean,1.974e-03 std | Epoch: [4][310/1047] | Time 2.106 (2.203) | Data 0.860 (1.057) | Loss 0.4681 (0.5058) | acc 0.993 (0.984) | IoU 0.643 (0.642) | [7 days, 8:28:54]
Lr: 9.673e-03 | Update: 4.016e-04 mean,1.703e-03 std | Epoch: [4][320/1047] | Time 1.970 (2.202) | Data 0.778 (1.057) | Loss 0.4684 (0.5062) | acc 0.975 (0.984) | IoU 0.546 (0.641) | [7 days, 8:26:17]
Lr: 9.672e-03 | Update: 4.190e-04 mean,3.231e-03 std | Epoch: [4][330/1047] | Time 1.988 (2.202) | Data 0.788 (1.056) | Loss 0.4701 (0.5054) | acc 0.984 (0.984) | IoU 0.608 (0.641) | [7 days, 8:23:36]
Lr: 9.671e-03 | Update: 3.687e-03 mean,5.433e-02 std | Epoch: [4][340/1047] | Time 2.129 (2.201) | Data 0.972 (1.056) | Loss 0.6540 (0.5057) | acc 0.995 (0.984) | IoU 0.867 (0.641) | [7 days, 8:21:56]
Lr: 9.670e-03 | Update: 5.936e-04 mean,2.864e-03 std | Epoch: [4][350/1047] | Time 2.136 (2.201) | Data 1.090 (1.056) | Loss 0.5147 (0.5072) | acc 0.970 (0.984) | IoU 0.554 (0.640) | [7 days, 8:19:14]
Lr: 9.670e-03 | Update: 6.477e-04 mean,3.493e-03 std | Epoch: [4][360/1047] | Time 1.692 (2.201) | Data 0.624 (1.055) | Loss 0.3886 (0.5058) | acc 0.995 (0.984) | IoU 0.639 (0.639) | [7 days, 8:16:34]
Lr: 9.669e-03 | Update: 4.901e-04 mean,5.754e-03 std | Epoch: [4][370/1047] | Time 2.055 (2.201) | Data 0.941 (1.055) | Loss 0.5777 (0.5049) | acc 0.981 (0.984) | IoU 0.513 (0.643) | [7 days, 8:16:04]
Lr: 9.668e-03 | Update: 5.408e-04 mean,4.009e-03 std | Epoch: [4][380/1047] | Time 2.271 (2.201) | Data 1.055 (1.055) | Loss 0.5583 (0.5066) | acc 0.976 (0.984) | IoU 0.573 (0.642) | [7 days, 8:15:55]
Lr: 9.667e-03 | Update: 1.191e-03 mean,8.411e-03 std | Epoch: [4][390/1047] | Time 2.337 (2.201) | Data 1.190 (1.055) | Loss 0.6176 (0.5089) | acc 0.975 (0.984) | IoU 0.518 (0.639) | [7 days, 8:15:06]
Lr: 9.666e-03 | Update: 3.736e-04 mean,6.192e-04 std | Epoch: [4][400/1047] | Time 2.369 (2.201) | Data 1.228 (1.055) | Loss 0.5691 (0.5110) | acc 0.980 (0.984) | IoU 0.506 (0.638) | [7 days, 8:15:19]
Lr: 9.665e-03 | Update: 8.837e-04 mean,2.263e-03 std | Epoch: [4][410/1047] | Time 2.063 (2.201) | Data 0.832 (1.055) | Loss 0.6738 (0.5109) | acc 0.990 (0.984) | IoU 0.698 (0.639) | [7 days, 8:14:58]
Lr: 9.664e-03 | Update: 6.341e-04 mean,9.480e-04 std | Epoch: [4][420/1047] | Time 2.315 (2.201) | Data 1.111 (1.055) | Loss 0.4891 (0.5107) | acc 0.989 (0.984) | IoU 0.661 (0.639) | [7 days, 8:13:42]
Lr: 9.663e-03 | Update: 4.354e-04 mean,6.077e-04 std | Epoch: [4][430/1047] | Time 2.212 (2.201) | Data 1.120 (1.055) | Loss 0.5504 (0.5102) | acc 0.983 (0.984) | IoU 0.552 (0.638) | [7 days, 8:13:21]
Lr: 9.662e-03 | Update: 7.972e-04 mean,2.226e-03 std | Epoch: [4][440/1047] | Time 2.356 (2.201) | Data 1.219 (1.055) | Loss 0.5779 (0.5099) | acc 0.989 (0.984) | IoU 0.747 (0.639) | [7 days, 8:12:16]
Lr: 9.661e-03 | Update: 4.062e-04 mean,6.306e-04 std | Epoch: [4][450/1047] | Time 2.297 (2.201) | Data 1.068 (1.055) | Loss 0.5839 (0.5109) | acc 0.957 (0.984) | IoU 0.572 (0.638) | [7 days, 8:10:36]
Lr: 9.660e-03 | Update: 3.506e-04 mean,6.005e-04 std | Epoch: [4][460/1047] | Time 2.380 (2.201) | Data 1.265 (1.054) | Loss 0.7395 (0.5123) | acc 0.974 (0.983) | IoU 0.542 (0.636) | [7 days, 8:09:35]
Lr: 9.659e-03 | Update: 2.849e-04 mean,1.004e-03 std | Epoch: [4][470/1047] | Time 2.225 (2.201) | Data 0.958 (1.054) | Loss 0.2925 (0.5119) | acc 0.987 (0.983) | IoU 0.739 (0.637) | [7 days, 8:08:47]
Lr: 9.658e-03 | Update: 9.289e-04 mean,3.888e-03 std | Epoch: [4][480/1047] | Time 2.608 (2.201) | Data 1.317 (1.054) | Loss 0.4846 (0.5119) | acc 0.990 (0.983) | IoU 0.725 (0.636) | [7 days, 8:08:22]
Lr: 9.657e-03 | Update: 3.176e-04 mean,5.405e-04 std | Epoch: [4][490/1047] | Time 2.521 (2.201) | Data 1.137 (1.054) | Loss 0.5632 (0.5119) | acc 0.997 (0.984) | IoU 0.811 (0.636) | [7 days, 8:08:08]
Lr: 9.657e-03 | Update: 5.482e-04 mean,8.310e-04 std | Epoch: [4][500/1047] | Time 2.146 (2.201) | Data 1.038 (1.054) | Loss 0.6815 (0.5115) | acc 0.984 (0.984) | IoU 0.525 (0.638) | [7 days, 8:07:21]
Lr: 9.656e-03 | Update: 1.812e-04 mean,3.267e-04 std | Epoch: [4][510/1047] | Time 2.345 (2.201) | Data 1.049 (1.054) | Loss 0.3615 (0.5114) | acc 0.997 (0.984) | IoU 0.891 (0.638) | [7 days, 8:05:50]
Lr: 9.655e-03 | Update: 2.406e-04 mean,3.466e-04 std | Epoch: [4][520/1047] | Time 2.167 (2.201) | Data 1.044 (1.054) | Loss 0.4227 (0.5112) | acc 0.989 (0.984) | IoU 0.587 (0.639) | [7 days, 8:04:22]
Lr: 9.654e-03 | Update: 7.979e-04 mean,2.519e-03 std | Epoch: [4][530/1047] | Time 2.713 (2.201) | Data 1.378 (1.054) | Loss 0.5466 (0.5106) | acc 0.987 (0.984) | IoU 0.637 (0.639) | [7 days, 8:04:09]
Lr: 9.653e-03 | Update: 4.523e-04 mean,8.184e-04 std | Epoch: [4][540/1047] | Time 2.273 (2.201) | Data 1.045 (1.054) | Loss 0.3582 (0.5092) | acc 0.997 (0.984) | IoU 0.806 (0.640) | [7 days, 8:04:18]
Lr: 9.652e-03 | Update: 2.037e-04 mean,4.462e-04 std | Epoch: [4][550/1047] | Time 2.142 (2.201) | Data 0.912 (1.053) | Loss 0.3378 (0.5099) | acc 0.994 (0.984) | IoU 0.727 (0.641) | [7 days, 8:03:08]
Lr: 9.651e-03 | Update: 2.635e-04 mean,4.114e-04 std | Epoch: [4][560/1047] | Time 1.687 (2.201) | Data 0.660 (1.053) | Loss 0.5710 (0.5111) | acc 0.950 (0.984) | IoU 0.488 (0.639) | [7 days, 7:59:18]
Lr: 9.650e-03 | Update: 3.195e-04 mean,5.124e-04 std | Epoch: [4][570/1047] | Time 1.988 (2.200) | Data 0.848 (1.053) | Loss 0.5615 (0.5109) | acc 0.997 (0.984) | IoU 0.717 (0.639) | [7 days, 7:57:51]
Lr: 9.649e-03 | Update: 3.413e-04 mean,1.203e-03 std | Epoch: [4][580/1047] | Time 1.884 (2.200) | Data 0.742 (1.052) | Loss 0.4625 (0.5116) | acc 0.986 (0.984) | IoU 0.633 (0.638) | [7 days, 7:55:49]
Lr: 9.648e-03 | Update: 2.174e-04 mean,3.079e-04 std | Epoch: [4][590/1047] | Time 2.191 (2.200) | Data 1.012 (1.052) | Loss 0.3856 (0.5103) | acc 0.993 (0.984) | IoU 0.550 (0.639) | [7 days, 7:55:58]
Lr: 9.647e-03 | Update: 2.541e-04 mean,4.392e-04 std | Epoch: [4][600/1047] | Time 2.173 (2.200) | Data 1.146 (1.052) | Loss 0.4171 (0.5104) | acc 0.980 (0.984) | IoU 0.571 (0.640) | [7 days, 7:54:09]
Lr: 9.646e-03 | Update: 6.908e-04 mean,1.074e-03 std | Epoch: [4][610/1047] | Time 1.940 (2.200) | Data 0.777 (1.052) | Loss 0.8926 (0.5115) | acc 0.961 (0.984) | IoU 0.488 (0.638) | [7 days, 7:52:13]
Lr: 9.645e-03 | Update: 6.206e-04 mean,1.361e-03 std | Epoch: [4][620/1047] | Time 2.087 (2.200) | Data 0.957 (1.052) | Loss 0.7251 (0.5121) | acc 0.989 (0.984) | IoU 0.744 (0.638) | [7 days, 7:51:19]
Lr: 9.644e-03 | Update: 2.504e-04 mean,3.300e-04 std | Epoch: [4][630/1047] | Time 1.839 (2.200) | Data 0.741 (1.052) | Loss 0.4294 (0.5115) | acc 0.996 (0.984) | IoU 0.738 (0.639) | [7 days, 7:49:55]
Lr: 9.644e-03 | Update: 9.789e-04 mean,4.966e-03 std | Epoch: [4][640/1047] | Time 1.652 (2.199) | Data 0.580 (1.051) | Loss 0.5926 (0.5115) | acc 0.990 (0.984) | IoU 0.765 (0.639) | [7 days, 7:47:20]
Lr: 9.643e-03 | Update: 7.252e-04 mean,1.000e-03 std | Epoch: [4][650/1047] | Time 2.186 (2.199) | Data 0.991 (1.051) | Loss 0.7553 (0.5115) | acc 0.972 (0.984) | IoU 0.511 (0.638) | [7 days, 7:46:43]
Lr: 9.642e-03 | Update: 1.449e-04 mean,3.116e-04 std | Epoch: [4][660/1047] | Time 2.050 (2.199) | Data 0.975 (1.051) | Loss 0.2995 (0.5111) | acc 0.998 (0.984) | IoU 0.897 (0.638) | [7 days, 7:44:21]
Lr: 9.641e-03 | Update: 6.957e-04 mean,1.134e-03 std | Epoch: [4][670/1047] | Time 1.875 (2.198) | Data 0.704 (1.050) | Loss 0.5098 (0.5100) | acc 0.994 (0.984) | IoU 0.766 (0.640) | [7 days, 7:42:08]
Lr: 9.640e-03 | Update: 2.088e-04 mean,4.734e-04 std | Epoch: [4][680/1047] | Time 2.251 (2.198) | Data 1.174 (1.050) | Loss 0.5361 (0.5101) | acc 0.987 (0.984) | IoU 0.538 (0.639) | [7 days, 7:40:49]
Lr: 9.639e-03 | Update: 3.550e-04 mean,5.378e-04 std | Epoch: [4][690/1047] | Time 1.936 (2.198) | Data 0.913 (1.050) | Loss 0.6219 (0.5101) | acc 0.983 (0.984) | IoU 0.531 (0.640) | [7 days, 7:38:48]
Lr: 9.638e-03 | Update: 5.017e-04 mean,9.949e-04 std | Epoch: [4][700/1047] | Time 2.163 (2.198) | Data 1.064 (1.050) | Loss 0.5524 (0.5102) | acc 0.981 (0.984) | IoU 0.572 (0.640) | [7 days, 7:37:33]
Lr: 9.637e-03 | Update: 2.351e-04 mean,3.336e-04 std | Epoch: [4][710/1047] | Time 2.147 (2.198) | Data 1.071 (1.050) | Loss 0.5276 (0.5103) | acc 0.984 (0.984) | IoU 0.623 (0.640) | [7 days, 7:35:33]
Lr: 9.636e-03 | Update: 2.879e-04 mean,3.937e-04 std | Epoch: [4][720/1047] | Time 1.822 (2.197) | Data 0.672 (1.049) | Loss 0.5667 (0.5099) | acc 0.991 (0.985) | IoU 0.534 (0.640) | [7 days, 7:33:31]
Lr: 9.635e-03 | Update: 3.896e-04 mean,1.304e-03 std | Epoch: [4][730/1047] | Time 1.963 (2.197) | Data 0.782 (1.049) | Loss 0.4252 (0.5096) | acc 0.997 (0.985) | IoU 0.679 (0.640) | [7 days, 7:32:04]
Lr: 9.634e-03 | Update: 3.090e-04 mean,8.384e-04 std | Epoch: [4][740/1047] | Time 2.277 (2.197) | Data 1.147 (1.049) | Loss 0.4079 (0.5094) | acc 0.997 (0.985) | IoU 0.866 (0.641) | [7 days, 7:30:43]
Lr: 9.633e-03 | Update: 1.096e-03 mean,9.060e-03 std | Epoch: [4][750/1047] | Time 2.030 (2.197) | Data 0.997 (1.049) | Loss 0.5042 (0.5092) | acc 0.988 (0.985) | IoU 0.667 (0.641) | [7 days, 7:28:18]
Lr: 9.632e-03 | Update: 3.396e-04 mean,1.695e-03 std | Epoch: [4][760/1047] | Time 1.967 (2.196) | Data 0.733 (1.048) | Loss 0.4182 (0.5090) | acc 0.994 (0.985) | IoU 0.625 (0.641) | [7 days, 7:26:20]
Lr: 9.632e-03 | Update: 4.849e-04 mean,1.877e-03 std | Epoch: [4][770/1047] | Time 2.093 (2.196) | Data 0.926 (1.048) | Loss 0.6426 (0.5093) | acc 0.992 (0.985) | IoU 0.589 (0.642) | [7 days, 7:24:27]
Lr: 9.631e-03 | Update: 3.503e-04 mean,6.121e-04 std | Epoch: [4][780/1047] | Time 2.047 (2.196) | Data 0.708 (1.048) | Loss 0.5450 (0.5089) | acc 0.986 (0.985) | IoU 0.530 (0.642) | [7 days, 7:22:37]
Lr: 9.630e-03 | Update: 2.624e-04 mean,1.095e-03 std | Epoch: [4][790/1047] | Time 2.242 (2.196) | Data 0.937 (1.047) | Loss 0.4608 (0.5092) | acc 0.990 (0.985) | IoU 0.626 (0.642) | [7 days, 7:21:01]
Lr: 9.629e-03 | Update: 3.205e-04 mean,1.772e-03 std | Epoch: [4][800/1047] | Time 2.135 (2.195) | Data 1.002 (1.047) | Loss 0.3903 (0.5089) | acc 0.995 (0.985) | IoU 0.659 (0.641) | [7 days, 7:18:33]
Lr: 9.628e-03 | Update: 2.979e-04 mean,7.278e-04 std | Epoch: [4][810/1047] | Time 2.184 (2.195) | Data 1.138 (1.047) | Loss 0.4513 (0.5081) | acc 0.994 (0.985) | IoU 0.618 (0.642) | [7 days, 7:16:20]
Lr: 9.627e-03 | Update: 1.366e-04 mean,2.531e-04 std | Epoch: [4][820/1047] | Time 2.265 (2.195) | Data 0.999 (1.046) | Loss 0.2740 (0.5076) | acc 0.999 (0.985) | IoU 0.878 (0.643) | [7 days, 7:14:25]
Lr: 9.626e-03 | Update: 4.809e-04 mean,1.246e-03 std | Epoch: [4][830/1047] | Time 2.107 (2.195) | Data 1.001 (1.046) | Loss 0.3954 (0.5075) | acc 0.997 (0.985) | IoU 0.867 (0.643) | [7 days, 7:12:48]
Lr: 9.625e-03 | Update: 1.631e-04 mean,2.636e-04 std | Epoch: [4][840/1047] | Time 2.318 (2.194) | Data 1.248 (1.046) | Loss 0.6162 (0.5077) | acc 0.960 (0.985) | IoU 0.484 (0.643) | [7 days, 7:10:00]
Lr: 9.624e-03 | Update: 3.290e-04 mean,6.160e-04 std | Epoch: [4][850/1047] | Time 2.006 (2.194) | Data 0.894 (1.046) | Loss 0.4959 (0.5079) | acc 0.995 (0.985) | IoU 0.707 (0.642) | [7 days, 7:08:07]
Lr: 9.623e-03 | Update: 4.540e-04 mean,8.986e-04 std | Epoch: [4][860/1047] | Time 2.237 (2.194) | Data 0.869 (1.045) | Loss 0.3868 (0.5076) | acc 0.996 (0.985) | IoU 0.814 (0.643) | [7 days, 7:06:45]
Lr: 9.622e-03 | Update: 2.762e-04 mean,5.601e-04 std | Epoch: [4][870/1047] | Time 2.235 (2.193) | Data 1.230 (1.045) | Loss 0.3861 (0.5065) | acc 0.995 (0.985) | IoU 0.706 (0.643) | [7 days, 7:05:23]
Lr: 9.621e-03 | Update: 3.662e-04 mean,6.076e-04 std | Epoch: [4][880/1047] | Time 2.299 (2.193) | Data 1.177 (1.045) | Loss 0.4688 (0.5058) | acc 0.989 (0.986) | IoU 0.571 (0.644) | [7 days, 7:03:08]
Lr: 9.620e-03 | Update: 3.963e-04 mean,5.627e-04 std | Epoch: [4][890/1047] | Time 2.093 (2.193) | Data 1.028 (1.045) | Loss 0.5378 (0.5063) | acc 0.972 (0.986) | IoU 0.534 (0.643) | [7 days, 7:01:05]
Lr: 9.620e-03 | Update: 7.183e-04 mean,3.608e-03 std | Epoch: [4][900/1047] | Time 1.948 (2.193) | Data 0.806 (1.044) | Loss 0.6768 (0.5067) | acc 0.982 (0.985) | IoU 0.542 (0.643) | [7 days, 6:59:16]
Lr: 9.619e-03 | Update: 2.070e-04 mean,3.327e-04 std | Epoch: [4][910/1047] | Time 2.019 (2.192) | Data 0.764 (1.044) | Loss 0.3758 (0.5066) | acc 0.994 (0.986) | IoU 0.652 (0.643) | [7 days, 6:57:34]
Lr: 9.618e-03 | Update: 5.240e-04 mean,2.144e-03 std | Epoch: [4][920/1047] | Time 2.272 (2.192) | Data 1.212 (1.044) | Loss 0.4448 (0.5067) | acc 0.988 (0.986) | IoU 0.615 (0.643) | [7 days, 6:56:24]
Lr: 9.617e-03 | Update: 9.009e-04 mean,9.457e-03 std | Epoch: [4][930/1047] | Time 2.128 (2.192) | Data 1.088 (1.044) | Loss 0.5084 (0.5061) | acc 0.993 (0.986) | IoU 0.688 (0.643) | [7 days, 6:54:07]
Lr: 9.616e-03 | Update: 2.608e-04 mean,6.322e-04 std | Epoch: [4][940/1047] | Time 2.092 (2.192) | Data 0.835 (1.043) | Loss 0.3983 (0.5057) | acc 0.992 (0.986) | IoU 0.639 (0.643) | [7 days, 6:52:23]
Lr: 9.615e-03 | Update: 1.123e-03 mean,4.338e-03 std | Epoch: [4][950/1047] | Time 2.013 (2.192) | Data 0.903 (1.043) | Loss 0.9113 (0.5059) | acc 0.978 (0.986) | IoU 0.540 (0.643) | [7 days, 6:50:53]
Lr: 9.614e-03 | Update: 2.835e-04 mean,1.598e-03 std | Epoch: [4][960/1047] | Time 2.166 (2.191) | Data 1.105 (1.043) | Loss 0.4732 (0.5061) | acc 0.990 (0.986) | IoU 0.559 (0.643) | [7 days, 6:48:58]
Lr: 9.613e-03 | Update: 3.397e-04 mean,1.105e-03 std | Epoch: [4][970/1047] | Time 2.321 (2.191) | Data 1.061 (1.043) | Loss 0.4939 (0.5052) | acc 0.996 (0.986) | IoU 0.691 (0.643) | [7 days, 6:48:08]
Lr: 9.612e-03 | Update: 3.893e-04 mean,1.787e-03 std | Epoch: [4][980/1047] | Time 2.236 (2.191) | Data 1.128 (1.042) | Loss 0.3857 (0.5040) | acc 0.997 (0.986) | IoU 0.917 (0.645) | [7 days, 6:46:18]
Lr: 9.611e-03 | Update: 2.880e-04 mean,4.466e-04 std | Epoch: [4][990/1047] | Time 1.661 (2.191) | Data 0.615 (1.042) | Loss 0.3574 (0.5033) | acc 0.991 (0.986) | IoU 0.706 (0.645) | [7 days, 6:44:20]
Lr: 9.610e-03 | Update: 2.977e-04 mean,6.836e-04 std | Epoch: [4][1000/1047] | Time 1.910 (2.191) | Data 0.734 (1.042) | Loss 0.4845 (0.5031) | acc 0.988 (0.986) | IoU 0.544 (0.645) | [7 days, 6:42:46]
Lr: 9.609e-03 | Update: 5.303e-04 mean,1.439e-03 std | Epoch: [4][1010/1047] | Time 2.065 (2.190) | Data 0.896 (1.042) | Loss 0.4403 (0.5028) | acc 0.996 (0.986) | IoU 0.868 (0.646) | [7 days, 6:40:51]
Lr: 9.608e-03 | Update: 2.216e-04 mean,9.197e-04 std | Epoch: [4][1020/1047] | Time 2.084 (2.190) | Data 0.928 (1.041) | Loss 0.3274 (0.5026) | acc 0.985 (0.986) | IoU 0.634 (0.645) | [7 days, 6:39:00]
Lr: 9.608e-03 | Update: 4.243e-04 mean,8.205e-04 std | Epoch: [4][1030/1047] | Time 2.185 (2.190) | Data 0.835 (1.041) | Loss 0.4356 (0.5027) | acc 0.997 (0.986) | IoU 0.871 (0.645) | [7 days, 6:37:22]
Lr: 9.607e-03 | Update: 2.915e-04 mean,7.807e-04 std | Epoch: [4][1040/1047] | Time 2.109 (2.190) | Data 1.069 (1.041) | Loss 0.4633 (0.5033) | acc 0.990 (0.986) | IoU 0.597 (0.645) | [7 days, 6:36:02]
Best mean iou in training set so far, save model!
Validation:: 100%|████████████████████████████| 508/508 [11:49<00:00,  1.40s/it]
********************************************************************************
Validation set:
Time avg per batch 1.769
Loss avg 1.4384
Jaccard avg 0.5764
WCE avg 0.8620
Acc avg 0.691302
IoU avg 0.350770
IoU class 0 [unlabeled] = 0.000000
IoU class 1 [static] = 0.690206
IoU class 2 [moving] = 0.011333
********************************************************************************
Lr: 9.606e-03 | Update: 4.164e-04 mean,9.854e-04 std | Epoch: [5][0/1047] | Time 1.770 (2.189) | Data 0.540 (1.041) | Loss 0.5355 (0.5355) | acc 0.986 (0.986) | IoU 0.601 (0.601) | [7 days, 4:24:26]
Lr: 9.605e-03 | Update: 2.432e-04 mean,4.918e-04 std | Epoch: [5][10/1047] | Time 2.132 (2.189) | Data 1.110 (1.040) | Loss 0.6028 (0.5228) | acc 0.983 (0.987) | IoU 0.521 (0.626) | [7 days, 4:22:00]
Lr: 9.604e-03 | Update: 1.339e-04 mean,2.131e-04 std | Epoch: [5][20/1047] | Time 2.259 (2.189) | Data 1.113 (1.040) | Loss 0.3681 (0.4944) | acc 0.998 (0.990) | IoU 0.796 (0.659) | [7 days, 4:20:18]
Lr: 9.603e-03 | Update: 5.042e-04 mean,7.548e-04 std | Epoch: [5][30/1047] | Time 2.165 (2.189) | Data 0.986 (1.040) | Loss 0.4550 (0.4827) | acc 0.996 (0.992) | IoU 0.707 (0.682) | [7 days, 4:18:33]
Lr: 9.602e-03 | Update: 6.915e-04 mean,1.591e-03 std | Epoch: [5][40/1047] | Time 1.755 (2.188) | Data 0.663 (1.040) | Loss 0.6817 (0.4760) | acc 0.994 (0.992) | IoU 0.733 (0.689) | [7 days, 4:16:00]
Lr: 9.601e-03 | Update: 7.513e-04 mean,9.719e-03 std | Epoch: [5][50/1047] | Time 2.243 (2.188) | Data 1.112 (1.039) | Loss 0.2761 (0.4599) | acc 0.997 (0.993) | IoU 0.624 (0.691) | [7 days, 4:13:50]
Lr: 9.600e-03 | Update: 1.684e-04 mean,2.438e-04 std | Epoch: [5][60/1047] | Time 2.142 (2.187) | Data 0.948 (1.039) | Loss 0.3875 (0.4573) | acc 0.988 (0.992) | IoU 0.582 (0.695) | [7 days, 4:10:58]
Lr: 9.600e-03 | Update: 4.722e-04 mean,1.972e-03 std | Epoch: [5][70/1047] | Time 1.910 (2.187) | Data 0.746 (1.039) | Loss 0.4500 (0.4568) | acc 0.992 (0.992) | IoU 0.757 (0.697) | [7 days, 4:08:42]
Lr: 9.599e-03 | Update: 3.893e-04 mean,5.540e-04 std | Epoch: [5][80/1047] | Time 1.879 (2.186) | Data 0.864 (1.038) | Loss 0.4514 (0.4662) | acc 0.988 (0.991) | IoU 0.547 (0.683) | [7 days, 4:05:40]
Lr: 9.598e-03 | Update: 1.114e-04 mean,2.453e-04 std | Epoch: [5][90/1047] | Time 1.780 (2.186) | Data 0.612 (1.038) | Loss 0.2699 (0.4657) | acc 0.995 (0.991) | IoU 0.755 (0.682) | [7 days, 4:03:02]
Lr: 9.597e-03 | Update: 4.185e-04 mean,1.024e-03 std | Epoch: [5][100/1047] | Time 1.813 (2.186) | Data 0.697 (1.037) | Loss 0.4434 (0.4657) | acc 0.994 (0.991) | IoU 0.756 (0.684) | [7 days, 4:00:21]

fengluodb · 2022-09-07T06:15:40Z

I have found this problem. When I disable the softpool, the net will be train normal. My sofrpool version is 1.1.
When I debug, I found that.

MotionSeg3D/modules/SalsaNextWithMotionAttention.py

Lines 161 to 167 in dc4c95f

    
           down2c, down2b = self.resBlock3(down1c) 
        
           if self.use_attention == "MGA": 
        
               down2c = self.encoder_attention_module_MGA_tmc(down2c, RI_down2c, self.conv1x1_layer2_channel_wise, self.conv1x1_layer2_spatial) 
        
           elif self.use_attention == "Add": 
        
               down2c += RI_down2c 
        
           down3c, down3b = self.resBlock4(down2c)

donw2b will be Nan, and this will cause down3b become Nan, finally the output will become Nan.

MaxChanger · 2022-09-07T06:31:25Z

Hi, glad to hear this. I check the softpool version in my pc, it is also 1.1
Maybe you can make test to test the correctness of softpool? as provided in its readme.

$ git clone https://github.com/alexandrosstergiou/SoftPool.git
$ cd SoftPool-master/pytorch
$ make install
--- (optional) ---
$ make test

Another quick test is to use the checkpoint I provided, inference on your machine, perform eval to see if the result metric is normal, and visualize the results of MOS.

byw, softpool did not appear nan in my so many experiments.

MaxChanger · 2022-09-07T07:06:54Z

I check the value of in_vol. I don't konw whether the value is normal, but they are so small

As for this issue, the dim of in_vol is [bs, channel, h, w]. The channel numer is range dim + residual number.
That is, range dim is 5, and residual number is 8, so if your bs=8, the dim is [8, 13, 64, 2048]. I am not sure which dim you shown.

fengluodb · 2022-09-07T07:29:04Z

I check the value of in_vol. I don't konw whether the value is normal, but they are so small

As for this issue, the dim of in_vol is [bs, channel, h, w]. The channel numer is range dim + residual number. That is, range dim is 5, and residual number is 8, so if your bs=8, the dim is [8, 13, 64, 2048]. I am not sure which dim you shown.

Thank you for your reply. I think my data is ok. I try it with SalasNext and SalsaNextWithMotionAttention without softpool, they work well.

MaxChanger · 2022-09-07T07:53:52Z

Hi @fengluodb, if Iyou are very sure that it is a softpool problem, you can refer to alexandrosstergiou/SoftPool#12 and alexandrosstergiou/adaPool#2.

I have tested it on 4~5 different types of GPU servers, and I have not found this issue in nearly a hundred experiments. Thank you for your feedback!

MaxChanger · 2022-09-07T11:27:31Z

@fengluodb Can you tell me which version of python, pytorch, cuda, cudnn you are using？
As the authors of SoftPool suggest, pytorch version ≥ 1.7 may be a better choice.

fengluodb · 2022-09-07T11:40:09Z

@fengluodb Can you tell me which version of python, pytorch, cuda, cudnn you are using？ As the authors of SoftPool
suggest, pytorch version ≥ 1.7 may be a better choice.

My create conda enviroment as readme. My cuda version is 11.4. I don't know the cudnn version. As for softpool, the test output is following.

output log

--- Initial checks for forward ---

Checking 1D CPU ...
PASSED
Checking 1D GPU ...
PASSED
Checking 1D CPU-GPU output similarities ...
PASSED

Checking 2D CPU ...
PASSED
Checking 2D GPU ...
PASSED
Checking 2D CPU-GPU output similarities ...
PASSED

Checking 3D CPU ...
PASSED
Checking 3D GPU ...
PASSED
Checking 3D CPU-GPU output similarities ...
PASSED

--- Initial checks for backward ---

Checking 1D CPU ...
PASSED
Checking 1D GPU ...
PASSED
Checking 1D grad similarities ...
PASSED

Checking 2D CPU ...
PASSED
Checking 2D GPU ...
PASSED
Checking 2D grad similarities ...
PASSED

Checking 3D CPU ...
PASSED
Checking 3D GPU ...
PASSED
Checking 3D grad similarities ...
PASSED

TESTS COMPLETED

--- Profiling checks ---
SoftPool1d (CPU) [foward]

               Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg    # of Calls

   aten::avg_pool2d        18.39%       8.211ms        20.36%       9.093ms      45.464us           200  
          aten::exp        11.30%       5.045ms        18.21%       8.130ms      40.652us           200  
         aten::mul_        10.59%       4.727ms        16.79%       7.497ms      37.484us           200  
        aten::clamp         8.98%       4.011ms        15.33%       6.846ms      17.115us           400  
          aten::mul         8.53%       3.810ms         9.08%       4.055ms      40.550us           100  
         aten::div_         7.80%       3.483ms        16.51%       7.370ms      36.852us           200  
        aten::copy_         5.27%       2.354ms         5.27%       2.354ms       7.847us           300  
    aten::unsqueeze         4.52%       2.018ms         5.32%       2.377ms      11.887us           200  
          aten::sum         4.44%       1.985ms         5.73%       2.560ms      25.603us           100  
      aten::squeeze         3.93%       1.755ms         4.46%       1.993ms       9.963us           200  
   aten::avg_pool1d         2.98%       1.331ms        33.13%      14.793ms      73.967us           200  
        aten::empty         2.61%       1.164ms         2.61%       1.164ms       1.663us           700  
      aten::resize_         2.56%       1.143ms         2.56%       1.143ms       2.286us           500  
           aten::to         2.09%     931.874us         6.20%       2.770ms      13.848us           200  
        aten::clone         1.90%     846.345us         4.58%       2.043ms      20.432us           100  
aten::empty_strided         1.52%     680.656us         1.52%     680.656us       2.269us           300  
   aten::as_strided         1.34%     597.453us         1.34%     597.453us       1.494us           400  
        aten::fill_         0.80%     357.872us         0.80%     357.872us       3.579us           100  
   aten::contiguous         0.46%     204.474us         0.46%     204.474us       1.022us           200

Self CPU time total: 44.654ms

SoftPool1d (CPU) [forward + backward]

                           Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg    # of Calls

                    aten::copy_        12.79%      21.061ms        12.84%      21.142ms       9.610us          2200  
                      aten::mul        12.30%      20.248ms        18.65%      30.705ms      27.914us          1100  
               aten::avg_pool2d         6.20%      10.205ms         6.93%      11.419ms      57.096us           200  
                    aten::empty         4.14%       6.821ms         4.14%       6.821ms       1.895us          3600  
                      aten::div         3.75%       6.175ms         4.04%       6.651ms      22.171us           300  
                     aten::mul_         3.68%       6.061ms         5.70%       9.389ms      46.943us           200  
      aten::avg_pool2d_backward         3.49%       5.750ms         6.51%      10.718ms      53.588us           200  
                      aten::exp         3.35%       5.524ms         6.25%      10.293ms      51.466us           200  
                      aten::sum         3.28%       5.397ms         4.21%       6.936ms      23.122us           300  
                    aten::clamp         3.26%       5.374ms         5.33%       8.772ms      21.929us           400  
                     aten::div_         2.96%       4.871ms         6.19%      10.187ms      50.935us           200  
                       aten::le         2.87%       4.720ms        10.86%      17.876ms      44.690us           400  
            aten::empty_strided         2.79%       4.588ms         2.79%       4.588ms       3.277us          1400  
                  aten::squeeze         2.72%       4.474ms         3.12%       5.141ms      10.281us           500  
                       aten::to         2.71%       4.456ms        12.19%      20.077ms      20.077us          1000  
               aten::as_strided         2.42%       3.980ms         2.42%       3.980ms       2.653us          1500  
                    aten::fill_         2.03%       3.340ms         2.03%       3.340ms       3.712us           900  
                       aten::ge         1.91%       3.138ms         5.87%       9.673ms      24.184us           400  
                    aten::clone         1.72%       2.834ms         4.35%       7.168ms      17.921us           400  
    torch::autograd::CopySlices         1.69%       2.791ms        22.93%      37.753ms     125.843us           300  
                aten::unsqueeze         1.66%       2.732ms         1.94%       3.193ms      15.965us           200  
                  aten::resize_         1.63%       2.678ms         1.63%       2.678ms       2.231us          1200  
              AsStridedBackward         1.42%       2.343ms         5.93%       9.762ms      48.812us           200  
      aten::_local_scalar_dense         1.33%       2.182ms         1.33%       2.182ms       5.455us           400  
                      aten::add         1.28%       2.113ms         1.49%       2.447ms      12.233us           200  
                  ClampBackward         1.28%       2.103ms        17.15%      28.236ms     141.181us           200  
                   MulBackward0         1.19%       1.967ms        12.55%      20.662ms      68.873us           300  
               aten::avg_pool1d         1.12%       1.849ms        11.70%      19.270ms      96.351us           200  
                    aten::zero_         0.93%       1.526ms         2.40%       3.956ms       6.593us           600  
                   DivBackward0         0.92%       1.511ms         7.06%      11.623ms     116.228us           100  
               aten::empty_like         0.89%       1.460ms         1.48%       2.445ms       4.890us           500  
                      aten::neg         0.80%       1.318ms         2.08%       3.429ms      17.144us           200  
                     aten::add_         0.67%       1.109ms         0.67%       1.109ms      11.205us            99  
                    aten::zeros         0.64%       1.054ms         2.09%       3.439ms      17.193us           200  
                     aten::item         0.53%     872.504us         1.85%       3.055ms       7.636us           400  
                     aten::conj         0.46%     762.023us         0.46%     762.023us       1.270us           600  
              AvgPool2DBackward         0.46%     759.096us         6.97%      11.477ms      57.384us           200  
               aten::zeros_like         0.46%     754.499us         1.98%       3.257ms      16.285us           200  
                   aten::expand         0.39%     650.275us         0.50%     824.254us       8.243us           100  
                  aten::type_as         0.30%     493.399us         3.06%       5.046ms      25.232us           200  
torch::autograd::AccumulateGrad         0.29%     479.494us         0.97%       1.592ms      15.921us           100  
               aten::contiguous         0.29%     478.323us         0.29%     478.323us       1.196us           400  
               aten::resize_as_         0.28%     453.311us         0.36%     597.930us       2.990us           200  
             UnsqueezeBackward0         0.25%     418.392us         1.17%       1.924ms       9.622us           200  
                    ExpBackward         0.20%     327.909us         1.53%       2.519ms      25.193us           100  
                   SumBackward1         0.15%     245.377us         0.65%       1.070ms      10.696us           100  
     torch::autograd::GraphRoot         0.09%     140.844us         0.09%     140.844us       1.408us           100  
                   aten::stride         0.05%      80.645us         0.05%      80.645us       0.806us           100  
                   aten::detach         0.00%       2.040us         0.00%       3.248us       3.248us             1  
                         detach         0.00%       1.208us         0.00%       1.208us       1.208us             1

Self CPU time total: 164.672ms

SoftPool1d (CUDA-inplace) [foward]

               Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg     Self CUDA   Self CUDA %    CUDA total  CUDA time avg    # of Calls

          aten::exp        34.00%      17.998ms        64.82%      34.312ms     171.562us      17.053ms        29.14%      33.194ms     165.971us           200  
        aten::empty         1.86%     983.295us         1.86%     983.295us       1.405us       0.000us         0.00%       0.000us       0.000us           700  
      aten::resize_         1.97%       1.041ms         1.97%       1.041ms       2.083us       0.000us         0.00%       0.000us       0.000us           500  
          aten::sum         4.41%       2.335ms         4.79%       2.534ms      25.344us       1.140ms         1.95%       1.140ms      11.398us           100  
        aten::clamp        11.76%       6.227ms        18.47%       9.776ms      24.439us       2.547ms         4.35%       3.372ms       8.430us           400  
          aten::mul         5.22%       2.765ms         5.58%       2.953ms      29.530us       3.209ms         5.48%       3.209ms      32.092us           100  
   aten::avg_pool1d         3.61%       1.909ms        20.77%      10.998ms      54.988us       3.139ms         5.36%      10.332ms      51.661us           200  
    aten::unsqueeze         2.16%       1.142ms         2.58%       1.366ms       6.830us       0.000us         0.00%       0.000us       0.000us           200  
   aten::as_strided         0.68%     360.524us         0.68%     360.524us       0.901us       0.000us         0.00%       0.000us       0.000us           400  
   aten::avg_pool2d         9.81%       5.195ms        12.44%       6.588ms      32.939us       6.765ms        11.56%       7.194ms      35.968us           200  
   aten::contiguous         1.42%     751.756us         1.42%     751.756us       3.759us     429.047us         0.73%     429.047us       2.145us           200  
      aten::squeeze         1.89%     998.304us         2.14%       1.135ms       5.675us       0.000us         0.00%       0.000us       0.000us           200  
         aten::mul_         8.99%       4.757ms         8.99%       4.757ms      23.784us      10.691ms        18.27%      10.691ms      53.453us           200  
         aten::div_         5.48%       2.902ms        14.30%       7.570ms      37.850us       7.819ms        13.36%      16.381ms      81.905us           200  
        aten::clone         3.32%       1.757ms         6.75%       3.573ms      35.730us     831.475us         1.42%       5.733ms      57.334us           100  
aten::empty_strided         0.61%     323.094us         0.61%     323.094us       3.231us       0.000us         0.00%       0.000us       0.000us           100  
        aten::copy_         2.82%       1.493ms         2.82%       1.493ms      14.930us       4.902ms         8.38%       4.902ms      49.019us           100

Self CPU time total: 52.938ms
CUDA time total: 58.524ms

SoftPool1d (CUDA-inplace) [forward + backward]

                           Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg     Self CUDA   Self CUDA %    CUDA total  CUDA time avg    # of Calls

                      aten::exp         3.62%      10.571ms         5.81%      16.960ms      84.800us      11.089ms         3.81%      15.853ms      79.267us           200  
                    aten::empty         3.33%       9.728ms         3.33%       9.728ms       3.040us       0.000us         0.00%       0.000us       0.000us          3200  
                  aten::resize_         1.53%       4.460ms         1.53%       4.460ms       3.716us       0.000us         0.00%       0.000us       0.000us          1200  
                      aten::sum         1.87%       5.466ms         2.08%       6.075ms      60.749us       5.920ms         2.03%       5.920ms      59.198us           100  
                    aten::clamp         4.15%      12.114ms         6.41%      18.716ms      46.790us      13.094ms         4.49%      18.192ms      45.481us           400  
                      aten::mul        10.57%      30.883ms        11.67%      34.098ms      30.998us      32.913ms        11.30%      32.913ms      29.921us          1100  
               aten::avg_pool1d         1.23%       3.581ms         7.24%      21.135ms     105.673us       8.517ms         2.92%      26.575ms     132.874us           200  
                aten::unsqueeze         0.80%       2.337ms         0.98%       2.853ms      14.263us       0.000us         0.00%       0.000us       0.000us           200  
               aten::as_strided         1.43%       4.166ms         1.43%       4.166ms       2.778us       0.000us         0.00%       0.000us       0.000us          1500  
               aten::avg_pool2d         3.40%       9.924ms         4.24%      12.395ms      61.973us      14.699ms         5.04%      18.058ms      90.291us           200  
               aten::contiguous         1.00%       2.921ms         1.00%       2.921ms       4.868us       5.040ms         1.73%       5.040ms       8.400us           600  
                  aten::squeeze         1.29%       3.780ms         1.53%       4.469ms       8.937us       0.000us         0.00%       0.000us       0.000us           500  
                     aten::mul_         3.10%       9.068ms         3.10%       9.068ms      45.338us       8.755ms         3.00%       8.755ms      43.773us           200  
                     aten::div_         1.87%       5.474ms         4.91%      14.328ms      71.639us       5.283ms         1.81%      13.915ms      69.575us           200  
                    aten::clone         3.57%      10.421ms         6.85%      20.009ms      50.022us      12.233ms         4.20%      21.537ms      53.844us           400  
            aten::empty_strided         1.29%       3.776ms         1.29%       3.776ms       6.294us       0.000us         0.00%       0.000us       0.000us           600  
                    aten::copy_        12.93%      37.761ms        12.93%      37.761ms      26.972us      39.360ms        13.51%      39.360ms      28.114us          1400  
     torch::autograd::GraphRoot         0.22%     640.713us         0.22%     640.713us       6.407us     587.597us         0.20%     587.597us       5.876us           100  
                  ClampBackward         2.63%       7.679ms        22.83%      66.698ms     333.492us       7.093ms         2.43%      65.204ms     326.022us           200  
                       aten::le         5.10%      14.911ms        10.99%      32.109ms      80.273us      16.411ms         5.63%      32.732ms      81.830us           400  
                     aten::item         1.28%       3.732ms         1.94%       5.661ms      14.152us       3.703ms         1.27%       5.567ms      13.918us           400  
      aten::_local_scalar_dense         0.66%       1.929ms         0.66%       1.929ms       4.822us       1.864ms         0.64%       1.864ms       4.661us           400  
                       aten::ge         3.08%       9.007ms         6.88%      20.094ms      50.234us       9.433ms         3.24%      19.435ms      48.587us           400  
                  aten::type_as         0.68%       1.980ms         5.52%      16.121ms      80.604us       1.917ms         0.66%      16.036ms      80.180us           200  
                       aten::to         0.82%       2.399ms         4.84%      14.141ms      70.704us       3.527ms         1.21%      14.119ms      70.594us           200  
              AsStridedBackward         1.48%       4.329ms         9.45%      27.611ms     138.056us       6.552ms         2.25%      26.728ms     133.641us           200  
                    aten::zeros         0.77%       2.245ms         5.24%      15.304ms      76.520us       3.077ms         1.06%      14.960ms      74.798us           200  
                    aten::zero_         2.31%       6.755ms         5.59%      16.325ms      40.812us       6.262ms         2.15%      15.911ms      39.778us           400  
                    aten::fill_         3.28%       9.570ms         3.28%       9.570ms      23.926us       9.649ms         3.31%       9.649ms      24.123us           400  
    torch::autograd::CopySlices         2.99%       8.723ms        23.90%      69.815ms     232.716us      11.281ms         3.87%      69.749ms     232.497us           300  
               aten::empty_like         0.47%       1.364ms         1.01%       2.943ms       5.885us       0.000us         0.00%       0.000us       0.000us           500  
                   DivBackward0         1.52%       4.428ms         6.90%      20.159ms     201.590us       4.296ms         1.47%      19.448ms     194.477us           100  
                      aten::div         2.54%       7.410ms         2.81%       8.211ms      27.370us       7.934ms         2.72%       7.934ms      26.446us           300  
                     aten::conj         1.44%       4.219ms         1.44%       4.219ms       7.031us       4.018ms         1.38%       4.018ms       6.697us           600  
                      aten::neg         1.21%       3.536ms         1.98%       5.775ms      28.874us       3.828ms         1.31%       5.554ms      27.771us           200  
                   MulBackward0         2.00%       5.837ms         7.37%      21.519ms      71.731us       5.770ms         1.98%      20.899ms      69.663us           300  
              AvgPool2DBackward         0.78%       2.277ms         8.55%      24.961ms     124.803us       2.173ms         0.75%      24.384ms     121.918us           200  
      aten::avg_pool2d_backward         3.74%      10.915ms         7.77%      22.684ms     113.418us      10.814ms         3.71%      22.211ms     111.053us           200  
               aten::zeros_like         1.12%       3.277ms         2.96%       8.637ms      43.186us       4.374ms         1.50%       8.403ms      42.014us           200  
               aten::resize_as_         0.41%       1.194ms         0.47%       1.362ms       6.810us       1.313ms         0.45%       1.313ms       6.564us           200  
             UnsqueezeBackward0         0.41%       1.188ms         0.88%       2.557ms      12.786us       2.431ms         0.83%       2.431ms      12.155us           200  
                   SumBackward1         0.32%     945.430us         0.60%       1.742ms      17.420us       1.707ms         0.59%       1.707ms      17.069us           100  
                   aten::expand         0.21%     615.382us         0.27%     796.571us       7.966us       0.000us         0.00%       0.000us       0.000us           100  
                    ExpBackward         0.35%       1.034ms         1.31%       3.824ms      38.235us       1.014ms         0.35%       3.751ms      37.509us           100  
torch::autograd::AccumulateGrad         0.43%       1.267ms         1.21%       3.534ms      35.338us       1.258ms         0.43%       3.458ms      34.581us           100  
                   aten::detach         0.00%       9.989us         0.00%      13.758us      13.758us       4.098us         0.00%       6.145us       6.145us             1  
                         detach         0.00%       3.769us         0.00%       3.769us       3.769us       2.047us         0.00%       2.047us       2.047us             1  
                     aten::add_         0.77%       2.253ms         0.77%       2.253ms      22.760us       2.193ms         0.75%       2.193ms      22.156us            99

Self CPU time total: 292.102ms
CUDA time total: 291.388ms

SoftPool1d (CUDA) [foward]

                     Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg     Self CUDA   Self CUDA %    CUDA total  CUDA time avg    # of Calls

          CUDA_SOFTPOOL1d         4.75%       3.755ms        10.88%       8.592ms      85.924us       3.196ms         4.12%      68.318ms     683.182us           100  
          aten::new_zeros         1.11%     873.973us         6.12%       4.837ms      48.372us     802.959us         1.03%      65.122ms     651.221us           100  
              aten::zeros         1.14%     899.206us         5.02%       3.963ms      39.632us       1.275ms         1.64%      64.319ms     643.191us           100  
              aten::empty         0.80%     632.947us         0.80%     632.947us       2.110us       0.000us         0.00%       0.000us       0.000us           300  
              aten::zero_         1.92%       1.518ms         3.36%       2.658ms      26.585us       1.388ms         1.79%      63.044ms     630.438us           100  
              aten::fill_         1.44%       1.140ms         1.44%       1.140ms      11.401us      61.656ms        79.39%      61.656ms     616.556us           100  
              aten::isnan         1.74%       1.372ms         5.12%       4.041ms      40.413us       1.202ms         1.55%       3.366ms      33.659us           100  
                 aten::ne         2.92%       2.305ms         5.47%       4.320ms      21.600us       2.164ms         2.79%       3.514ms      17.572us           200  
            aten::resize_         0.51%     402.674us         0.51%     402.674us       2.013us       0.000us         0.00%       0.000us       0.000us           200  
                aten::any         2.97%       2.350ms         3.45%       2.723ms      27.233us       2.186ms         2.82%       2.186ms      21.863us           100  
         aten::as_strided         0.14%     107.817us         0.14%     107.817us       1.078us       0.000us         0.00%       0.000us       0.000us           100  
         aten::is_nonzero         0.92%     728.025us        80.56%      63.649ms     636.492us     664.097us         0.86%       3.792ms      37.925us           100  
               aten::item         0.92%     724.141us        79.64%      62.921ms     629.211us     872.494us         1.12%       3.128ms      31.284us           100  
aten::_local_scalar_dense        78.72%      62.197ms        78.72%      62.197ms     621.970us       2.256ms         2.90%       2.256ms      22.559us           100

Self CPU time total: 79.006ms
CUDA time total: 77.663ms

SoftPool1d (CUDA) [forward + backward]

                           Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg     Self CUDA   Self CUDA %    CUDA total  CUDA time avg    # of Calls

                CUDA_SOFTPOOL1d         7.67%       6.218ms        16.94%      13.737ms     137.371us       6.167ms         7.64%      13.630ms     136.301us           100  
                aten::new_zeros         1.58%       1.284ms         9.27%       7.519ms      75.188us       1.229ms         1.52%       7.463ms      74.631us           100  
                    aten::zeros         1.61%       1.302ms         7.69%       6.235ms      62.350us       2.045ms         2.53%       6.234ms      62.343us           100  
                    aten::empty         2.10%       1.700ms         2.10%       1.700ms       3.401us       0.000us         0.00%       0.000us       0.000us           500  
                    aten::zero_         4.21%       3.414ms         8.54%       6.929ms      34.643us       3.245ms         4.02%       6.931ms      34.653us           200  
                    aten::fill_         5.82%       4.717ms         5.82%       4.717ms      15.724us       4.943ms         6.13%       4.943ms      16.476us           300  
                    aten::isnan         4.97%       4.027ms        14.18%      11.498ms      57.491us       3.988ms         4.94%      11.434ms      57.170us           200  
                       aten::ne         7.80%       6.325ms        14.55%      11.799ms      29.497us       7.446ms         9.23%      11.926ms      29.814us           400  
                  aten::resize_         1.41%       1.141ms         1.41%       1.141ms       2.851us       0.000us         0.00%       0.000us       0.000us           400  
                      aten::any         4.25%       3.449ms         5.03%       4.076ms      40.760us       4.058ms         5.03%       4.058ms      40.582us           100  
               aten::as_strided         0.79%     643.204us         0.79%     643.204us       1.072us       0.000us         0.00%       0.000us       0.000us           600  
               aten::is_nonzero         1.25%       1.011ms         7.19%       5.833ms      58.334us     983.131us         1.22%       5.728ms      57.283us           100  
                     aten::item         1.70%       1.376ms         5.95%       4.823ms      48.228us       1.345ms         1.67%       4.745ms      47.452us           100  
      aten::_local_scalar_dense         4.25%       3.447ms         4.25%       3.447ms      34.473us       3.400ms         4.21%       3.400ms      34.004us           100  
     torch::autograd::GraphRoot         0.74%     601.811us         0.74%     601.811us       6.018us     536.910us         0.67%     536.910us       5.369us           100  
        CUDA_SOFTPOOL1dBackward        13.52%      10.965ms        57.95%      46.994ms     469.940us      10.928ms        13.54%      46.901ms     469.007us           100  
               aten::zeros_like         1.54%       1.251ms         6.81%       5.526ms      55.255us       2.795ms         3.46%       5.536ms      55.364us           100  
               aten::empty_like         1.12%     909.715us         1.91%       1.551ms      15.507us       0.000us         0.00%       0.000us       0.000us           100  
            aten::empty_strided         0.79%     640.962us         0.79%     640.962us       6.410us       0.000us         0.00%       0.000us       0.000us           100  
            aten::scalar_tensor         1.48%       1.201ms         3.45%       2.799ms      27.985us       1.517ms         1.88%       2.774ms      27.740us           100  
                       aten::to         0.64%     518.175us         0.64%     518.175us       5.182us     509.974us         0.63%     509.974us       5.100us           100  
               aten::index_put_         2.38%       1.928ms        26.65%      21.610ms     216.099us       1.959ms         2.43%      21.619ms     216.186us           100  
         aten::_index_put_impl_         6.50%       5.274ms        24.27%      19.682ms     196.819us       5.351ms         6.63%      19.660ms     196.597us           100  
                  aten::nonzero         9.13%       7.403ms        12.58%      10.198ms     101.984us       8.217ms        10.18%      10.198ms     101.980us           100  
               aten::contiguous         0.60%     483.090us         0.60%     483.090us       4.831us     452.509us         0.56%     452.509us       4.525us           100  
                        aten::t         0.25%     203.003us         0.76%     614.080us       6.141us       0.000us         0.00%       0.000us       0.000us           100  
                aten::transpose         0.31%     250.354us         0.51%     411.077us       4.111us       0.000us         0.00%       0.000us       0.000us           100  
                     aten::set_         1.93%       1.566ms         1.93%       1.566ms      15.661us       1.529ms         1.89%       1.529ms      15.289us           100  
                   aten::select         2.11%       1.715ms         2.39%       1.934ms       6.448us       2.016ms         2.50%       2.016ms       6.721us           300  
                   aten::stride         0.17%     139.151us         0.17%     139.151us       0.464us       0.000us         0.00%       0.000us       0.000us           300  
                  aten::reshape         2.01%       1.629ms         2.54%       2.060ms       6.866us       2.094ms         2.60%       2.094ms       6.980us           300  
                     aten::view         0.53%     430.451us         0.53%     430.451us       1.435us       0.000us         0.00%       0.000us       0.000us           300  
torch::autograd::AccumulateGrad         1.53%       1.242ms         4.84%       3.925ms      39.247us       1.259ms         1.56%       3.927ms      39.271us           100  
                   aten::detach         0.02%      16.176us         0.03%      25.531us      25.531us      17.408us         0.02%      26.624us      26.624us             1  
                         detach         0.01%       9.355us         0.01%       9.355us       9.355us       9.216us         0.01%       9.216us       9.216us             1  
                     aten::add_         3.28%       2.657ms         3.28%       2.657ms      26.838us       2.642ms         3.27%       2.642ms      26.686us            99

Self CPU time total: 81.088ms
CUDA time total: 80.682ms

SoftPool2d (CPU) [foward]

               Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg    # of Calls

   aten::avg_pool2d        44.88%     130.344ms        45.25%     131.425ms     657.124us           200  
          aten::mul        14.90%      43.282ms        15.21%      44.173ms     441.728us           100  
          aten::exp        11.71%      34.011ms        22.93%      66.583ms     332.915us           200  
          aten::sum        11.31%      32.837ms        12.74%      37.006ms     370.064us           100  
        aten::clamp         5.33%      15.481ms        10.10%      29.335ms      73.337us           400  
         aten::mul_         4.48%      13.010ms         5.39%      15.642ms      78.210us           200  
        aten::copy_         2.22%       6.436ms         2.22%       6.436ms      21.453us           300  
         aten::div_         1.68%       4.883ms         5.19%      15.064ms      75.320us           200  
        aten::fill_         1.30%       3.777ms         1.30%       3.777ms      37.767us           100  
        aten::empty         0.72%       2.098ms         0.72%       2.098ms       2.998us           700  
      aten::resize_         0.45%       1.317ms         0.45%       1.317ms       2.633us           500  
           aten::to         0.37%       1.063ms         0.91%       2.632ms      13.162us           200  
        aten::clone         0.30%     864.535us         2.23%       6.490ms      64.900us           100  
aten::empty_strided         0.26%     758.530us         0.26%     758.530us       2.528us           300  
   aten::contiguous         0.09%     257.936us         0.09%     257.936us       1.290us           200

Self CPU time total: 290.420ms

SoftPool2d (CPU) [forward + backward]

                           Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg    # of Calls

                      aten::mul        25.87%     204.252ms        26.48%     209.002ms     190.002us          1100  
               aten::avg_pool2d        14.87%     117.381ms        15.00%     118.385ms     591.925us           200  
                      aten::sum        11.98%      94.530ms        13.13%     103.651ms     345.504us           300  
      aten::avg_pool2d_backward        11.67%      92.083ms        16.04%     126.639ms     633.195us           200  
                    aten::fill_         4.69%      37.003ms         4.69%      37.003ms      52.861us           700  
                      aten::add         4.49%      35.413ms         4.55%      35.917ms     179.587us           200  
                     aten::add_         3.97%      31.340ms         3.97%      31.340ms     316.565us            99  
                    aten::copy_         3.39%      26.765ms         3.39%      26.765ms      24.332us          1100  
                      aten::exp         3.06%      24.124ms         5.83%      46.012ms     230.059us           200  
                      aten::div         2.30%      18.146ms         2.44%      19.298ms      64.325us           300  
                    aten::clamp         2.24%      17.677ms         4.23%      33.397ms      83.492us           400  
                       aten::le         1.81%      14.264ms         4.05%      31.967ms      79.917us           400  
                     aten::mul_         1.65%      13.008ms         1.99%      15.705ms      78.523us           200  
                      aten::neg         1.48%      11.722ms         2.92%      23.068ms     115.338us           200  
                       aten::ge         1.17%       9.218ms         2.71%      21.404ms      53.510us           400  
                    aten::empty         0.89%       7.049ms         0.89%       7.049ms       2.274us          3100  
                     aten::div_         0.58%       4.558ms         2.26%      17.838ms      89.192us           200  
                       aten::to         0.56%       4.408ms         3.20%      25.222ms      25.222us          1000  
               aten::zeros_like         0.42%       3.335ms         3.56%      28.087ms     140.433us           200  
            aten::empty_strided         0.39%       3.073ms         0.39%       3.073ms       2.794us          1100  
                  ClampBackward         0.37%       2.913ms        10.33%      81.537ms     407.687us           200  
                  aten::resize_         0.30%       2.331ms         0.30%       2.331ms       1.942us          1200  
                   MulBackward0         0.28%       2.234ms        13.35%     105.347ms     351.157us           300  
                   DivBackward0         0.23%       1.842ms         5.59%      44.125ms     441.248us           100  
                     aten::conj         0.14%       1.106ms         0.14%       1.106ms       1.843us           600  
                   aten::expand         0.14%       1.083ms         0.19%       1.505ms      15.053us           100  
                     aten::item         0.12%     984.346us         0.19%       1.482ms       3.706us           400  
                    aten::clone         0.12%     908.544us         1.26%       9.933ms      99.330us           100  
              AvgPool2DBackward         0.11%     895.169us        16.16%     127.534ms     637.670us           200  
torch::autograd::AccumulateGrad         0.11%     854.916us         4.08%      32.201ms     322.005us           100  
                  aten::type_as         0.11%     842.494us         2.24%      17.689ms      88.446us           200  
                    aten::zero_         0.09%     730.088us         3.73%      29.467ms      73.666us           400  
               aten::empty_like         0.07%     570.192us         0.13%     989.100us       4.946us           200  
      aten::_local_scalar_dense         0.06%     498.030us         0.06%     498.030us       1.245us           400  
               aten::resize_as_         0.06%     483.137us         0.08%     608.411us       3.042us           200  
               aten::contiguous         0.05%     424.397us         0.05%     424.397us       1.061us           400  
               aten::as_strided         0.05%     421.954us         0.05%     421.954us       4.220us           100  
                   SumBackward1         0.05%     408.965us         0.24%       1.914ms      19.142us           100  
                    ExpBackward         0.05%     399.722us         4.71%      37.183ms     371.833us           100  
     torch::autograd::GraphRoot         0.01%     106.732us         0.01%     106.732us       1.067us           100  
                   aten::detach         0.00%       3.334us         0.00%       5.624us       5.624us             1  
                         detach         0.00%       2.290us         0.00%       2.290us       2.290us             1

Self CPU time total: 789.392ms

SoftPool2d (CUDA-inplace) [foward]

               Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg     Self CUDA   Self CUDA %    CUDA total  CUDA time avg    # of Calls

          aten::exp         9.60%       3.238ms        14.66%       4.948ms      24.739us       5.298ms        14.05%       8.979ms      44.897us           200  
        aten::empty         3.13%       1.057ms         3.13%       1.057ms       1.510us       0.000us         0.00%       0.000us       0.000us           700  
      aten::resize_         2.96%     998.373us         2.96%     998.373us       1.997us       0.000us         0.00%       0.000us       0.000us           500  
          aten::sum         6.92%       2.336ms         7.53%       2.542ms      25.416us       1.860ms         4.93%       1.860ms      18.596us           100  
        aten::clamp        18.16%       6.126ms        27.83%       9.391ms      23.478us       6.438ms        17.07%       9.717ms      24.292us           400  
          aten::mul         7.28%       2.457ms         7.94%       2.679ms      26.791us       5.347ms        14.18%       5.347ms      53.473us           100  
   aten::avg_pool2d        17.97%       6.064ms        22.11%       7.459ms      37.296us       7.084ms        18.79%       8.110ms      40.551us           200  
   aten::contiguous         2.27%     766.255us         2.27%     766.255us       3.831us       1.026ms         2.72%       1.026ms       5.130us           200  
         aten::mul_        13.06%       4.405ms        13.06%       4.405ms      22.024us       3.286ms         8.71%       3.286ms      16.430us           200  
         aten::div_         7.92%       2.672ms        21.93%       7.398ms      36.990us       3.525ms         9.35%       9.193ms      45.967us           200  
        aten::clone         5.34%       1.803ms        10.73%       3.620ms      36.200us       2.191ms         5.81%       3.842ms      38.420us           100  
aten::empty_strided         0.93%     313.773us         0.93%     313.773us       3.138us       0.000us         0.00%       0.000us       0.000us           100  
        aten::copy_         4.46%       1.503ms         4.46%       1.503ms      15.034us       1.651ms         4.38%       1.651ms      16.507us           100

Self CPU time total: 33.740ms
CUDA time total: 37.706ms

SoftPool2d (CUDA-inplace) [forward + backward]

                           Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg     Self CUDA   Self CUDA %    CUDA total  CUDA time avg    # of Calls

                      aten::exp         5.18%       9.833ms         8.33%      15.826ms      79.130us       7.730ms         4.09%      10.076ms      50.381us           200  
                    aten::empty         4.15%       7.875ms         4.15%       7.875ms       2.916us       0.000us         0.00%       0.000us       0.000us          2700  
                  aten::resize_         2.89%       5.493ms         2.89%       5.493ms       4.578us       0.000us         0.00%       0.000us       0.000us          1200  
                      aten::sum         5.54%      10.527ms         5.94%      11.286ms     112.855us      12.912ms         6.83%      12.912ms     129.117us           100  
                    aten::clamp         5.86%      11.136ms         9.12%      17.322ms      43.304us      14.055ms         7.43%      19.616ms      49.039us           400  
                      aten::mul        14.10%      26.777ms        15.84%      30.091ms      27.355us      34.927ms        18.48%      34.927ms      31.752us          1100  
               aten::avg_pool2d         4.73%       8.980ms         5.97%      11.335ms      56.676us      10.938ms         5.79%      11.350ms      56.750us           200  
               aten::contiguous         1.43%       2.709ms         1.43%       2.709ms       4.515us       1.356ms         0.72%       1.356ms       2.260us           600  
                     aten::mul_         3.79%       7.193ms         3.79%       7.193ms      35.963us       8.552ms         4.52%       8.552ms      42.762us           200  
                     aten::div_         2.28%       4.333ms         6.57%      12.481ms      62.407us       1.535ms         0.81%       8.392ms      41.959us           200  
                    aten::clone         1.52%       2.892ms         3.31%       6.291ms      62.914us       3.127ms         1.65%       5.944ms      59.443us           100  
            aten::empty_strided         1.06%       2.015ms         1.06%       2.015ms       6.716us       0.000us         0.00%       0.000us       0.000us           300  
                    aten::copy_         4.02%       7.631ms         4.02%       7.631ms      25.436us       4.936ms         2.61%       4.936ms      16.453us           300  
     torch::autograd::GraphRoot         0.33%     620.214us         0.33%     620.214us       6.202us     216.029us         0.11%     216.029us       2.160us           100  
                  ClampBackward         4.64%       8.814ms        28.78%      54.663ms     273.313us       2.622ms         1.39%      55.556ms     277.780us           200  
                       aten::le         5.00%       9.505ms        12.45%      23.658ms      59.144us      16.807ms         8.89%      27.659ms      69.148us           400  
                     aten::item         1.76%       3.336ms         2.64%       5.018ms      12.546us       6.232ms         3.30%       9.314ms      23.286us           400  
      aten::_local_scalar_dense         0.89%       1.682ms         0.89%       1.682ms       4.206us       3.082ms         1.63%       3.082ms       7.705us           400  
                       aten::ge         4.27%       8.110ms         9.61%      18.248ms      45.621us      12.217ms         6.46%      27.512ms      68.779us           400  
                  aten::type_as         0.94%       1.784ms         5.41%      10.280ms      51.402us       3.124ms         1.65%       8.365ms      41.826us           200  
                       aten::to         1.18%       2.250ms         4.47%       8.497ms      42.483us       3.122ms         1.65%       5.241ms      26.204us           200  
                   DivBackward0         2.22%       4.222ms         9.99%      18.983ms     189.827us       1.709ms         0.90%      11.700ms     117.001us           100  
                      aten::div         3.71%       7.055ms         4.13%       7.837ms      26.123us       6.697ms         3.54%       6.697ms      22.324us           300  
                     aten::conj         1.47%       2.800ms         1.47%       2.800ms       4.667us       1.251ms         0.66%       1.251ms       2.085us           600  
                      aten::neg         1.73%       3.281ms         2.82%       5.366ms      26.828us       1.724ms         0.91%       2.595ms      12.974us           200  
                   MulBackward0         3.02%       5.731ms         9.56%      18.154ms      60.512us       2.439ms         1.29%      16.324ms      54.412us           300  
              AvgPool2DBackward         1.14%       2.173ms        10.50%      19.948ms      99.741us     881.554us         0.47%      21.519ms     107.597us           200  
      aten::avg_pool2d_backward         4.08%       7.757ms         9.36%      17.775ms      88.876us      15.876ms         8.40%      20.638ms     103.189us           200  
               aten::zeros_like         1.05%       1.997ms         3.78%       7.173ms      35.867us     838.473us         0.44%       3.396ms      16.978us           200  
               aten::empty_like         0.27%     513.069us         0.58%       1.096ms       5.482us       0.000us         0.00%       0.000us       0.000us           200  
                    aten::zero_         0.94%       1.785ms         2.15%       4.080ms      20.398us     854.051us         0.45%       2.557ms      12.785us           200  
                    aten::fill_         1.21%       2.294ms         1.21%       2.294ms      11.471us       1.703ms         0.90%       1.703ms       8.515us           200  
               aten::resize_as_         0.57%       1.087ms         0.64%       1.216ms       6.081us     421.857us         0.22%     421.857us       2.109us           200  
                   SumBackward1         0.31%     586.405us         0.73%       1.394ms      13.939us       2.484ms         1.31%       2.484ms      24.842us           100  
                   aten::expand         0.31%     579.555us         0.43%     807.540us       8.075us       0.000us         0.00%       0.000us       0.000us           100  
               aten::as_strided         0.12%     227.985us         0.12%     227.985us       2.280us       0.000us         0.00%       0.000us       0.000us           100  
                    ExpBackward         0.56%       1.065ms         2.00%       3.798ms      37.983us     449.494us         0.24%       9.924ms      99.236us           100  
torch::autograd::AccumulateGrad         0.63%       1.189ms         1.74%       3.299ms      32.990us     502.772us         0.27%       4.222ms      42.219us           100  
                   aten::detach         0.01%      12.837us         0.01%      20.257us      20.257us       4.096us         0.00%       6.144us       6.144us             1  
                         detach         0.00%       7.420us         0.00%       7.420us       7.420us       2.048us         0.00%       2.048us       2.048us             1  
                     aten::add_         1.10%       2.090ms         1.10%       2.090ms      21.113us       3.713ms         1.96%       3.713ms      37.505us            99

Self CPU time total: 189.949ms
CUDA time total: 189.042ms

SoftPool2d (CUDA) [foward]

                     Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg     Self CUDA   Self CUDA %    CUDA total  CUDA time avg    # of Calls

          CUDA_SOFTPOOL2d        19.07%       3.941ms        42.95%       8.876ms      88.761us       5.498ms        26.20%      10.451ms     104.506us           100  
          aten::new_zeros         4.41%     910.495us        23.88%       4.936ms      49.356us     884.025us         4.21%       4.953ms      49.526us           100  
              aten::zeros         4.40%     908.897us        19.48%       4.025ms      40.251us       1.362ms         6.49%       4.069ms      40.686us           100  
              aten::empty         3.22%     665.494us         3.22%     665.494us       2.218us       0.000us         0.00%       0.000us       0.000us           300  
              aten::zero_         7.59%       1.568ms        13.00%       2.686ms      26.860us       1.392ms         6.63%       2.706ms      27.061us           100  
              aten::fill_         5.41%       1.118ms         5.41%       1.118ms      11.183us       1.314ms         6.26%       1.314ms      13.139us           100  
              aten::isnan         6.96%       1.438ms        20.50%       4.237ms      42.368us       1.062ms         5.06%       3.023ms      30.229us           100  
                 aten::ne        11.75%       2.428ms        21.94%       4.534ms      22.670us       1.961ms         9.35%       3.363ms      16.815us           200  
            aten::resize_         2.01%     414.651us         2.01%     414.651us       2.073us       0.000us         0.00%       0.000us       0.000us           200  
                aten::any        14.90%       3.078ms        16.84%       3.480ms      34.804us       3.491ms        16.64%       3.491ms      34.908us           100  
         aten::as_strided         0.60%     123.640us         0.60%     123.640us       1.236us       0.000us         0.00%       0.000us       0.000us           100  
         aten::is_nonzero         3.61%     745.143us        19.71%       4.074ms      40.736us     746.173us         3.56%       4.018ms      40.182us           100  
               aten::item         3.73%     771.765us        16.11%       3.328ms      33.285us     772.022us         3.68%       3.272ms      32.721us           100  
aten::_local_scalar_dense        12.37%       2.557ms        12.37%       2.557ms      25.567us       2.500ms        11.91%       2.500ms      25.000us           100

Self CPU time total: 20.667ms
CUDA time total: 20.983ms

SoftPool2d (CUDA) [forward + backward]

                           Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg     Self CUDA   Self CUDA %    CUDA total  CUDA time avg    # of Calls

                CUDA_SOFTPOOL2d         6.60%       9.711ms        13.68%      20.140ms     201.404us      20.770ms        13.37%      27.739ms     277.390us           100  
                aten::new_zeros         1.22%       1.799ms         7.09%      10.429ms     104.289us       1.905ms         1.23%       6.969ms      69.695us           100  
                    aten::zeros         1.22%       1.796ms         5.86%       8.630ms      86.296us       2.286ms         1.47%       5.065ms      50.646us           100  
                    aten::empty         1.69%       2.488ms         1.69%       2.488ms       4.977us       0.000us         0.00%       0.000us       0.000us           500  
                    aten::zero_         3.14%       4.616ms         6.38%       9.395ms      46.977us       2.495ms         1.61%      29.036ms     145.179us           200  
                    aten::fill_         4.33%       6.376ms         4.33%       6.376ms      21.252us      30.959ms        19.93%      30.959ms     103.196us           300  
                    aten::isnan         3.30%       4.862ms        10.11%      14.879ms      74.396us       2.057ms         1.32%       7.611ms      38.056us           200  
                       aten::ne         5.69%       8.376ms        10.93%      16.083ms      40.208us       5.554ms         3.58%      10.189ms      25.472us           400  
                  aten::resize_         1.12%       1.646ms         1.12%       1.646ms       4.115us       0.000us         0.00%       0.000us       0.000us           400  
                      aten::any         3.42%       5.029ms         4.04%       5.945ms      59.447us       8.453ms         5.44%       8.453ms      84.531us           100  
               aten::as_strided         0.64%     944.993us         0.64%     944.993us       1.350us       0.000us         0.00%       0.000us       0.000us           700  
               aten::is_nonzero         0.69%       1.022ms        16.48%      24.252ms     242.525us       1.103ms         0.71%       6.906ms      69.059us           100  
                     aten::item         0.99%       1.464ms        15.78%      23.230ms     232.300us       1.118ms         0.72%       5.803ms      58.029us           100  
      aten::_local_scalar_dense        14.79%      21.766ms        14.79%      21.766ms     217.657us       4.685ms         3.02%       4.685ms      46.854us           100  
     torch::autograd::GraphRoot         0.41%     603.156us         0.41%     603.156us       6.032us     578.529us         0.37%     578.529us       5.785us           100  
        CUDA_SOFTPOOL2dBackward        10.12%      14.893ms        56.92%      83.778ms     837.783us      12.982ms         8.36%     104.140ms       1.041ms           100  
               aten::zeros_like         1.09%       1.609ms         4.63%       6.816ms      68.157us       2.731ms         1.76%      28.988ms     289.883us           100  
               aten::empty_like         0.31%     452.563us         1.03%       1.514ms      15.136us       0.000us         0.00%       0.000us       0.000us           100  
            aten::empty_strided         0.72%       1.061ms         0.72%       1.061ms      10.611us       0.000us         0.00%       0.000us       0.000us           100  
            aten::scalar_tensor         1.04%       1.538ms         2.49%       3.663ms      36.630us       7.719ms         4.97%      12.136ms     121.364us           100  
                       aten::to         0.45%     666.836us         0.45%     666.836us       6.668us     216.123us         0.14%     216.123us       2.161us           100  
               aten::index_put_         1.95%       2.872ms        34.31%      50.496ms     504.964us     858.000us         0.55%      45.627ms     456.273us           100  
         aten::_index_put_impl_         5.22%       7.681ms        32.36%      47.624ms     476.245us       9.113ms         5.87%      44.769ms     447.693us           100  
                  aten::nonzero        20.03%      29.478ms        22.34%      32.887ms     328.873us      25.905ms        16.68%      26.794ms     267.940us           100  
               aten::contiguous         0.42%     616.545us         0.42%     616.545us       6.165us     210.938us         0.14%     210.938us       2.109us           100  
                        aten::t         0.15%     227.831us         0.46%     670.662us       6.707us       0.000us         0.00%       0.000us       0.000us           100  
                aten::transpose         0.18%     267.303us         0.30%     442.831us       4.428us       0.000us         0.00%       0.000us       0.000us           100  
                     aten::set_         1.32%       1.945ms         1.32%       1.945ms      19.446us     677.857us         0.44%     677.857us       6.779us           100  
                   aten::select         1.95%       2.877ms         2.23%       3.280ms       8.200us     916.401us         0.59%     916.401us       2.291us           400  
                   aten::stride         0.12%     175.333us         0.12%     175.333us       0.438us       0.000us         0.00%       0.000us       0.000us           400  
                  aten::reshape         1.89%       2.787ms         2.37%       3.494ms       8.735us       7.946ms         5.12%       7.946ms      19.866us           400  
                     aten::view         0.48%     706.643us         0.48%     706.643us       1.767us       0.000us         0.00%       0.000us       0.000us           400  
torch::autograd::AccumulateGrad         1.10%       1.613ms         3.29%       4.838ms      48.377us     419.837us         0.27%       4.099ms      40.991us           100  
                   aten::detach         0.01%      10.582us         0.01%      16.573us      16.573us       4.096us         0.00%       6.144us       6.144us             1  
                         detach         0.00%       5.991us         0.00%       5.991us       5.991us       2.048us         0.00%       2.048us       2.048us             1  
                     aten::add_         2.18%       3.208ms         2.18%       3.208ms      32.402us       3.673ms         2.36%       3.673ms      37.102us            99

Self CPU time total: 147.192ms
CUDA time total: 155.337ms

SoftPool3d (CPU) [foward]

               Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg    # of Calls

   aten::avg_pool3d        31.04%     828.545ms        31.13%     830.776ms       4.154ms           200  
          aten::exp        30.66%     818.304ms        61.32%        1.637s       8.183ms           200  
          aten::mul        22.75%     607.218ms        22.84%     609.557ms       6.096ms           100  
          aten::sum        10.01%     267.203ms        10.38%     277.128ms       2.771ms           100  
        aten::clamp         1.73%      46.063ms         3.38%      90.118ms     225.295us           400  
         aten::mul_         1.32%      35.322ms         1.49%      39.739ms     198.693us           200  
        aten::copy_         1.27%      33.817ms         1.27%      33.817ms     112.724us           300  
         aten::div_         0.40%      10.629ms         2.00%      53.342ms     266.712us           200  
        aten::fill_         0.35%       9.330ms         0.35%       9.330ms      93.297us           100  
        aten::empty         0.17%       4.464ms         0.17%       4.464ms       6.378us           700  
      aten::resize_         0.13%       3.522ms         0.13%       3.522ms       7.044us           500  
           aten::to         0.06%       1.647ms         0.17%       4.417ms      22.083us           200  
aten::empty_strided         0.05%       1.438ms         0.05%       1.438ms       4.793us           300  
        aten::clone         0.04%       1.047ms         1.26%      33.533ms     335.325us           100  
   aten::contiguous         0.02%     544.987us         0.02%     544.987us       2.725us           200

Self CPU time total: 2.669s

SoftPool3d (CPU) [forward + backward]

                           Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg    # of Calls

                      aten::mul        32.00%        2.534s        32.17%        2.547s       2.316ms          1100  
                      aten::exp         9.81%     776.988ms        19.63%        1.554s       7.772ms           200  
               aten::avg_pool3d         9.77%     773.349ms         9.79%     775.439ms       3.877ms           200  
                    aten::fill_         8.86%     701.494ms         8.86%     701.494ms       1.002ms           700  
      aten::avg_pool3d_backward         8.84%     699.745ms        17.49%        1.385s       6.923ms           200  
                      aten::add         7.98%     631.929ms         8.04%     636.900ms       3.185ms           200  
                      aten::sum         7.69%     608.998ms         8.04%     636.892ms       2.123ms           300  
torch::autograd::AccumulateGrad         5.42%     429.000ms         9.81%     776.467ms       7.765ms           100  
                     aten::add_         4.39%     347.462ms         4.39%     347.462ms       3.510ms            99  
                      aten::div         0.95%      75.268ms         0.97%      76.867ms     256.224us           300  
                    aten::copy_         0.93%      73.511ms         0.93%      73.511ms      66.828us          1100  
                      aten::neg         0.66%      51.953ms         1.33%     105.173ms     525.864us           200  
                    aten::clamp         0.61%      48.641ms         1.21%      95.916ms     239.789us           400  
                       aten::le         0.45%      35.813ms         0.97%      76.633ms     191.583us           400  
                     aten::mul_         0.41%      32.260ms         0.45%      35.774ms     178.869us           200  
                    aten::empty         0.31%      24.770ms         0.31%      24.770ms       7.990us          3100  
                       aten::ge         0.23%      18.400ms         0.51%      40.156ms     100.390us           400  
                     aten::div_         0.14%      11.408ms         0.67%      53.072ms     265.361us           200  
                  aten::resize_         0.08%       6.537ms         0.08%       6.537ms       5.447us          1200  
                       aten::to         0.07%       5.358ms         0.66%      52.383ms      52.383us          1000  
                   MulBackward0         0.06%       4.834ms        15.40%        1.220s       4.066ms           300  
            aten::empty_strided         0.05%       4.203ms         0.05%       4.203ms       3.821us          1100  
                  ClampBackward         0.05%       4.085ms         2.65%     209.941ms       1.050ms           200  
               aten::zeros_like         0.04%       3.468ms         6.51%     515.548ms       2.578ms           200  
                   DivBackward0         0.03%       2.540ms         2.52%     199.853ms       1.999ms           100  
                     aten::conj         0.02%       1.618ms         0.02%       1.618ms       2.697us           600  
              AvgPool3DBackward         0.02%       1.426ms        17.50%        1.386s       6.930ms           200  
                     aten::item         0.02%       1.403ms         0.03%       2.033ms       5.083us           400  
                   aten::expand         0.01%       1.083ms         0.02%       1.475ms      14.752us           100  
                    aten::clone         0.01%       1.053ms         0.40%      31.742ms     317.419us           100  
                    aten::zero_         0.01%       1.014ms         8.55%     676.785ms       1.692ms           400  
                  aten::type_as         0.01%     823.768us         0.53%      42.220ms     211.102us           200  
               aten::resize_as_         0.01%     744.948us         0.01%     940.909us       4.705us           200  
                    ExpBackward         0.01%     730.343us         7.43%     588.339ms       5.883ms           100  
               aten::contiguous         0.01%     721.015us         0.01%     721.015us       1.803us           400  
               aten::empty_like         0.01%     665.327us         0.04%       3.410ms      17.052us           200  
      aten::_local_scalar_dense         0.01%     630.038us         0.01%     630.038us       1.575us           400  
               aten::as_strided         0.00%     391.738us         0.00%     391.738us       3.917us           100  
                   SumBackward1         0.00%     389.240us         0.02%       1.864ms      18.644us           100  
     torch::autograd::GraphRoot         0.00%     129.415us         0.00%     129.415us       1.294us           100  
                         detach         0.00%       3.344us         0.00%       3.344us       3.344us             1  
                   aten::detach         0.00%       1.994us         0.00%       5.338us       5.338us             1

Self CPU time total: 7.919s

SoftPool3d (CUDA-inplace) [foward]

               Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg     Self CUDA   Self CUDA %    CUDA total  CUDA time avg    # of Calls

          aten::exp        20.71%      22.414ms        22.39%      24.228ms     121.140us      37.999ms        24.24%      75.155ms     375.777us           200  
        aten::empty         1.01%       1.097ms         1.01%       1.097ms       1.566us       0.000us         0.00%       0.000us       0.000us           700  
      aten::resize_         1.06%       1.149ms         1.06%       1.149ms       2.298us       0.000us         0.00%       0.000us       0.000us           500  
          aten::sum         2.30%       2.491ms         2.51%       2.711ms      27.114us      16.009ms        10.21%      16.009ms     160.091us           100  
        aten::clamp        24.93%      26.979ms        28.12%      30.430ms      76.074us       9.319ms         5.95%      12.378ms      30.945us           400  
          aten::mul         4.38%       4.743ms         4.60%       4.975ms      49.755us      28.481ms        18.17%      28.481ms     284.805us           100  
   aten::avg_pool3d        13.44%      14.542ms        19.00%      20.559ms     102.796us      37.724ms        24.07%      39.011ms     195.057us           200  
   aten::contiguous         0.70%     753.110us         0.70%     753.110us       3.766us     407.441us         0.26%     407.441us       2.037us           200  
      aten::reshape         3.67%       3.976ms         4.21%       4.551ms      11.378us     879.679us         0.56%     879.679us       2.199us           400  
         aten::view         0.53%     575.313us         0.53%     575.313us       1.438us       0.000us         0.00%       0.000us       0.000us           400  
         aten::mul_         8.34%       9.025ms         8.34%       9.025ms      45.123us       5.654ms         3.61%       5.654ms      28.268us           200  
         aten::div_        13.21%      14.302ms        20.02%      21.667ms     108.335us       7.846ms         5.01%      27.464ms     137.318us           200  
        aten::clone         3.89%       4.206ms         5.72%       6.187ms      61.872us       5.343ms         3.41%      12.420ms     124.200us           100  
aten::empty_strided         0.37%     395.789us         0.37%     395.789us       3.958us       0.000us         0.00%       0.000us       0.000us           100  
        aten::copy_         1.46%       1.585ms         1.46%       1.585ms      15.849us       7.077ms         4.52%       7.077ms      70.769us           100

Self CPU time total: 108.232ms
CUDA time total: 156.739ms

SoftPool3d (CUDA-inplace) [forward + backward]

                           Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg     Self CUDA   Self CUDA %    CUDA total  CUDA time avg    # of Calls

                      aten::exp         3.93%      10.763ms         6.42%      17.550ms      87.749us      16.808ms         5.76%      32.768ms     163.839us           200  
                    aten::empty         4.12%      11.270ms         4.12%      11.270ms       4.174us       0.000us         0.00%       0.000us       0.000us          2700  
                  aten::resize_         2.18%       5.967ms         2.18%       5.967ms       4.972us       0.000us         0.00%       0.000us       0.000us          1200  
                      aten::sum         1.96%       5.368ms         2.24%       6.134ms      61.340us       9.082ms         3.11%       9.082ms      90.820us           100  
                    aten::clamp         4.07%      11.142ms         6.51%      17.817ms      44.542us       5.259ms         1.80%       8.587ms      21.468us           400  
                      aten::mul        16.01%      43.801ms        17.85%      48.829ms      44.390us     103.777ms        35.58%     103.777ms      94.343us          1100  
               aten::avg_pool3d         4.31%      11.782ms         7.71%      21.092ms     105.459us      16.439ms         5.64%      18.522ms      92.610us           200  
               aten::contiguous         0.90%       2.472ms         0.90%       2.472ms       6.181us       1.086ms         0.37%       1.086ms       2.716us           400  
                  aten::reshape         2.82%       7.703ms         3.96%      10.836ms      13.545us       3.666ms         1.26%       3.666ms       4.582us           800  
                     aten::view         1.15%       3.133ms         1.15%       3.133ms       3.916us       0.000us         0.00%       0.000us       0.000us           800  
                     aten::mul_         2.78%       7.615ms         2.78%       7.615ms      38.075us       3.494ms         1.20%       3.494ms      17.470us           200  
                     aten::div_         1.69%       4.613ms         4.94%      13.517ms      67.584us       3.364ms         1.15%      11.663ms      58.317us           200  
                    aten::clone         1.14%       3.132ms         2.53%       6.931ms      69.309us     963.624us         0.33%       5.691ms      56.915us           100  
            aten::empty_strided         0.98%       2.675ms         0.98%       2.675ms       8.916us       0.000us         0.00%       0.000us       0.000us           300  
                    aten::copy_         3.76%      10.276ms         3.76%      10.276ms      34.254us       8.912ms         3.06%       8.912ms      29.706us           300  
     torch::autograd::GraphRoot         0.24%     654.515us         0.24%     654.515us       6.545us     263.158us         0.09%     263.158us       2.632us           100  
                  ClampBackward         3.98%      10.875ms        29.44%      80.529ms     402.646us       3.453ms         1.18%      33.688ms     168.438us           200  
                       aten::le         5.35%      14.643ms        12.12%      33.168ms      82.921us       5.534ms         1.90%      11.910ms      29.775us           400  
                     aten::item         1.90%       5.202ms         2.86%       7.826ms      19.564us       2.038ms         0.70%       3.046ms       7.614us           400  
      aten::_local_scalar_dense         0.96%       2.624ms         0.96%       2.624ms       6.560us       1.008ms         0.35%       1.008ms       2.519us           400  
                       aten::ge         4.59%      12.555ms        10.48%      28.679ms      71.698us       5.526ms         1.89%      12.016ms      30.039us           400  
                  aten::type_as         0.98%       2.684ms         5.64%      15.433ms      77.167us       1.182ms         0.41%       7.023ms      35.117us           200  
                       aten::to         1.31%       3.596ms         4.66%      12.749ms      63.745us       1.657ms         0.57%       5.841ms      29.205us           200  
                   DivBackward0         2.32%       6.334ms        10.72%      29.333ms     293.334us       2.307ms         0.79%      22.848ms     228.484us           100  
                      aten::div         3.94%      10.791ms         4.40%      12.035ms      40.118us      10.792ms         3.70%      10.792ms      35.972us           300  
                     aten::conj         1.58%       4.335ms         1.58%       4.335ms       7.225us       1.832ms         0.63%       1.832ms       3.054us           600  
                      aten::neg         1.80%       4.935ms         3.00%       8.215ms      41.073us       4.377ms         1.50%       7.362ms      36.808us           200  
                   MulBackward0         3.16%       8.643ms        11.55%      31.600ms     105.335us       4.006ms         1.37%      61.606ms     205.352us           300  
              AvgPool3DBackward         1.21%       3.300ms        15.38%      42.087ms     210.433us       2.024ms         0.69%      49.966ms     249.831us           200  
      aten::avg_pool3d_backward         5.45%      14.922ms        14.18%      38.787ms     193.933us      17.507ms         6.00%      47.942ms     239.712us           200  
               aten::zeros_like         1.11%       3.044ms         4.00%      10.952ms      54.759us       5.305ms         1.82%      16.535ms      82.677us           200  
               aten::empty_like         0.27%     734.177us         0.63%       1.721ms       8.606us       0.000us         0.00%       0.000us       0.000us           200  
                    aten::zero_         1.96%       5.367ms         4.31%      11.784ms      29.459us       3.047ms         1.04%      21.574ms      53.934us           400  
                    aten::fill_         2.35%       6.416ms         2.35%       6.416ms      16.041us      18.527ms         6.35%      18.527ms      46.318us           400  
               aten::resize_as_         0.61%       1.665ms         0.69%       1.880ms       9.401us     887.768us         0.30%     887.768us       4.439us           200  
                   SumBackward1         0.33%     911.682us         0.74%       2.026ms      20.261us     444.419us         0.15%     444.419us       4.444us           100  
                   aten::expand         0.29%     799.662us         0.41%       1.114ms      11.145us       0.000us         0.00%       0.000us       0.000us           100  
               aten::as_strided         0.12%     314.789us         0.12%     314.789us       3.148us       0.000us         0.00%       0.000us       0.000us           100  
                    ExpBackward         0.60%       1.636ms         2.06%       5.639ms      56.387us     563.026us         0.19%      17.475ms     174.755us           100  
torch::autograd::AccumulateGrad         0.66%       1.796ms         1.78%       4.880ms      48.800us     426.041us         0.15%      26.526ms     265.256us           100  
                   aten::detach         0.01%      25.378us         0.01%      38.282us      38.282us       4.096us         0.00%       6.144us       6.144us             1  
                         detach         0.00%      12.904us         0.00%      12.904us      12.904us       2.048us         0.00%       2.048us       2.048us             1  
                     aten::add_         1.11%       3.045ms         1.11%       3.045ms      30.761us      26.093ms         8.95%      26.093ms     263.570us            99

Self CPU time total: 273.574ms
CUDA time total: 291.654ms

SoftPool3d (CUDA) [foward]

                     Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg     Self CUDA   Self CUDA %    CUDA total  CUDA time avg    # of Calls

          CUDA_SOFTPOOL3d         1.66%       4.105ms         3.68%       9.089ms      90.892us      18.581ms         7.54%     238.650ms       2.386ms           100  
          aten::new_zeros         0.36%     896.518us         2.02%       4.984ms      49.844us     762.784us         0.31%     220.069ms       2.201ms           100  
              aten::zeros         0.37%     910.308us         1.65%       4.088ms      40.879us       1.205ms         0.49%     219.306ms       2.193ms           100  
              aten::empty         0.28%     692.563us         0.28%     692.563us       2.309us       0.000us         0.00%       0.000us       0.000us           300  
              aten::zero_         0.63%       1.559ms         1.10%       2.720ms      27.203us       1.204ms         0.49%     218.101ms       2.181ms           100  
              aten::fill_         0.47%       1.161ms         0.47%       1.161ms      11.608us     216.897ms        88.02%     216.897ms       2.169ms           100  
              aten::isnan         0.57%       1.411ms         1.70%       4.210ms      42.102us     874.523us         0.35%       2.832ms      28.324us           100  
                 aten::ne         0.98%       2.415ms         1.83%       4.531ms      22.653us       1.958ms         0.79%       3.483ms      17.413us           200  
            aten::resize_         0.18%     433.717us         0.18%     433.717us       2.169us       0.000us         0.00%       0.000us       0.000us           200  
                aten::any         1.22%       3.023ms         1.39%       3.446ms      34.464us       1.526ms         0.62%       1.526ms      15.259us           100  
         aten::as_strided         0.06%     138.248us         0.06%     138.248us       1.382us       0.000us         0.00%       0.000us       0.000us           100  
         aten::is_nonzero         0.28%     698.505us        93.22%     230.327ms       2.303ms     523.520us         0.21%       3.404ms      34.040us           100  
               aten::item         0.32%     790.430us        92.94%     229.629ms       2.296ms     561.143us         0.23%       2.880ms      28.804us           100  
aten::_local_scalar_dense        92.62%     228.838ms        92.62%     228.838ms       2.288ms       2.319ms         0.94%       2.319ms      23.193us           100

Self CPU time total: 247.073ms
CUDA time total: 246.412ms

SoftPool3d (CUDA) [forward + backward]

                           Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg     Self CUDA   Self CUDA %    CUDA total  CUDA time avg    # of Calls

                CUDA_SOFTPOOL3d         6.35%       8.317ms        13.43%      17.585ms     175.854us      17.743ms        12.81%      21.915ms     219.153us           100  
                aten::new_zeros         1.19%       1.562ms         7.08%       9.268ms      92.680us     587.050us         0.42%       4.173ms      41.727us           100  
                    aten::zeros         1.20%       1.571ms         5.89%       7.706ms      77.060us     765.266us         0.55%       3.586ms      35.856us           100  
                    aten::empty         1.68%       2.198ms         1.68%       2.198ms       4.397us       0.000us         0.00%       0.000us       0.000us           500  
                    aten::zero_         3.15%       4.130ms         6.53%       8.547ms      42.733us       2.176ms         1.57%      12.736ms      63.680us           200  
                    aten::fill_         4.49%       5.872ms         4.49%       5.872ms      19.572us      10.989ms         7.94%      10.989ms      36.629us           300  
                    aten::isnan         3.51%       4.591ms        10.66%      13.953ms      69.765us       1.721ms         1.24%      14.218ms      71.091us           200  
                       aten::ne         6.01%       7.862ms        11.55%      15.124ms      37.811us      12.497ms         9.03%      24.158ms      60.396us           400  
                  aten::resize_         1.19%       1.555ms         1.19%       1.555ms       3.888us       0.000us         0.00%       0.000us       0.000us           400  
                      aten::any         3.81%       4.990ms         4.44%       5.813ms      58.132us       2.068ms         1.49%       2.068ms      20.685us           100  
               aten::as_strided         0.79%       1.035ms         0.79%       1.035ms       1.294us       0.000us         0.00%       0.000us       0.000us           800  
               aten::is_nonzero         0.83%       1.081ms        11.73%      15.361ms     153.611us     755.014us         0.55%       4.808ms      48.076us           100  
                     aten::item         1.15%       1.512ms        10.91%      14.281ms     142.806us     936.134us         0.68%       4.053ms      40.526us           100  
      aten::_local_scalar_dense         9.75%      12.769ms         9.75%      12.769ms     127.686us       3.116ms         2.25%       3.116ms      31.164us           100  
     torch::autograd::GraphRoot         0.49%     646.309us         0.49%     646.309us       6.463us     581.246us         0.42%     581.246us       5.812us           100  
        CUDA_SOFTPOOL3dBackward        11.43%      14.966ms        61.03%      79.901ms     799.012us      24.126ms        17.42%      80.413ms     804.128us           100  
               aten::zeros_like         1.14%       1.497ms         4.76%       6.227ms      62.270us       2.421ms         1.75%      12.336ms     123.363us           100  
               aten::empty_like         0.31%     408.229us         0.97%       1.267ms      12.667us       0.000us         0.00%       0.000us       0.000us           100  
            aten::empty_strided         0.66%     858.428us         0.66%     858.428us       8.584us       0.000us         0.00%       0.000us       0.000us           100  
            aten::scalar_tensor         1.11%       1.458ms         2.57%       3.367ms      33.673us     446.447us         0.32%     875.508us       8.755us           100  
                       aten::to         0.48%     625.716us         0.48%     625.716us       6.257us     246.771us         0.18%     246.771us       2.468us           100  
               aten::index_put_         1.93%       2.531ms        36.37%      47.618ms     476.179us       1.231ms         0.89%      31.412ms     314.122us           100  
         aten::_index_put_impl_         6.68%       8.745ms        34.44%      45.087ms     450.867us       4.121ms         2.98%      30.181ms     301.814us           100  
                  aten::nonzero        18.53%      24.260ms        21.05%      27.553ms     275.533us      21.363ms        15.43%      22.271ms     222.710us           100  
               aten::contiguous         0.43%     563.838us         0.43%     563.838us       5.638us     250.919us         0.18%     250.919us       2.509us           100  
                        aten::t         0.17%     227.859us         0.50%     659.881us       6.599us       0.000us         0.00%       0.000us       0.000us           100  
                aten::transpose         0.20%     263.598us         0.33%     432.022us       4.320us       0.000us         0.00%       0.000us       0.000us           100  
                     aten::set_         1.44%       1.889ms         1.44%       1.889ms      18.888us     657.498us         0.47%     657.498us       6.575us           100  
                   aten::select         2.88%       3.764ms         3.26%       4.273ms       8.545us       1.477ms         1.07%       1.477ms       2.953us           500  
                   aten::stride         0.18%     233.265us         0.18%     233.265us       0.467us       0.000us         0.00%       0.000us       0.000us           500  
                  aten::reshape         2.53%       3.310ms         3.19%       4.179ms       8.358us       2.312ms         1.67%       2.312ms       4.624us           500  
                     aten::view         0.66%     868.518us         0.66%     868.518us       1.737us       0.000us         0.00%       0.000us       0.000us           500  
torch::autograd::AccumulateGrad         1.23%       1.611ms         3.63%       4.749ms      47.493us       1.020ms         0.74%      25.878ms     258.776us           100  
                   aten::detach         0.02%      23.460us         0.03%      36.994us      36.994us      21.504us         0.02%      35.840us      35.840us             1  
                         detach         0.01%      13.534us         0.01%      13.534us      13.534us      14.336us         0.01%      14.336us      14.336us             1  
                     aten::add_         2.37%       3.101ms         2.37%       3.101ms      31.326us      24.822ms        17.93%      24.822ms     250.725us            99

Self CPU time total: 130.912ms
CUDA time total: 138.466ms

SoftPool1d [forward + backward]

----------- C P U ------------
Self CPU time total: 164.672ms

-- C U D A - I N P L A C E ---
Self CPU time total: 292.102msCUDA time total: 291.388ms

---------- C U D A -----------
Self CPU time total: 81.088msCUDA time total: 80.682ms

SoftPool2d [forward + backward]

----------- C P U ------------
Self CPU time total: 789.392ms

-- C U D A - I N P L A C E ---
Self CPU time total: 189.949msCUDA time total: 189.042ms

---------- C U D A -----------
Self CPU time total: 147.192msCUDA time total: 155.337ms

SoftPool3d [forward + backward]

----------- C P U ------------
Self CPU time total: 7.919s

-- C U D A - I N P L A C E ---
Self CPU time total: 273.574msCUDA time total: 291.654ms

---------- C U D A -----------
Self CPU time total: 130.912msCUDA time total: 138.466ms

--- Tests finished ---

huixiancheng · 2022-09-19T10:58:05Z

If the training is done with mixed precision (amp), nan may be caused by numerical overflow.

Terminal-K · 2022-11-18T10:13:52Z

Hi, I also encountered this problem

Lr: 1.390e-03 | Update: nan mean,nan std | Epoch: [0][96/698] | Time 2.741 (2.997) | Data 0.078 (0.239) | Loss nan (nan) | acc 0.000 (0.000) | IoU 0.000 (0.000) | [3 days, 22:02:11]

Lr: 1.562e-03 | Update: nan mean,nan std | Epoch: [0][108/698] | Time 2.728 (2.956) | Data 0.077 (0.221) | Loss nan (nan) | acc 0.000 (0.000) | IoU 0.000 (0.000) | [3 days, 20:18:25]

Lr: 1.734e-03 | Update: nan mean,nan std | Epoch: [0][120/698] | Time 2.578 (2.917) | Data 0.076 (0.207) | Loss nan (nan) | acc 0.000 (0.000) | IoU 0.000 (0.000) | [3 days, 18:45:15]

I just use SemanticKitti dataset and my running recommand is
python train.py -d DATAROOT -ac ./train_yaml/mos_coarse_stage.yml -l log/ours_motionseg3d -dc config/labels/semantic-kitti-all.yaml

My environment is following:
python=3.8
cuda=11.1
torch=1.10.2
softpool=1.1
4*Tesla V100

Pytorch version meets the requirements, while the training loss is still nan.

MaxChanger · 2022-12-09T15:52:21Z

Hi @Terminal-K @fengluodb @huixiancheng , I finally found the cause of this problem!
I ignored the version of softpool, its author updated the repo on 2022/04/07, and our project was implemented before that, using its historical version, the commits id is 2d2ec6d.

When I use the new version d056ab8 of softpool code for inference, nan will also appear, just like what you encounter in the process of inference or train.

The specific reason may need to carefully check the softpool code and discuss with the author, but rolling back the softpool version is a quick solution to this problem.

git clone https://github.com/alexandrosstergiou/SoftPool.git
cd SoftPool
git checkout 2d2ec6d # rollback to 2d2ec6dca10b7683ffd41061a27910d67816bfa5
cd pytorch
make install
--- (optional) ---
make test

If you have any questions, please contact me again.

Terminal-K · 2022-12-09T16:33:49Z

Thanks for your patience, i'll try it again:)

MaxChanger mentioned this issue Sep 7, 2022

find out nan in tensor alexandrosstergiou/SoftPool#12

Closed

fengluodb closed this as completed Sep 7, 2022

MaxChanger mentioned this issue Dec 1, 2022

model output is NAN for toy dataset #9

Closed

MaxChanger mentioned this issue Dec 9, 2022

About the pretrained model #11

Closed

MaxChanger added the bug Something isn't working label Dec 9, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

loss is nan #6

loss is nan #6

fengluodb commented Sep 6, 2022

MaxChanger commented Sep 6, 2022

fengluodb commented Sep 6, 2022

MaxChanger commented Sep 6, 2022

fengluodb commented Sep 6, 2022 •

edited

Loading

MaxChanger commented Sep 6, 2022 •

edited

Loading

fengluodb commented Sep 6, 2022

MaxChanger commented Sep 6, 2022

fengluodb commented Sep 6, 2022

MaxChanger commented Sep 6, 2022

fengluodb commented Sep 6, 2022

fengluodb commented Sep 6, 2022

MaxChanger commented Sep 6, 2022 •

edited

Loading

fengluodb commented Sep 6, 2022

fengluodb commented Sep 6, 2022

MaxChanger commented Sep 6, 2022

fengluodb commented Sep 7, 2022

MaxChanger commented Sep 7, 2022

MaxChanger commented Sep 7, 2022

fengluodb commented Sep 7, 2022

MaxChanger commented Sep 7, 2022 •

edited

Loading

MaxChanger commented Sep 7, 2022

fengluodb commented Sep 7, 2022 •

edited

Loading

huixiancheng commented Sep 19, 2022

Terminal-K commented Nov 18, 2022 •

edited

Loading

MaxChanger commented Dec 9, 2022

Terminal-K commented Dec 9, 2022

loss is nan #6

loss is nan #6

Comments

fengluodb commented Sep 6, 2022

MaxChanger commented Sep 6, 2022

fengluodb commented Sep 6, 2022

MaxChanger commented Sep 6, 2022

fengluodb commented Sep 6, 2022 • edited Loading

MaxChanger commented Sep 6, 2022 • edited Loading

fengluodb commented Sep 6, 2022

MaxChanger commented Sep 6, 2022

fengluodb commented Sep 6, 2022

MaxChanger commented Sep 6, 2022

fengluodb commented Sep 6, 2022

fengluodb commented Sep 6, 2022

MaxChanger commented Sep 6, 2022 • edited Loading

fengluodb commented Sep 6, 2022

fengluodb commented Sep 6, 2022

MaxChanger commented Sep 6, 2022

fengluodb commented Sep 7, 2022

MaxChanger commented Sep 7, 2022

MaxChanger commented Sep 7, 2022

fengluodb commented Sep 7, 2022

MaxChanger commented Sep 7, 2022 • edited Loading

MaxChanger commented Sep 7, 2022

fengluodb commented Sep 7, 2022 • edited Loading

huixiancheng commented Sep 19, 2022

Terminal-K commented Nov 18, 2022 • edited Loading

MaxChanger commented Dec 9, 2022

Terminal-K commented Dec 9, 2022

fengluodb commented Sep 6, 2022 •

edited

Loading

MaxChanger commented Sep 6, 2022 •

edited

Loading

MaxChanger commented Sep 6, 2022 •

edited

Loading

MaxChanger commented Sep 7, 2022 •

edited

Loading

fengluodb commented Sep 7, 2022 •

edited

Loading

Terminal-K commented Nov 18, 2022 •

edited

Loading