[task 2023-12-20T16:01:46.690Z] [2023-12-20 16:01:46] [Trainer] [INFO] Reading original for epoch 110
[task 2023-12-20T16:01:50.054Z] [2023-12-20 16:01:50] [Trainer] [INFO] Reading original for epoch 111
[task 2023-12-20T16:01:53.445Z] [2023-12-20 16:01:53] [Trainer] [INFO] Reading original for epoch 112
[task 2023-12-20T16:01:56.771Z] [2023-12-20 16:01:56] [Trainer] [INFO] Reading original for epoch 113
[task 2023-12-20T16:02:35.811Z] [2023-12-20 16:02:35] [training] Batches are processed as 1 process(es) x 4 devices/process
[task 2023-12-20T16:02:35.831Z] [2023-12-20 16:02:35] [memory] Reserving 34 MB, device gpu0
[task 2023-12-20T16:02:35.831Z] [2023-12-20 16:02:35] [memory] Reserving 34 MB, device gpu1
[task 2023-12-20T16:02:35.831Z] [2023-12-20 16:02:35] [memory] Reserving 34 MB, device gpu3
[task 2023-12-20T16:02:35.832Z] [2023-12-20 16:02:35] [memory] Reserving 34 MB, device gpu2
[task 2023-12-20T16:02:35.902Z] [2023-12-20 16:02:35] [memory] Reserving 34 MB, device gpu0
[task 2023-12-20T16:02:35.954Z] [2023-12-20 16:02:35] [memory] Reserving 34 MB, device gpu1
[task 2023-12-20T16:02:35.988Z] [2023-12-20 16:02:35] [memory] Reserving 34 MB, device gpu3
[task 2023-12-20T16:02:36.034Z] [2023-12-20 16:02:36] [memory] Reserving 34 MB, device gpu2
[task 2023-12-20T16:02:36.829Z] [2023-12-20 16:02:36] Allocating memory for general optimizer shards
[task 2023-12-20T16:02:36.829Z] [2023-12-20 16:02:36] Parameter type float32, optimization type float32, casting types false
[task 2023-12-20T16:02:36.829Z] [2023-12-20 16:02:36] [memory] Reserving 8 MB, device gpu3
[task 2023-12-20T16:02:36.829Z] [2023-12-20 16:02:36] Parameter type float32, optimization type float32, casting types false
[task 2023-12-20T16:02:36.829Z] [2023-12-20 16:02:36] [memory] Reserving 8 MB, device gpu2
[task 2023-12-20T16:02:36.829Z] [2023-12-20 16:02:36] [memory] Reserving 8 MB, device gpu0
[task 2023-12-20T16:02:36.829Z] [2023-12-20 16:02:36] [memory] Reserving 8 MB, device gpu1
[task 2023-12-20T16:02:36.832Z] [2023-12-20 16:02:36] Allocating memory for Adam-specific shards
[task 2023-12-20T16:02:36.832Z] [2023-12-20 16:02:36] [memory] Reserving 17 MB, device gpu3
[task 2023-12-20T16:02:36.833Z] [2023-12-20 16:02:36] [memory] Reserving 17 MB, device gpu1
[task 2023-12-20T16:02:36.834Z] [2023-12-20 16:02:36] [memory] Reserving 17 MB, device gpu2
[task 2023-12-20T16:02:36.834Z] [2023-12-20 16:02:36] [memory] Reserving 17 MB, device gpu0
[task 2023-12-20T16:02:36.837Z] [2023-12-20 16:02:36] Ep. 1 : Up. 1 : Sen. 1,520 : Cost 0.35829252 : Time 434.08s : 570.25 words/s : gNorm 35.1306 : L.r. 1.8750e-08
[task 2023-12-20T16:02:36.950Z] [2023-12-20 16:02:36] [Trainer] [INFO] Reading original for epoch 114
[task 2023-12-20T16:02:37.203Z] [2023-12-20 16:02:37] Ep. 1 : Up. 2 : Sen. 2,736 : Cost 0.31103832 : Time 0.37s : 660343.34 words/s : gNorm 38.4629 : L.r. 3.7500e-08
[task 2023-12-20T16:02:37.509Z] [2023-12-20 16:02:37] Ep. 1 : Up. 3 : Sen. 4,704 : Cost 0.45516947 : Time 0.31s : 894133.48 words/s : gNorm 37.1530 : L.r. 5.6250e-08
[task 2023-12-20T16:02:37.850Z] [2023-12-20 16:02:37] Ep. 1 : Up. 4 : Sen. 6,672 : Cost 0.41129625 : Time 0.34s : 783250.06 words/s : gNorm 35.2627 : L.r. 7.5000e-08
[task 2023-12-20T16:02:38.179Z] [2023-12-20 16:02:38] Ep. 1 : Up. 5 : Sen. 9,304 : Cost 0.55869937 : Time 0.33s : 631791.73 words/s : gNorm 35.8894 : L.r. 9.3750e-08
[task 2023-12-20T16:02:38.605Z] [2023-12-20 16:02:38] Ep. 1 : Up. 6 : Sen. 10,952 : Cost 0.27760094 : Time 0.43s : 579648.11 words/s : gNorm 35.8150 : L.r. 1.1250e-07
[task 2023-12-20T16:02:38.976Z] [2023-12-20 16:02:38] Ep. 1 : Up. 7 : Sen. 12,600 : Cost 0.30948466 : Time 0.37s : 697927.66 words/s : gNorm 35.9055 : L.r. 1.3125e-07
[task 2023-12-20T16:02:39.388Z] [2023-12-20 16:02:39] Ep. 1 : Up. 8 : Sen. 14,008 : Cost 0.29586118 : Time 0.41s : 584519.03 words/s : gNorm 35.9367 : L.r. 1.5000e-07
[task 2023-12-20T16:02:39.673Z] [2023-12-20 16:02:39] Ep. 1 : Up. 9 : Sen. 16,640 : Cost 0.87608075 : Time 0.28s : 347581.71 words/s : gNorm 34.0804 : L.r. 1.6875e-07
[task 2023-12-20T16:02:39.960Z] [2023-12-20 16:02:39] Ep. 1 : Up. 10 : Sen. 20,056 : Cost 0.71641898 : Time 0.29s : 1010508.76 words/s : gNorm 33.9601 : L.r. 1.8750e-07
[task 2023-12-20T16:02:40.116Z] [2023-12-20 16:02:40] [Trainer] [INFO] Reading original for epoch 115
[task 2023-12-20T16:02:40.297Z] [2023-12-20 16:02:40] Ep. 1 : Up. 11 : Sen. 22,224 : Cost 0.46978521 : Time 0.34s : 835891.12 words/s : gNorm 34.2215 : L.r. 2.0625e-07
[task 2023-12-20T16:02:40.633Z] [2023-12-20 16:02:40] Ep. 1 : Up. 12 : Sen. 23,528 : Cost 0.33826894 : Time 0.34s : 702515.17 words/s : gNorm 34.3421 : L.r. 2.2500e-07
[task 2023-12-20T16:02:40.992Z] [2023-12-20 16:02:40] Ep. 1 : Up. 13 : Sen. 25,920 : Cost 0.43727690 : Time 0.36s : 630155.58 words/s : gNorm 33.6542 : L.r. 2.4375e-07
[task 2023-12-20T16:02:41.350Z] [2023-12-20 16:02:41] Ep. 1 : Up. 14 : Sen. 28,312 : Cost 0.41895959 : Time 0.36s : 789302.46 words/s : gNorm 34.3530 : L.r. 2.6250e-07
[task 2023-12-20T16:02:41.689Z] [2023-12-20 16:02:41] Ep. 1 : Up. 15 : Sen. 29,720 : Cost 0.31884900 : Time 0.34s : 731555.40 words/s : gNorm 35.5653 : L.r. 2.8125e-07
[task 2023-12-20T16:02:42.014Z] [2023-12-20 16:02:42] Ep. 1 : Up. 16 : Sen. 31,688 : Cost 0.39861014 : Time 0.33s : 846883.90 words/s : gNorm 36.3300 : L.r. 3.0000e-07
[task 2023-12-20T16:02:42.371Z] [2023-12-20 16:02:42] Ep. 1 : Up. 17 : Sen. 32,904 : Cost 0.26450431 : Time 0.36s : 671093.69 words/s : gNorm 37.4739 : L.r. 3.1875e-07
[task 2023-12-20T16:02:42.703Z] [2023-12-20 16:02:42] Ep. 1 : Up. 18 : Sen. 34,424 : Cost 0.34111276 : Time 0.33s : 736406.41 words/s : gNorm 37.7942 : L.r. 3.3750e-07
[task 2023-12-20T16:02:43.010Z] [2023-12-20 16:02:43] Ep. 1 : Up. 19 : Sen. 38,312 : Cost 0.70943439 : Time 0.31s : 1000975.98 words/s : gNorm 37.4241 : L.r. 3.5625e-07
[task 2023-12-20T16:02:43.251Z] [2023-12-20 16:02:43] [Trainer] [INFO] Reading original for epoch 116
[task 2023-12-20T16:02:43.361Z] [2023-12-20 16:02:43] Ep. 1 : Up. 20 : Sen. 39,720 : Cost 0.29177508 : Time 0.35s : 707568.07 words/s : gNorm 37.9653 : L.r. 3.7500e-07
[task 2023-12-20T16:02:43.696Z] [2023-12-20 16:02:43] Ep. 1 : Up. 21 : Sen. 41,240 : Cost 0.32154128 : Time 0.34s : 729269.28 words/s : gNorm 38.0836 : L.r. 3.9375e-07
[task 2023-12-20T16:02:44.043Z] [2023-12-20 16:02:44] Ep. 1 : Up. 22 : Sen. 42,888 : Cost 0.33382386 : Time 0.35s : 734118.76 words/s : gNorm 37.8119 : L.r. 4.1250e-07
[task 2023-12-20T16:02:44.382Z] [2023-12-20 16:02:44] Ep. 1 : Up. 23 : Sen. 44,296 : Cost 0.33948040 : Time 0.34s : 738878.41 words/s : gNorm 37.5384 : L.r. 4.3125e-07
[task 2023-12-20T16:02:44.735Z] [2023-12-20 16:02:44] Ep. 1 : Up. 24 : Sen. 45,512 : Cost 0.24956438 : Time 0.35s : 659311.17 words/s : gNorm 38.0852 : L.r. 4.5000e-07
[task 2023-12-20T16:02:45.043Z] [2023-12-20 16:02:45] Ep. 1 : Up. 25 : Sen. 48,928 : Cost 0.60070020 : Time 0.31s : 942750.68 words/s : gNorm 37.9049 : L.r. 4.6875e-07
[task 2023-12-20T16:02:45.044Z] [2023-12-20 16:02:45] Saving model weights and runtime parameters to /home/ubuntu/tasks/task_170308757707336/artifacts/model.npz
[task 2023-12-20T16:02:45.252Z] [2023-12-20 16:02:45] Saving Adam parameters
[task 2023-12-20T16:02:45.680Z] [2023-12-20 16:02:45] [training] Saving training checkpoint to /home/ubuntu/tasks/task_170308757707336/artifacts/model.npz and /home/ubuntu/tasks/task_170308757707336/artifacts/model.npz.optimizer.npz
[task 2023-12-20T16:02:46.327Z] [2023-12-20 16:02:46] [Trainer] [INFO] Reading original for epoch 117
[task 2023-12-20T16:02:46.415Z] [2023-12-20 16:02:46] Ep. 1 : Up. 26 : Sen. 51,096 : Cost 0.36897981 : Time 1.37s : 202230.98 words/s : gNorm 37.9473 : L.r. 4.8750e-07
[task 2023-12-20T16:02:46.731Z] [2023-12-20 16:02:46] Ep. 1 : Up. 27 : Sen. 53,064 : Cost 0.36203054 : Time 0.32s : 836455.64 words/s : gNorm 38.1723 : L.r. 5.0625e-07
[task 2023-12-20T16:02:47.087Z] [2023-12-20 16:02:47] Ep. 1 : Up. 28 : Sen. 55,696 : Cost 0.45072937 : Time 0.36s : 743436.01 words/s : gNorm 37.7709 : L.r. 5.2500e-07
[task 2023-12-20T16:02:47.410Z] [2023-12-20 16:02:47] Ep. 1 : Up. 29 : Sen. 57,000 : Cost 0.32245305 : Time 0.32s : 729462.94 words/s : gNorm 37.5855 : L.r. 5.4375e-07
[task 2023-12-20T16:02:47.786Z] [2023-12-20 16:02:47] Ep. 1 : Up. 30 : Sen. 58,520 : Cost 0.28100342 : Time 0.38s : 530481.43 words/s : gNorm 37.5505 : L.r. 5.6250e-07
[task 2023-12-20T16:02:48.137Z] [2023-12-20 16:02:48] Ep. 1 : Up. 31 : Sen. 60,320 : Cost 0.32259452 : Time 0.35s : 764282.94 words/s : gNorm 37.5245 : L.r. 5.8125e-07
[task 2023-12-20T16:02:48.492Z] [2023-12-20 16:02:48] Ep. 1 : Up. 32 : Sen. 61,840 : Cost 0.29403517 : Time 0.36s : 709787.52 words/s : gNorm 37.4015 : L.r. 6.0000e-07
[task 2023-12-20T16:02:48.831Z] [2023-12-20 16:02:48] Ep. 1 : Up. 33 : Sen. 63,144 : Cost 0.29029113 : Time 0.34s : 709219.61 words/s : gNorm 37.2604 : L.r. 6.1875e-07
[task 2023-12-20T16:02:49.126Z] [2023-12-20 16:02:49] Ep. 1 : Up. 34 : Sen. 66,144 : Cost 0.59547281 : Time 0.30s : 955456.58 words/s : gNorm 37.0679 : L.r. 6.3750e-07
[task 2023-12-20T16:02:49.393Z] [2023-12-20 16:02:49] Ep. 1 : Up. 35 : Sen. 70,704 : Cost 0.87668622 : Time 0.27s : 1092451.86 words/s : gNorm 37.2957 : L.r. 6.5625e-07
[task 2023-12-20T16:02:49.539Z] [2023-12-20 16:02:49] [Trainer] [INFO] Reading original for epoch 118
[task 2023-12-20T16:02:49.680Z] [2023-12-20 16:02:49] Ep. 1 : Up. 36 : Sen. 73,704 : Cost 0.56593966 : Time 0.29s : 994188.23 words/s : gNorm 36.9528 : L.r. 6.7500e-07
[task 2023-12-20T16:02:49.988Z] [2023-12-20 16:02:49] Ep. 1 : Up. 37 : Sen. 75,872 : Cost 0.39948454 : Time 0.31s : 871829.53 words/s : gNorm 36.8387 : L.r. 6.9375e-07
[task 2023-12-20T16:02:50.280Z] [2023-12-20 16:02:50] Ep. 1 : Up. 38 : Sen. 78,872 : Cost 0.54701596 : Time 0.29s : 947451.61 words/s : gNorm 36.4006 : L.r. 7.1250e-07
[task 2023-12-20T16:02:50.615Z] [2023-12-20 16:02:50] Ep. 1 : Up. 39 : Sen. 80,672 : Cost 0.31074080 : Time 0.33s : 785136.96 words/s : gNorm 36.4163 : L.r. 7.3125e-07
[task 2023-12-20T16:02:50.966Z] [2023-12-20 16:02:50] Ep. 1 : Up. 40 : Sen. 84,560 : Cost 0.54136819 : Time 0.35s : 842738.79 words/s : gNorm 36.2921 : L.r. 7.5000e-07
[task 2023-12-20T16:02:51.264Z] [2023-12-20 16:02:51] Ep. 1 : Up. 41 : Sen. 86,728 : Cost 0.37497389 : Time 0.30s : 909267.82 words/s : gNorm 36.3746 : L.r. 7.6875e-07
[task 2023-12-20T16:02:51.562Z] [2023-12-20 16:02:51] Ep. 1 : Up. 42 : Sen. 88,376 : Cost 0.35949805 : Time 0.30s : 835253.49 words/s : gNorm 36.1966 : L.r. 7.8750e-07
[task 2023-12-20T16:02:51.916Z] [2023-12-20 16:02:51] Ep. 1 : Up. 43 : Sen. 90,344 : Cost 0.32054105 : Time 0.35s : 756353.18 words/s : gNorm 36.0594 : L.r. 8.0625e-07
[task 2023-12-20T16:02:52.258Z] [2023-12-20 16:02:52] Ep. 1 : Up. 44 : Sen. 91,864 : Cost 0.27476776 : Time 0.34s : 750404.34 words/s : gNorm 35.9946 : L.r. 8.2500e-07
[task 2023-12-20T16:02:52.611Z] [2023-12-20 16:02:52] Ep. 1 : Up. 45 : Sen. 94,496 : Cost 0.36605355 : Time 0.35s : 793346.19 words/s : gNorm 35.7437 : L.r. 8.4375e-07
[task 2023-12-20T16:02:52.845Z] [2023-12-20 16:02:52] [Trainer] [INFO] Reading original for epoch 119
[task 2023-12-20T16:02:52.997Z] [2023-12-20 16:02:52] Ep. 1 : Up. 46 : Sen. 96,144 : Cost 0.22041114 : Time 0.39s : 663246.74 words/s : gNorm 35.7451 : L.r. 8.6250e-07
[task 2023-12-20T16:02:53.425Z] [2023-12-20 16:02:53] Ep. 1 : Up. 47 : Sen. 97,664 : Cost 0.21787170 : Time 0.43s : 569499.33 words/s : gNorm 35.5000 : L.r. 8.8125e-07
[task 2023-12-20T16:02:53.830Z] [2023-12-20 16:02:53] Ep. 1 : Up. 48 : Sen. 99,072 : Cost 0.18394145 : Time 0.40s : 622678.91 words/s : gNorm 35.4537 : L.r. 9.0000e-07
[task 2023-12-20T16:02:54.172Z] [2023-12-20 16:02:54] Ep. 1 : Up. 49 : Sen. 100,720 : Cost 0.28881773 : Time 0.34s : 771105.80 words/s : gNorm 35.2365 : L.r. 9.1875e-07
[task 2023-12-20T16:02:54.508Z] [2023-12-20 16:02:54] Ep. 1 : Up. 50 : Sen. 102,128 : Cost 0.25611338 : Time 0.34s : 733147.02 words/s : gNorm 35.1031 : L.r. 9.3750e-07
[task 2023-12-20T16:02:54.509Z] [2023-12-20 16:02:54] Saving model weights and runtime parameters to /home/ubuntu/tasks/task_170308757707336/artifacts/model.npz
[task 2023-12-20T16:02:54.833Z] [2023-12-20 16:02:54] Saving Adam parameters
[task 2023-12-20T16:02:55.199Z] [2023-12-20 16:02:55] [training] Saving training checkpoint to /home/ubuntu/tasks/task_170308757707336/artifacts/model.npz and /home/ubuntu/tasks/task_170308757707336/artifacts/model.npz.optimizer.npz
[task 2023-12-20T16:02:56.058Z] [2023-12-20 16:02:56] [Trainer] [INFO] Reading original for epoch 120
[task 2023-12-20T16:02:56.689Z] [2023-12-20 16:02:56] Training finished
[task 2023-12-20T16:02:57.238Z] [2023-12-20 16:02:57] [valid] First sentence's tokens as scored:
[task 2023-12-20T16:02:57.238Z] [2023-12-20 16:02:57] [valid] Decoding validation set with SentencePieceVocab for scoring
[task 2023-12-20T16:02:57.239Z] [2023-12-20 16:02:57] [valid]   Hyp: � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � �
[task 2023-12-20T16:02:57.239Z] [2023-12-20 16:02:57] [valid]   Ref: T h e m a i n a w a r d - t h e " G o l d e n S h e l l " - w a s a w a r d e d t o t h e d r a m a " B e t w e e n t h e T w o W a t e r s " , d i r e c t e d b y I s a k i L a c u e s t a .
[task 2023-12-20T16:02:59.294Z] [2023-12-20 16:02:59] [Trainer] [INFO] Reading original for epoch 121
[task 2023-12-20T16:03:02.648Z] [2023-12-20 16:03:02] [Trainer] [INFO] Reading original for epoch 122
[task 2023-12-20T16:03:05.540Z] [2023-12-20 16:03:05] Saving model weights and runtime parameters to /home/ubuntu/tasks/task_170308757707336/artifacts/model.npz.best-chrf.npz
[task 2023-12-20T16:03:05.642Z] [2023-12-20 16:03:05] [valid] Ep. 1 : Up. 50 : chrf : 0.109387 : new best
[task 2023-12-20T16:03:05.946Z] [2023-12-20 16:03:05] Saving model weights and runtime parameters to /home/ubuntu/tasks/task_170308757707336/artifacts/model.npz.best-ce-mean-words.npz
[task 2023-12-20T16:03:05.974Z] [2023-12-20 16:03:05] [Trainer] [INFO] Reading original for epoch 123
[task 2023-12-20T16:03:06.047Z] [2023-12-20 16:03:06] [valid] Ep. 1 : Up. 50 : ce-mean-words : 7.42303 : new best
[task 2023-12-20T16:03:09.321Z] [2023-12-20 16:03:09] [Trainer] [INFO] Reading original for epoch 124
[task 2023-12-20T16:03:12.682Z] [2023-12-20 16:03:12] [Trainer] [INFO] Reading original for epoch 125
[task 2023-12-20T16:03:13.547Z] [2023-12-20 16:03:13] Saving model weights and runtime parameters to /home/ubuntu/tasks/task_170308757707336/artifacts/model.npz.best-bleu-detok.npz
[task 2023-12-20T16:03:13.651Z] [2023-12-20 16:03:13] [valid] Ep. 1 : Up. 50 : bleu-detok : 0 : new best
[task 2023-12-20T16:03:13.651Z] [2023-12-20 16:03:13] Saving model weights and runtime parameters to /home/ubuntu/tasks/task_170308757707336/artifacts/model.npz
[task 2023-12-20T16:03:13.889Z] [2023-12-20 16:03:13] Saving Adam parameters
[task 2023-12-20T16:03:14.300Z] [2023-12-20 16:03:14] [training] Saving training checkpoint to /home/ubuntu/tasks/task_170308757707336/artifacts/model.npz and /home/ubuntu/tasks/task_170308757707336/artifacts/model.npz.optimizer.npz
[task 2023-12-20T16:03:16.078Z] [2023-12-20 16:03:16] [Trainer] [INFO] Reading original for epoch 126
[task 2023-12-20T16:03:19.379Z] [2023-12-20 16:03:19] [Trainer] [INFO] Reading original for epoch 127
[task 2023-12-20T16:03:22.681Z] [2023-12-20 16:03:22] [Trainer] [INFO] Reading original for epoch 128
[task 2023-12-20T16:03:26.118Z] [2023-12-20 16:03:26] [Trainer] [INFO] Reading original for epoch 129
[task 2023-12-20T16:03:29.497Z] [2023-12-20 16:03:29] [Trainer] [INFO] Reading original for epoch 130
[task 2023-12-20T16:03:32.954Z] [2023-12-20 16:03:32] [Trainer] [INFO] Reading original for epoch 131
[task 2023-12-20T16:03:36.455Z] [2023-12-20 16:03:36] [Trainer] [INFO] Reading original for epoch 132
[task 2023-12-20T16:03:40.015Z] [2023-12-20 16:03:40] [Trainer] [INFO] Reading original for epoch 133
[task 2023-12-20T16:03:43.541Z] [2023-12-20 16:03:43] [Trainer] [INFO] Reading original for epoch 134
[task 2023-12-20T16:03:47.010Z] [2023-12-20 16:03:47] [Trainer] [INFO] Reading original for epoch 135
[task 2023-12-20T16:03:50.473Z] [2023-12-20 16:03:50] [Trainer] [INFO] Reading original for epoch 136
[task 2023-12-20T16:03:53.943Z] [2023-12-20 16:03:53] [Trainer] [INFO] Reading original for epoch 137
[task 2023-12-20T16:03:57.368Z] [2023-12-20 16:03:57] [Trainer] [INFO] Reading original for epoch 138
[task 2023-12-20T16:04:00.734Z] [2023-12-20 16:04:00] [Trainer] [INFO] Reading original for epoch 139
[task 2023-12-20T16:04:04.066Z] [2023-12-20 16:04:04] [Trainer] [INFO] Reading original for epoch 140
[task 2023-12-20T16:04:07.400Z] [2023-12-20 16:04:07] [Trainer] [INFO] Reading original for epoch 141
[task 2023-12-20T16:04:10.711Z] [2023-12-20 16:04:10] [Trainer] [INFO] Reading original for epoch 142
[task 2023-12-20T16:04:14.013Z] [2023-12-20 16:04:14] [Trainer] [INFO] Reading original for epoch 143
[task 2023-12-20T16:04:17.710Z] [2023-12-20 16:04:17] [Trainer] [INFO] Reading original for epoch 144
[task 2023-12-20T16:04:21.235Z] [2023-12-20 16:04:21] [Trainer] [INFO] Reading original for epoch 145
[task 2023-12-20T16:04:24.849Z] [2023-12-20 16:04:24] [Trainer] [INFO] Reading original for epoch 146
[task 2023-12-20T16:04:28.462Z] [2023-12-20 16:04:28] [Trainer] [INFO] Reading original for epoch 147
[task 2023-12-20T16:04:32.067Z] [2023-12-20 16:04:32] [Trainer] [INFO] Reading original for epoch 148
[task 2023-12-20T16:04:35.715Z] [2023-12-20 16:04:35] [Trainer] [INFO] Reading original for epoch 149
[task 2023-12-20T16:04:39.329Z] [2023-12-20 16:04:39] [Trainer] [INFO] Reading original for epoch 150
[task 2023-12-20T16:04:42.934Z] [2023-12-20 16:04:42] [Trainer] [INFO] Reading original for epoch 151
[task 2023-12-20T16:04:46.580Z] [2023-12-20 16:04:46] [Trainer] [INFO] Reading original for epoch 152
[task 2023-12-20T16:04:50.197Z] [2023-12-20 16:04:50] [Trainer] [INFO] Reading original for epoch 153
[task 2023-12-20T16:04:53.924Z] [2023-12-20 16:04:53] [Trainer] [INFO] Reading original for epoch 154
[task 2023-12-20T16:04:57.591Z] [2023-12-20 16:04:57] [Trainer] [INFO] Reading original for epoch 155
[task 2023-12-20T16:05:01.232Z] [2023-12-20 16:05:01] [Trainer] [INFO] Reading original for epoch 156
[task 2023-12-20T16:05:04.775Z] [2023-12-20 16:05:04] [Trainer] [INFO] Reading original for epoch 157
[task 2023-12-20T16:05:08.321Z] [2023-12-20 16:05:08] [Trainer] [INFO] Reading original for epoch 158
[task 2023-12-20T16:05:11.821Z] [2023-12-20 16:05:11] [Trainer] [INFO] Reading original for epoch 159
[task 2023-12-20T16:05:15.371Z] [2023-12-20 16:05:15] [Trainer] [INFO] Reading original for epoch 160
[task 2023-12-20T16:05:18.879Z] [2023-12-20 16:05:18] [Trainer] [INFO] Reading original for epoch 161
[task 2023-12-20T16:05:22.360Z] [2023-12-20 16:05:22] [Trainer] [INFO] Reading original for epoch 162
[task 2023-12-20T16:05:25.830Z] [2023-12-20 16:05:25] [Trainer] [INFO] Reading original for epoch 163
[task 2023-12-20T16:05:29.289Z] [2023-12-20 16:05:29] [Trainer] [INFO] Reading original for epoch 164
[task 2023-12-20T16:05:32.661Z] [2023-12-20 16:05:32] [Trainer] [INFO] Reading original for epoch 165
[task 2023-12-20T16:05:35.982Z] [2023-12-20 16:05:35] [Trainer] [INFO] Reading original for epoch 166
[task 2023-12-20T16:05:39.335Z] [2023-12-20 16:05:39] [Trainer] [INFO] Reading original for epoch 167
[task 2023-12-20T16:05:42.694Z] [2023-12-20 16:05:42] [Trainer] [INFO] Reading original for epoch 168
[task 2023-12-20T16:05:46.017Z] [2023-12-20 16:05:46] [Trainer] [INFO] Reading original for epoch 169
[task 2023-12-20T16:05:49.343Z] [2023-12-20 16:05:49] [Trainer] [INFO] Reading original for epoch 170
[task 2023-12-20T16:05:52.708Z] [2023-12-20 16:05:52] [Trainer] [INFO] Reading original for epoch 171
[task 2023-12-20T16:05:56.034Z] [2023-12-20 16:05:56] [Trainer] [INFO] Reading original for epoch 172
[task 2023-12-20T16:05:59.371Z] [2023-12-20 16:05:59] [Trainer] [INFO] Reading original for epoch 173
[task 2023-12-20T16:06:03.676Z] [2023-12-20 16:06:03] [Trainer] [INFO] Reading original for epoch 174
[task 2023-12-20T16:06:07.419Z] [2023-12-20 16:06:07] [Trainer] [INFO] Reading original for epoch 175
[task 2023-12-20T16:06:11.198Z] [2023-12-20 16:06:11] [Trainer] [INFO] Reading original for epoch 176
[task 2023-12-20T16:06:15.006Z] [2023-12-20 16:06:15] [Trainer] [INFO] Reading original for epoch 177
[task 2023-12-20T16:06:18.841Z] [2023-12-20 16:06:18] [Trainer] [INFO] Reading original for epoch 178
[task 2023-12-20T16:06:22.636Z] [2023-12-20 16:06:22] [Trainer] [INFO] Reading original for epoch 179
[task 2023-12-20T16:06:26.400Z] [2023-12-20 16:06:26] [Trainer] [INFO] Reading original for epoch 180
[task 2023-12-20T16:06:30.184Z] [2023-12-20 16:06:30] [Trainer] [INFO] Reading original for epoch 181
[task 2023-12-20T16:06:33.962Z] [2023-12-20 16:06:33] [Trainer] [INFO] Reading original for epoch 182
[task 2023-12-20T16:06:37.717Z] [2023-12-20 16:06:37] [Trainer] [INFO] Reading original for epoch 183
[task 2023-12-20T16:06:41.474Z] [2023-12-20 16:06:41] [Trainer] [INFO] Reading original for epoch 184
[task 2023-12-20T16:06:45.243Z] [2023-12-20 16:06:45] [Trainer] [INFO] Reading original for epoch 185
[task 2023-12-20T16:06:49.021Z] [2023-12-20 16:06:49] [Trainer] [INFO] Reading original for epoch 186
[task 2023-12-20T16:06:52.774Z] [2023-12-20 16:06:52] [Trainer] [INFO] Reading original for epoch 187
[task 2023-12-20T16:06:56.548Z] [2023-12-20 16:06:56] [Trainer] [INFO] Reading original for epoch 188
[task 2023-12-20T16:07:00.322Z] [2023-12-20 16:07:00] [Trainer] [INFO] Reading original for epoch 189
[task 2023-12-20T16:07:04.092Z] [2023-12-20 16:07:04] [Trainer] [INFO] Reading original for epoch 190
[task 2023-12-20T16:07:07.846Z] [2023-12-20 16:07:07] [Trainer] [INFO] Reading original for epoch 191
[task 2023-12-20T16:07:11.616Z] [2023-12-20 16:07:11] [Trainer] [INFO] Reading original for epoch 192
[task 2023-12-20T16:07:15.439Z] [2023-12-20 16:07:15] [Trainer] [INFO] Reading original for epoch 193
[task 2023-12-20T16:07:19.259Z] [2023-12-20 16:07:19] [Trainer] [INFO] Reading original for epoch 194
[task 2023-12-20T16:07:23.116Z] [2023-12-20 16:07:23] [Trainer] [INFO] Reading original for epoch 195
[task 2023-12-20T16:07:26.968Z] [2023-12-20 16:07:26] [Trainer] [INFO] Reading original for epoch 196
[task 2023-12-20T16:07:30.795Z] [2023-12-20 16:07:30] [Trainer] [INFO] Reading original for epoch 197
[task 2023-12-20T16:07:34.651Z] [2023-12-20 16:07:34] [Trainer] [INFO] Reading original for epoch 198
[task 2023-12-20T16:07:38.473Z] [2023-12-20 16:07:38] [Trainer] [INFO] Reading original for epoch 199
[task 2023-12-20T16:07:42.340Z] [2023-12-20 16:07:42] [Trainer] [INFO] Reading original for epoch 200
[task 2023-12-20T16:07:46.104Z] [2023-12-20 16:07:46] [Trainer] [INFO] Reading original for epoch 201
[task 2023-12-20T16:07:49.861Z] [2023-12-20 16:07:49] [Trainer] [INFO] Reading original for epoch 202
[task 2023-12-20T16:07:53.611Z] [2023-12-20 16:07:53] [Trainer] [INFO] Reading original for epoch 203
[task 2023-12-20T16:07:57.375Z] [2023-12-20 16:07:57] [Trainer] [INFO] Reading original for epoch 204
[task 2023-12-20T16:08:01.121Z] [2023-12-20 16:08:01] [Trainer] [INFO] Reading original for epoch 205
[task 2023-12-20T16:08:04.892Z] [2023-12-20 16:08:04] [Trainer] [INFO] Reading original for epoch 206
[task 2023-12-20T16:08:08.634Z] [2023-12-20 16:08:08] [Trainer] [INFO] Reading original for epoch 207
[task 2023-12-20T16:08:12.431Z] [2023-12-20 16:08:12] [Trainer] [INFO] Reading original for epoch 208
[task 2023-12-20T16:08:16.137Z] [2023-12-20 16:08:16] [Trainer] [INFO] Reading original for epoch 209
[task 2023-12-20T16:08:19.792Z] [2023-12-20 16:08:19] [Trainer] [INFO] Reading original for epoch 210
[task 2023-12-20T16:08:23.467Z] [2023-12-20 16:08:23] [Trainer] [INFO] Reading original for epoch 211
[task 2023-12-20T16:08:27.144Z] [2023-12-20 16:08:27] [Trainer] [INFO] Reading original for epoch 212
[task 2023-12-20T16:08:30.794Z] [2023-12-20 16:08:30] [Trainer] [INFO] Reading original for epoch 213
[task 2023-12-20T16:08:34.391Z] [2023-12-20 16:08:34] [Trainer] [INFO] Reading original for epoch 214
[task 2023-12-20T16:08:37.969Z] [2023-12-20 16:08:37] [Trainer] [INFO] Reading original for epoch 215
[task 2023-12-20T16:08:41.430Z] [2023-12-20 16:08:41] [Trainer] [INFO] Reading original for epoch 216
[task 2023-12-20T16:08:44.865Z] [2023-12-20 16:08:44] [Trainer] [INFO] Reading original for epoch 217
[task 2023-12-20T16:08:48.334Z] [2023-12-20 16:08:48] [Trainer] [INFO] Reading original for epoch 218
[task 2023-12-20T16:08:51.841Z] [2023-12-20 16:08:51] [Trainer] [INFO] Reading original for epoch 219
[task 2023-12-20T16:08:55.361Z] [2023-12-20 16:08:55] [Trainer] [INFO] Reading original for epoch 220
[task 2023-12-20T16:08:58.851Z] [2023-12-20 16:08:58] [Trainer] [INFO] Reading original for epoch 221
[task 2023-12-20T16:09:02.343Z] [2023-12-20 16:09:02] [Trainer] [INFO] Reading original for epoch 222
[task 2023-12-20T16:09:05.853Z] [2023-12-20 16:09:05] [Trainer] [INFO] Reading original for epoch 223
[task 2023-12-20T16:09:09.369Z] [2023-12-20 16:09:09] [Trainer] [INFO] Reading original for epoch 224
[task 2023-12-20T16:09:12.849Z] [2023-12-20 16:09:12] [Trainer] [INFO] Reading original for epoch 225
[task 2023-12-20T16:09:16.355Z] [2023-12-20 16:09:16] [Trainer] [INFO] Reading original for epoch 226
[task 2023-12-20T16:09:19.896Z] [2023-12-20 16:09:19] [Trainer] [INFO] Reading original for epoch 227
[task 2023-12-20T16:10:04.697Z] [2023-12-20 16:10:04] [Trainer] [INFO] trainer stopped reading input
[task 2023-12-20T16:10:04.787Z] + cp /home/ubuntu/tasks/task_170308757707336/artifacts/model.npz.best-chrf.npz /home/ubuntu/tasks/task_170308757707336/artifacts/final.model.npz.best-chrf.npz
[task 2023-12-20T16:10:04.826Z] + cp /home/ubuntu/tasks/task_170308757707336/artifacts/model.npz.best-chrf.npz.decoder.yml /home/ubuntu/tasks/task_170308757707336/artifacts/final.model.npz.best-chrf.npz.decoder.yml
[task 2023-12-20T16:10:04.828Z] + echo '### Model training is completed: /home/ubuntu/tasks/task_170308757707336/artifacts'
[task 2023-12-20T16:10:04.829Z] ### Model training is completed: /home/ubuntu/tasks/task_170308757707336/artifacts
[task 2023-12-20T16:10:04.829Z] + echo '###### Done: Training a model'
[task 2023-12-20T16:10:04.829Z] ###### Done: Training a model
[task 2023-12-20T16:10:04.829Z] + '[' None == None ']'
[task 2023-12-20T16:10:04.829Z] + cp /home/ubuntu/tasks/task_170308757707336/fetches/vocab.spm /home/ubuntu/tasks/task_170308757707336/artifacts
[fetches 2023-12-20T16:10:04.832Z] removing /home/ubuntu/tasks/task_170308757707336/fetches
[fetches 2023-12-20T16:10:05.182Z] finished
[taskcluster 2023-12-20T16:10:05.193Z]    Exit Code: 0
[taskcluster 2023-12-20T16:10:05.193Z]    User Time: 18m44.429668s
[taskcluster 2023-12-20T16:10:05.193Z]  Kernel Time: 1m55.476847s
[taskcluster 2023-12-20T16:10:05.193Z]    Wall Time: 16m25.112482767s
[taskcluster 2023-12-20T16:10:05.193Z]       Result: SUCCEEDED
[taskcluster 2023-12-20T16:10:05.193Z] === Task Finished ===
[taskcluster 2023-12-20T16:10:05.193Z] Task Duration: 16m25.114299409s
[taskcluster 2023-12-20T16:10:05.266Z] Uploading artifact public/build/config.opustrainer.yml from file /home/ubuntu/tasks/task_170308757707336/artifacts/config.opustrainer.yml with content encoding "gzip", mime type "application/x-yaml" and expiry 2024-03-19T15:03:53.175Z
[taskcluster 2023-12-20T16:10:05.727Z] Uploading artifact public/build/config.opustrainer.yml.state from file /home/ubuntu/tasks/task_170308757707336/artifacts/config.opustrainer.yml.state with content encoding "gzip", mime type "application/octet-stream" and expiry 2024-03-19T15:03:53.175Z
[taskcluster 2023-12-20T16:10:05.892Z] Uploading artifact public/build/devset.out from file /home/ubuntu/tasks/task_170308757707336/artifacts/devset.out with content encoding "gzip", mime type "application/octet-stream" and expiry 2024-03-19T15:03:53.175Z
[taskcluster 2023-12-20T16:10:06.066Z] Uploading artifact public/build/final.model.npz.best-chrf.npz from file /home/ubuntu/tasks/task_170308757707336/artifacts/final.model.npz.best-chrf.npz with content encoding "gzip", mime type "application/octet-stream" and expiry 2024-03-19T15:03:53.175Z
[taskcluster 2023-12-20T16:10:08.157Z] Uploading artifact public/build/final.model.npz.best-chrf.npz.decoder.yml from file /home/ubuntu/tasks/task_170308757707336/artifacts/final.model.npz.best-chrf.npz.decoder.yml with content encoding "gzip", mime type "application/x-yaml" and expiry 2024-03-19T15:03:53.175Z
[taskcluster 2023-12-20T16:10:08.320Z] Uploading artifact public/build/model.npz from file /home/ubuntu/tasks/task_170308757707336/artifacts/model.npz with content encoding "gzip", mime type "application/octet-stream" and expiry 2024-03-19T15:03:53.175Z
[taskcluster 2023-12-20T16:10:10.280Z] Uploading artifact public/build/model.npz.best-bleu-detok.npz from file /home/ubuntu/tasks/task_170308757707336/artifacts/model.npz.best-bleu-detok.npz with content encoding "gzip", mime type "application/octet-stream" and expiry 2024-03-19T15:03:53.175Z
[taskcluster 2023-12-20T16:10:12.271Z] Uploading artifact public/build/model.npz.best-bleu-detok.npz.decoder.yml from file /home/ubuntu/tasks/task_170308757707336/artifacts/model.npz.best-bleu-detok.npz.decoder.yml with content encoding "gzip", mime type "application/x-yaml" and expiry 2024-03-19T15:03:53.175Z
[taskcluster 2023-12-20T16:10:12.435Z] Uploading artifact public/build/model.npz.best-ce-mean-words.npz from file /home/ubuntu/tasks/task_170308757707336/artifacts/model.npz.best-ce-mean-words.npz with content encoding "gzip", mime type "application/octet-stream" and expiry 2024-03-19T15:03:53.175Z
[taskcluster 2023-12-20T16:10:14.237Z] Uploading artifact public/build/model.npz.best-ce-mean-words.npz.decoder.yml from file /home/ubuntu/tasks/task_170308757707336/artifacts/model.npz.best-ce-mean-words.npz.decoder.yml with content encoding "gzip", mime type "application/x-yaml" and expiry 2024-03-19T15:03:53.175Z
[taskcluster 2023-12-20T16:10:14.393Z] Uploading artifact public/build/model.npz.best-chrf.npz from file /home/ubuntu/tasks/task_170308757707336/artifacts/model.npz.best-chrf.npz with content encoding "gzip", mime type "application/octet-stream" and expiry 2024-03-19T15:03:53.175Z
[taskcluster 2023-12-20T16:10:16.222Z] Uploading artifact public/build/model.npz.best-chrf.npz.decoder.yml from file /home/ubuntu/tasks/task_170308757707336/artifacts/model.npz.best-chrf.npz.decoder.yml with content encoding "gzip", mime type "application/x-yaml" and expiry 2024-03-19T15:03:53.175Z
[taskcluster 2023-12-20T16:10:16.378Z] Uploading artifact public/build/model.npz.decoder.yml from file /home/ubuntu/tasks/task_170308757707336/artifacts/model.npz.decoder.yml with content encoding "gzip", mime type "application/x-yaml" and expiry 2024-03-19T15:03:53.175Z
[taskcluster 2023-12-20T16:10:16.540Z] Uploading artifact public/build/model.npz.optimizer.npz from file /home/ubuntu/tasks/task_170308757707336/artifacts/model.npz.optimizer.npz with content encoding "gzip", mime type "application/octet-stream" and expiry 2024-03-19T15:03:53.175Z
[taskcluster 2023-12-20T16:10:23.785Z] Uploading artifact public/build/model.npz.progress.yml from file /home/ubuntu/tasks/task_170308757707336/artifacts/model.npz.progress.yml with content encoding "gzip", mime type "application/x-yaml" and expiry 2024-03-19T15:03:53.175Z
[taskcluster 2023-12-20T16:10:23.988Z] Uploading artifact public/build/model.npz.yml from file /home/ubuntu/tasks/task_170308757707336/artifacts/model.npz.yml with content encoding "gzip", mime type "application/x-yaml" and expiry 2024-03-19T15:03:53.175Z
[taskcluster 2023-12-20T16:10:24.442Z] Uploading artifact public/build/opustrainer.log from file /home/ubuntu/tasks/task_170308757707336/artifacts/opustrainer.log with content encoding "gzip", mime type "text/plain" and expiry 2024-03-19T15:03:53.175Z
[taskcluster 2023-12-20T16:10:25.324Z] Uploading artifact public/build/train.log from file /home/ubuntu/tasks/task_170308757707336/artifacts/train.log with content encoding "gzip", mime type "text/plain" and expiry 2024-03-19T15:03:53.175Z
[taskcluster 2023-12-20T16:10:25.483Z] Uploading artifact public/build/valid.log from file /home/ubuntu/tasks/task_170308757707336/artifacts/valid.log with content encoding "gzip", mime type "text/plain" and expiry 2024-03-19T15:03:53.175Z
[taskcluster 2023-12-20T16:10:25.646Z] Uploading artifact public/build/vocab.spm from file /home/ubuntu/tasks/task_170308757707336/artifacts/vocab.spm with content encoding "gzip", mime type "application/x-source-rpm" and expiry 2024-03-19T15:03:53.175Z
[taskcluster 2023-12-20T16:10:25.801Z] [mounts] Preserving cache: Moving "/home/ubuntu/tasks/task_170308757707336/checkouts" to "/home/ubuntu/caches/a6k4jbs8QMei5FtLdWD3MA"
[taskcluster 2023-12-20T16:10:25.857Z] Uploading link artifact public/logs/live.log to artifact public/logs/live_backing.log with expiry 2024-03-19T15:03:53.175Z