[task 2024-02-28T03:23:27.453Z] [2024-02-28 03:23:27] [memory] Reserving 34 MB, device gpu1
[task 2024-02-28T03:23:27.458Z] [2024-02-28 03:23:27] [memory] Reserving 34 MB, device gpu3
[task 2024-02-28T03:23:27.768Z] [2024-02-28 03:23:27] Quantizing the model to 8-bits
[task 2024-02-28T03:23:27.768Z] [2024-02-28 03:23:27] [memory] Reserving 34 MB, device gpu0
[task 2024-02-28T03:23:27.768Z] [2024-02-28 03:23:27] Quantizing the model to 8-bits
[task 2024-02-28T03:23:27.768Z] [2024-02-28 03:23:27] Quantizing the model to 8-bits
[task 2024-02-28T03:23:27.768Z] [2024-02-28 03:23:27] [memory] Reserving 34 MB, device gpu2
[task 2024-02-28T03:23:27.768Z] [2024-02-28 03:23:27] Quantizing the model to 8-bits
[task 2024-02-28T03:23:27.768Z] [2024-02-28 03:23:27] [memory] Reserving 34 MB, device gpu1
[task 2024-02-28T03:23:27.768Z] [2024-02-28 03:23:27] [memory] Reserving 34 MB, device gpu3
[task 2024-02-28T03:23:27.771Z] [2024-02-28 03:23:27] [memory] Reserving 4 B, device gpu2
[task 2024-02-28T03:23:27.771Z] [2024-02-28 03:23:27] [memory] Reserving 4 B, device gpu0
[task 2024-02-28T03:23:27.771Z] [2024-02-28 03:23:27] [memory] Reserving 4 B, device gpu1
[task 2024-02-28T03:23:27.771Z] [2024-02-28 03:23:27] [memory] Reserving 4 B, device gpu3
[task 2024-02-28T03:23:28.161Z] [2024-02-28 03:23:28] Allocating memory for general optimizer shards
[task 2024-02-28T03:23:28.161Z] [2024-02-28 03:23:28] Parameter type float32, optimization type float32, casting types false
[task 2024-02-28T03:23:28.161Z] [2024-02-28 03:23:28] [memory] Reserving 8 MB, device gpu3
[task 2024-02-28T03:23:28.161Z] [2024-02-28 03:23:28] [memory] Reserving 8 MB, device gpu2
[task 2024-02-28T03:23:28.161Z] [2024-02-28 03:23:28] [memory] Reserving 8 MB, device gpu1
[task 2024-02-28T03:23:28.161Z] [2024-02-28 03:23:28] [memory] Reserving 8 MB, device gpu0
[task 2024-02-28T03:23:28.166Z] [2024-02-28 03:23:28] Allocating memory for Adam-specific shards
[task 2024-02-28T03:23:28.166Z] [2024-02-28 03:23:28] [memory] Reserving 17 MB, device gpu3
[task 2024-02-28T03:23:28.166Z] [2024-02-28 03:23:28] [memory] Reserving 17 MB, device gpu0
[task 2024-02-28T03:23:28.167Z] [2024-02-28 03:23:28] [memory] Reserving 17 MB, device gpu1
[task 2024-02-28T03:23:28.167Z] [2024-02-28 03:23:28] [memory] Reserving 17 MB, device gpu2
[task 2024-02-28T03:23:28.180Z] [2024-02-28 03:23:28] Ep. 1 : Up. 1 : Sen. 3,296 : Cost 0.54241848 : Time 341.73s : 530.47 words/s : gNorm 26.3176 : L.r. 1.8750e-08
[task 2024-02-28T03:23:28.860Z] [2024-02-28 03:23:28] Ep. 1 : Up. 2 : Sen. 5,728 : Cost 0.32275912 : Time 0.68s : 513329.09 words/s : gNorm 25.0529 : L.r. 3.7500e-08
[task 2024-02-28T03:23:29.191Z] [2024-02-28 03:23:29] Ep. 1 : Up. 3 : Sen. 10,147 : Cost 0.81093085 : Time 0.33s : 547009.32 words/s : gNorm 21.5916 : L.r. 5.6250e-08
[task 2024-02-28T03:23:29.439Z] [2024-02-28 03:23:29] [Trainer] [INFO] Reading original for epoch 115
[task 2024-02-28T03:23:29.975Z] [2024-02-28 03:23:29] Ep. 1 : Up. 4 : Sen. 12,755 : Cost 0.33538195 : Time 0.78s : 600880.70 words/s : gNorm 23.8704 : L.r. 7.5000e-08
[task 2024-02-28T03:23:30.583Z] [2024-02-28 03:23:30] Ep. 1 : Up. 5 : Sen. 15,363 : Cost 0.39121661 : Time 0.61s : 558332.72 words/s : gNorm 26.5554 : L.r. 9.3750e-08
[task 2024-02-28T03:23:30.904Z] [2024-02-28 03:23:30] Ep. 1 : Up. 6 : Sen. 17,795 : Cost 0.58755869 : Time 0.32s : 90589.55 words/s : gNorm 26.3403 : L.r. 1.1250e-07
[task 2024-02-28T03:23:31.355Z] [2024-02-28 03:23:31] Ep. 1 : Up. 7 : Sen. 28,707 : Cost 1.51458764 : Time 0.45s : 48331.18 words/s : gNorm 27.3855 : L.r. 1.3125e-07
[task 2024-02-28T03:23:31.778Z] [2024-02-28 03:23:31] [Trainer] [INFO] Reading original for epoch 116
[task 2024-02-28T03:23:32.066Z] [2024-02-28 03:23:32] Ep. 1 : Up. 8 : Sen. 31,747 : Cost 0.38079676 : Time 0.71s : 671272.84 words/s : gNorm 27.3568 : L.r. 1.5000e-07
[task 2024-02-28T03:23:32.789Z] [2024-02-28 03:23:32] Ep. 1 : Up. 9 : Sen. 34,179 : Cost 0.35937697 : Time 0.72s : 572047.60 words/s : gNorm 27.4596 : L.r. 1.6875e-07
[task 2024-02-28T03:23:33.445Z] [2024-02-28 03:23:33] Ep. 1 : Up. 10 : Sen. 36,611 : Cost 0.37976804 : Time 0.66s : 550267.20 words/s : gNorm 26.9389 : L.r. 1.8750e-07
[task 2024-02-28T03:23:33.961Z] [2024-02-28 03:23:33] Ep. 1 : Up. 11 : Sen. 39,907 : Cost 0.51229376 : Time 0.52s : 619674.48 words/s : gNorm 26.4935 : L.r. 2.0625e-07
[task 2024-02-28T03:23:34.091Z] [2024-02-28 03:23:34] [Trainer] [INFO] Reading original for epoch 117
[task 2024-02-28T03:23:34.288Z] [2024-02-28 03:23:34] Ep. 1 : Up. 12 : Sen. 42,947 : Cost 0.79194504 : Time 0.33s : 454299.24 words/s : gNorm 25.6776 : L.r. 2.2500e-07
[task 2024-02-28T03:23:34.863Z] [2024-02-28 03:23:34] Ep. 1 : Up. 13 : Sen. 46,547 : Cost 0.46608835 : Time 0.57s : 648503.63 words/s : gNorm 25.7590 : L.r. 2.4375e-07
[task 2024-02-28T03:23:35.372Z] [2024-02-28 03:23:35] Ep. 1 : Up. 14 : Sen. 52,547 : Cost 0.82117558 : Time 0.51s : 58926.44 words/s : gNorm 25.4581 : L.r. 2.6250e-07
[task 2024-02-28T03:23:35.955Z] [2024-02-28 03:23:35] Ep. 1 : Up. 15 : Sen. 54,979 : Cost 0.26748526 : Time 0.58s : 475824.58 words/s : gNorm 26.2616 : L.r. 2.8125e-07
[task 2024-02-28T03:23:36.437Z] [2024-02-28 03:23:36] [Trainer] [INFO] Reading original for epoch 118
[task 2024-02-28T03:23:36.473Z] [2024-02-28 03:23:36] Ep. 1 : Up. 16 : Sen. 59,983 : Cost 0.65274936 : Time 0.52s : 917251.98 words/s : gNorm 26.3040 : L.r. 3.0000e-07
[task 2024-02-28T03:23:37.032Z] [2024-02-28 03:23:37] Ep. 1 : Up. 17 : Sen. 70,895 : Cost 1.23228383 : Time 0.56s : 58586.71 words/s : gNorm 26.6416 : L.r. 3.1875e-07
[task 2024-02-28T03:23:37.360Z] [2024-02-28 03:23:37] Ep. 1 : Up. 18 : Sen. 73,503 : Cost 0.54736507 : Time 0.33s : 336766.25 words/s : gNorm 26.2256 : L.r. 3.3750e-07
[task 2024-02-28T03:23:37.668Z] [2024-02-28 03:23:37] Ep. 1 : Up. 19 : Sen. 76,044 : Cost 0.49865890 : Time 0.31s : 131934.94 words/s : gNorm 26.0507 : L.r. 3.5625e-07
[task 2024-02-28T03:23:38.198Z] [2024-02-28 03:23:38] Ep. 1 : Up. 20 : Sen. 83,820 : Cost 0.98737669 : Time 0.53s : 44082.41 words/s : gNorm 26.3864 : L.r. 3.7500e-07
[task 2024-02-28T03:23:38.783Z] [2024-02-28 03:23:38] Ep. 1 : Up. 21 : Sen. 86,252 : Cost 0.32868114 : Time 0.59s : 536467.07 words/s : gNorm 26.7536 : L.r. 3.9375e-07
[task 2024-02-28T03:23:38.865Z] [2024-02-28 03:23:38] [Trainer] [INFO] Reading original for epoch 119
[task 2024-02-28T03:23:39.408Z] [2024-02-28 03:23:39] Ep. 1 : Up. 22 : Sen. 90,188 : Cost 0.46841949 : Time 0.63s : 673675.93 words/s : gNorm 26.8839 : L.r. 4.1250e-07
[task 2024-02-28T03:23:39.718Z] [2024-02-28 03:23:39] Ep. 1 : Up. 23 : Sen. 93,319 : Cost 0.77706558 : Time 0.31s : 830136.49 words/s : gNorm 27.7201 : L.r. 4.3125e-07
[task 2024-02-28T03:23:40.406Z] [2024-02-28 03:23:40] Ep. 1 : Up. 24 : Sen. 95,751 : Cost 0.34299019 : Time 0.69s : 578554.78 words/s : gNorm 27.6749 : L.r. 4.5000e-07
[task 2024-02-28T03:23:41.166Z] [2024-02-28 03:23:41] Ep. 1 : Up. 25 : Sen. 99,351 : Cost 0.37297297 : Time 0.76s : 691493.27 words/s : gNorm 27.5715 : L.r. 4.6875e-07
[task 2024-02-28T03:23:41.167Z] [2024-02-28 03:23:41] Saving model weights and runtime parameters to /home/ubuntu/tasks/task_170909010451186/artifacts/model.npz
[task 2024-02-28T03:23:41.223Z] [2024-02-28 03:23:41] [Trainer] [INFO] Reading original for epoch 120
[task 2024-02-28T03:23:41.374Z] [2024-02-28 03:23:41] Saving Adam parameters
[task 2024-02-28T03:23:41.739Z] [2024-02-28 03:23:41] [training] Saving training checkpoint to /home/ubuntu/tasks/task_170909010451186/artifacts/model.npz and /home/ubuntu/tasks/task_170909010451186/artifacts/model.npz.optimizer.npz
[task 2024-02-28T03:23:42.735Z] [2024-02-28 03:23:42] Ep. 1 : Up. 26 : Sen. 101,783 : Cost 0.30836076 : Time 1.57s : 183615.95 words/s : gNorm 27.3930 : L.r. 4.8750e-07
[task 2024-02-28T03:23:43.419Z] [2024-02-28 03:23:43] Ep. 1 : Up. 27 : Sen. 104,391 : Cost 0.36938065 : Time 0.68s : 586027.72 words/s : gNorm 27.2634 : L.r. 5.0625e-07
[task 2024-02-28T03:23:43.608Z] [2024-02-28 03:23:43] [Trainer] [INFO] Reading original for epoch 121
[task 2024-02-28T03:23:43.988Z] [2024-02-28 03:23:43] Ep. 1 : Up. 28 : Sen. 109,655 : Cost 0.62209702 : Time 0.57s : 687920.04 words/s : gNorm 27.1203 : L.r. 5.2500e-07
[task 2024-02-28T03:23:44.456Z] [2024-02-28 03:23:44] Ep. 1 : Up. 29 : Sen. 113,771 : Cost 0.57520545 : Time 0.47s : 879723.78 words/s : gNorm 27.2197 : L.r. 5.4375e-07
[task 2024-02-28T03:23:45.236Z] [2024-02-28 03:23:45] Ep. 1 : Up. 30 : Sen. 116,379 : Cost 0.30919984 : Time 0.78s : 502817.99 words/s : gNorm 27.2659 : L.r. 5.6250e-07
[task 2024-02-28T03:23:45.551Z] [2024-02-28 03:23:45] Ep. 1 : Up. 31 : Sen. 118,811 : Cost 0.67550319 : Time 0.31s : 58540.46 words/s : gNorm 26.5457 : L.r. 5.8125e-07
[task 2024-02-28T03:23:46.115Z] [2024-02-28 03:23:46] [Trainer] [INFO] Reading original for epoch 122
[task 2024-02-28T03:23:46.133Z] [2024-02-28 03:23:46] Ep. 1 : Up. 32 : Sen. 122,107 : Cost 0.43396029 : Time 0.58s : 641185.02 words/s : gNorm 26.5811 : L.r. 6.0000e-07
[task 2024-02-28T03:23:46.859Z] [2024-02-28 03:23:46] Ep. 1 : Up. 33 : Sen. 125,147 : Cost 0.38467780 : Time 0.73s : 646443.59 words/s : gNorm 26.4008 : L.r. 6.1875e-07
[task 2024-02-28T03:23:47.489Z] [2024-02-28 03:23:47] Ep. 1 : Up. 34 : Sen. 127,755 : Cost 0.29372877 : Time 0.63s : 477705.89 words/s : gNorm 26.4142 : L.r. 6.3750e-07
[task 2024-02-28T03:23:48.038Z] [2024-02-28 03:23:48] Ep. 1 : Up. 35 : Sen. 130,187 : Cost 0.38035065 : Time 0.55s : 385193.32 words/s : gNorm 26.5497 : L.r. 6.5625e-07
[task 2024-02-28T03:23:48.515Z] [2024-02-28 03:23:48] [Trainer] [INFO] Reading original for epoch 123
[task 2024-02-28T03:23:48.750Z] [2024-02-28 03:23:48] Ep. 1 : Up. 36 : Sen. 133,227 : Cost 0.36352062 : Time 0.71s : 693977.84 words/s : gNorm 26.7238 : L.r. 6.7500e-07
[task 2024-02-28T03:23:49.384Z] [2024-02-28 03:23:49] Ep. 1 : Up. 37 : Sen. 138,491 : Cost 0.56643552 : Time 0.63s : 913524.16 words/s : gNorm 26.7543 : L.r. 6.9375e-07
[task 2024-02-28T03:23:49.991Z] [2024-02-28 03:23:49] Ep. 1 : Up. 38 : Sen. 141,787 : Cost 0.42993459 : Time 0.61s : 536426.16 words/s : gNorm 26.6379 : L.r. 7.1250e-07
[task 2024-02-28T03:23:50.453Z] [2024-02-28 03:23:50] Ep. 1 : Up. 39 : Sen. 144,219 : Cost 0.42056540 : Time 0.46s : 15786.84 words/s : gNorm 27.3340 : L.r. 7.3125e-07
[task 2024-02-28T03:23:50.846Z] [2024-02-28 03:23:50] Ep. 1 : Up. 40 : Sen. 150,219 : Cost 0.80184966 : Time 0.39s : 760264.57 words/s : gNorm 27.1203 : L.r. 7.5000e-07
[task 2024-02-28T03:23:50.960Z] [2024-02-28 03:23:50] [Trainer] [INFO] Reading original for epoch 124
[task 2024-02-28T03:23:51.421Z] [2024-02-28 03:23:51] Ep. 1 : Up. 41 : Sen. 161,131 : Cost 1.18918145 : Time 0.57s : 56963.63 words/s : gNorm 27.2269 : L.r. 7.6875e-07
[task 2024-02-28T03:23:52.151Z] [2024-02-28 03:23:52] Ep. 1 : Up. 42 : Sen. 164,171 : Cost 0.44604170 : Time 0.73s : 670220.32 words/s : gNorm 27.3105 : L.r. 7.8750e-07
[task 2024-02-28T03:23:52.567Z] [2024-02-28 03:23:52] Ep. 1 : Up. 43 : Sen. 168,107 : Cost 0.70884520 : Time 0.42s : 499363.50 words/s : gNorm 27.2083 : L.r. 8.0625e-07
[task 2024-02-28T03:23:53.078Z] [2024-02-28 03:23:53] Ep. 1 : Up. 44 : Sen. 170,765 : Cost 0.44699258 : Time 0.51s : 755143.59 words/s : gNorm 27.2904 : L.r. 8.2500e-07
[task 2024-02-28T03:23:53.432Z] [2024-02-28 03:23:53] [Trainer] [INFO] Reading original for epoch 125
[task 2024-02-28T03:23:53.855Z] [2024-02-28 03:23:53] Ep. 1 : Up. 45 : Sen. 173,581 : Cost 0.34527850 : Time 0.78s : 637379.04 words/s : gNorm 27.2897 : L.r. 8.4375e-07
[task 2024-02-28T03:23:54.221Z] [2024-02-28 03:23:54] Ep. 1 : Up. 46 : Sen. 178,876 : Cost 0.96309614 : Time 0.37s : 942567.48 words/s : gNorm 27.1439 : L.r. 8.6250e-07
[task 2024-02-28T03:23:54.978Z] [2024-02-28 03:23:54] Ep. 1 : Up. 47 : Sen. 181,916 : Cost 0.36130330 : Time 0.76s : 665091.99 words/s : gNorm 27.1253 : L.r. 8.8125e-07
[task 2024-02-28T03:23:55.337Z] [2024-02-28 03:23:55] Ep. 1 : Up. 48 : Sen. 184,732 : Cost 0.47160715 : Time 0.36s : 215497.54 words/s : gNorm 26.9563 : L.r. 9.0000e-07
[task 2024-02-28T03:23:55.772Z] [2024-02-28 03:23:55] Ep. 1 : Up. 49 : Sen. 188,028 : Cost 0.45829058 : Time 0.43s : 359854.92 words/s : gNorm 26.8827 : L.r. 9.1875e-07
[task 2024-02-28T03:23:55.920Z] [2024-02-28 03:23:55] [Trainer] [INFO] Reading original for epoch 126
[task 2024-02-28T03:23:56.181Z] [2024-02-28 03:23:56] Ep. 1 : Up. 50 : Sen. 191,324 : Cost 0.54040915 : Time 0.41s : 437040.48 words/s : gNorm 26.8997 : L.r. 9.3750e-07
[task 2024-02-28T03:23:56.181Z] [2024-02-28 03:23:56] Saving model weights and runtime parameters to /home/ubuntu/tasks/task_170909010451186/artifacts/model.npz
[task 2024-02-28T03:23:56.452Z] [2024-02-28 03:23:56] Saving Adam parameters
[task 2024-02-28T03:23:56.811Z] [2024-02-28 03:23:56] [training] Saving training checkpoint to /home/ubuntu/tasks/task_170909010451186/artifacts/model.npz and /home/ubuntu/tasks/task_170909010451186/artifacts/model.npz.optimizer.npz
[task 2024-02-28T03:23:58.114Z] [2024-02-28 03:23:58] Training finished
[task 2024-02-28T03:23:58.435Z] [2024-02-28 03:23:58] [Trainer] [INFO] Reading original for epoch 127
[task 2024-02-28T03:23:58.490Z] [2024-02-28 03:23:58] [valid] First sentence's tokens as scored:
[task 2024-02-28T03:23:58.490Z] [2024-02-28 03:23:58] [valid] Decoding validation set with SentencePieceVocab for scoring
[task 2024-02-28T03:23:58.491Z] [2024-02-28 03:23:58] [valid]   Hyp: & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & &
[task 2024-02-28T03:23:58.491Z] [2024-02-28 03:23:58] [valid]   Ref: A C C O R D I N G T O D A I L Y M A I L , H U N T ’ S C O M M E N T C O N C E R N E D T H E D I F F I C U L T I E S I N B R E X I T T A L K S .
[task 2024-02-28T03:24:00.993Z] [2024-02-28 03:24:00] [Trainer] [INFO] Reading original for epoch 128
[task 2024-02-28T03:24:03.626Z] [2024-02-28 03:24:03] [Trainer] [INFO] Reading original for epoch 129
[task 2024-02-28T03:24:06.272Z] [2024-02-28 03:24:06] [Trainer] [INFO] Reading original for epoch 130
[task 2024-02-28T03:24:08.821Z] [2024-02-28 03:24:08] [Trainer] [INFO] Reading original for epoch 131
[task 2024-02-28T03:24:11.415Z] [2024-02-28 03:24:11] [Trainer] [INFO] Reading original for epoch 132
[task 2024-02-28T03:24:14.040Z] [2024-02-28 03:24:14] [Trainer] [INFO] Reading original for epoch 133
[task 2024-02-28T03:24:15.566Z] [2024-02-28 03:24:15] Saving model weights and runtime parameters to /home/ubuntu/tasks/task_170909010451186/artifacts/model.npz.best-chrf.npz
[task 2024-02-28T03:24:15.683Z] [2024-02-28 03:24:15] [valid] Ep. 1 : Up. 50 : chrf : 0.0312806 : new best
[task 2024-02-28T03:24:16.347Z] [2024-02-28 03:24:16] Saving model weights and runtime parameters to /home/ubuntu/tasks/task_170909010451186/artifacts/model.npz.best-ce-mean-words.npz
[task 2024-02-28T03:24:16.449Z] [2024-02-28 03:24:16] [valid] Ep. 1 : Up. 50 : ce-mean-words : 7.32263 : new best
[task 2024-02-28T03:24:16.698Z] [2024-02-28 03:24:16] [Trainer] [INFO] Reading original for epoch 134
[task 2024-02-28T03:24:19.335Z] [2024-02-28 03:24:19] [Trainer] [INFO] Reading original for epoch 135
[task 2024-02-28T03:24:21.965Z] [2024-02-28 03:24:21] [Trainer] [INFO] Reading original for epoch 136
[task 2024-02-28T03:24:24.582Z] [2024-02-28 03:24:24] [Trainer] [INFO] Reading original for epoch 137
[task 2024-02-28T03:24:27.200Z] [2024-02-28 03:24:27] [Trainer] [INFO] Reading original for epoch 138
[task 2024-02-28T03:24:29.830Z] [2024-02-28 03:24:29] [Trainer] [INFO] Reading original for epoch 139
[task 2024-02-28T03:24:32.488Z] [2024-02-28 03:24:32] [Trainer] [INFO] Reading original for epoch 140
[task 2024-02-28T03:24:32.887Z] [2024-02-28 03:24:32] Saving model weights and runtime parameters to /home/ubuntu/tasks/task_170909010451186/artifacts/model.npz.best-bleu-detok.npz
[task 2024-02-28T03:24:32.989Z] [2024-02-28 03:24:32] [valid] Ep. 1 : Up. 50 : bleu-detok : 0 : new best
[task 2024-02-28T03:24:32.989Z] [2024-02-28 03:24:32] Saving model weights and runtime parameters to /home/ubuntu/tasks/task_170909010451186/artifacts/model.npz
[task 2024-02-28T03:24:33.327Z] [2024-02-28 03:24:33] Saving Adam parameters
[task 2024-02-28T03:24:33.796Z] [2024-02-28 03:24:33] [training] Saving training checkpoint to /home/ubuntu/tasks/task_170909010451186/artifacts/model.npz and /home/ubuntu/tasks/task_170909010451186/artifacts/model.npz.optimizer.npz
[task 2024-02-28T03:24:35.138Z] [2024-02-28 03:24:35] [Trainer] [INFO] Reading original for epoch 141
[task 2024-02-28T03:24:37.721Z] [2024-02-28 03:24:37] [Trainer] [INFO] Reading original for epoch 142
[task 2024-02-28T03:24:40.338Z] [2024-02-28 03:24:40] [Trainer] [INFO] Reading original for epoch 143
[task 2024-02-28T03:24:43.273Z] [2024-02-28 03:24:43] [Trainer] [INFO] Reading original for epoch 144
[task 2024-02-28T03:24:45.847Z] [2024-02-28 03:24:45] [Trainer] [INFO] Reading original for epoch 145
[task 2024-02-28T03:24:48.467Z] [2024-02-28 03:24:48] [Trainer] [INFO] Reading original for epoch 146
[task 2024-02-28T03:24:51.153Z] [2024-02-28 03:24:51] [Trainer] [INFO] Reading original for epoch 147
[task 2024-02-28T03:24:53.896Z] [2024-02-28 03:24:53] [Trainer] [INFO] Reading original for epoch 148
[task 2024-02-28T03:24:56.648Z] [2024-02-28 03:24:56] [Trainer] [INFO] Reading original for epoch 149
[task 2024-02-28T03:24:59.418Z] [2024-02-28 03:24:59] [Trainer] [INFO] Reading original for epoch 150
[task 2024-02-28T03:25:02.187Z] [2024-02-28 03:25:02] [Trainer] [INFO] Reading original for epoch 151
[task 2024-02-28T03:25:04.985Z] [2024-02-28 03:25:04] [Trainer] [INFO] Reading original for epoch 152
[task 2024-02-28T03:25:07.765Z] [2024-02-28 03:25:07] [Trainer] [INFO] Reading original for epoch 153
[task 2024-02-28T03:25:10.515Z] [2024-02-28 03:25:10] [Trainer] [INFO] Reading original for epoch 154
[task 2024-02-28T03:25:13.232Z] [2024-02-28 03:25:13] [Trainer] [INFO] Reading original for epoch 155
[task 2024-02-28T03:25:15.945Z] [2024-02-28 03:25:15] [Trainer] [INFO] Reading original for epoch 156
[task 2024-02-28T03:25:18.660Z] [2024-02-28 03:25:18] [Trainer] [INFO] Reading original for epoch 157
[task 2024-02-28T03:25:21.379Z] [2024-02-28 03:25:21] [Trainer] [INFO] Reading original for epoch 158
[task 2024-02-28T03:25:24.057Z] [2024-02-28 03:25:24] [Trainer] [INFO] Reading original for epoch 159
[task 2024-02-28T03:25:26.748Z] [2024-02-28 03:25:26] [Trainer] [INFO] Reading original for epoch 160
[task 2024-02-28T03:25:29.430Z] [2024-02-28 03:25:29] [Trainer] [INFO] Reading original for epoch 161
[task 2024-02-28T03:25:32.103Z] [2024-02-28 03:25:32] [Trainer] [INFO] Reading original for epoch 162
[task 2024-02-28T03:25:34.726Z] [2024-02-28 03:25:34] [Trainer] [INFO] Reading original for epoch 163
[task 2024-02-28T03:25:37.332Z] [2024-02-28 03:25:37] [Trainer] [INFO] Reading original for epoch 164
[task 2024-02-28T03:25:39.940Z] [2024-02-28 03:25:39] [Trainer] [INFO] Reading original for epoch 165
[task 2024-02-28T03:25:42.548Z] [2024-02-28 03:25:42] [Trainer] [INFO] Reading original for epoch 166
[task 2024-02-28T03:25:45.208Z] [2024-02-28 03:25:45] [Trainer] [INFO] Reading original for epoch 167
[task 2024-02-28T03:25:47.811Z] [2024-02-28 03:25:47] [Trainer] [INFO] Reading original for epoch 168
[task 2024-02-28T03:25:50.426Z] [2024-02-28 03:25:50] [Trainer] [INFO] Reading original for epoch 169
[task 2024-02-28T03:25:53.059Z] [2024-02-28 03:25:53] [Trainer] [INFO] Reading original for epoch 170
[task 2024-02-28T03:25:55.693Z] [2024-02-28 03:25:55] [Trainer] [INFO] Reading original for epoch 171
[task 2024-02-28T03:25:58.319Z] [2024-02-28 03:25:58] [Trainer] [INFO] Reading original for epoch 172
[task 2024-02-28T03:26:00.956Z] [2024-02-28 03:26:00] [Trainer] [INFO] Reading original for epoch 173
[task 2024-02-28T03:26:04.317Z] [2024-02-28 03:26:04] [Trainer] [INFO] Reading original for epoch 174
[task 2024-02-28T03:26:07.013Z] [2024-02-28 03:26:07] [Trainer] [INFO] Reading original for epoch 175
[task 2024-02-28T03:26:09.796Z] [2024-02-28 03:26:09] [Trainer] [INFO] Reading original for epoch 176
[task 2024-02-28T03:26:12.665Z] [2024-02-28 03:26:12] [Trainer] [INFO] Reading original for epoch 177
[task 2024-02-28T03:26:15.555Z] [2024-02-28 03:26:15] [Trainer] [INFO] Reading original for epoch 178
[task 2024-02-28T03:26:18.486Z] [2024-02-28 03:26:18] [Trainer] [INFO] Reading original for epoch 179
[task 2024-02-28T03:26:21.441Z] [2024-02-28 03:26:21] [Trainer] [INFO] Reading original for epoch 180
[task 2024-02-28T03:26:24.385Z] [2024-02-28 03:26:24] [Trainer] [INFO] Reading original for epoch 181
[task 2024-02-28T03:26:27.299Z] [2024-02-28 03:26:27] [Trainer] [INFO] Reading original for epoch 182
[task 2024-02-28T03:26:30.197Z] [2024-02-28 03:26:30] [Trainer] [INFO] Reading original for epoch 183
[task 2024-02-28T03:26:33.094Z] [2024-02-28 03:26:33] [Trainer] [INFO] Reading original for epoch 184
[task 2024-02-28T03:26:35.990Z] [2024-02-28 03:26:35] [Trainer] [INFO] Reading original for epoch 185
[task 2024-02-28T03:26:38.923Z] [2024-02-28 03:26:38] [Trainer] [INFO] Reading original for epoch 186
[task 2024-02-28T03:26:41.808Z] [2024-02-28 03:26:41] [Trainer] [INFO] Reading original for epoch 187
[task 2024-02-28T03:26:44.695Z] [2024-02-28 03:26:44] [Trainer] [INFO] Reading original for epoch 188
[task 2024-02-28T03:26:47.605Z] [2024-02-28 03:26:47] [Trainer] [INFO] Reading original for epoch 189
[task 2024-02-28T03:26:50.481Z] [2024-02-28 03:26:50] [Trainer] [INFO] Reading original for epoch 190
[task 2024-02-28T03:26:53.395Z] [2024-02-28 03:26:53] [Trainer] [INFO] Reading original for epoch 191
[task 2024-02-28T03:26:56.258Z] [2024-02-28 03:26:56] [Trainer] [INFO] Reading original for epoch 192
[task 2024-02-28T03:26:59.115Z] [2024-02-28 03:26:59] [Trainer] [INFO] Reading original for epoch 193
[task 2024-02-28T03:27:01.965Z] [2024-02-28 03:27:01] [Trainer] [INFO] Reading original for epoch 194
[task 2024-02-28T03:27:04.795Z] [2024-02-28 03:27:04] [Trainer] [INFO] Reading original for epoch 195
[task 2024-02-28T03:27:07.648Z] [2024-02-28 03:27:07] [Trainer] [INFO] Reading original for epoch 196
[task 2024-02-28T03:27:10.513Z] [2024-02-28 03:27:10] [Trainer] [INFO] Reading original for epoch 197
[task 2024-02-28T03:27:13.331Z] [2024-02-28 03:27:13] [Trainer] [INFO] Reading original for epoch 198
[task 2024-02-28T03:27:16.127Z] [2024-02-28 03:27:16] [Trainer] [INFO] Reading original for epoch 199
[task 2024-02-28T03:27:18.957Z] [2024-02-28 03:27:18] [Trainer] [INFO] Reading original for epoch 200
[task 2024-02-28T03:27:21.817Z] [2024-02-28 03:27:21] [Trainer] [INFO] Reading original for epoch 201
[task 2024-02-28T03:27:24.596Z] [2024-02-28 03:27:24] [Trainer] [INFO] Reading original for epoch 202
[task 2024-02-28T03:27:27.403Z] [2024-02-28 03:27:27] [Trainer] [INFO] Reading original for epoch 203
[task 2024-02-28T03:27:30.173Z] [2024-02-28 03:27:30] [Trainer] [INFO] Reading original for epoch 204
[task 2024-02-28T03:27:32.974Z] [2024-02-28 03:27:32] [Trainer] [INFO] Reading original for epoch 205
[task 2024-02-28T03:27:35.762Z] [2024-02-28 03:27:35] [Trainer] [INFO] Reading original for epoch 206
[task 2024-02-28T03:27:38.534Z] [2024-02-28 03:27:38] [Trainer] [INFO] Reading original for epoch 207
[task 2024-02-28T03:27:41.318Z] [2024-02-28 03:27:41] [Trainer] [INFO] Reading original for epoch 208
[task 2024-02-28T03:27:44.017Z] [2024-02-28 03:27:44] [Trainer] [INFO] Reading original for epoch 209
[task 2024-02-28T03:27:46.732Z] [2024-02-28 03:27:46] [Trainer] [INFO] Reading original for epoch 210
[task 2024-02-28T03:27:49.370Z] [2024-02-28 03:27:49] [Trainer] [INFO] Reading original for epoch 211
[task 2024-02-28T03:27:51.994Z] [2024-02-28 03:27:51] [Trainer] [INFO] Reading original for epoch 212
[task 2024-02-28T03:27:54.622Z] [2024-02-28 03:27:54] [Trainer] [INFO] Reading original for epoch 213
[task 2024-02-28T03:27:57.242Z] [2024-02-28 03:27:57] [Trainer] [INFO] Reading original for epoch 214
[task 2024-02-28T03:27:59.896Z] [2024-02-28 03:27:59] [Trainer] [INFO] Reading original for epoch 215
[task 2024-02-28T03:28:02.545Z] [2024-02-28 03:28:02] [Trainer] [INFO] Reading original for epoch 216
[task 2024-02-28T03:28:05.181Z] [2024-02-28 03:28:05] [Trainer] [INFO] Reading original for epoch 217
[task 2024-02-28T03:28:07.817Z] [2024-02-28 03:28:07] [Trainer] [INFO] Reading original for epoch 218
[task 2024-02-28T03:28:10.485Z] [2024-02-28 03:28:10] [Trainer] [INFO] Reading original for epoch 219
[task 2024-02-28T03:28:13.135Z] [2024-02-28 03:28:13] [Trainer] [INFO] Reading original for epoch 220
[task 2024-02-28T03:28:15.768Z] [2024-02-28 03:28:15] [Trainer] [INFO] Reading original for epoch 221
[task 2024-02-28T03:28:18.413Z] [2024-02-28 03:28:18] [Trainer] [INFO] Reading original for epoch 222
[task 2024-02-28T03:28:21.072Z] [2024-02-28 03:28:21] [Trainer] [INFO] Reading original for epoch 223
[task 2024-02-28T03:28:23.741Z] [2024-02-28 03:28:23] [Trainer] [INFO] Reading original for epoch 224
[task 2024-02-28T03:28:26.377Z] [2024-02-28 03:28:26] [Trainer] [INFO] Reading original for epoch 225
[task 2024-02-28T03:28:29.015Z] [2024-02-28 03:28:29] [Trainer] [INFO] Reading original for epoch 226
[task 2024-02-28T03:28:31.666Z] [2024-02-28 03:28:31] [Trainer] [INFO] Reading original for epoch 227
[task 2024-02-28T03:28:34.322Z] [2024-02-28 03:28:34] [Trainer] [INFO] Reading original for epoch 228
[task 2024-02-28T03:29:14.498Z] [2024-02-28 03:29:14] [Trainer] [INFO] trainer stopped reading input
[task 2024-02-28T03:29:14.581Z] + cp /home/ubuntu/tasks/task_170909010451186/artifacts/model.npz.best-chrf.npz /home/ubuntu/tasks/task_170909010451186/artifacts/final.model.npz.best-chrf.npz
[task 2024-02-28T03:29:14.619Z] + cp /home/ubuntu/tasks/task_170909010451186/artifacts/model.npz.best-chrf.npz.decoder.yml /home/ubuntu/tasks/task_170909010451186/artifacts/final.model.npz.best-chrf.npz.decoder.yml
[task 2024-02-28T03:29:14.621Z] + echo '### Model training is completed: /home/ubuntu/tasks/task_170909010451186/artifacts'
[task 2024-02-28T03:29:14.621Z] ### Model training is completed: /home/ubuntu/tasks/task_170909010451186/artifacts
[task 2024-02-28T03:29:14.621Z] + echo '###### Done: Training a model'
[task 2024-02-28T03:29:14.621Z] ###### Done: Training a model
[task 2024-02-28T03:29:14.621Z] + '[' None == None ']'
[task 2024-02-28T03:29:14.621Z] + cp /home/ubuntu/tasks/task_170909010451186/fetches/vocab.spm /home/ubuntu/tasks/task_170909010451186/artifacts
[fetches 2024-02-28T03:29:14.624Z] removing /home/ubuntu/tasks/task_170909010451186/fetches
[fetches 2024-02-28T03:29:14.964Z] finished
[taskcluster 2024-02-28T03:29:14.975Z]    Exit Code: 0
[taskcluster 2024-02-28T03:29:14.975Z]    User Time: 16m55.930711s
[taskcluster 2024-02-28T03:29:14.975Z]  Kernel Time: 1m32.737247s
[taskcluster 2024-02-28T03:29:14.975Z]    Wall Time: 14m9.503503723s
[taskcluster 2024-02-28T03:29:14.975Z]       Result: SUCCEEDED
[taskcluster 2024-02-28T03:29:14.975Z] === Task Finished ===
[taskcluster 2024-02-28T03:29:14.975Z] Task Duration: 14m9.505480918s
[taskcluster 2024-02-28T03:29:17.558Z] Uploading artifact public/build/config.opustrainer.yml from file /home/ubuntu/tasks/task_170909010451186/artifacts/config.opustrainer.yml with content encoding "gzip", mime type "application/x-yaml" and expiry 2024-05-28T02:23:32.960Z
[taskcluster 2024-02-28T03:29:17.803Z] Uploading artifact public/build/config.opustrainer.yml.state from file /home/ubuntu/tasks/task_170909010451186/artifacts/config.opustrainer.yml.state with content encoding "gzip", mime type "application/octet-stream" and expiry 2024-05-28T02:23:32.960Z
[taskcluster 2024-02-28T03:29:17.980Z] Uploading artifact public/build/devset.out from file /home/ubuntu/tasks/task_170909010451186/artifacts/devset.out with content encoding "gzip", mime type "application/octet-stream" and expiry 2024-05-28T02:23:32.960Z
[taskcluster 2024-02-28T03:29:18.168Z] Uploading artifact public/build/final.model.npz.best-chrf.npz from file /home/ubuntu/tasks/task_170909010451186/artifacts/final.model.npz.best-chrf.npz with content encoding "gzip", mime type "application/octet-stream" and expiry 2024-05-28T02:23:32.960Z
[taskcluster 2024-02-28T03:29:20.102Z] Uploading artifact public/build/final.model.npz.best-chrf.npz.decoder.yml from file /home/ubuntu/tasks/task_170909010451186/artifacts/final.model.npz.best-chrf.npz.decoder.yml with content encoding "gzip", mime type "application/x-yaml" and expiry 2024-05-28T02:23:32.960Z
[taskcluster 2024-02-28T03:29:20.289Z] Uploading artifact public/build/model.npz from file /home/ubuntu/tasks/task_170909010451186/artifacts/model.npz with content encoding "gzip", mime type "application/octet-stream" and expiry 2024-05-28T02:23:32.960Z
[taskcluster 2024-02-28T03:29:22.183Z] Uploading artifact public/build/model.npz.best-bleu-detok.npz from file /home/ubuntu/tasks/task_170909010451186/artifacts/model.npz.best-bleu-detok.npz with content encoding "gzip", mime type "application/octet-stream" and expiry 2024-05-28T02:23:32.960Z
[taskcluster 2024-02-28T03:29:24.053Z] Uploading artifact public/build/model.npz.best-bleu-detok.npz.decoder.yml from file /home/ubuntu/tasks/task_170909010451186/artifacts/model.npz.best-bleu-detok.npz.decoder.yml with content encoding "gzip", mime type "application/x-yaml" and expiry 2024-05-28T02:23:32.960Z
[taskcluster 2024-02-28T03:29:24.235Z] Uploading artifact public/build/model.npz.best-ce-mean-words.npz from file /home/ubuntu/tasks/task_170909010451186/artifacts/model.npz.best-ce-mean-words.npz with content encoding "gzip", mime type "application/octet-stream" and expiry 2024-05-28T02:23:32.960Z
[taskcluster 2024-02-28T03:29:26.129Z] Uploading artifact public/build/model.npz.best-ce-mean-words.npz.decoder.yml from file /home/ubuntu/tasks/task_170909010451186/artifacts/model.npz.best-ce-mean-words.npz.decoder.yml with content encoding "gzip", mime type "application/x-yaml" and expiry 2024-05-28T02:23:32.960Z
[taskcluster 2024-02-28T03:29:26.347Z] Uploading artifact public/build/model.npz.best-chrf.npz from file /home/ubuntu/tasks/task_170909010451186/artifacts/model.npz.best-chrf.npz with content encoding "gzip", mime type "application/octet-stream" and expiry 2024-05-28T02:23:32.960Z
[taskcluster 2024-02-28T03:29:28.311Z] Uploading artifact public/build/model.npz.best-chrf.npz.decoder.yml from file /home/ubuntu/tasks/task_170909010451186/artifacts/model.npz.best-chrf.npz.decoder.yml with content encoding "gzip", mime type "application/x-yaml" and expiry 2024-05-28T02:23:32.960Z
[taskcluster 2024-02-28T03:29:28.485Z] Uploading artifact public/build/model.npz.decoder.yml from file /home/ubuntu/tasks/task_170909010451186/artifacts/model.npz.decoder.yml with content encoding "gzip", mime type "application/x-yaml" and expiry 2024-05-28T02:23:32.960Z
[taskcluster 2024-02-28T03:29:28.742Z] Uploading artifact public/build/model.npz.optimizer.npz from file /home/ubuntu/tasks/task_170909010451186/artifacts/model.npz.optimizer.npz with content encoding "gzip", mime type "application/octet-stream" and expiry 2024-05-28T02:23:32.960Z
[taskcluster 2024-02-28T03:29:36.705Z] Uploading artifact public/build/model.npz.progress.yml from file /home/ubuntu/tasks/task_170909010451186/artifacts/model.npz.progress.yml with content encoding "gzip", mime type "application/x-yaml" and expiry 2024-05-28T02:23:32.960Z
[taskcluster 2024-02-28T03:29:36.898Z] Uploading artifact public/build/model.npz.yml from file /home/ubuntu/tasks/task_170909010451186/artifacts/model.npz.yml with content encoding "gzip", mime type "application/x-yaml" and expiry 2024-05-28T02:23:32.960Z
[taskcluster 2024-02-28T03:29:37.089Z] Uploading artifact public/build/opustrainer.log from file /home/ubuntu/tasks/task_170909010451186/artifacts/opustrainer.log with content encoding "gzip", mime type "text/plain" and expiry 2024-05-28T02:23:32.960Z
[taskcluster 2024-02-28T03:29:37.314Z] Uploading artifact public/build/train.log from file /home/ubuntu/tasks/task_170909010451186/artifacts/train.log with content encoding "gzip", mime type "text/plain" and expiry 2024-05-28T02:23:32.960Z
[taskcluster 2024-02-28T03:29:37.499Z] Uploading artifact public/build/valid.log from file /home/ubuntu/tasks/task_170909010451186/artifacts/valid.log with content encoding "gzip", mime type "text/plain" and expiry 2024-05-28T02:23:32.960Z
[taskcluster 2024-02-28T03:29:37.708Z] Uploading artifact public/build/vocab.spm from file /home/ubuntu/tasks/task_170909010451186/artifacts/vocab.spm with content encoding "gzip", mime type "application/x-source-rpm" and expiry 2024-05-28T02:23:32.960Z
[taskcluster 2024-02-28T03:29:37.859Z] [mounts] Preserving cache: Moving "/home/ubuntu/tasks/task_170909010451186/checkouts" to "/home/ubuntu/caches/faV2MkqfSoSbJlmhMMWkow"
[taskcluster 2024-02-28T03:29:37.919Z] Uploading link artifact public/logs/live.log to artifact public/logs/live_backing.log with expiry 2024-05-28T02:23:32.960Z