-
Notifications
You must be signed in to change notification settings - Fork 0
/
readmefirst.html
1328 lines (1278 loc) · 54.9 KB
/
readmefirst.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8" />
<meta name="generator" content="pandoc" />
<meta http-equiv="X-UA-Compatible" content="IE=EDGE" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="author" content="Dragos Bandur" />
<meta name="date" content="2021-08-17" />
<title>Update Datasets With replacer</title>
<script src="data:application/javascript;base64,Ly8gUGFuZG9jIDIuOSBhZGRzIGF0dHJpYnV0ZXMgb24gYm90aCBoZWFkZXIgYW5kIGRpdi4gV2UgcmVtb3ZlIHRoZSBmb3JtZXIgKHRvCi8vIGJlIGNvbXBhdGlibGUgd2l0aCB0aGUgYmVoYXZpb3Igb2YgUGFuZG9jIDwgMi44KS4KZG9jdW1lbnQuYWRkRXZlbnRMaXN0ZW5lcignRE9NQ29udGVudExvYWRlZCcsIGZ1bmN0aW9uKGUpIHsKICB2YXIgaHMgPSBkb2N1bWVudC5xdWVyeVNlbGVjdG9yQWxsKCJkaXYuc2VjdGlvbltjbGFzcyo9J2xldmVsJ10gPiA6Zmlyc3QtY2hpbGQiKTsKICB2YXIgaSwgaCwgYTsKICBmb3IgKGkgPSAwOyBpIDwgaHMubGVuZ3RoOyBpKyspIHsKICAgIGggPSBoc1tpXTsKICAgIGlmICghL15oWzEtNl0kL2kudGVzdChoLnRhZ05hbWUpKSBjb250aW51ZTsgIC8vIGl0IHNob3VsZCBiZSBhIGhlYWRlciBoMS1oNgogICAgYSA9IGguYXR0cmlidXRlczsKICAgIHdoaWxlIChhLmxlbmd0aCA+IDApIGgucmVtb3ZlQXR0cmlidXRlKGFbMF0ubmFtZSk7CiAgfQp9KTsK"></script>
<script src="data:application/javascript;base64,JChkb2N1bWVudCkucmVhZHkoZnVuY3Rpb24oKXsKICAgIGlmICh0eXBlb2YgJCgnW2RhdGEtdG9nZ2xlPSJ0b29sdGlwIl0nKS50b29sdGlwID09PSAnZnVuY3Rpb24nKSB7CiAgICAgICAgJCgnW2RhdGEtdG9nZ2xlPSJ0b29sdGlwIl0nKS50b29sdGlwKCk7CiAgICB9CiAgICBpZiAoJCgnW2RhdGEtdG9nZ2xlPSJwb3BvdmVyIl0nKS5wb3BvdmVyID09PSAnZnVuY3Rpb24nKSB7CiAgICAgICAgJCgnW2RhdGEtdG9nZ2xlPSJwb3BvdmVyIl0nKS5wb3BvdmVyKCk7CiAgICB9Cn0pOwo="></script>
<link href="data:text/css,%0A%2Elightable%2Dminimal%20%7B%0Aborder%2Dcollapse%3A%20separate%3B%0Aborder%2Dspacing%3A%2016px%201px%3B%0Awidth%3A%20100%25%3B%0Amargin%2Dbottom%3A%2010px%3B%0A%7D%0A%2Elightable%2Dminimal%20td%20%7B%0Amargin%2Dleft%3A%205px%3B%0Amargin%2Dright%3A%205px%3B%0A%7D%0A%2Elightable%2Dminimal%20th%20%7B%0Amargin%2Dleft%3A%205px%3B%0Amargin%2Dright%3A%205px%3B%0A%7D%0A%2Elightable%2Dminimal%20thead%20tr%3Alast%2Dchild%20th%20%7B%0Aborder%2Dbottom%3A%202px%20solid%20%2300000050%3B%0Aempty%2Dcells%3A%20hide%3B%0A%7D%0A%2Elightable%2Dminimal%20tbody%20tr%3Afirst%2Dchild%20td%20%7B%0Apadding%2Dtop%3A%200%2E5em%3B%0A%7D%0A%2Elightable%2Dminimal%2Elightable%2Dhover%20tbody%20tr%3Ahover%20%7B%0Abackground%2Dcolor%3A%20%23f5f5f5%3B%0A%7D%0A%2Elightable%2Dminimal%2Elightable%2Dstriped%20tbody%20tr%3Anth%2Dchild%28even%29%20%7B%0Abackground%2Dcolor%3A%20%23f5f5f5%3B%0A%7D%0A%2Elightable%2Dclassic%20%7B%0Aborder%2Dtop%3A%200%2E16em%20solid%20%23111111%3B%0Aborder%2Dbottom%3A%200%2E16em%20solid%20%23111111%3B%0Awidth%3A%20100%25%3B%0Amargin%2Dbottom%3A%2010px%3B%0Amargin%3A%2010px%205px%3B%0A%7D%0A%2Elightable%2Dclassic%20tfoot%20tr%20td%20%7B%0Aborder%3A%200%3B%0A%7D%0A%2Elightable%2Dclassic%20tfoot%20tr%3Afirst%2Dchild%20td%20%7B%0Aborder%2Dtop%3A%200%2E14em%20solid%20%23111111%3B%0A%7D%0A%2Elightable%2Dclassic%20caption%20%7B%0Acolor%3A%20%23222222%3B%0A%7D%0A%2Elightable%2Dclassic%20td%20%7B%0Apadding%2Dleft%3A%205px%3B%0Apadding%2Dright%3A%205px%3B%0Acolor%3A%20%23222222%3B%0A%7D%0A%2Elightable%2Dclassic%20th%20%7B%0Apadding%2Dleft%3A%205px%3B%0Apadding%2Dright%3A%205px%3B%0Afont%2Dweight%3A%20normal%3B%0Acolor%3A%20%23222222%3B%0A%7D%0A%2Elightable%2Dclassic%20thead%20tr%3Alast%2Dchild%20th%20%7B%0Aborder%2Dbottom%3A%200%2E10em%20solid%20%23111111%3B%0A%7D%0A%2Elightable%2Dclassic%2Elightable%2Dhover%20tbody%20tr%3Ahover%20%7B%0Abackground%2Dcolor%3A%20%23F9EEC1%3B%0A%7D%0A%2Elightable%2Dclassic%2Elightable%2Dstriped%20tbody%20tr%3Anth%2Dchild%28even%29%20%7B%0Abackground%2Dcolor%3A%20%23f5f5f5%3B%0A%7D%0A%2Elightable%2Dclassic%2D2%20%7B%0Aborder%2Dtop%3A%203px%20double%20%23111111%3B%0Aborder%2Dbottom%3A%203px%20double%20%23111111%3B%0Awidth%3A%20100%25%3B%0Amargin%2Dbottom%3A%2010px%3B%0A%7D%0A%2Elightable%2Dclassic%2D2%20tfoot%20tr%20td%20%7B%0Aborder%3A%200%3B%0A%7D%0A%2Elightable%2Dclassic%2D2%20tfoot%20tr%3Afirst%2Dchild%20td%20%7B%0Aborder%2Dtop%3A%203px%20double%20%23111111%3B%0A%7D%0A%2Elightable%2Dclassic%2D2%20caption%20%7B%0Acolor%3A%20%23222222%3B%0A%7D%0A%2Elightable%2Dclassic%2D2%20td%20%7B%0Apadding%2Dleft%3A%205px%3B%0Apadding%2Dright%3A%205px%3B%0Acolor%3A%20%23222222%3B%0A%7D%0A%2Elightable%2Dclassic%2D2%20th%20%7B%0Apadding%2Dleft%3A%205px%3B%0Apadding%2Dright%3A%205px%3B%0Afont%2Dweight%3A%20normal%3B%0Acolor%3A%20%23222222%3B%0A%7D%0A%2Elightable%2Dclassic%2D2%20tbody%20tr%3Alast%2Dchild%20td%20%7B%0Aborder%2Dbottom%3A%203px%20double%20%23111111%3B%0A%7D%0A%2Elightable%2Dclassic%2D2%20thead%20tr%3Alast%2Dchild%20th%20%7B%0Aborder%2Dbottom%3A%201px%20solid%20%23111111%3B%0A%7D%0A%2Elightable%2Dclassic%2D2%2Elightable%2Dhover%20tbody%20tr%3Ahover%20%7B%0Abackground%2Dcolor%3A%20%23F9EEC1%3B%0A%7D%0A%2Elightable%2Dclassic%2D2%2Elightable%2Dstriped%20tbody%20tr%3Anth%2Dchild%28even%29%20%7B%0Abackground%2Dcolor%3A%20%23f5f5f5%3B%0A%7D%0A%2Elightable%2Dmaterial%20%7B%0Amin%2Dwidth%3A%20100%25%3B%0Awhite%2Dspace%3A%20nowrap%3B%0Atable%2Dlayout%3A%20fixed%3B%0Afont%2Dfamily%3A%20Roboto%2C%20sans%2Dserif%3B%0Aborder%3A%201px%20solid%20%23EEE%3B%0Aborder%2Dcollapse%3A%20collapse%3B%0Amargin%2Dbottom%3A%2010px%3B%0A%7D%0A%2Elightable%2Dmaterial%20tfoot%20tr%20td%20%7B%0Aborder%3A%200%3B%0A%7D%0A%2Elightable%2Dmaterial%20tfoot%20tr%3Afirst%2Dchild%20td%20%7B%0Aborder%2Dtop%3A%201px%20solid%20%23EEE%3B%0A%7D%0A%2Elightable%2Dmaterial%20th%20%7B%0Aheight%3A%2056px%3B%0Apadding%2Dleft%3A%2016px%3B%0Apadding%2Dright%3A%2016px%3B%0A%7D%0A%2Elightable%2Dmaterial%20td%20%7B%0Aheight%3A%2052px%3B%0Apadding%2Dleft%3A%2016px%3B%0Apadding%2Dright%3A%2016px%3B%0Aborder%2Dtop%3A%201px%20solid%20%23eeeeee%3B%0A%7D%0A%2Elightable%2Dmaterial%2Elightable%2Dhover%20tbody%20tr%3Ahover%20%7B%0Abackground%2Dcolor%3A%20%23f5f5f5%3B%0A%7D%0A%2Elightable%2Dmaterial%2Elightable%2Dstriped%20tbody%20tr%3Anth%2Dchild%28even%29%20%7B%0Abackground%2Dcolor%3A%20%23f5f5f5%3B%0A%7D%0A%2Elightable%2Dmaterial%2Elightable%2Dstriped%20tbody%20td%20%7B%0Aborder%3A%200%3B%0A%7D%0A%2Elightable%2Dmaterial%2Elightable%2Dstriped%20thead%20tr%3Alast%2Dchild%20th%20%7B%0Aborder%2Dbottom%3A%201px%20solid%20%23ddd%3B%0A%7D%0A%2Elightable%2Dmaterial%2Ddark%20%7B%0Amin%2Dwidth%3A%20100%25%3B%0Awhite%2Dspace%3A%20nowrap%3B%0Atable%2Dlayout%3A%20fixed%3B%0Afont%2Dfamily%3A%20Roboto%2C%20sans%2Dserif%3B%0Aborder%3A%201px%20solid%20%23FFFFFF12%3B%0Aborder%2Dcollapse%3A%20collapse%3B%0Amargin%2Dbottom%3A%2010px%3B%0Abackground%2Dcolor%3A%20%23363640%3B%0A%7D%0A%2Elightable%2Dmaterial%2Ddark%20tfoot%20tr%20td%20%7B%0Aborder%3A%200%3B%0A%7D%0A%2Elightable%2Dmaterial%2Ddark%20tfoot%20tr%3Afirst%2Dchild%20td%20%7B%0Aborder%2Dtop%3A%201px%20solid%20%23FFFFFF12%3B%0A%7D%0A%2Elightable%2Dmaterial%2Ddark%20th%20%7B%0Aheight%3A%2056px%3B%0Apadding%2Dleft%3A%2016px%3B%0Apadding%2Dright%3A%2016px%3B%0Acolor%3A%20%23FFFFFF60%3B%0A%7D%0A%2Elightable%2Dmaterial%2Ddark%20td%20%7B%0Aheight%3A%2052px%3B%0Apadding%2Dleft%3A%2016px%3B%0Apadding%2Dright%3A%2016px%3B%0Acolor%3A%20%23FFFFFF%3B%0Aborder%2Dtop%3A%201px%20solid%20%23FFFFFF12%3B%0A%7D%0A%2Elightable%2Dmaterial%2Ddark%2Elightable%2Dhover%20tbody%20tr%3Ahover%20%7B%0Abackground%2Dcolor%3A%20%23FFFFFF12%3B%0A%7D%0A%2Elightable%2Dmaterial%2Ddark%2Elightable%2Dstriped%20tbody%20tr%3Anth%2Dchild%28even%29%20%7B%0Abackground%2Dcolor%3A%20%23FFFFFF12%3B%0A%7D%0A%2Elightable%2Dmaterial%2Ddark%2Elightable%2Dstriped%20tbody%20td%20%7B%0Aborder%3A%200%3B%0A%7D%0A%2Elightable%2Dmaterial%2Ddark%2Elightable%2Dstriped%20thead%20tr%3Alast%2Dchild%20th%20%7B%0Aborder%2Dbottom%3A%201px%20solid%20%23FFFFFF12%3B%0A%7D%0A%2Elightable%2Dpaper%20%7B%0Awidth%3A%20100%25%3B%0Amargin%2Dbottom%3A%2010px%3B%0Acolor%3A%20%23444%3B%0A%7D%0A%2Elightable%2Dpaper%20tfoot%20tr%20td%20%7B%0Aborder%3A%200%3B%0A%7D%0A%2Elightable%2Dpaper%20tfoot%20tr%3Afirst%2Dchild%20td%20%7B%0Aborder%2Dtop%3A%201px%20solid%20%2300000020%3B%0A%7D%0A%2Elightable%2Dpaper%20thead%20tr%3Alast%2Dchild%20th%20%7B%0Acolor%3A%20%23666%3B%0Avertical%2Dalign%3A%20bottom%3B%0Aborder%2Dbottom%3A%201px%20solid%20%2300000020%3B%0Aline%2Dheight%3A%201%2E15em%3B%0Apadding%3A%2010px%205px%3B%0A%7D%0A%2Elightable%2Dpaper%20td%20%7B%0Avertical%2Dalign%3A%20middle%3B%0Aborder%2Dbottom%3A%201px%20solid%20%2300000010%3B%0Aline%2Dheight%3A%201%2E15em%3B%0Apadding%3A%207px%205px%3B%0A%7D%0A%2Elightable%2Dpaper%2Elightable%2Dhover%20tbody%20tr%3Ahover%20%7B%0Abackground%2Dcolor%3A%20%23F9EEC1%3B%0A%7D%0A%2Elightable%2Dpaper%2Elightable%2Dstriped%20tbody%20tr%3Anth%2Dchild%28even%29%20%7B%0Abackground%2Dcolor%3A%20%2300000008%3B%0A%7D%0A%2Elightable%2Dpaper%2Elightable%2Dstriped%20tbody%20td%20%7B%0Aborder%3A%200%3B%0A%7D%0A" rel="stylesheet" />
<style type="text/css">
code{white-space: pre-wrap;}
span.smallcaps{font-variant: small-caps;}
span.underline{text-decoration: underline;}
div.column{display: inline-block; vertical-align: top; width: 50%;}
div.hanging-indent{margin-left: 1.5em; text-indent: -1.5em;}
ul.task-list{list-style: none;}
</style>
<link rel="stylesheet" href="data:text/css,body%20%7B%0Abackground%2Dcolor%3A%20%23fff%3B%0Amargin%3A%201em%20auto%3B%0Amax%2Dwidth%3A%20700px%3B%0Aoverflow%3A%20visible%3B%0Apadding%2Dleft%3A%202em%3B%0Apadding%2Dright%3A%202em%3B%0Afont%2Dfamily%3A%20%22Open%20Sans%22%2C%20%22Helvetica%20Neue%22%2C%20Helvetica%2C%20Arial%2C%20sans%2Dserif%3B%0Afont%2Dsize%3A%2014px%3B%0Aline%2Dheight%3A%201%2E35%3B%0A%7D%0A%23TOC%20%7B%0Aclear%3A%20both%3B%0Amargin%3A%200%200%2010px%2010px%3B%0Apadding%3A%204px%3B%0Awidth%3A%20400px%3B%0Aborder%3A%201px%20solid%20%23CCCCCC%3B%0Aborder%2Dradius%3A%205px%3B%0Abackground%2Dcolor%3A%20%23f6f6f6%3B%0Afont%2Dsize%3A%2013px%3B%0Aline%2Dheight%3A%201%2E3%3B%0A%7D%0A%23TOC%20%2Etoctitle%20%7B%0Afont%2Dweight%3A%20bold%3B%0Afont%2Dsize%3A%2015px%3B%0Amargin%2Dleft%3A%205px%3B%0A%7D%0A%23TOC%20ul%20%7B%0Apadding%2Dleft%3A%2040px%3B%0Amargin%2Dleft%3A%20%2D1%2E5em%3B%0Amargin%2Dtop%3A%205px%3B%0Amargin%2Dbottom%3A%205px%3B%0A%7D%0A%23TOC%20ul%20ul%20%7B%0Amargin%2Dleft%3A%20%2D2em%3B%0A%7D%0A%23TOC%20li%20%7B%0Aline%2Dheight%3A%2016px%3B%0A%7D%0Atable%20%7B%0Amargin%3A%201em%20auto%3B%0Aborder%2Dwidth%3A%201px%3B%0Aborder%2Dcolor%3A%20%23DDDDDD%3B%0Aborder%2Dstyle%3A%20outset%3B%0Aborder%2Dcollapse%3A%20collapse%3B%0A%7D%0Atable%20th%20%7B%0Aborder%2Dwidth%3A%202px%3B%0Apadding%3A%205px%3B%0Aborder%2Dstyle%3A%20inset%3B%0A%7D%0Atable%20td%20%7B%0Aborder%2Dwidth%3A%201px%3B%0Aborder%2Dstyle%3A%20inset%3B%0Aline%2Dheight%3A%2018px%3B%0Apadding%3A%205px%205px%3B%0A%7D%0Atable%2C%20table%20th%2C%20table%20td%20%7B%0Aborder%2Dleft%2Dstyle%3A%20none%3B%0Aborder%2Dright%2Dstyle%3A%20none%3B%0A%7D%0Atable%20thead%2C%20table%20tr%2Eeven%20%7B%0Abackground%2Dcolor%3A%20%23f7f7f7%3B%0A%7D%0Ap%20%7B%0Amargin%3A%200%2E5em%200%3B%0A%7D%0Ablockquote%20%7B%0Abackground%2Dcolor%3A%20%23f6f6f6%3B%0Apadding%3A%200%2E25em%200%2E75em%3B%0A%7D%0Ahr%20%7B%0Aborder%2Dstyle%3A%20solid%3B%0Aborder%3A%20none%3B%0Aborder%2Dtop%3A%201px%20solid%20%23777%3B%0Amargin%3A%2028px%200%3B%0A%7D%0Adl%20%7B%0Amargin%2Dleft%3A%200%3B%0A%7D%0Adl%20dd%20%7B%0Amargin%2Dbottom%3A%2013px%3B%0Amargin%2Dleft%3A%2013px%3B%0A%7D%0Adl%20dt%20%7B%0Afont%2Dweight%3A%20bold%3B%0A%7D%0Aul%20%7B%0Amargin%2Dtop%3A%200%3B%0A%7D%0Aul%20li%20%7B%0Alist%2Dstyle%3A%20circle%20outside%3B%0A%7D%0Aul%20ul%20%7B%0Amargin%2Dbottom%3A%200%3B%0A%7D%0Apre%2C%20code%20%7B%0Abackground%2Dcolor%3A%20%23f7f7f7%3B%0Aborder%2Dradius%3A%203px%3B%0Acolor%3A%20%23333%3B%0Awhite%2Dspace%3A%20pre%2Dwrap%3B%20%0A%7D%0Apre%20%7B%0Aborder%2Dradius%3A%203px%3B%0Amargin%3A%205px%200px%2010px%200px%3B%0Apadding%3A%2010px%3B%0A%7D%0Apre%3Anot%28%5Bclass%5D%29%20%7B%0Abackground%2Dcolor%3A%20%23f7f7f7%3B%0A%7D%0Acode%20%7B%0Afont%2Dfamily%3A%20Consolas%2C%20Monaco%2C%20%27Courier%20New%27%2C%20monospace%3B%0Afont%2Dsize%3A%2085%25%3B%0A%7D%0Ap%20%3E%20code%2C%20li%20%3E%20code%20%7B%0Apadding%3A%202px%200px%3B%0A%7D%0Adiv%2Efigure%20%7B%0Atext%2Dalign%3A%20center%3B%0A%7D%0Aimg%20%7B%0Abackground%2Dcolor%3A%20%23FFFFFF%3B%0Apadding%3A%202px%3B%0Aborder%3A%201px%20solid%20%23DDDDDD%3B%0Aborder%2Dradius%3A%203px%3B%0Aborder%3A%201px%20solid%20%23CCCCCC%3B%0Amargin%3A%200%205px%3B%0A%7D%0Ah1%20%7B%0Amargin%2Dtop%3A%200%3B%0Afont%2Dsize%3A%2035px%3B%0Aline%2Dheight%3A%2040px%3B%0A%7D%0Ah2%20%7B%0Aborder%2Dbottom%3A%204px%20solid%20%23f7f7f7%3B%0Apadding%2Dtop%3A%2010px%3B%0Apadding%2Dbottom%3A%202px%3B%0Afont%2Dsize%3A%20145%25%3B%0A%7D%0Ah3%20%7B%0Aborder%2Dbottom%3A%202px%20solid%20%23f7f7f7%3B%0Apadding%2Dtop%3A%2010px%3B%0Afont%2Dsize%3A%20120%25%3B%0A%7D%0Ah4%20%7B%0Aborder%2Dbottom%3A%201px%20solid%20%23f7f7f7%3B%0Amargin%2Dleft%3A%208px%3B%0Afont%2Dsize%3A%20105%25%3B%0A%7D%0Ah5%2C%20h6%20%7B%0Aborder%2Dbottom%3A%201px%20solid%20%23ccc%3B%0Afont%2Dsize%3A%20105%25%3B%0A%7D%0Aa%20%7B%0Acolor%3A%20%230033dd%3B%0Atext%2Ddecoration%3A%20none%3B%0A%7D%0Aa%3Ahover%20%7B%0Acolor%3A%20%236666ff%3B%20%7D%0Aa%3Avisited%20%7B%0Acolor%3A%20%23800080%3B%20%7D%0Aa%3Avisited%3Ahover%20%7B%0Acolor%3A%20%23BB00BB%3B%20%7D%0Aa%5Bhref%5E%3D%22http%3A%22%5D%20%7B%0Atext%2Ddecoration%3A%20underline%3B%20%7D%0Aa%5Bhref%5E%3D%22https%3A%22%5D%20%7B%0Atext%2Ddecoration%3A%20underline%3B%20%7D%0A%0Acode%20%3E%20span%2Ekw%20%7B%20color%3A%20%23555%3B%20font%2Dweight%3A%20bold%3B%20%7D%20%0Acode%20%3E%20span%2Edt%20%7B%20color%3A%20%23902000%3B%20%7D%20%0Acode%20%3E%20span%2Edv%20%7B%20color%3A%20%2340a070%3B%20%7D%20%0Acode%20%3E%20span%2Ebn%20%7B%20color%3A%20%23d14%3B%20%7D%20%0Acode%20%3E%20span%2Efl%20%7B%20color%3A%20%23d14%3B%20%7D%20%0Acode%20%3E%20span%2Ech%20%7B%20color%3A%20%23d14%3B%20%7D%20%0Acode%20%3E%20span%2Est%20%7B%20color%3A%20%23d14%3B%20%7D%20%0Acode%20%3E%20span%2Eco%20%7B%20color%3A%20%23888888%3B%20font%2Dstyle%3A%20italic%3B%20%7D%20%0Acode%20%3E%20span%2Eot%20%7B%20color%3A%20%23007020%3B%20%7D%20%0Acode%20%3E%20span%2Eal%20%7B%20color%3A%20%23ff0000%3B%20font%2Dweight%3A%20bold%3B%20%7D%20%0Acode%20%3E%20span%2Efu%20%7B%20color%3A%20%23900%3B%20font%2Dweight%3A%20bold%3B%20%7D%20%0Acode%20%3E%20span%2Eer%20%7B%20color%3A%20%23a61717%3B%20background%2Dcolor%3A%20%23e3d2d2%3B%20%7D%20%0A" type="text/css" />
</head>
<body>
<h1 class="title toc-ignore">Update Datasets With replacer</h1>
<h4 class="author">Dragos Bandur</h4>
<h4 class="date">2021-08-17</h4>
<div id="TOC">
<ul>
<li><a href="#introduction">Introduction</a></li>
<li><a href="#definitions">Definitions</a></li>
<li><a href="#example">Example</a></li>
<li><a href="#user-made-index-in-lookup">User-made Index In Lookup</a></li>
<li><a href="#functions-intended-for-user">Functions Intended For User</a></li>
<li><a href="#internal-workflow">Internal Workflow</a></li>
<li><a href="#screen-output-example">Screen Output Example</a></li>
<li><a href="#example-datasets">Example Datasets</a></li>
<li><a href="#session-info">Session Info</a></li>
</ul>
</div>
<div id="introduction" class="section level1">
<h1>Introduction</h1>
<p><font font_family="courier new"><code>replacer</code></font>, a value replacement utility currently based on package <a href="https://CRAN.R-project.org/package=data.table"><font family="courier"><code>data.table</code></font></a>, is intended for outside-database update of data. It requires the preparation of a <em>lookup</em> file which is a list of replacement <em>requests</em> with and, in special circumstances, without an <em>index</em> column.</p>
<p>The input and output data files are comma-separated values file format (<em>csv</em>). Various other file formats can therefore be processed upon conversion.</p>
<p>Data processing such as cleanup, file format conversion and the appendage of new data are outside the scope of this utility.</p>
<p>There could be several lookup files associated with one data file and vice versa which will be batch-processed.</p>
<p>This vignette defines the terms related to replacement and to the elements of the lookup file, and details the procedure of creating an efficient lookup file, suggesting ways to maintain it relevant. Continues by introducing the two User-intended functions for single file and batch file processing and their internal workflow. Finally, it presents a screen output result obtained with a lookup without index and describes the datasets used in examples.</p>
<div id="motivation" class="section level2">
<h2>Motivation</h2>
<p>This utility is designed for beginners to <code>R</code> working with outside-database files. It facilitates complex dataset updates with minimal prompt time.</p>
<p>The choice for <code>data.table</code> is motivated by its concise and sound code, effectiveness and efficiency in processing large and complex datasets, and its independence on packages, or <em>verse</em>s, <a href="https://www.rdocumentation.org/packages/data.table/versions/1.14.0">other than base <code>R</code> itself</a>; this utility will only install <code>data.table</code> on User’s machine if it had not been there already.</p>
<p>Familiarity with basic <code>R</code> commands is necessary. Familiarity with <code>data.table</code> is not, yet strongly recommended!</p>
</div>
</div>
<div id="definitions" class="section level1" style="margin-top:30px;">
<h1>Definitions</h1>
</div>
<p>The following terms are specific to this utility:</p>
<ul>
<li><p><strong>lookup</strong> table: a csv format data file containing at least 4 standard columns with standard column names:</p>
<ul>
<li>“<strong>vars</strong>”: names of data columns involved in value replacement</li>
<li>“<strong>oldVals</strong>”: values set for replacement found in the data file</li>
<li>“<strong>newVals</strong>”: replacement values associated with <strong>oldVals</strong></li>
<li>“<strong>id</strong>”: index containing row numbers and special values such as 0, NA (or empty)</li>
</ul></li>
<li><p><strong>simple (or 1:1) replacement</strong>: one value from <strong>newVals</strong> replaces one value from <strong>oldVals</strong></p></li>
<li><p><strong>multiple (or 1:many) replacement</strong>: one value from <strong>newVals</strong> replaces identical values from <strong>oldVals</strong></p></li>
<li><p><strong>split</strong>: in this context, recursive partition of multiple replacements into simple and multiple replacements</p></li>
<li><p><strong>generic value</strong>: restricted replacement value used for multiple replacements of missing data</p></li>
<li><p><strong>involved column</strong>: data column with the name listed in <strong>vars</strong> column of lookup</p></li>
<li><p><strong>request</strong>: implicit information containing value type, value missingness, value uniqueness, value location, replacement type, replacement value in one row spanning the standard columns of lookup</p></li>
</ul>
<p>Lookup may contain a combination of simple and multiple replacements, simple replacements only, multiple replacements only (i.e. missing only, duplicated values only, or mixed) as needed. Multiple replacements can be split as needed.</p>
<p>On special circumstances, column “<strong>id</strong>” may be absent from lookup while the presence of the other three standard columns is mandatory. Modifying standard columns’ names will result in error. Any extra columns in lookup will be automatically removed during internal conformance checks on lookup.</p>
<p>Multiple replacements apply to missing or duplicated non-missing values. While replacement values of duplicated values could be regarded as generic, they are entirely decided by the User. They are not restricted in this sense.</p>
<p>In this version, replacements of missing data are restricted to words formed with the word roots <em>MIS|mis, PRE|pre, UN|un, ABS|abs</em> and the adverbs <em>YES|yes, NO|no</em> for missing values of type character, and with a number equal to- or larger than 3 integer <strong>9</strong>s (i.e. 999, 9999 etc.) for missing values of double or integer types.</p>
<p>Short sentences beginning with any word root or adverb above may be accepted.</p>
<div id="example" class="section level1" style="margin-top:30px;">
<h1>Example</h1>
</div>
<p>Typical data replacement workflow: the initial data file and the corresponding lookup file (here named <em>“data.csv”</em> and <em>“lookup.csv”</em>) are saved in a directory, here named <em>dir</em>. Once the packaged is installed, the following commands should be typed at the <code>R</code> prompt:</p>
<pre><code>> require(replacer)
> dir = 'C:/path/to/directory'
> replaceVals(dir) ## to update the data</code></pre>
<p>Results will show on screen (Section “Screen Output Example” presents the output of a different data/lookup pair). A csv file similar to <em>Out-Data</em> is saved on <em>dir</em>.</p>
<div style="margin-bottom:430px;">
<table class="table table-striped" style="font-size: 10px; width: auto !important; float: left; margin-right: 10px;">
<caption style="font-size: initial !important;">
<em>In-Data</em>
</caption>
<thead>
<tr>
<th style="text-align:right;">
Rw
</th>
<th style="text-align:left;">
a
</th>
<th style="text-align:right;">
b
</th>
<th style="text-align:right;">
uninvlvd
</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align:right;">
1
</td>
<td style="text-align:left;">
aa
</td>
<td style="text-align:right;">
7.2
</td>
<td style="text-align:right;">
0.6
</td>
</tr>
<tr>
<td style="text-align:right;">
2
</td>
<td style="text-align:left;">
</td>
<td style="text-align:right;">
17.6
</td>
<td style="text-align:right;">
0.1
</td>
</tr>
<tr>
<td style="text-align:right;">
3
</td>
<td style="text-align:left;">
cc
</td>
<td style="text-align:right;">
1.6
</td>
<td style="text-align:right;">
1.6
</td>
</tr>
<tr>
<td style="text-align:right;">
4
</td>
<td style="text-align:left;">
dd
</td>
<td style="text-align:right;">
0.8
</td>
<td style="text-align:right;">
1.2
</td>
</tr>
<tr>
<td style="text-align:right;">
5
</td>
<td style="text-align:left;">
ee
</td>
<td style="text-align:right;">
-14.0
</td>
<td style="text-align:right;">
5.0
</td>
</tr>
<tr>
<td style="text-align:right;">
6
</td>
<td style="text-align:left;">
cc
</td>
<td style="text-align:right;">
0.1
</td>
<td style="text-align:right;">
</td>
</tr>
<tr>
<td style="text-align:right;">
7
</td>
<td style="text-align:left;">
</td>
<td style="text-align:right;">
1.6
</td>
<td style="text-align:right;">
</td>
</tr>
<tr>
<td style="text-align:right;">
8
</td>
<td style="text-align:left;">
</td>
<td style="text-align:right;">
-5.0
</td>
<td style="text-align:right;">
3.3
</td>
</tr>
<tr>
<td style="text-align:right;">
9
</td>
<td style="text-align:left;">
mm
</td>
<td style="text-align:right;">
1.6
</td>
<td style="text-align:right;">
9.6
</td>
</tr>
<tr>
<td style="text-align:right;">
10
</td>
<td style="text-align:left;">
</td>
<td style="text-align:right;">
5.2
</td>
<td style="text-align:right;">
</td>
</tr>
<tr>
<td style="text-align:right;">
11
</td>
<td style="text-align:left;">
</td>
<td style="text-align:right;">
</td>
<td style="text-align:right;">
</td>
</tr>
<tr>
<td style="text-align:right;">
12
</td>
<td style="text-align:left;">
cc
</td>
<td style="text-align:right;">
</td>
<td style="text-align:right;">
</td>
</tr>
</tbody>
</table>
<table class="table table-hover" style="font-size: 10px; width: auto !important; float: left; margin-right: 10px;">
<caption style="font-size: initial !important;">
Lookup
</caption>
<thead>
<tr>
<th style="text-align:right;">
Rw
</th>
<th style="text-align:left;">
vars
</th>
<th style="text-align:left;">
oldVals
</th>
<th style="text-align:left;">
newVals
</th>
<th style="text-align:right;">
id
</th>
<th style="text-align:left;">
source
</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align:right;">
1
</td>
<td style="text-align:left;">
a
</td>
<td style="text-align:left;">
</td>
<td style="text-align:left;">
ZZ
</td>
<td style="text-align:right;">
2
</td>
<td style="text-align:left;">
source1
</td>
</tr>
<tr>
<td style="text-align:right;">
2
</td>
<td style="text-align:left;">
a
</td>
<td style="text-align:left;">
</td>
<td style="text-align:left;">
CC
</td>
<td style="text-align:right;">
8
</td>
<td style="text-align:left;">
source2
</td>
</tr>
<tr>
<td style="text-align:right;">
3
</td>
<td style="text-align:left;">
a
</td>
<td style="text-align:left;">
cc
</td>
<td style="text-align:left;">
DD
</td>
<td style="text-align:right;">
0
</td>
<td style="text-align:left;">
source3
</td>
</tr>
<tr>
<td style="text-align:right;">
4
</td>
<td style="text-align:left;">
a
</td>
<td style="text-align:left;">
mm
</td>
<td style="text-align:left;">
UU
</td>
<td style="text-align:right;">
9
</td>
<td style="text-align:left;">
source4
</td>
</tr>
<tr>
<td style="text-align:right;">
5
</td>
<td style="text-align:left;">
a
</td>
<td style="text-align:left;">
cc
</td>
<td style="text-align:left;">
CA
</td>
<td style="text-align:right;">
12
</td>
<td style="text-align:left;">
source5
</td>
</tr>
<tr>
<td style="text-align:right;">
6
</td>
<td style="text-align:left;">
a
</td>
<td style="text-align:left;">
</td>
<td style="text-align:left;">
YES
</td>
<td style="text-align:right;">
</td>
<td style="text-align:left;">
source6
</td>
</tr>
<tr>
<td style="text-align:right;">
7
</td>
<td style="text-align:left;">
b
</td>
<td style="text-align:left;">
1.607
</td>
<td style="text-align:left;">
1.1
</td>
<td style="text-align:right;">
0
</td>
<td style="text-align:left;">
source7
</td>
</tr>
<tr>
<td style="text-align:right;">
8
</td>
<td style="text-align:left;">
b
</td>
<td style="text-align:left;">
1.607
</td>
<td style="text-align:left;">
8.8
</td>
<td style="text-align:right;">
7
</td>
<td style="text-align:left;">
source8
</td>
</tr>
<tr>
<td style="text-align:right;">
9
</td>
<td style="text-align:left;">
b
</td>
<td style="text-align:left;">
</td>
<td style="text-align:left;">
999
</td>
<td style="text-align:right;">
</td>
<td style="text-align:left;">
source9
</td>
</tr>
</tbody>
</table>
<table class="table table-striped" style="font-size: 10px; width: auto !important; float: left; margin-right: 10px;">
<caption style="font-size: initial !important;">
<em>Out-Data</em>
</caption>
<thead>
<tr>
<th style="text-align:right;">
Rw
</th>
<th style="text-align:left;">
a
</th>
<th style="text-align:right;">
b
</th>
<th style="text-align:right;">
uninvlvd
</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align:right;">
1
</td>
<td style="text-align:left;">
aa
</td>
<td style="text-align:right;">
7.2
</td>
<td style="text-align:right;">
0.6
</td>
</tr>
<tr>
<td style="text-align:right;">
2
</td>
<td style="text-align:left;">
ZZ
</td>
<td style="text-align:right;">
17.6
</td>
<td style="text-align:right;">
0.1
</td>
</tr>
<tr>
<td style="text-align:right;">
3
</td>
<td style="text-align:left;">
DD
</td>
<td style="text-align:right;">
1.1
</td>
<td style="text-align:right;">
1.6
</td>
</tr>
<tr>
<td style="text-align:right;">
4
</td>
<td style="text-align:left;">
dd
</td>
<td style="text-align:right;">
0.8
</td>
<td style="text-align:right;">
1.2
</td>
</tr>
<tr>
<td style="text-align:right;">
5
</td>
<td style="text-align:left;">
ee
</td>
<td style="text-align:right;">
-14.0
</td>
<td style="text-align:right;">
5.0
</td>
</tr>
<tr>
<td style="text-align:right;">
6
</td>
<td style="text-align:left;">
DD
</td>
<td style="text-align:right;">
0.1
</td>
<td style="text-align:right;">
</td>
</tr>
<tr>
<td style="text-align:right;">
7
</td>
<td style="text-align:left;">
YES
</td>
<td style="text-align:right;">
8.8
</td>
<td style="text-align:right;">
</td>
</tr>
<tr>
<td style="text-align:right;">
8
</td>
<td style="text-align:left;">
CC
</td>
<td style="text-align:right;">
-5.0
</td>
<td style="text-align:right;">
3.3
</td>
</tr>
<tr>
<td style="text-align:right;">
9
</td>
<td style="text-align:left;">
UU
</td>
<td style="text-align:right;">
1.1
</td>
<td style="text-align:right;">
9.6
</td>
</tr>
<tr>
<td style="text-align:right;">
10
</td>
<td style="text-align:left;">
YES
</td>
<td style="text-align:right;">
5.2
</td>
<td style="text-align:right;">
</td>
</tr>
<tr>
<td style="text-align:right;">
11
</td>
<td style="text-align:left;">
YES
</td>
<td style="text-align:right;">
999.0
</td>
<td style="text-align:right;">
</td>
</tr>
<tr>
<td style="text-align:right;">
12
</td>
<td style="text-align:left;">
CA
</td>
<td style="text-align:right;">
999.0
</td>
<td style="text-align:right;">
</td>
</tr>
</tbody>
</table>
</div>
<p>This small example covers the complete set of lookup requests with index present. Column “<strong>Rw</strong>” is here for convenience only. It should not be manually created.</p>
<p>The left-hand side table <em>In-Data</em> contains 3 columns of data. The two columns (“<strong>a</strong>” and “<strong>b</strong>”), involved in replacements, are of character respectively, of numeric data types and both contain missing, duplicates and unique values. The “<strong>uninvlvd</strong>” column in <em>In-Data</em>, is not listed in lookup’s <strong>vars</strong> therefore, not involved in replacements and remains unchanged in <em>Out-Data</em>. Non-standard column <strong>source</strong> in <em>Lookup</em> is not standard hence, is automatically removed.</p>
<p>The <em>Out-Data</em> result requires a User-made lookup file with index; the replacement requests are as follows:</p>
<ul>
<li>simple replacements: missing and non-missing values at rows 2, 7, 8, 9 and 12</li>
<li>multiple replacements:
<ul>
<li>of duplicated values (<strong>id</strong> = <strong>0</strong>) at rows 3, 6, and 9</li>
<li>of missing values (<strong>id</strong> = <strong>empty</strong>) at rows 7, 10, 11 and 12</li>
</ul></li>
<li>split replacements: for example, column “<strong>a</strong>” undergoes a split on the duplicated value “cc” present in 3 rows of <em>In-Data</em>; a potential multiple replacement of length 3 was split in a length 2 multiple replacement at rows 3 and 6 replaced by “DD” and a simple replacement at row 12 replaced by “CA.”</li>
</ul>
<div id="user-made-index-in-lookup" class="section level1" style="margin-top:30px;">
<h1>User-made Index In Lookup</h1>
</div>
<p>A User-made index may or, may not be necessary in lookup:</p>
<div id="lookup-with-user-made-index" class="section level2">
<h2>Lookup With User-made Index</h2>
<p>A User-made index is the safest data value replacement procedure; it requires little insight in the structure of the data and insures fast review of large datasets. When the full range of request types including splitting are present, the index should contain the types of values presented in the above example, in the lookup column <strong>id</strong>.</p>
<div id="zero-value-present" class="section level3">
<h3>Zero Value Present</h3>
<p>There could be several distinct sets of duplicated values in any data column. If the purpose is multiple replacements of any/all of these sets (or subsets of these sets) then, for each subset the replacement request should take only one row in lookup, having <strong>0</strong> as index value. Corresponding replacement values are at User’s discretion.</p>
</div>
<div id="empty-na-value-present" class="section level3">
<h3>Empty (NA) Value Present</h3>
<p>Same rules as above apply to multiple replacement of sets of missing values with two differences: 1) the corresponding index value is empty or NA, and 2) the generic value is restricted to a word or short sentence starting with any word root or adverb listed in “Definitions.” For generic values of numeric type, data type preservation (see <strong>Note 2</strong>) is not required (i.e. it is not required to enter 9999.0 instead of 9999 in the <strong>id</strong> row corresponding to missing values of double data type).</p>
<p>Further versions may bring more generic value customization if necessary.</p>
</div>
<div id="row-number-present" class="section level3">
<h3>Row Number Present</h3>
<p>To request simple replacements, whether having unique, duplicated or missing values in <strong>oldVals</strong> column, the index should contain the corresponding row number in the data file. Sorting lookup by <strong>vars</strong> would maintain focus on one data column at a time.</p>
<p>Standard columns <strong>oldVals</strong> and <strong>newVals</strong> in lookup, revert to the character data type upon completion since they contain combined numeric and character values. This behavior is normal.</p>
<p>In conclusion, a User-made index covers all types of replacements, including splits.</p>
</div>
</div>
<div id="lookup-without-user-made-index" class="section level2">
<h2>Lookup Without User-made Index</h2>
<p>There are special circumstances when the index is absent in lookup:</p>
<div id="index-must-be-absent-in-lookup" class="section level3">
<h3>Index <span style="color: red;">Must</span> Be Absent In Lookup</h3>
<p>There is no need for index when lookup contains <em>only</em> multiple replacement requests of missing and/or duplicated values (Section “Example Datasets”). An extension of this case regarding splitting on missing values is explained in the sub-section below.</p>
</div>
<div id="index-may-be-absent-in-lookup" class="section level3">
<h3>Index <span style="color: red;">May</span> Be Absent In Lookup</h3>
<p>This case requires a deeper insight in the structure of data file; when used standalone, helper function <code>whichDups()</code>, part of this utility, finds the duplicated values in all data columns.</p>
<p>The requirements for an effective lookup table without index are:</p>
<ul>
<li><p>rows in lookup <em>must</em> have the same succession the values set for replacement have in any involved data column</p></li>
<li><p>simple replacements: if values set for replacement within any involved data column are unique (either missing or not), simple replacements without index are possible</p></li>
<li><p>mixed simple and multiple replacements <span style="color: red;"><em>which do not undergo</em></span> splitting on any involved data column: if any involved data column contains unique, duplicated and/or missing values <em>and</em> some or all of the duplicated and/ missing values are set for multiple replacements(or not at all), replacements without index are possible</p></li>
<li><p>splitting: not implemented for duplicated values. For missing values splitting is allowed (see <strong>Note 3</strong>)</p></li>
</ul>
<p><strong>Note 1</strong> Whether the index is present or not in lookup, at the time of run the row order in the data file is identical to the row order in the data file at the completion of lookup. If necessary, an index named <em>other than id</em> could be built inside the dataset. Re-ordering the dataset by this index before run will retrieve the row order at the completion of lookup and will keep lookup relevant for future updates.</p>
<p><strong>Note 2</strong> User should preserve the data type between values set for replacement and their replacements (i.e. integer/double to integer/double etc.). If an error message similar to “<em>Item 2 is type integer but the first item is type double. Please coerce …</em>” appears on screen, it means that 1) the requirement was violated and that 2) the helper function <code>con2fcoales()</code>, part of this utility, was unable to execute the necessary conversion. In cases like this, the User should start by investigating the data and the associated lookup.</p>
<p><strong>Note 3</strong> Due to their special type, splitting on missing values <em>across</em> data columns is allowed when index is absent, i.e. if one involved data column contains one missing value and other involved columns contain missing values set (or not) for multiple replacements, splitting is allowed and the unique missing value can be replaced with a non-generic replacement value.</p>
</div>
</div>
<div id="functions-intended-for-user" class="section level1" style="margin-top:30px;">
<h1>Functions Intended For User</h1>
</div>
<p>There are two functions designed for direct use and set to display a series of messages and comments informing the User on the computational path taken and on findings along the way. While the messages can be turned off, the comments remain visible.</p>
<p>Single file replacements are processed with function <code>replaceVals()</code>. When data and lookup files are named simply <em>“data.csv”</em> and <em>“lookup.csv”</em> the function only requires the path to the directory where these files are stored, written as length <strong>1</strong> quoted character, with forward “<strong>/</strong>” or, double backward “<strong>\\</strong>” slash and without end slash.</p>
<p>At start, this function performs a series of conformance checks on lookup, excludes non-standard columns then separates the data into involved and uninvolved columns, rejoining them on exit.</p>
<p>Batch file replacements are processed with function <code>bReplace()</code>. When messages are set to <strong>TRUE</strong> it offers an almost full range of messages/comments for each data/lookup input pair and request type. During run <code>bReplace()</code> calls <code>replaceVals()</code> function as many times as data/lookup pairs are on the list, displaying a counted list of messages, comments and data for each input pair.</p>
<p>Both functions save their updated data to the directory above, in csv file format.</p>
<div id="internal-workflow" class="section level1" style="margin-top:30px;">
<h1>Internal Workflow</h1>
</div>
<p>Helper function <code>sReplace()</code> (described in Manual) is the data replacement workhorse for this utility. When called by <code>replaceVals()</code>, it firstly checks for index presence in lookup. Upon the result, the function moves along the branches of a decision tree:</p>
<div id="if-the-index-is-found-absent" class="section level2">
<h2>If The Index Is Found Absent</h2>
<p>The function starts by identifying duplicated and/or missing values within the involved columns as well as eventual splits on missing values.</p>
<p>If lookup requires multiple and simple replacements, the function separates lookup into 3 subsets: 1) of multiple replacements for duplicated values, for which it later creates an internal index, 2) multiple replacements for missing values for which an internal index is not necessary, and 3) the remainder subset containing unique values, including unique missing values, for which it also creates an internal index.</p>
<p>In case of splitting on duplicated values with no index the function stops with an error.</p>
<div id="index-for-duplicated-values" class="section level3">
<h3>Index For Duplicated Values</h3>
<p>The function creates an internal index of row numbers corresponding to all elements of distinct subsets of duplicated values found within each involved data column and loops the function <code>data.table::set()</code> for replacements.</p>
</div>
<div id="no-index-for-missing-values" class="section level3">
<h3>No Index For Missing Values</h3>
<p>As mentioned above, no index is created for multiple replacements of missing values as there is only one generic value per data column (this utility uses no information on missing data beside the data type. For a different approach on this subject the User is directed to <em>data imputation</em> literature).</p>
<p>The subset of missing values is then <em>reshaped</em>, and the columns are <em>coalesced</em> (please check <code>data.table</code> Manual for these terms) with corresponding data columns, for each generic value present in lookup.</p>
</div>
<div id="index-for-unique-values" class="section level3">
<h3>Index For Unique Values</h3>
<p>As stated above, simple replacements of unique values without User-made index are possible. Once the internal index is created, this subset is <em>reshaped</em>, <em>joined</em> with the data on index and then, the corresponding columns are <em>coalesced</em>.</p>
</div>
</div>
<div id="if-the-index-is-found-present" class="section level2">
<h2>If The Index Is Found Present</h2>
<p>The function subsets the lookup using the special index values <strong>0</strong> and/or <strong>NA</strong> (or empty). At maximum, 3 subsets are formed as above. The replacement process is similar with the process used for absent index with the difference that unique values already have the User-made index of row numbers.</p>
<p>In conclusion, whether single or batch file processing, all request types are processed in one run while the User can monitor the internal workflow.</p>
</div>
<div id="screen-output-example" class="section level1" style="margin-top:30px;">
<h1>Screen Output Example</h1>
</div>
<div style="margin-bottom:20px;">
<p>Presented below is an update of the “Chile” dataset from package <code>carData</code>.</p>
<p>Type at the <code>R</code> prompt:</p>
<pre><code>> help(Chile, package = "carData")
> require(replacer)
> dir = system.file("extdata", package = "replacer")
> replaceVals(dir, 'chile.csv', 'chile_nadup.csv', save = FALSE)</code></pre>
</div>
<div style="margin-bottom:30px;">
<pre><code>reading data from: C:/R/R-4.1.0/library/replacer/extdata ...
completed reading data ...
checking standard columns in lookup ...
starting replacements ...
found no request for 1:1 replacements ...
searching for 1:many replacements ...
found request for multiple duplicated value replacments: creating index ...
replacing multiple duplicated values ...
found request for multiple missing value replacements: replacing ...
helper function has completed!
$` updated_chile_using_chile_nadup`
region population sex age education income statusquo vote
1: North 200000 Male 65 Primary 35000 1.00820 Y
2: North 200000 Male 29 Post-secondary 7500 -1.29617 N
3: North 200000 Female 38 Primary 15000 1.23072 Y
4: North 200000 Female 49 Primary 35000 -1.03163 N
5: North 200000 Female 23 Secondary 35000 -1.10496 N
---
2696: Metro Santiago 15000 Male 42 Primary 15000 -1.26247 N
2697: Metro Santiago 15000 Female 28 Primary 15000 1.32950 Y
2698: Metro Santiago 15000 Female 44 Primary 75000 1.42045 Y
2699: Metro Santiago 15000 Male 21 Secondary 75000 0.18315 PREFERS NO ANSWER
2700: Metro Santiago 15000 Male 20 Post-secondary 35000 1.38179 Y
$` multiple_dups_repl_counts`
vars oldVals newVals education population region sex
1: education P Primary 1107 NA NA NA
2: education PS Post-secondary 462 NA NA NA
3: education S Secondary 1120 NA NA NA
4: population 175000 200000 NA 140 NA NA
5: population 25000 50000 NA 360 NA NA
6: region C Central NA NA 600 NA
7: region M Metro Santiago NA NA 100 NA
8: region N North NA NA 322 NA
9: region S South NA NA 718 NA
10: region SA City of Santiago NA NA 960 NA
11: sex F Female NA NA NA 1379
12: sex M Male NA NA NA 1321
$` multiple_NAs_repl_counts`
region population sex age education income statusquo vote
0 0 0 1 11 98 17 168 </code></pre>
</div>
<p>This update replaced the abbreviations used in the source data with full names and complete words, as well as, updated the <strong>population</strong> totals on few regions with fictitious values. For exemplification, missing values in column <strong>vote</strong> were replaced by a short sentence using the word root “<em>PRE</em>” listed above.</p>
<p>The requests were processed through a lookup table without User-made index.</p>
<p>The first output block named “<em>updated_chile_using_chile_nadup</em>” displays the head and the tail of the updated data file.</p>
<p>To view the complete updated dataset on screen, type:</p>
<pre><code>> upData = replaceVals(dir, 'chile.csv', 'chile_nadup.csv', save = FALSE)[[1]]
> View(upData)</code></pre>
<p>Next block, named “<em>multiple_dups_repl_counts</em>,” displays a count of duplicated value updates, processed on each involved column, by value and replacement value. In this example, the left-hand side of this block contains the complete cases of lookup.</p>
<p>Finally, the third block named “<em>multiple_NAs_repl_counts</em>” shows counts of missing values replaced by generic values within respective involved columns. In other cases, unrequested replacements within any of the involved columns will be commented in the screen output (see Examples <code>replaceVal()</code>).</p>
<p>All block names change upon input and request type.</p>
<p>In case of error, a first read of the displayed messages/comments should hint of where in the process the error was triggered.</p>
<div id="example-datasets" class="section level1" style="margin-top:30px;">
<h1>Example Datasets</h1>
</div>
<p>The datasets in this package cover situations encountered in practice. They contain fictitious values inserted for exemplification purpose with the exception of “<em>chile</em>” dataset, a copy of the “Chile” dataset from package <code>carData</code>; some value replacements within the associated lookup files are also fictitious.</p>
<p>The batch-file example (function <code>bReplace()</code>) includes data and associated lookup files presented below. Duplicated values and counts of missing values are shown for all these datasets. Column name indexing appears when several distinctive subsets of duplicated values are present in respective column.</p>
<div id="data-and-associated-lookup-files-with-index" class="section level2">
<h2>Data And Associated Lookup Files With Index</h2>
<div style="margin-bottom:10px;">
<ul>
<li>The “<em>data_id</em>” dataset</li>
</ul>
<div style="margin-bottom:30px;">
<table>
<tr valign="top">
<td>
<table>
<caption>
Duplicated
</caption>
<thead>
<tr>
<th style="text-align:center;">
a
</th>
<th style="text-align:center;">
c
</th>
<th style="text-align:center;">
d1
</th>
<th style="text-align:center;">
d2
</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align:center;">
cc
</td>
<td style="text-align:center;">
0.737
</td>
<td style="text-align:center;">
15
</td>
<td style="text-align:center;">
8
</td>
</tr>
</tbody>
</table>
</td>
<td>
<table>
<caption>
Missing
</caption>
<thead>
<tr>
<th style="text-align:center;">
a
</th>
<th style="text-align:center;">
b
</th>
<th style="text-align:center;">
c
</th>
<th style="text-align:center;">
d
</th>
<th style="text-align:center;">
e
</th>
<th style="text-align:center;">
f
</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align:center;">
4
</td>
<td style="text-align:center;">
2
</td>
<td style="text-align:center;">
3
</td>
<td style="text-align:center;">
2
</td>
<td style="text-align:center;">
2
</td>
<td style="text-align:center;">
3
</td>
</tr>
</tbody>
</table>
</td>
<tr>
</table>
</div>
<p>Associated lookup files:</p>
<p>“<em>lookup_id.csv</em>”: mixed simple/multiple replacements of unique, duplicated and missing data, similar to the first example. The file contains standard columns only.</p>
<p>“<em>lookup_idsimple.csv</em>”: simple replacements without <strong>0</strong> or <strong>NA</strong> values in “<strong>id</strong>.” The file contains standard columns only.</p>
<ul>
<li>The ‘<em>chile</em>’ dataset. Since these data contain large subsets of duplicated values, only a very small sample is shown below.</li>
</ul>
<div style="margin-bottom:30px;">
<table>
<tr valign="top">
<td>
<table>
<caption>
Duplicated
</caption>
<thead>
<tr>
<th style="text-align:center;">
statusquo27
</th>
<th style="text-align:center;">
vote4
</th>
<th style="text-align:center;">
income4
</th>
<th style="text-align:center;">
statusquo96
</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align:center;">
-1.11878
</td>
<td style="text-align:center;">
A
</td>
<td style="text-align:center;">