Use class translation in construction and group construction by group id instead of description

[Qrox]

Summary

SUMMARY: I18N "Use class translation in construction and group construction by group id instead of description"

Purpose of change

Use class translation in construction for translation correctness and text style check. Replace grouping by descriptions with grouping by dedicated group ids.

Describe the solution

1. Change the type of construction::pre_note to translation and update related code.
2. Add construction group type and replace construction description with group id.
3. Update construction json files using python script (see "additional context" for the full code).
   3.1 Construction descriptions are converted to ids by replacing consecutive or single non-alphabet/digit characters with a underscore, and then stripping any starting and ending underscores. The code also checks that a converted id always points to the exact same description, to avoid having conflicting ids from different descriptions (no such case is reported though).
   3.2 The ids are written back to construction objects, while the ids and the corresponding descriptions are written to the mod dir as construction group objects. The descriptions are stripped of ending period for consistency.
   3.3 Manually remove a repeated construction group (cut_grass) from CRT_EXPANSION (the script does not do this because it does not take mod dependency into account)
4. Update string extraction script and documentation.

Testing

Ran the unit test and no text style error was reported.
Tested in game and construction descriptions/pre notes were correctly translated and displayed.
Ran the string extraction script and the strings were correctly extracted.

Additional context

python code used to update construction json files

#!/usr/bin/env python3

import glob
import json
import os
import re
import subprocess
import sys


def dump_json_and_lint(content, path):
    with open(path, 'w', encoding='utf-8') as fs:
        json.dump(content, fs, indent=2)
    json_formatter_name = glob.glob(
        'tools/format/json_formatter.[ec]*')
    assert len(json_formatter_name) == 1
    subprocess.run([json_formatter_name[0], path],
                   stdout=subprocess.DEVNULL)


def main(argv):
    mod_dirs = {
        "data/core",
        "data/json",
    }

    with os.scandir("data/mods") as it:
        for entry in it:
            if entry.is_dir():
                mod_dirs.add(os.path.join("data/mods", entry.name))

    json_filename = re.compile("[^.].*\\.json")

    group_to_desc_map = dict()
    mod_to_groups_map = dict()

    for mod_dir in mod_dirs:
        mod_to_groups_map[mod_dir] = set()
        print("walking dir {}".format(mod_dir))
        for root, dirs, files in os.walk(mod_dir):
            for file in files:
                json_path = os.path.join(root, file)
                content = None
                changed = False
                if json_filename.match(file):
                    try:
                        with open(json_path, 'r', encoding='utf-8') as fs:
                            content = json.load(fs)
                    except Exception:
                        sys.stderr.write('Error parsing %r\n' % json_path)
                        raise
                if type(content) is list:
                    for j in range(len(content)):
                        obj = content[j]
                        if not (type(obj) is dict and
                                "type" in obj and obj["type"] == "construction" and
                                "description" in obj):
                            continue
                        group = re.sub(r'[^\w\d]+', '_', obj["description"].lower()).strip('_')
                        assert group not in group_to_desc_map or group_to_desc_map[group] == obj["description"]
                        group_to_desc_map[group] = obj["description"]
                        mod_to_groups_map[mod_dir].add(group)
                        new_obj = dict()
                        for k, v in obj.items():
                            if k == "description":
                                new_obj["group"] = group
                            else:
                                new_obj[k] = v
                        content[j] = new_obj
                        if not changed:
                            changed = True
                            print("updating {}".format(json_path))
                if changed:
                    dump_json_and_lint(content, json_path)

        if mod_to_groups_map[mod_dir]:
            mod_groups_json = [
                {
                    "type": "construction_group",
                    "id": group,
                    "name": re.sub(r'\.$', '', group_to_desc_map[group]),
                } for group in sorted(mod_to_groups_map[mod_dir])
            ]
            dump_json_and_lint(mod_groups_json, os.path.join(mod_dir, "construction_group.json"))


if __name__ == "__main__":
    main(sys.argv[1:])

[kevingranade]

If you put the human readable group name in the construction "group" field, can't you use just that to populate the group entries and not have a distinct json entry for them? I only ask because 1,200 lines of json and growing is nothing to sneeze at...

[Qrox]

Multiple constructions may belong to the same group, and the group name is the same for these constructions (it's only displayed once for all constructions in a group), if the names were specified directly in construction json entries, the code cannot easily decide which name to use, and people also have to edit every string when they want to edit the group name. The 1200 lines are mostly json boilerplate (type, id, braces, etc)