Update non-english documentation/use Hugo's builtin i18n system for docs

[nodiscc]

• Gitea version (or commit ref):
  • Not relevant

Description

Hi, as discussed in #6348, gitea documentation in french language (and probably others) is outdated. As most of the work happens on the english documentation (example recent patch) docs in other languages quickly fall out of date (missing or inaccurate sections) with no indication.

Hugo is used to build the documentation at https://docs.gitea.io/, and Hugo's multilingual feature can manage translations of strings and replaces missing strings with the default language string

Example:

docs/content/doc/installation/from-binary.en-us.md:

{{ i18n "install_readonly" }}

docs/themes/gitea/i18n/en-us.toml:

[install_readonly]
other = "If you don't want the web installer to be able to write the config file at all, it is also possible to make the config file read-only for the gitea user (owner/group `root:root`, mode `0660`), and set `INSTALL_LOCK = true`. In that case all database configuration details must be set beforehand in the config file, as well as the `SECRET_KEY` and `INTERNAL_TOKEN` values. See the [command line documentation]({{< relref "doc/usage/command-line.en-us.md" >}}) for information on using `gitea generate secret INTERNAL_TOKEN`."

docs/themes/gitea/i18n/fr-fr.toml:

[install_readonly]
other = "Si vous ne souhaitez pas que l'installeur puisse écrire dans le fichier de configuration, il est possible de placer le fichier de config en lecture seul pour l'utilisateur gitea (utilisateur/group `root:root`, mode `0660`), et de définir `INSTALL_LOCK = true`. Dans ce cas les détails de configuration doivent être définis d'avance dans le fichier de config, ainsi que les valeurs `SECRET_KEY` and `INTERNAL_TOKEN`. Voir la documentation sur la [ligne de commande]({{< relref "doc/usage/command-line.fr-.md" >}}) pour des informations sur l'utilisation de  `gitea generate secret INTERNAL_TOKEN`."

This change would require converting converting current .md page contents (in all languages) to .toml. I would be happy to help on french docs but I understand most languages would need to be redone...

This would fix the "page is outdated but there is no warning" problem. An (easier but less clean) alternative is manually adding a warning banner on outdated localized pages.

Edit: It would allow managing documentation from Crowdin along with the main gitea localization.

[zeripath]

I would be very happy if we had proper internationalised docs. I have one reservation though, anything that makes writing documentation harder may mean that it doesn't happen.

Is there any way to automate the conversion from a say a normal md to an internationalised toml? Then we get the benefit of easy to write documentation with internationalisation?

[lunny]

@nodiscc currently we also implemented that with some scripts and didn't use builtin i18n. Why we should convert to builtin i18n?

[nodiscc]

Regarding difficulty of writing documentation: using builtin i18n would allow managing the translations through Crowdin which looks easy to use for translators (difficulty on par with the current process, fork the repo, update markdown, sen a pull request...)

Is there any way to automate the conversion from a say a normal md to an internationalised toml?

In the current state I think the translated docs would have to be deleted, and a translation from scratch from en-US docs would have to be started (there is too much divergence).

Why we should convert to builtin i18n?

• It prevents localized pages from lagging behind the english docs
  • If an english paragraph changes, corresponding localized paragraphs are marked as untranslated and displayed in their complete form in english (instead of just staying partial, translated info without warnings).
  • If an english paragraph is added, it is also added (in english) to localized pages (instead of the page just remaining incomplete)
• It allows translators to use the same workflow as the rest of Gitea localization.

[zeripath]

@nodiscc I don't think you understood me. I'm not talking about the translated documentation - I'm talking about how you go about writing the original documentation.

Writing translatable strings in code is a non-trivial task. It involves abstracting your strings out to a label codename and keeping the variables separate until you reach the UI layer whereupon you produce a human readable string. Every time you want to change anything you need to change it in two places - one of which is semantically unrelated to the code you're editing, (tooling of course should help but go doesn't appear to have very good ide tooling for this.) Coming up with a good label for the translations is also non-trivial.

Now if you are suggesting that when we write documentation - a task which many people find difficult to remember to do, presumably because not only is its position outside of the code, but also is difficult in its own right - we have to do a similar game then that's not ideal. We need to choose a mechanism that is as easy as possible for a documenter because otherwise it won't happen.

[ghost]

Setup trans to use 2 stage pipeline

1. Google translator
2. Human translator

This cuts the work by 90% in keeping translations up to date.

You really don't need crowdin.
We used it but it was much easier with Hugo to use a basic pipeline.

Each page of the we site has a link to open the same data inside the git repo.
The user just working on their own branch and then pushes a or via the GUI.

That's all you need

[nodiscc]

Every time you want to change anything you need to change it in two places - one of which is semantically unrelated to the code you're editing,

@zeripath You are right. My solution is not ideal.

@gedw99 How would that work?

Setup trans to use 2 stage pipeline

I imagine you meant Travis CI. Travis would detect in a Pull Request that it affects the documentation, run a machine translation for all languages on the diff, and attach the output to the PR comments? Then the output would have to be reviewed manually by a translator for each language, and added to the Pull Request? This looks tedious too.

[lunny]

@gedw99 right. Good idea. We could also add automatically translations on some step of drone before uploading it to crowdin. That should reduce many translation operations. Anyone know any drone plugin could do that?

[ghost]

You can use Ci but you will need to code up some basic stuff too.

2. You MUST have a cache that is file based, not DB based.
   File based allows the git and CI system to be able to see changes and drive the notifications.

• This is so you get to run the whole trans process and if the cache has a record you use that, and if not you hit google tran.
• likewise at the human stage, if the user finds a mistake in the google trans, then when they fix it, the system must update the cache.

3. Setup google tran

• https://github.com/bregydoc/gtranslate

4. Setup a email notification for human so they knwo when work is needed

5. Now the Human stage

• If they correct a machine trans, it must update the cache !
• I think Crowdin can do that. But this is why i said its easier to just put up a basic web site and do it yourself with email notifications prompting the humans.

Hope this is making some sense. It pretty simple really.

I have been using this. It solves the i10n stuff like dates, currencies, etc etc etc.
Its uses ARB, which i really like as its very concise. Other formats are creaking old and highly verbose.
The way Plurals work with ARB is very concise for example.
This use protobufs to handle that messy stuff and codegens the golang and Dart code for everything needed at runtime. HUGO can also use this !!!
https://github.com/empirefox/protoc-gen-dart-ext/blob/master/Makefile

[joeblew99]

@lunny

I have setup a repo here to do it.
https://github.com/joeblew99/gitea-trans

I need to add a few more bits and docs to show how it works

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs during the next 2 weeks. Thank you for your contributions.

[stale]

This issue has been automatically closed because of inactivity. You can re-open it if needed.

[nodiscc]

This looks like something hard to implement correctly, so the issue might as well be closed for the moment.

I still recommend that you include a short banner at the beginning of every translated page. Some of them were badly out of date, and it will confuse users trying to follow old setup procedures, trying to find info on recent features etc

Cheers