Using Weblate for translations

[ann0see]

^{Originally posted by ignotus666 July 26, 2022}
@jamulussoftware/maindevelopers @jamulussoftware/translators

In #2675 the use of Weblate for translations was proposed. It basically means moving the translation process to an online tool where translators can login, do their work, and need not worry about using GitHub, PRs, forks, etc. as Weblate can be configured to take care of that, removing what is often a big barrier. If you’ve never used GH before and are not technologically minded, just reading through the TRANSLATING.md file is enough to make you faint. With Weblate, that document could practically be reduced to “Log in to Weblate, click on a doc and translate it. The End”. If for whatever reason a translator prefers to continue using the current workflow, there is nothing stopping them – no one would be forced to use Weblate.

So far the response by translators to the proposal has been positive. However, such a move would involve some changes on the admin side, mainly:

• PRs would arrive by doc, not by author/language. For the app, this would mean all translations would go into the same PR (unless they are merged as they arrive, though this could still mean more than one language in the same PR).
• Language reviews should preferably happen on Weblate. They could theoretically take place on the PR here, but it'd be better if the language could be corrected before getting to that stage - and less complicated for translators.
• As @BLumia commented:

we'll need to pay more attention to the source strings that contains arguments (e.g. "About %1") and make use of the 2nd disambiguation argument in QObject::tr() to tell translator what's the argument means, since unlike using QtLinguist with full source code fetched, the translation platform won't show the related source code in the translation page.

• Preferably, the .nsi files should also be included, but that involves creating .po files for them. The question is, where should these .po files live and what process do we follow to reconvert them into .nsi files? There is already the po4a script 'infrastructure' for doing these conversions on the website repo. Could they live there and we push the converted files from there to the main repo?
• … possibly other stuff I can't think of now.

All in all, this would vastly reduce complexity for (in particular new) translators, but would require some changes, mainly on the main repo. For the website repo, aside from how PRs are submitted, it would be quite seamless. An option could be to first use it for the website docs and over time include the app stuff (and meanwhile explore in more depth the best way to go about it).

BTW: In order to qualify for free hosting, one the requirements is that the main devs give the go-ahead to use Weblate. I'm not sure how that happens but saying so on this thread (so it can be later linked back to) might suffice. Once the project has been created, we have 14 days to actually set it up - and once that's done we submit a request for libre hosting.

Anyway - please share your thoughts, questions, etc.

[ann0see]

There is already the po4a script 'infrastructure' for doing these conversions on the website repo. Could they live there and we push the converted files from there to the main repo?

Not a fan of mixing App related stuff in the website repo. We could probably re use it, but probably it would be better to have them at a shared place. Other repo, …

[ann0see]

BTW: In order to qualify for free hosting, one the requirements is that the main devs give the go-ahead to use Weblate.

I‘d certainly be ok with that even though it might mean a change in the workflow and maybe the release process.

[ignotus666]

Ok, so I've gone ahead and created a Jamulus project on Weblate here. The app files have not been added, only the website files are there for now. It will be in trial mode for two weeks; we have within that period to request libre hosting, but before I do so I'd like to get the thumbs-up from all the main devs if possible. These are the requirements (I don't think we have any problem meeting them):

Translated content has to be released under a license approved by OSI or recognized as libre by FSF.
All the source code has to be publicly available in a supported version control system.
Translations have to be in the same repository as the source code, or under the same project/organization as the source code repository.
Using Weblate has been approved by the upstream project.
Mention you use Weblate for translations in the README. It’ll attract new translators and also help Weblate to be free for more projects.
The project has had active development for at least three months.

Also: we have to enable a webhook on the repo so Weblate can keep files updated. I can do that but just so you know. Some READMEs describing the translation process would also need updating (pretty much simplified, I hope).

[ann0see]

@jamulussoftware/maindevelopers vould you please approve/comment on the use of weblate? They need our OK.

[pljones]

Oooh, it'll make reviewing things easier if the reviewer is allowed to mark things like the {% include_relative Include-Shared-Commands.md %} strings as not requiring translation. Then the (real) untranslated bits will stand out.

(Interestingly, the default is that Dismiss failing check is granted to the Translate role.)

Can the project be "Protected" whilst in "demo mode"?

I guess it'll take a little readjustment for those familiar with the existing translations process to get acquainted with the new UX but it looks pretty nice to me (as I won't have to start installing a bunch of tools to make reviews easy).

(That's a "Yes, subject to it being as good as it initially looks.")

[ignotus666]

I don't think strings can be marked as "non-translatable" in .po files other than via a comment that has to be added to the relevant string (it can be done to any language in Weblate and the comment is visible in all other languages). In any event, I don't think this has proven to be an issue up until now - translators have generally managed to identify strings that do not require translation.

The checks are informative and do not actually block a translated string from getting published. They can be customised by adding/removing them from a list or by creating your own. I haven't really explored them that much but I think it can be potentially interesting.

Can the project be "Protected" whilst in "demo mode"?

I've locked it while things get set up. Hadn't found that before ;)

I think the main advantage is the removal of all the GitHub kerfuffle for translators. The UX is like Poedit on steroids but fairly easy to get used to. A thing to watch out for I think is how easily (or not) conflicts arise and how to pace merges of PRs coming in from Weblate. It kind of encourages a "rolling release" model where translators log in and work away at stuff at their own pace - unless we lock it in between freezes.

There was talk recently about a website update at some point in the near future - it would be a good chance to test it out.

[trebmuh]

Nice to read. Can't wait for the repo to be unlocked for translation!

[hoffie]

@ignotus666 Thanks for leading this effort! I'm OK with it, so YES for Weblate from me as well.

Personally, I hadn't heard about Weblate before. I'm always a bit skeptical about introducing more reliance on "free" services (we already rely heavily on Github, of course), but I guess it's fine in this case as other notable Open Source projects do rely on Weblate as well. It also seems that multiple active translators do like that change so I guess it's mainly their votes which should be heard! :)

If for whatever reason a translator prefers to continue using the current workflow, there is nothing stopping them – no one would be forced to use Weblate.

Such a fall back or secondary path is always good to have. :)

[jujudusud]

It is also ok for me to use weblate.
Just go ahead :-)

[melcon]

This seems to be my cue to start a pt_BR for website as well :) Looking forward for it
BTW, I'm already watching the project there. Just waiting for the notification get started.

[ignotus666]

There's just a couple of things that need doing first (didn't have time today...) and then we should be good to go.

@melcon that would be great! Also to test how adding a new language works out with weblate - make sure you open an issue first as explained here. Just open the issue and disregard everything about forking and PRs as that shouldn't be needed any more... hopefully!

[ann0see]

@ignotus666 I think the only problem I see with the PR from Weblate is that our E-Mails are exposed now. I think that should change - we might want to open an issue on the weblate repo.

[ann0see]

Actually, there's an issue on the weblate Repo: WeblateOrg/weblate#6508

[ignotus666]

Yes. While that gets addressed by weblate, maybe we could have a script that deletes that line or replaces it leaving the email empty. It'll still be there in the commit history though...

[ann0see]

It'll still be there in the commit history though...

Unless we do nasty force-pushes...

[pljones]

Does this "big" issue still need to be open or can it be closed as "done" and any problems with Weblate addressed separately / through discussions / other channels?

[ann0see]

I think it should probably stay open until we have set up weblate for the App too?

@ignotus666 would you say it’s feasible to set up weblate for the app repo too?

[trebmuh]

It'd be great !

[ann0see]

I've set something up, but it still needs some investigation

[henkdegroot]

I've set something up, but it still needs some investigation

Looks nice to me. Guess we need something for the windows installer as well.

[ann0see]

I've installed weblate.git.squash

[ann0see]

Ok. Windows installer is also set up partly. We might need to rename the .nsi files to fit to some standard.

[ann0see]

@ignotus666 I think now I know why I'm still not 100% happy with the commit message on the website: It should follow imperative style: "Update language web translation from Weblate"

See https://www.theserverside.com/video/Follow-these-git-commit-message-guidelines

[ann0see]

How can I change the commit message for all components at once?

[ignotus666]

We also have to look at why it forked from my repo. We might need to merge what we have, delete the component and try again.

[ann0see]

I'm going to all components and change the commit message manually now.
For the other problem, yes, probably that's the easiest way.

[ann0see]

Ok. Done.

[ignotus666]

@ann0see did you enable the webhook on the main jamulus repo?

[ann0see]

Yes

[ignotus666]

Ok. Just wondering if it had to do with the mysterious fork. It was activated on my repo (from when I did the testing) though I don't know how it could be related.

[ann0see]

@henkdegroot there's a NSIS to .po file script on the NSIS wiki: https://nsis.sourceforge.io/Creating_language_files_and_integrating_with_MUI#Another_solution:_Native_PO_File_support

Edit: Probably the script is here: https://github.com/multitheftauto/mtasa-blue/tree/master/utils/localization
multitheftauto/mtasa-blue#2525 is a related PR, but I think it would be fair to notify them if we use their scripts (which we should be able to since they're GPLv3 (still we must state it))

[henkdegroot]

Can someone explain where the translation text shown in Weblate is sourced from?

I just did an lupdate local and got 1 untranslated string (Korean from the util.cpp).
So I would like to know is Weblate taking the .ts file as the source or the actual source code?
If it is taking the .ts file, we probably need to get a process in-place to auto update the ts file on each PR/merge?

EDIT: Hmmm...looking more closely at the Weblate information, it does seem to use the .ts file. So I would say it would help to update the ts files more often, unless you want to wait for a string freeze or something like that before you accept any translation work.

[henkdegroot]

@henkdegroot there's a NSIS to .po file script on the NSIS wiki: https://nsis.sourceforge.io/Creating_language_files_and_integrating_with_MUI#Another_solution:_Native_PO_File_support

Edit: Probably the script is here: https://github.com/multitheftauto/mtasa-blue/tree/master/utils/localization multitheftauto/mtasa-blue#2525 is a related PR, but I think it would be fair to notify them if we use their scripts (which we should be able to since they're GPLv3 (still we must state it))

I came across that as well, thought it was a bit overkill for this....but from a translation point of view might be worth doing.

[henkdegroot]

Another thing I noticed for the app translation, the variables are not very clear in the GUI:

How do I know what %1 will be and what will %2 be?

[pljones]

How do I know what %1 will be and what will %2 be?

There should be a comment in the translation source saying what each value is. Does Weblate show that comment? (Qt tr takes a second param, the comment; I don't know if the .ui files do the same but they generally don't have parameters. I don't know how well covered the code is, though.)

[BLumia]

Does Weblate show that comment?

If it has a comment, it will show the comment.

(Screenshot of Source string screen, should be also in the translating page as well)

I don't know if the .ui files do the same but they generally don't have parameters.

There is a comment property in the designer to fill. It will look like this if we fill that property:

jamulus/src/serverdlgbase.ui

Lines 229 to 235 in 73ba38c

	<item>
	<widget class="QLabel" name="lblLanguage">
	<property name="text">
	<string extracomment="Consider add &quot;(Lang)&quot; as a suffix to ensure that users who selected the wrong language find the correct button">Language</string>
	</property>
	</widget>
	</item>

[henkdegroot]

So I checked one string for which there is a comment provided, and indeed it shows:

This means we need to update all instances which use the substitution like this and add an appropriate comment.

And comments can be added in the ui file as well, as you already show in the code and this is what it looks like on Weblate:

[ignotus666]

Now that the app translation files have been added to Weblate, the translation instructions on the main repo should be updated. The approach for the website translations has been to recommend Weblate but leave open the option of making PRs. We assume that someone preferring to submit a PR knows what they're doing so don't provide detailed instructions. @softins created a superb guide that should probably be kept somewhere, but translation instructions can now practically be shortened to "log in to Weblate and translate your language", which I think makes it more accessible.

Then there's the issue of adding new languages. It's simple enough for the website as it's all automated, but I think it's not as straightforward for the app translation (I'm not too familiar with the specific steps beyond creating new .ts and .nsi files). Maybe we could explore the possibility of at least automating the creation of these two files similarly to how it's done on the website (the translator opens an issue with the language code in the title, and an admin adds a specific label that triggers the file creation) to keep things coherent across both repos and make it easier for translators?

[ann0see]

Pasting a few notes for adding a new .ts file here:

Adding a new language to Jamulus

1. Open an issue on jamulussoftware/jamulus
2. Fork the jamulus repo
3. Create a branch on your fork
4. Add translation file to TRANSLATIONS variable: src/translation/translation_xx_XX.ts in alphabetical order and .qm file to DISTFILES
5. Add translation to resources.qrc
6. Run lupdate Jamulus.pro and lrelease Jamulus.pro
7. Add, commit and push the edited files

[ann0see]

Adding a new language to the installer:

1. Duplicate src/translation/wininstaller/en_UK.nsi to src/translation/wininstaller/xx_XX.nsi
2. Add language with the correct language code to src/translation/installerlng.nsi
3. Rename ${LANG_ENGLISH} to the respective value you set in installerlng.nsi in xx_XX.nsi

[pljones]

5. Add translation to ressources.qrc

Typo.

[comradekingu]

1. I don't know what the problem with having e-mail in Git commit logs are, and it is in https://hosted.weblate.org/legal/ so that is above board.

2. The process of adding files first, that go unpopulated and add to language selectors with no actual translation behind them isn't good. Not sure if that exists for a whole version before it gets fixed. Not a big problem though.

My big problem is how much of a hurdle this is to get a translation started.
Weblate has support for making .ts files
https://hosted.weblate.org/settings/jamulus/jamulus-app/#files

I imagine either Weblate has some post-script to run, or it can be added to the CI/CD to add the rest of the files.

3. For some reason only the app and the Windows installer are GPLv2 or later
   https://hosted.weblate.org/projects/jamulus/#information
   whereas the rest are CC-BY-SA-4.0.

From assumptive memory, I think CC-BY-SA-4.0 is only compatible with GPLv3(+), so a hmmm there.

4. I'd worry now about getting some fixes in for the source strings.
   At first glance it could do with a lot of polish, which in turn makes translation less of a needlessly staggered ordeal.
   I don't have the time to do a lot of Git-stuff, but I can edit the source strings directly in WL, or edit just the text as PRs.

[ignotus666]

@comradekingu I'll try to address some of your points.

2. Up until now, translation was done during a specific period prior to release, so this problem didn't generally arise (speaking mainly about the app itself, not the website), because when someone requested a new language the translation was usually completed by the release date. Weblate might change this slightly, but it shouldn't be too much of a problem.

I might be missing something, but I can't see how Weblate can create .ts files for a new language. There's more involved than just that (see ann0see's post above).

3. Long story short: the application code (including GUI texts) is under a different licence to the website docs.

4. You are welcome to suggest improvements, but if you want to propose changes to the GUI texts it should be done either via comments on Weblate or (preferably) PRs here. Besides, editing source text in .ts files on Weblate would be pointless as that text is extracted from the application's source files, which is where it has to be changed.

[ann0see]

The process of adding files first, that go unpopulated and add to language selectors with no actual translation behind them isn't good. Not sure if that exists for a whole version before it gets fixed. Not a big problem though.

That's only true for the time between the merge of the PR containing empty .ts files and the merge of the PR containing your translations from Weblate

[ann0see]

I don't know what the problem with having e-mail in Git commit logs are,

Privacy and spam.

[henkdegroot]

I believe the Weblate process should be a bit further integrated in the release process.

Here some thoughts (hoping most of them can be automated) :

• Include an announcement on Weblate to inform translators that we are getting ready for release
• Make the generation of the PR automated (translators can then use the artifacts from the PR to test the results, this needs some further work for the app (.ts) files as the related qm files should be generated automatic and be used in the build).
• Update documentation to include references to the Weblate
• Add information for Weblate to the "generated" issues for the translation (mainly for reference, as the Weblate only translators won't see that anyway).
• Allow a "translator" role to link the translation github issue to the Weblate PR, when Weblate is used for the translation.

This is also to prevent doing the work twice...as without the PR, it will be complex to get my translation tested in the app and website. If the PR exists, then you can download/clone the PR and test using that.

[ann0see]

Include an announcement on Weblate to inform translators that we are getting ready for release

I did that. Feel free to open a PR on the website with a suggestion to the release process (If it's not added yet)

Make the generation of the PR automated (translators can then use the artifacts from the PR to test the results, this needs some further work for the app (.ts) files as the related qm files should be generated automatic and be used in the build).

This should be part of another separate issue, I think.

Update documentation to include references to the Weblate
Add information for Weblate to the "generated" issues for the translation (mainly for reference, as the Weblate only translators won't see that anyway).

#2880

Allow a "translator" role to link the translation github issue to the Weblate PR, when Weblate is used for the translation.

Not sure if I understand this correctly. Could you please elaborate? We have jamulussoftware/translators?

[ann0see]

Closing for now, as we should open separate actionable issues.