Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

i18n workflow: force compile fuzzy #103

Closed
22 of 43 tasks
mb-wali opened this issue Jan 18, 2022 · 9 comments
Closed
22 of 43 tasks

i18n workflow: force compile fuzzy #103

mb-wali opened this issue Jan 18, 2022 · 9 comments

Comments

@ghost
Copy link

ghost commented Aug 17, 2023

Can we check the status of this issue?

  1. based on this list it seems, that we have packages where we force fuzzy and packages, where this is not the case. Is this information still correct?

  2. since this lists is already some days old, I remember, that there were new packages added, what's the status with these? The documentation mentions the fuzzy: https://inveniordm.docs.cern.ch/develop/howtos/i18n/#configuration
    Related: (transifex) make a list of modules that invenioRDM - uses #99

  3. are we sure that we need/want to force it?

    • Maybe there is a better way, by fixing fuzzy (when using transifex)?
  • I can/will try to check transifex in the upcoming weeks.

  • I would be happy to get some opinions/experience of the force-use.
    Maybe: @utnapischtim @rekt-hard @zzacharo

  • When discussed point 3. I could try to get the changes in all packages.

@rekt-hard
Copy link
Contributor

rekt-hard commented Aug 23, 2023

  1. All packages which contain translatable strings should have the config. In the above list, not all packages have translations, which is probably the reason why not everything is checked. We can go through the packages and check.
  2. New packages should have force fuzzy enabled by default. Still should check though.
  3. It would probably require to go back and analyze where the fuzzy is being added and why. Then, we should also check which errors would be caught if we do not force it and if it affects us.

@ghost
Copy link

ghost commented Aug 28, 2023

I think earliest end of September I can investigate that.

@ghost
Copy link

ghost commented Sep 27, 2023

wrt. point 3:

  • as described here i18n workflow: translation files (.mo) are not being created invenio-app-rdm#230

  • and tested and read in the docs:

  • at least, as long as we have the header we get the fuzzy

  • an alternative for this header-fuzzy is to use python setup.py extract_messages --omit-header

  • but then we loose the header information

  • I didn't check whether there are other ways how fuzzy could get in our files

  • the moment a file contains fuzzy this file will not compiled at all anymore - without any warning or error messages (others are complaining about this, too)

  • this makes it hard to detect the issue

  • therefore, using fuzzy in setup.cfg sounds the best way to go, for now:

    [compile_catalog]
    use-fuzzy = True   
    

wrt. point 2:

wrt. point 1:

I will try go through all packages and check for the fuzzy and report back.

@ghost
Copy link

ghost commented Sep 27, 2023

Starting with the following:

from Pipfile.lock with invenio-app-rdm "=12.0.0b2.dev35"
all packages containing invenio in their name

checked | has fuzzy | okay without fuzzy

@ghost
Copy link

ghost commented Sep 27, 2023

based on this:

have fuzzy:

  • invenio-access
  • invenio-accounts
  • invenio-app-rdm
  • invenio-banners
  • invenio-communities
  • invenio-drafts-resources
  • invenio-formatter
  • invenio-i18n
  • invenio-notifications
  • invenio-oaiserver
  • invenio-oauth2server
  • invenio-oauthclient
  • invenio-pidstore
  • invenio-rdm-records
  • invenio-records
  • invenio-records-permissions
  • invenio-records-resources
  • invenio-requests
  • invenio-search-ui
  • invenio-theme
  • invenio-userprofiles
  • invenio-users-resources
  • invenio-vocabularies
  • invenio-webhooks

no fuzzy but likely no problem:

  • flask-collect-invenio
  • flask-kvsession-invenio
  • invenio-app
  • invenio-assets
  • invenio-base
  • invenio-cache
  • invenio-celery
  • invenio-config
  • invenio-db
  • invenio-jsonschemas
  • invenio-logging
  • invenio-mail
  • invenio-queues
  • invenio-records-files
  • invenio-rest
  • invenio-search
  • invenio-stats

no fuzzy and more likely a problem:

  • invenio-files-rest
  • invenio-github
  • flask-security-invenio
  • invenio-admin
  • invenio-administration
  • invenio-pages
  • invenio-previewer

no fuzzy and maybe a problem:

  • invenio-indexer
  • invenio-records-rest
  • invenio-records-ui

not checked

  • cookiecutter-invenio-rdm
  • cookiecutter-invenio-module
  • marshmallow-utils
  • react-searchkit

@ghost
Copy link

ghost commented Oct 4, 2023

wrt 'no fuzzy but likely no problem:'

within each repo searched for _(, format, gettext, I18N, i18next, Trans

result:

flask-collect-invenio:
- found: python 'format'
- but no traces of translation intents

flask-kvsession:
- found _(...) in docs-folder
- but no traces of translation intents

invenio-app:
- found: 'gettext' and 'I18N' in documentation
- found: python 'format' in test_helpers
- but no traces of translation intents


invenio-assets:
- found: 'gettext' and 'I18N' in docs
- found: python 'format'
- but no traces of translation intents

invenio-base:
- found: 'gettext' and 'I18N' in docs
- but no traces of translation intents

invenio-cache:
- found: python 'format'
- found: 'gettext' and 'I18N' in docs
- but no traces of translation intents

invenio-celery:
- found: python 'format'
- found: 'gettext' and 'I18N' in docs
- but no traces of translation intents

invenio-config:
- found: python 'format'
- found: 'gettext' and 'I18N' in docs
- but no traces of translation intents

invenio-db:
- found: python 'format'
- found: 'gettext' and 'I18N' in docs
- but no traces of translation intents

invenio-jsonschemas:
- found: python 'format'
- found: 'gettext' and 'I18N' in docs
- but no traces of translation intents

invenio-logging:
- found: python 'format'
- found: 'gettext' and 'I18N' in docs
- but no traces of translation intents

invenio-mail:
- found: python 'format'
- found: 'gettext' and 'I18N' in docs
- but no traces of translation intents

invenio-queue:
- found: python 'format'
- found: 'gettext' and 'I18N' in docs
- but no traces of translation intents
- .tx/config:9:# TODO: Transifex integration

invenio-records-files:
- found: python 'format'
- found: 'gettext' and 'I18N' in docs
- but no traces of translation intents

invenio-rest:
- found: python 'format'
- found: 'gettext' and 'I18N' in docs
- but no traces of translation intents

invenio-search:
- found: python 'format'
- found: 'gettext' and 'I18N' in docs
- but no traces of translation intents

invenio-stats:
- found: python 'format'
- found: 'gettext' and 'I18N' in docs
- but no traces of translation intents
- .tx/config:9:# TODO: Transifex integration

@ghost
Copy link

ghost commented Oct 4, 2023

Additionally found: https://github.com/inveniosoftware/cookiecutter-invenio-module/blob/6decff43538c9e317175bfed33878fe7b0843a53/%7B%7B%20cookiecutter.project_shortname%20%7D%7D/setup.cfg#L80

Conclusion:

  • waiting for the PRs to get merged
  • there is the possibility that some packages do not enable translation, although text will get to the user (maybe invenio-stats?). However, it is hard to see, which sentences will be seen by whom.
  • Therefore, we will wait and check in the gui whether untranslated sentences will show up anywhere and fix it then.

mb-wali pushed a commit to inveniosoftware/invenio-files-rest that referenced this issue Nov 27, 2023
mb-wali pushed a commit to inveniosoftware/invenio-github that referenced this issue Nov 27, 2023
mb-wali pushed a commit to inveniosoftware/flask-security-fork that referenced this issue Nov 27, 2023
mb-wali pushed a commit to inveniosoftware/invenio-admin that referenced this issue Nov 27, 2023
mb-wali pushed a commit to inveniosoftware/invenio-administration that referenced this issue Nov 27, 2023
mb-wali pushed a commit to inveniosoftware/invenio-records-ui that referenced this issue Nov 27, 2023
mb-wali pushed a commit to inveniosoftware/invenio-records-rest that referenced this issue Nov 27, 2023
mb-wali pushed a commit to inveniosoftware/invenio-indexer that referenced this issue Nov 27, 2023
mb-wali pushed a commit to inveniosoftware/invenio-previewer that referenced this issue Nov 27, 2023
mb-wali pushed a commit to inveniosoftware/invenio-pages that referenced this issue Nov 27, 2023
@ghost
Copy link

ghost commented Mar 21, 2024

All PRs are merged. For the moment we should be good. Let's close it. @zzacharo

alejandromumo added a commit to alejandromumo/zenodo-rdm that referenced this issue Aug 7, 2024
📁 invenio-app-rdm (13.0.0b0.dev10 -> 13.0.0b0.dev11 )

    📦 release: v13.0.0b0.dev11
    deposit: pass record to publish button
    CI: switch to centralised workflows
    templates: landing page custom field macro refactor
    custom_fields: allow to search vocab

    * closes CERNDocumentServer/cds-rdm#180
    requests: fix small ui bugs

    * closes CERNDocumentServer/cds-rdm#176
    migration: add vocabulary updates
    migration: add parent doi handling

    * list points which were tested

📁 invenio-banners (3.0.2 -> 3.1.0 🌈)

    📦 release: v3.1.0
    http headers: use and adjust vnd.inveniordm.v1+json http accept header

    * closes zenodo/rdm-project#598

📁 invenio-communities (14.5.1 -> 14.5.2 🐛)

    📦 release: v14.5.2
    user_moderation: dispatch Celery tasks for each community operation
    fix(logo): not fully deleted

    * logo file hasn't been deleted. this prevented a new file upload,
      because of an sqlalchemy.exc.IntegrityError:
      (psycopg2.errors.UniqueViolation) error.
    subcommunities: fix missing double-quote in email templates

📁 invenio-i18n (2.1.1 -> 2.1.2 🐛)

    release: v2.1.2
    ext: allow BABEL_DEFAULT_LOCALE overridability

📁 invenio-pages (4.0.1 -> 4.1.0 🌈)

    📦 release: v4.1.0
    ci: use reusable workflows
    http headers: use and adjust vnd.inveniordm.v1+json http accept header

    * closes zenodo/rdm-project#598
    setup: update dependencies

📁 invenio-rdm-records (11.5.0 -> 11.6.0 🌈)

    📦 release: v11.6.0
    creatibutors: fix buttons
    permissions: change error handler for resolving pid permission denied

    * closes inveniosoftware/invenio-app-rdm#2685
    record inclusion: use system identity to accept inclusion request when can_include_directly

    * closes CERNDocumentServer/cds-rdm#176
    user_moderation: improve DB queries and use Celery tasks
    fix: use index to distinguish type

    * the problem with is_published is that drafts created from records will
      not be recogniced correctly

    * using the index is a valid solution but it is not a nice
      implementation.
    results: added support for drafts

    * Added support for drafts in the results list
    fix(community): set branding

    * the set branding didn't work at all. it didn't work for rebranding if
      a default already exists and it didn't work if no branding exists at
      all.

    * the default property of the CommunitiesRelationManager needs a string.
      it can't handle a dict.

📁 invenio-records-ui (1.2.0 -> 1.2.1 🐛)

    release: v1.2.1
    views: add __qualname__ to partial wrapped view func
    config: fix extract messages err
    i18n-global: add compile-catalog fuzzy (inveniosoftware/invenio-records-ui#100)

    Refs inveniosoftware/invenio-i18n#103
    fix change sh to bash to use newer syntax
    fix res.location changed structure, localhost not more part of it
    fix add missing configuration variables

    NOTE:
    the bugfix seams as not complete. the real bug should be fixed somewhere else.
    it was too easy to add only those variables.
    global: clean test infrastructure
    increase minimal python version to 3.7
    move check_manifest configuration to setup.cfg.

    concentrate the configuration of all calls in one place
    fix docs compatibilty problem with Sphinx>=5.0.0
    add .git-blame-ignore-revs
    migrate to use black as opinionated auto formater
    migrate setup.py to setup.cfg

📁 invenio-search (2.4.0 -> 2.4.1 🐛)

    release: v2.4.1
    fix: avoid closing db session

    Using `with self.app.app_context():` closes the current DB session on
    `__exit__`, causing a `sqlalchemy.orm.exc.DetachedInstanceError` when
    accessing any property of the object expected in the session. This error
    occurs only the first time index_templates are loaded, such as when
    indexing the very first record.
    Subsequent accesses won't trigger this issue due to the @cached_property,
    preventing reloading of index_templates.
    ci: upgrade tests

📁 invenio-vocabularies (4.2.0 -> 4.3.0 🌈)

    📦 release: v4.3.0
    names: make names_exclude_regex configurable
    setup: add regex as dependency
    names: validate entry full names
    ci: use reusable workflows
    names: keep original datastream config

    * allow to pass since
    names: update writer service call
    names: add orcid public data sync

    * Adds delete all values of a vocab to CLI
alejandromumo added a commit to zenodo/zenodo-rdm that referenced this issue Aug 7, 2024
📁 invenio-app-rdm (13.0.0b0.dev10 -> 13.0.0b0.dev11 )

    📦 release: v13.0.0b0.dev11
    deposit: pass record to publish button
    CI: switch to centralised workflows
    templates: landing page custom field macro refactor
    custom_fields: allow to search vocab

    * closes CERNDocumentServer/cds-rdm#180
    requests: fix small ui bugs

    * closes CERNDocumentServer/cds-rdm#176
    migration: add vocabulary updates
    migration: add parent doi handling

    * list points which were tested

📁 invenio-banners (3.0.2 -> 3.1.0 🌈)

    📦 release: v3.1.0
    http headers: use and adjust vnd.inveniordm.v1+json http accept header

    * closes zenodo/rdm-project#598

📁 invenio-communities (14.5.1 -> 14.5.2 🐛)

    📦 release: v14.5.2
    user_moderation: dispatch Celery tasks for each community operation
    fix(logo): not fully deleted

    * logo file hasn't been deleted. this prevented a new file upload,
      because of an sqlalchemy.exc.IntegrityError:
      (psycopg2.errors.UniqueViolation) error.
    subcommunities: fix missing double-quote in email templates

📁 invenio-i18n (2.1.1 -> 2.1.2 🐛)

    release: v2.1.2
    ext: allow BABEL_DEFAULT_LOCALE overridability

📁 invenio-pages (4.0.1 -> 4.1.0 🌈)

    📦 release: v4.1.0
    ci: use reusable workflows
    http headers: use and adjust vnd.inveniordm.v1+json http accept header

    * closes zenodo/rdm-project#598
    setup: update dependencies

📁 invenio-rdm-records (11.5.0 -> 11.6.0 🌈)

    📦 release: v11.6.0
    creatibutors: fix buttons
    permissions: change error handler for resolving pid permission denied

    * closes inveniosoftware/invenio-app-rdm#2685
    record inclusion: use system identity to accept inclusion request when can_include_directly

    * closes CERNDocumentServer/cds-rdm#176
    user_moderation: improve DB queries and use Celery tasks
    fix: use index to distinguish type

    * the problem with is_published is that drafts created from records will
      not be recogniced correctly

    * using the index is a valid solution but it is not a nice
      implementation.
    results: added support for drafts

    * Added support for drafts in the results list
    fix(community): set branding

    * the set branding didn't work at all. it didn't work for rebranding if
      a default already exists and it didn't work if no branding exists at
      all.

    * the default property of the CommunitiesRelationManager needs a string.
      it can't handle a dict.

📁 invenio-records-ui (1.2.0 -> 1.2.1 🐛)

    release: v1.2.1
    views: add __qualname__ to partial wrapped view func
    config: fix extract messages err
    i18n-global: add compile-catalog fuzzy (inveniosoftware/invenio-records-ui#100)

    Refs inveniosoftware/invenio-i18n#103
    fix change sh to bash to use newer syntax
    fix res.location changed structure, localhost not more part of it
    fix add missing configuration variables

    NOTE:
    the bugfix seams as not complete. the real bug should be fixed somewhere else.
    it was too easy to add only those variables.
    global: clean test infrastructure
    increase minimal python version to 3.7
    move check_manifest configuration to setup.cfg.

    concentrate the configuration of all calls in one place
    fix docs compatibilty problem with Sphinx>=5.0.0
    add .git-blame-ignore-revs
    migrate to use black as opinionated auto formater
    migrate setup.py to setup.cfg

📁 invenio-search (2.4.0 -> 2.4.1 🐛)

    release: v2.4.1
    fix: avoid closing db session

    Using `with self.app.app_context():` closes the current DB session on
    `__exit__`, causing a `sqlalchemy.orm.exc.DetachedInstanceError` when
    accessing any property of the object expected in the session. This error
    occurs only the first time index_templates are loaded, such as when
    indexing the very first record.
    Subsequent accesses won't trigger this issue due to the @cached_property,
    preventing reloading of index_templates.
    ci: upgrade tests

📁 invenio-vocabularies (4.2.0 -> 4.3.0 🌈)

    📦 release: v4.3.0
    names: make names_exclude_regex configurable
    setup: add regex as dependency
    names: validate entry full names
    ci: use reusable workflows
    names: keep original datastream config

    * allow to pass since
    names: update writer service call
    names: add orcid public data sync

    * Adds delete all values of a vocab to CLI
slint added a commit to zenodo/zenodo-rdm that referenced this issue Nov 11, 2024
📁 invenio-access (2.0.0 -> 2.1.0 🌈)

    release: v2.1.0
    permissions:  add system permisssion

    * closes CERNDocumentServer/cds-rdm#240

📁 invenio-admin (1.4.1 -> 1.5.1 🌈)

    📦 release: v1.4.1
    docs: fix intersphinx config
    setup: pin wtforms

    3.2.0 makes a breaking change https://wtforms.readthedocs.io/en/3.2.x/changes/
    📦 release: v1.5.1
    setup: pin wtforms

    3.2.0 makes a breaking change https://wtforms.readthedocs.io/en/3.2.x/changes/
    release: v1.5.0
    setup: add invenio-app dependency
    fix: before_app_first_request deprecation
    i18n-global: add compile-catalog fuzzy (inveniosoftware/invenio-admin#104)

    Refs inveniosoftware/invenio-i18n#103

📁 invenio-administration (2.8.2 -> 2.9.0 🌈)

    📦 release: v2.9.0
    admin: change default # of results from 10 to 20
    fix: make tooltip required
    bug: fix type for BoolFormatter value

📁 invenio-app-rdm (13.0.0b1.dev16 -> 13.0.0b1.dev17 🌈)

    release: v13.0.0b1.dev17
    setup: bump major invenio-rdm-records
    admin: improve files and stats display
    bug: add message to browse page when no subcoms
    admin: fix mobile display via css instead of html
    admin: increase title length and fix mobile
    admin: link GitHub and ORCiD icons to sites
    admin: add link to user in records and drafts
    upgrades: add v13 script

    * closes CERNDocumentServer/cds-rdm#240
    global: remove invenio-admin

    * invenio-admin has been deprecated since a while in favour of
      invenio-administration
    landing_page: add block user option
    pass the whole APP_RDM_USER_DASHBOARD_ROUTES object
    rename variable to be more precise
    pass config to deposit_create and _edit

📁 invenio-communities (17.3.2 -> 17.4.0 🌈)

    📦 release: v17.4.0
    admin: change default # of results from 10 to 20
    refactor: remove showManualEntry argument
    UI/UX: improve display of ROR information in select field
    refactor: pass community_ui dict instead of community object

📁 invenio-rdm-records (15.7.1 -> 16.0.1 ⚠️)

    📦 release: v16.0.1
    deposit-ui: fix creator affiliations selection display

    * Fixes a bug where the selected affiliations from the dropdown do not
      appear inside the input box.
    release: v16.0.0
    siglal: rename temporarily_connected_to to connected_to

    * See blinker PR: https://github.com/pallets-eco/blinker/pull/172/files#diff-6d65d4c3783f73a7e9f606573e156ed1f0597054715ac512004cb353d8a9886cL227
    identifiers: relax validation rules for deduplication
    records: upgrade jsonschema version

    * adds internal notes
    record: add internal_notes

    * components: add internal notes cmp
    records: upgrade jsonschema version

    * adds internal notes
    record: add internal_notes

    * components: add internal notes cmp
    metadata: add internal notes
    fix: translation only applied on app start

    * the problem was that with gettext the translations for facets
      have been only calculated on boot time.

    * using lazy_gettext instead fixes that, because it calculates
      the translation at the request level

    community submission: remove notification about request auto-accept

    * closes CERNDocumentServer/cds-rdm#203

📁 invenio-records-resources (6.3.1 -> 6.4.0 🌈)

    release: v6.4.0
    fix: translation only applied on app start

    * the problem was that with gettext the translations for facets
      have been only calculated on boot time.

    * using lazy_gettext instead fixes that, because it calculates
      the translation at the request level

    queryparser: add RestrictedTerm to exclude allowed phrases by permission
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants