-
Notifications
You must be signed in to change notification settings - Fork 528
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix part of #3728, #3729, #3730: Introduce infrastructure to support app & content string translations #3794
Fix part of #3728, #3729, #3730: Introduce infrastructure to support app & content string translations #3794
Conversation
There are a lot of details to cover here--see the PR for the complete context.
- Add missing codeowner - Add support for configuring base branch reference - Update CI for dev/alpha AAB builds to use 'develop' since there's no origin configured by default in the workflows
This is needed to open a PR on GitHub. This commit is being made so that the PR can start off in a broken Actions state. This also initially disables most non-Bazel workflows to make workflow iteration faster and less impacting on other team members.
This introduces a new mechanism for passing lists of tests to sharded test targets in CI, and hooks it up. No actual sharding is occurring yet. This led to some simplifications in the CI workflow since the script can be more dynamic in computing the full list of targets (which also works around a previous bug with instrumentation tests being run). Java proto lite also needed to be upgraded for the scripts to be able to use it. More testing/documentation needed as this functionality continues to expand.
This simply partitions bucketed groups of targets into chunks of 10 for each run. Only 3 buckets are currently retained to test sharding in CI before introducing full support.
Fixes some caching bucket and output bugs. Also, introduces while loop & keep_going to introduce resilience against app test build failures (or just test failures in general).
Also, enable other workflows. Note that CI shouldn't fully pass yet since some documentation and testing needs to be added yet, but this is meant to be a more realistic test of the CI environment before the PR is finished.
Adds a human-readable prefix to make the shards look a bit nicer. Also, adds more fine-tuned partitioning to bucket & reduce shard counts to improve overall timing. Will need to be tested in CI.
A newly computed variable wasn't updated to be used in an earlier change.
Add docstrings for proto.
…/oppia-android into add-bundles-proguard-build-flavors
See #3757 (comment) for context.
Neither 'mv -t' nor piping to mv work on OSX so we needed to find an alternative (in this case just trying to move everything). This actually works a bit better since it's doing a per-file move rather than accommodating for files that shouldn't be moved (which isn't an issue since the destination directory is different than the one containing the AAB file).
Documentation, thorough tests, and detailed description of these changes are still needed.
Also includes fixing circular dependency issue by splitting out some of the locale components to be part of utility rather than domain (so that utiltiy and other packages can depend on MachineLocale).
@vinitamurthi & @rt4914 PTAL for codeowners. @seanlip PTAL for codeowners change. @anandwana001 PTAL at the PR as a whole for correctness. |
SIGSEGV on StateFragmentLocalTest. Happens to me locally, too. Retrying Bazel CI. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM for codeowners.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM for code-owner files.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Overall LGTM, All the test are passing on checks that LGTM too.
1. Introduce proper API compatibility for LocaleController 2. Ensure TranslationController is scoped (breaks test in downstream PR)
@anandwana001 can you PTAL at e879c04 and let me know if it looks good to you? It was a late change after your approval & I'd like to make sure it gets some reviewer visibility. |
SIGSEGV on StateFragmentTest caused a CI flake. :( This is #2844 showing up again even though we thought it went away. My guess is that this was reintroduced because of sharding (& might actually be a separate issue). Restarting CI to unblock this PR, but we'll need to look into this. |
@BenHenning I wanted to check with you before merging this. are we ok to merge this now? |
Hi @BenHenning, this PR is ready to be merged. Please address any remaining comments prior to merging, and feel free to merge this PR once the CI checks pass and you're happy with it. Thanks! |
Thanks for the review everyone! @vinitamurthi I'll merge this momentarily. I want to make sure downstream PRs are in a good in-between state before doing that. |
LGTM |
Thanks @anandwana001! Will be merging this momentarily. Edit: since develop changed & this is a very large PR, I'm pulling the latest changes into this branch & reverifying CI before merging. |
All CI checks are passing, and there are no open unresolved comment threads. Thanks again all for the reviews! |
Explanation
Fixes part of #3728
Fixes part of #3729
Fixes part of #3730
Overview
This PR introduces the core support necessary for localization (particularly, automatic language selection), as described by this design document (note that this document is now a bit outdated). In particular:
OppiaLocale
& subclasses to replace uses of Android's Locale class (regex patterns will be coming in later PRs to enforce this requirement) to ensure properly localization choices are made throughout the codebase, and to provide consistent locale-esque functionality (such as for case conversion & time/date formatting)LocaleController
&TranslationController
) to support various future operations (see later PRs) that are necessary for proper localization supportSpecial high-level considerations
It's worth noting two things:
Regarding (1), much of the complexity comes from a number of factors:
Regarding (2), the importance of localization as critical infrastructure has driven an especially high standard for testing in this PR, including introducing a custom Robolectric shadow & multiple test fakes, each with their own test suites. While this is so thorough, it's meant to ensure a strong regression barrier for the critical infrastructure being introduced in this PR, and to lay a good foundation for future code refactoring as the codebase continues to evolve & scale.
High-level design considerations
This is slightly repeating some points in the design document, but for the purpose of review these may be useful:
ContentLocale
: for audio & written translations (generally all we care about is what language to choose; this may evolve in the future)MachineLocale
: for manipulating or creating machine-readable strings (such as for logs). Most case conversion will use this locale moving forward.DisplayLocale
: for retrieving & manipulating strings that will eventually be displayed to users (this one actually follows whatever the current Locale defined is)DataProvider
s are the key mechanism for computing locales since it allows monitoring changes to the default Android Locale instance, and to enforce app-configured language constraints (see 'configuration' section below)Language configuration
This PR is introducing a new pattern under the config/ folder: two textproto files which clearly defined the supported languages & regions. Some things to note about this:
Semi-related, but two configurations that are hardcoded:
Neither of these should actually affect users in practice, but it's possible to if things go wrong (& and a trivial default is needed).
Notes on testing
DisplayLocaleImplTest
makes use of test-only resources which means it can't run in Gradle (& actually needed to be omitted-see 'Gradle changes' section below). This is the pattern we want to use moving forward for test-only resources (as it's one of the major wins of using Bazel)AssetRepository
was refactored to an interface + module so that we could introduce a no-op version for testingLanguageConfigRetriever
when no textproto is present (the Gradle case). This led to updating most test suites (nearly all app module tests) to properly bindAssetRepository
in test Dagger graphs.TestAssetRepository
implementation, but this was reverted to be a no-op implementation to reduce the scope of the PR a bit. We can reference that commit & bring the changes back if they're desired in the future.FakeOppiaClock
was updated to force the Locale to UTC. I'm not sure why this didn't cause issues before, but Gradle sets the timezone to the user's local whereas Gradle sets it to UTC. Forcing to UTC ensure consistency between the two platforms & across all tests (& generally just seems more correct).Version/dependency & Gradle changes
api
)Codeowner changes
Static check exemptions
Test files:
AssetRepositoryImpl
: this is retaining the existing 'AssetRepository' exemption (though that's not being removed since it's now an interface and is legitimately exempted). This wasn't resolved in this PR to keep the PR slightly smaller since a lot of new tests would need to be introduced to properly test AssetRepository.OppiaBidiFormatter
&OppiaLocale
: interfaces that shouldn't require test files in this caseRegex checks: all of these are legitimate exemptions since the respective classes are hiding the patterns that are being prohibited (or testing those classes).
Other miscellaneous changes
BidiWrapper
was wrapped in a custom Oppia-specific implementation with regex checks to force usage of the wrapper so that tests can verify whether strings are actually wrapped (without relying on wrapping occurring or looking for implementation-specific mark characters)OppiaLocale.DisplayLocale
for these specific methods & their KDocs for more context. Note also that an intentional design choice was made to only allow strings in resource string formatting moving forward for strings that need to support bidirectional wrapping (this generally seems more correct).OppiaClock
had some minor code consolidation to simplify the prod/test implementationsbazel build //...
won't include those targets (building AABs is expensive, especially with Proguard running so this helps with local development). CI runs won't be affected since they directly reference the AAB targets (which is what "manual" does--it avoids the target being included in wildcard patterns).DateTimeUtil
incorrectly computes time-of-day, but we haven't noticed it since it defaults to evening (& it's the evening case that's wrong). This has been fixed inMachineLocaleImpl
& will replaceDateTimeUtil
in a future PR.Future changes
This PR is primarily focused on introducing the necessary support for automatic language selection for app strings, and to lay the groundwork for content & audio strings. Future PRs will introduce much of the remaining work for automatic selection (see the constituent issues that this PR is 'partly' fixing to follow along with that work). Longer-term, a full language selection UI will be introduced in the app. It's not expected that this domain layer will need to change much for that work to be completed.
Finally, while this & future PRs are introducing a lot of regex pattterns to enforce locale-specific best practices, additional custom CI checks are coming (see design doc) in coming weeks & months to ensure that the app strings themselves are properly set up for safe localization & are translated correctly (at least, for the things that we can easily verify as correct using automation).
Essential Checklist
For UI-specific PRs only
N/A