Enable toggling Case Encoding flag from C++ Train API #11
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
snukky
requested changes
Jul 14, 2021
snukky
approved these changes
Jul 14, 2021
emjotde
approved these changes
Jul 15, 2021
| @@ -153,6 +153,12 @@ util::Status SentencePieceTrainer::MergeSpecsFromArgs( | |||
| CHECK_OR_RETURN(absl::SimpleAtoi(value, &v)); | |||
| absl::SetFlag(&FLAGS_minloglevel, v); | |||
| continue; | |||
| } else if(key == "encode_unicode_case") { | |||
| bool encode_unicode_case; | |||
| std::istringstream(value) >> std::boolalpha >> encode_unicode_case; | |||
There was a problem hiding this comment.
I would have thought there is a better way of doing that, like std::stoi but with bool, but not really. So this is OK.
XapaJIaMnu
added a commit
to browsermt/sentencepiece
that referenced
this pull request
May 3, 2023
* Adding alternative project name for spm latest to prevent lib conflicts * Update cmake * Update CMakeFiles to allow for configurable artifact names * Enables --encode_unicode_case option for case-aware sentence piece (marian-nmt#10) * Enables --encode_unicode_case option for case-aware sentence piece * Example: This IS a TEST OF THE CASING gets converted internally to Tthis Uis a Atest of the casing before segmentation. * This is fully reversible. * Enable toggling Case Encoding flag from C++ Train API (marian-nmt#11) * Enable toggling Case Encoding flag from C++ Train API * Fixing issue with hardcoding truth value of encode_decode_case flag * Disable denormalizer flags (marian-nmt#13) Co-authored-by: Rohit Jain <[email protected]> * Fix Surface String to Token Mappings for Case Encoding (marian-nmt#12) Co-authored-by: Marcin Junczys-Dowmunt <[email protected]> Co-authored-by: Rohit Jain <[email protected]> * add one header file to installation * Rename VERSION to VERSION.txt * Rename VERSION to VERSION.txt Installing python package fails with below error. This change addresses this issue ``` × python setup.py egg_info did not run successfully. │ exit code: 1 ╰─> [10 lines of output] Traceback (most recent call last): File "<string>", line 2, in <module> File "<pip-setuptools-caller>", line 34, in <module> File "/home/alferre/code/sentencepiece/python/setup.py", line 111, in <module> version=version(), File "/home/alferre/code/sentencepiece/python/setup.py", line 36, in version with codecs.open('VERSION.txt', 'r', 'utf-8') as f: File "/opt/conda/envs/ptca/lib/python3.8/codecs.py", line 905, in open file = builtins.open(filename, mode, buffering) FileNotFoundError: [Errno 2] No such file or directory: 'VERSION.txt' [end of output] ``` --------- Co-authored-by: Rohit Jain <[email protected]> Co-authored-by: Rohit Jain <[email protected]> Co-authored-by: Marcin Junczys-Dowmunt <[email protected]> Co-authored-by: Roman Grundkiewicz <[email protected]> Co-authored-by: alexandremuzio <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Allow configuring encode_unicode_case flag from within the C++ Training API by fixing up argument parsing.