Add sudachi_projection option to BertJapaneseTokenizer

[hiroshi-matsuda-rit]

What does this PR do?

A new feature of SudachiPy v0.6.8 allows normalization of words based on Japanese morphological analysis.

WorksApplications/sudachi.rs#230

This morphology-based normalization functionality, named "projection" in SudachiPy, makes Japanese sub-tokenization more efficient and can improve transformer performance.

Very few changes are required to add sudachi_projection option to BertJapaneseTokenizer, and the models that do not specify sudachi_projection option can be used as before in environments using older versions of SudachiPy.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

• text models: @ArthurZucker and @younesbelkada

[hiroshi-matsuda-rit]

Example model for sudachi_projection option:
https://huggingface.co/hiroshi-matsuda-rit/bert-base-sudachitra-v11/blob/main/tokenizer_config.json#L16

[ArthurZucker]

Nice 🔥 Feel free to ping me for a review!

[hiroshi-matsuda-rit]

The errors which were raised from the test cases for jumanpp and mecab-unidic in local environment were not reproduced in circle-ci test environment, so I removed following section from the top description of this PR.

As a side note, some errors have been reported from the test cases for mecab-unidic and jumanpp when turning on RUN_CUSTOM_TOKENIZERS.
I would also like to update these outdated test cases in this PR if required.

$ RUN_CUSTOM_TOKENIZERS=1 pytest -rfs tests/models/bert_japanese/
...
======================================================================================= short test summary info =======================================================================================
FAILED tests/models/bert_japanese/test_tokenization_bert_japanese.py::BertJapaneseTokenizationTest::test_jumanpp_tokenizer - AssertionError: Lists differ: ['アップ[28 chars]', '\\ ', 'が', '\\ \\ ', '\\ ', '発売', 'さ', 'れた', '\\ ', '。'] != ['アップ[28 chars]', '\u3000', 'が', '\u3000', '\u3000', '\u3000'[28 chars]...
FAILED tests/models/bert_japanese/test_tokenization_bert_japanese.py::BertJapaneseTokenizationTest::test_jumanpp_tokenizer_ext - AssertionError: Lists differ: ['ありがとう', 'ございます', 'm', '(', '_', '\\ _', ')', 'm', '見つける', 'の[15 chars] '。'] != ['ありがとう', 'ございます', 'm(_ _)m', '見つける', 'の', 'が', '大...
FAILED tests/models/bert_japanese/test_tokenization_bert_japanese.py::BertJapaneseTokenizationTest::test_jumanpp_tokenizer_lower - AssertionError: Lists differ: ['アップ[28 chars]', '\\ ', 'が', '\\ \\ ', '\\ ', '発売', 'さ', 'れた', '\\ ', '。'] != ['アップ[28 chars]', '\u3000', 'が', '\u3000', '\u3000', '\u3000'[28 chars]...
FAILED tests/models/bert_japanese/test_tokenization_bert_japanese.py::BertJapaneseTokenizationTest::test_jumanpp_tokenizer_no_normalize - AssertionError: Lists differ: ['ｱ',[45 chars]', '\\ ', 'が', '\\ \\ ', '\\ ', '発売', 'さ', 'れた', '\u3000', '。'] != ['ｱ',[45 chars]', '\u3000', 'が', '\u3000', '\u3000', '\u3000'[28 chars] '。']
FAILED tests/models/bert_japanese/test_tokenization_bert_japanese.py::BertJapaneseTokenizationTest::test_jumanpp_tokenizer_trim_whitespace - AssertionError: Lists differ: ['アップ[21 chars]ne', '8', '\\', 'が', '\\ \\', '\\', '発売', 'さ', 'れた', '\\', '。'] != ['アップ[21 chars]ne', '8', 'が', '発売', 'さ', 'れた', '。']
FAILED tests/models/bert_japanese/test_tokenization_bert_japanese.py::BertJapaneseTokenizationTest::test_mecab_tokenizer_unidic - RuntimeError: The unidic dictionary itself is not found. See https://github.com/polm/unidic-py for installation.
SKIPPED [2] tests/test_tokenization_common.py:2394: test is PT+TF test
SKIPPED [2] tests/test_tokenization_common.py:2534: test is slow
SKIPPED [2] tests/test_tokenization_common.py:2499: test requires TensorFlow
SKIPPED [2] tests/test_tokenization_common.py:2448: test is slow
======================================================================== 6 failed, 193 passed, 8 skipped, 7 warnings in 4.52s =========================================================================

[hiroshi-matsuda-rit]

@ArthurZucker All tests passed! Please review and merge this PR.

[ArthurZucker]

Looks good to me! I am glad that the test did not change! Thanks you for adding this new feature to the Sudachi tokenizer

Let's add a small test where you pass sudachi_kwargs to a BertJapaneseTokenizer to make sure this is usable

[ArthurZucker]

I think we need a sudachi version check for this feature then! Instead of if else ! Something similar to version.parse(_torch_version) >= version.parse("2.1.1") (from:

transformers/src/transformers/utils/import_utils.py

Lines 265 to 277 in 64fa9f5

	def is_torch_sdpa_available():
	if not is_torch_available():
	return False
	elif _torch_version == "N/A":
	return False
	# NOTE: We require torch>=2.1 (and not torch>=2.0) to use SDPA in Transformers for two reasons:
	# - Allow the global use of the `scale` argument introduced in https://github.com/pytorch/pytorch/pull/95259
	# - Memory-efficient attention supports arbitrary attention_mask: https://github.com/pytorch/pytorch/pull/104310
	# NOTE: We require torch>=2.1.1 to avoid a numerical issue in SDPA with non-contiguous inputs: https://github.com/pytorch/pytorch/issues/112577
	return version.parse(_torch_version) >= version.parse("2.1.1")

with an exception if not supported

[hiroshi-matsuda-rit]

Let's add a small test where you pass sudachi_kwargs to a BertJapaneseTokenizer to make sure this is usable

I added some test cases which pass kwargs to word tokenizers.

[hiroshi-matsuda-rit]

I think we need a sudachi version check for this feature then!

Thanks for your suggestion. Let me explain the background.

I've been developing a Japanese Dependency parser GiNZA utilizing spacy-transformers and sudachipy, and its ELECTRA model has become widely used for Japanese text analysis in recent years.
I'm wondering that certain users of GiNZA ELECTRA model may hope to keep sudachipy as is but they need to upgrade transformers to use newer models together with GiNZA.

This is the reason why I wrote the if else clause.

[hiroshi-matsuda-rit]

@ArthurZucker Finally, I decided to pass projection arg to Dictionary.create() directly.
I would like to inform GiNZA users to update transformers and sudachipy at the same time.

[hiroshi-matsuda-rit]

@ArthurZucker What should I do for this type of errors? It seems I do not have a permission to rerun the failed actions.

FAILED tests/test_pipeline_mixin.py::ImageSegmentationPipelineTests::test_small_model_pt - requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

[ArthurZucker]

Yes if else is alright don't worry! Just wanted to make sure someone that tries to use the new feature with an older version would get an error saying he needs to update the sudachi version to use it!

[ArthurZucker]

LGTM, we need to make sure we test the sudachi version an warn users that would set a projection that is not supported!

[ArthurZucker]

as you mentioned this is not backward compatible as this was introduced later on in sudachi!

[ArthurZucker]

I would use if is_sudachi_projection_available here, and else use the backward compatible beahviour!

[ArthurZucker]

nice tests thanks!

[hiroshi-matsuda-rit]

I implemented is_sudachi_projection_available() and @require_sudachi_projection to check the sudachipy version.
Does this additional implementation meet your requirements? @ArthurZucker

[ArthurZucker]

Reviewing!

[ArthurZucker]

The test values should not be updated, but nice new test that was added!

[ArthurZucker]

we usually change this only for security reasons

[ArthurZucker]

I would use if is_sudachi_projection_available here, and else use the backward compatible beahviour!

[hiroshi-matsuda-rit]

@ArthurZucker Thanks! I just revised the changes as per your instructions.

[ArthurZucker]

That should be the last nit!

[hiroshi-matsuda-rit]

@ArthurZucker Thank you. I have completed the fix.

[ArthurZucker]

LGTM! Thank you for iterating and protecting the imports!

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.