[Backport to main]Add tokenizer and sparse encoding (#1301)

[zane-neo]

• add tokenizer and sparse encoding

• add tokenizer and sparse encoding

• add tokenizer and sparse encoding

• add tokenizer and sparse encoding

• add tokenizer and sparse encoding

• remove special token

• add filter

• try empty model

• remove warm up

• try empty model

• add block

• add log

• add log

• add log

• remove log

• remove pt file detect

• add log

• add functionName pipeline

• remove verify log

• skip special token in sparse encoding

• skip omit tokenize config

• skip omit tokenize config-change warm up logic

• reArch

• deduplicate

• omit ml config in sparse encoding

• add null config in warm up

• fix original test

• add tokenize ut half

• fix sparse encoding bug

• add UT for sparse encoding and tokenize

• remove useless framwork type

• common/src/test/java/org/opensearch/ml/common/input/MLInputTest.java

• change key for tokenize

• reArch DLModel

• reArch DLModel again

• response format

• tokenize only one output

• clean sparse output

• clean sparse output

• change UT number

• remove useless predict code

• remove useless part

• change tokenize way

• reArch add textEmbedding model

• add tokenize logic

• add abstract

• clear code

• fix it class

• fix it class

• add IT file

• reformulate

• reformulate remote inference

• reformulate remote inference

• reformulate remote inference json and array

• verify

• undo string utils

• skip dummy model

• skip dummy model

• skip dummy model

• skip dummy model

• skip dummy model

• skip dummy model

• add inner load Model

• rename variable

• add default for idf

• add ut for sparse encoding and tokenizer

• add close model

• change mock class

• remove buffer for sparse encoding output

• change tokenize model ready logic

• rewrite input functionName

• deduplicate

• change UT usage

• fix downloadAndSplit test

• fix Helper test

• remove meaningless change

• remove complie change

• rename

• fix typo error and simplify wrap code

• add comment

• using gson and remove useless close logic

• update comment and import problem

• add static idf name

• fix format problem

• extract an abstract model for sparse and dense sentence transformer translator

• fix typo error

• remove duplicate tokenizer file, fix import problem and add comment for tokenizer model

---

Description

Backport 31a4e25 from #1301

Issues Resolved

[List any issues this PR will resolve]

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[zane-neo]

Closing this since it's already backported