feat: use tokenizers to split words before doing entity detection

[Apollon77]

Pull Request Template

PR Checklist

• I have run npm test locally and all tests are passing.
• I have added/updated tests for any new behavior.
• If this is a significant change, an issue has already been created where the problem / solution was discussed: [N/A, or add link to issue here]

PR Description

This PR is a fix for an issue reported in the repository (need to check if I can find it again). The problem is that the entity extraction was using normal whitespaces till now and this is not working in languages like japanese which is not using whitespaces.

So this PR introduce the usage of the tokenizer to splitup the text into words and then do the entity extraction based on this text. It also has the neded logic to correctly map the "positions" then back to the original text, so that the positions contained in the entity details are correct

[sonarcloud]

SonarCloud Quality Gate failed.   

1 Bug
0 Vulnerabilities
1 Security Hotspot
5 Code Smells

No Coverage information
0.0% Duplication

[Apollon77]

@eric-lara For the sonarcloud "security hotspot" and "bug" ... it is a regex also used in other places ... Yes should be checked but I'm not really a regex expert ... Do you maybe have a proposal how to handle it?

BTW: Ready for review

[Apollon77]

@ericzon All PRs ready for review