[Backport 2.x] Upgrading ingest-attachment dependencies #3279
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![opensearch-trigger-bot[bot]](https://avatars.githubusercontent.com/u/80134844?s=60&v=4){kind=small}
* Upgrading Tika from 1.24.1 to 2.1.0 and bumping xmlbeans version This major version upgrade requires an explicit dependency on tika-parsers-standard-package to import the parser implementations, and an update to the namespace of RTFParser. Also, LanguageIdentifier has been deprecated and replaced by LanguageDetector. This change includes a bump in xmlbeans version from 3.0.1 to 3.1.0 Signed-off-by: Kartik Ganesh <[email protected]> * Upgrade Tika libraries from 2.1.0 to 2.2.0 This also requires a update of Apache Commons-IO from 2.7 to 2.11.0 Signed-off-by: Kartik Ganesh <[email protected]> * Upgrade Tika libraries from 2.2.0 to 2.2.1 Also update PDFBox to 2.0.25 as per Tika release notes Signed-off-by: Kartik Ganesh <[email protected]> * Upgraded Tika and xmlbeans libraries Tika libraries have been upgraded from 2.2.1 to 2.3.0. xmlbeans is now a subproject of POI, so POI was upgraded from 4.1.2 to 5.2.2. With POI 5.x the ooxml-schemas library has been moved to ooxml-lite/ooxml-full. Since ooxml-schemas no longer exists, the LICENSE and NOTICE files in the licenses/ directory have been removed. Finally, xmlbeans has been updated from 3.1.0 to 5.0.2 Signed-off-by: Kartik Ganesh <[email protected]> * (In progress) Added tika-langdetect Signed-off-by: Kartik Ganesh <[email protected]> * Upgrading tika libraries to 2.4.0 Signed-off-by: Kartik Ganesh <[email protected]> * Switched from tika-langdetect to tika-langdetect-optimaize To fix the license check, the mapping regex was expanded to tika-.* This now means the tika-core LICENSE and NOTICE files are no longer needed. Signed-off-by: Kartik Ganesh <[email protected]> * (Work in progress) Switching AttachmentProcessor to use OptimaizeLangDetector This is a concrete implementation of LanguageDetector. Using this requires bringing in the optimaize dependency. Signed-off-by: Kartik Ganesh <[email protected]> * Manually added LICENSE and NOTICE files for Optimaize language-detector Signed-off-by: Kartik Ganesh <[email protected]> * Move Optimaize dependency to runtimeOnly Also bring in transitive Guava dependency. This requires manual addition of LICENSE and NOTICE files as with other plugins. Signed-off-by: Kartik Ganesh <[email protected]> * Fix Optimaize langDetector to load models first before detecting Signed-off-by: Kartik Ganesh <[email protected]> * Fallback logic, and test updates Following the Tika library upgrade, some fallback logic is necessary: 1. "Author" is deprecated for MSOffice document parsing. It is recommended to use CREATOR from Tika Core Properties instead. 2. EPUB parsing no longer automatically extracts keywords. The convention to fall back to SUBJECT is now manually implemented in AttachmentProcessor Finally, unit tests have been upgraded to account for non-deterministic language results across library upgrades. Signed-off-by: Kartik Ganesh <[email protected]> * Drop Guava version from 31.1 to 18.0 This is the version that Optimaize 0.6 depends on, and it allows for a smaller ignoreViolations list Signed-off-by: Kartik Ganesh <[email protected]> * Fix ingest-attachment integration test to assert correct language Signed-off-by: Kartik Ganesh <[email protected]> (cherry picked from commit fc0f446)
kartg
approved these changes
May 10, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport fc0f446 from #3111