Extract dependencies from core list

[ZanSara]

Installing Haystack is really hard. It shouldn't be. This issue is about tracking efforts to remove dependencies from the core list and try-catch import statements.

Here is a list of all the dependencies we need to act on. Try-catch the dependencies aggressively and add an error message stating the dedicated extra to install. The goal is to be able to run import haystack without that dependency installed and having it fail only when a specific class is being initialized.

Steps

• Drop Python 3.7 support #4419
• Remove conditional statements from tiktoken (the issue has been resolved): build: Remove tiktoken alternative #4991
• Extract elasticsearch #4667
• Extract Transformers. The heaviest and harder to extract. Let's make a single extra transformers for all the packages involved. Packages to address are (likely). We decided to not remove transformers out of the core dependencies. Even HFInvocationLayer in PromptNode requires it. So the only thing one could do without transformers is PromptNode with OpenAI. Let's focus on other topics instead, such as making torch installation optional
  • transformers[torch] Related to Allow exclusion of torch GPU dependencies during installation #4233
  • huggingface-hub
  • sentence-transformers
  • See feat: optional transformers #5101
• Extract eval and statistical packages #4455: we can make a metrics extra
  • scipy
  • seqeval
  • mlflow
  • rapidfuzz
• Extract preprocessing and file conversion utils #4604: we make two extras, preprocessing and file-conversion
  • nltk
  • python-docx
  • langdetect
  • tika
• build: Move Azure's Form Recognizer dependency to extras #5096
  • pandas: It's not only required for tables but also for pipeline evaluation. eval results are stored in dataframes.
  • azure-ai-formrecognizer

Other libraries to investigate

• mmh3: Remove mmh3 dependency #4847
• protobuf: Install protobuf through transformers[sentencepiece] extra #4988
• dill: build: Remove dill dependency #4985
• qualtulum3: We're using its parser in our TableReader to check if all cells contain at least one numerical value and that all values share the same unit. That functionality could be become a part of Haystack's utility functions but it's more than just a few lines so I would keep it as is.
• tiktoken needed for ~3 times faster tokenization, for example with PromptNode (see build: Remove tiktoken alternative #4991)

Outcome

The resulting list of core dependencies will look like:

"requests",  # API calls
"pydantic",  # for the primitive classes
"pandas", # basic dataframes handling
"rank_bm25",  # for InMemoryDocumentStore
"scikit-learn", # tfidf, sklearnqueryclassifier and some other metrics  -> might be removed later
"tqdm",  # progress bars in model download and training scripts
"networkx",  # graphs library
"posthog",  # telemetry
"tenacity",  # retry decorator
"jsonschema",   # Schema validation
"more_itertools" # Utilities

The above happily installs in a couple of seconds. Let's get to it! 🚀