feat: introduce integration with Gradient AI

[mhaligowski]

Related Issues

Doesn't have have a related issue.

Proposed Changes:

Gradient AI is a self-managed cloud service for Large Language Models, offering fine-tuning and inference of open-source models and embeddings generation. This PR integrates haystack with Gradient for LLMs and embeddings by adding relevant generators and embedders.

How did you test it?

• unit testing,
• e2e test based on the RAG test.

Notes for the reviewer

• please note that we added a new optional dependency set that needs to be installed with pip install -e .[gradient],
• integration requires a generated access token, and tests assume set up environment variables. Please refer to our docs for SDK or for CLI. The CLI can set up environment variables with $(gradient env) command once it's configured.

Checklist

[x] I have read the contributors guidelines and the code of conduct
[x] I have updated the related issue with new insights and changes N/A
[x] I added unit tests and updated the docstrings
[x] I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test:. NOTE: to be squashed.
[x] I documented my code
[x] I ran pre-commit hooks and fixed any issue

[CLAassistant]

All committers have signed the CLA.

[coveralls]

Pull Request Test Coverage Report for Build 6659953333

• 0 of 0 changed or added relevant lines in 0 files are covered.
• 1 unchanged line in 1 file lost coverage.
• Overall coverage decreased (-0.007%) to 50.839%

---

Files with Coverage Reduction	New Missed Lines	%
utils/context_matching.py	1	95.7%

Totals
Change from base Build 6656519629:	-0.007%
Covered Lines:	13032
Relevant Lines:	25634

---

💛 - Coveralls

[julian-risch]

Great to see this PR! Looks quite good to me already. I was able to run the end-to-end test locally. 👍
I have some smaller change requests in the comments below. Please also generate a release note with reno as described here: https://github.com/deepset-ai/haystack/blob/main/CONTRIBUTING.md#release-notes You can put your text under preview in that note.

[mhaligowski]

Great to see this PR! Looks quite good to me already. I was able to run the end-to-end test locally. 👍 I have some smaller change requests in the comments below. Please also generate a release note with reno as described here: https://github.com/deepset-ai/haystack/blob/main/CONTRIBUTING.md#release-notes You can put your text under preview in that note.

Added the release notes! let me know

[ZanSara]

@mhaligowski Let me try to help you with the CI:

• The unit test failure is due to feat: upgrade canals to 0.10.1 #6309, so you need to update the serialization tests to match
• Linting is failing because it's still missing the gradientai dependency
• the e2e failure can be fixed by merging main into your branch, it's fixed in main.

Let me know if you need more help!

[ZanSara]

@mhaligowski here are the fixes: Preemo-Inc#12 Please merge them into your fork and let's see if they help.

One more thing: I noticed that you implemented to_dict, but not from_dict methods yet. Could you implement them? You can check the other embedders to see how it is done.

[mhaligowski]

applied the fixes, trying to keep up with your main :)

[ZanSara]

Left a fix

[ZanSara]

The preview package is gone, here is the fix:

Suggested change

	from haystack.preview import Pipeline, Document
	from haystack.components.embedders.gradient_document_embedder import GradientDocumentEmbedder
	from haystack.components.embedders.gradient_text_embedder import GradientTextEmbedder
	from haystack.preview.document_stores import InMemoryDocumentStore
	from haystack.preview.components.writers import DocumentWriter
	from haystack.preview.components.retrievers import InMemoryEmbeddingRetriever
	from haystack.preview.components.generators.gradient.base import GradientGenerator
	from haystack.components.builders.answer_builder import AnswerBuilder
	from haystack.components.builders.prompt_builder import PromptBuilder
	from haystack.core import Pipeline, Document
	from haystack.components.embedders.gradient_document_embedder import GradientDocumentEmbedder
	from haystack.components.embedders.gradient_text_embedder import GradientTextEmbedder
	from haystack.document_stores import InMemoryDocumentStore
	from haystack.components.writers import DocumentWriter
	from haystack.components.retrievers import InMemoryEmbeddingRetriever
	from haystack.components.generators.gradient.base import GradientGenerator
	from haystack.components.builders.answer_builder import AnswerBuilder
	from haystack.components.builders.prompt_builder import PromptBuilder

[mhaligowski]

looks like the GradientGenerator wasn't moved out of the preview anyways, so I moved it. All the Gradient tests are passing now in my box!

[masci]

Superseded by deepset-ai/haystack-core-integrations#72