feat(db): refactor db to support redis

[shreypandey]

Closes #782

📑 Description

✅ Checks

• My pull request adheres to the code style of this project
• My code requires changes to the documentation
• I have updated the documentation as required
• All the tests have passed
• Branch name follows type/descript (e.g. feature/add-llm-agents)
• Ready for code review

ℹ Additional Information

[shreypandey]

Hi @lwaekfjlk ,
I have created this PR for refactoring code for database clients as discussed in #428.
This adds the abstract class for database client, which can be extended by other clients such as redis in subsequent PRs.

Please review this PR once and let me know your thoughts on this.

[lwaekfjlk]

thanks, I will check later today

[shreypandey]

Hi @lwaekfjlk ,
I have updated the documentation to run the code. Please review the updated README.md for that.

[lwaekfjlk]

@shreypandey wow, your PR is amazing! Let's work together on more things!

[lwaekfjlk]

I just leave some little comments, generally looks amazingly good to me.

[lwaekfjlk]

what do you think is the advantage for using env compared with using export ENV_NAME.

For env, maybe add something like OPENAI_API_KEY="xxx"?

[shreypandey]

what do you think is the advantage for using env compared with using export ENV_NAME.

Hey @lwaekfjlk ,
When we integrate redis(or some other vectorDB), they generally have their set of ENV variables we need to provide.
This is similar to the scenario when we want to use any other llm provider(such as azure etc) apart from default openai in litellm.

So, to accommodate the increasing number of env variables, it'll be easier to set the env variables by sourcing the env file rather than individually setting each variable. In shell, the command to source the env file looks something like this:

$ set -a
$ source .env
$ set +a

Let me know your thoughts on this.

[shreypandey]

For env, maybe add something like OPENAI_API_KEY="xxx"?

The idea behind this is, that .env.template file is a placeholder file which specify all the env variable names the project is using. Users will create a .env file based on this template(they can just copy the entire template file).

Putting empty value denotes that the variable wasn't set by the user and can be handled appropriately in python code. Something like

api_key = os.getenv("OPENAI_API_KEY") # given empty string if variable is not set
asset api_key # Assertion error if API_KEY is not set, similar to not setting the value

Let me know your thoughts.

[shreypandey]

@shreypandey wow, your PR is amazing! Let's work together on more things!

Sure @lwaekfjlk ! Would love to work on more things!

[shreypandey]

Converting this to draft as waiting for #785 to merge. I will resolve the conflicts post merging #785 and reopen this.

[lwaekfjlk]

@shreypandey thanks for your efforts, I merged #785

[shreypandey]

Hi @lwaekfjlk,
I have resolved the conflicts and reopened the PR.
Please review the PR and let me know your thoughts on this. The PR is ready to merge now.

I have tested the code by running the test cases, examples and demo.

[lwaekfjlk]

sure. This is a big PR so I need to take some time and confirm on my side everything in bench works as well.

[shreypandey]

Sure @lwaekfjlk! Let me know your feedback and incase of any issues.

[lwaekfjlk]

This PR basically add an abstract layer to link to multiple types of basic support of DB (local / redis). The concrete Redis support needs to be implemented later.