Deployment

With all prerequisites are met, you are ready to deploy the Akcio system.

Configure

You can configure all arguments by modifying config.py to set up your system with default modules. You can find detailed instructions at Configuration Guide.

Click to see default configurations.

import os

################## LLM ##################
LLM_OPTION = os.getenv('LLM_OPTION', 'openai')  # select your LLM service
CHAT_CONFIG = {
    'openai': {
        'openai_model': 'gpt-3.5-turbo',
        'openai_api_key': None,  # will use environment  value 'OPENAI_API_KEY' if None
        'llm_kwargs': {
            'temperature': 0.8,
            # 'max_tokens': 200,
            }
    },
    'ernie': {
        'ernie_api_key': None, # If None, use environment  value 'ERNIE_API_KEY'
        'ernie_secret_key': None, # If None, use environment value 'ERNIE_SECRET_KEY'
        'llm_kwargs': {}
    },
    'minimax': {
        'minimax_model': 'abab5-chat',
        'minimax_api_key': None, # If None, use environment value 'MINIMAX_API_KEY'
        'minimax_group_id': None, # If None, use environment value 'MINIMAX_GROUP_ID'
        'llm_kwargs': {}
    },
    'dolly': {
        'dolly_model': 'databricks/dolly-v2-3b',
        'llm_kwargs': {'device': 'auto'}
    },
    'skychat': {
        'skychat_api_host': None, # If None, use default value 'sky-api.singularity-ai.com'
        'skychat_app_key': None, # If None, use environment value 'SKYCHAT_APP_KEY'
        'skychat_app_secret': None  # If None, use environment value 'SKYCHAT_APP_SECRET'
    },
    'dashscope': {
        'dashscope_model': 'qwen-plus-v1',
        'dashscope_api_key': None  # If None, use environment value 'DASHSCOPE_API_KEY'
    },
    'chatglm':{
        'chatglm_model': 'chatglm_130b',
        'chatglm_api_key': None  # If None, use environment value 'ZHIPUAI_API_KEY'
    }
}


################## Embedding ##################
TEXTENCODER_CONFIG = {
    'model': 'multi-qa-mpnet-base-cos-v1',
    'norm': True,
    'dim': 768
}


################## Store ##################
USE_SCALAR = os.getenv('USE_SCALAR', False)

# Vector db configs
VECTORDB_CONFIG = {
    'connection_args': {
        'uri': os.getenv('ZILLIZ_URI', 'https://localhost:19530'),
        'token': os.getenv('ZILLIZ_TOKEN', None)
        },
    'top_k': 10,
    'threshold': 0.6,
    'index_params': {
        'metric_type': 'IP',
        'index_type': 'IVF_FLAT',
        'params': {'nlist': 1024}
        }
}

# Scalar db configs
SCALARDB_CONFIG = {
    'connection_args': {
        'hosts': os.getenv('ES_HOSTS', 'https://localhost:9200'),
        'ca_certs': os.getenv('ES_CA_CERTS', None),
        'basic_auth': (os.getenv('ES_USER', 'user_name'), os.getenv('ES_PASSWORD', 'es_password'))
        },
}

# Memory db configs
MEMORYDB_CONFIG = {
    'connect_str': os.getenv('SQL_URI', 'sqlite:///./sqlite.db')
}


############### Rerank configs ##################
RERANK_CONFIG = {
    'rerank': True,
    'rerank_model': 'cross-encoder/ms-marco-MiniLM-L-12-v2',
    'threshold': 0.6
}

################## Data loader ##################
DATAPARSER_CONFIG = {
    'chunk_size': 300
}

QUESTIONGENERATOR_CONFIG = {
    'model_name': 'gpt-3.5-turbo',
    'temperature': 0,
}

To simply set up default modules with your environment, you just need to follow commands below:

• Set up LLM (eg. OpenAI)

  $ export OPENAI_API_KEY=your_keys_here

  Check how to SWITCH LLM.

  If you want to use another supported LLM service, you can change the LLM option and set up for it. Besides directly modifying the configuration file, you can also set up via environment variables. For example, to use Ernie instead of OpenAI, you need to change the option and set up Ernie API key & secret key:

  $ export LLM_OPTION=ernie
  $ export ERNIE_API_KEY=your_ernie_api_key
  $ export ERNIE_SECRET_KEY=your_ernie_secret_key

• Set up Vector Store (eg. Milvus or Zilliz Cloud)

  $ export ZILLIZ_URI=https://localhost:19530

• (Optional) Set up Scalar Store (eg. Elastic)

  To enable scalar store, set USE_SCALAR=True in configuration.

  $ export ES_HOSTS=https://localhost:9200
  $ export ES_CA_CERTS=path/to/es_ca_certs
  $ export ES_USER=elastic
  $ export ES_PASSWORD=your_es_password

• Set up Memory Store (eg. Postgresql)

  $ export SQL_URI=sqlite:///./sqlite.db

  Make sure you have the database service ready with the proper uri. Please note that LangChain mode only supports Postgresql for now.

Run

Option 1: FastAPI

The Akcio source code provides a script to run a FastAPI service with the default address localhost:8900.

1. Start with Towhee or LangChain:

$ python main.py --towhee
# $ python main.py --langchain

2. Visit via browser:

Visit https://localhost:8900/docs in browser to access the web service.

/: Check service status

/answer: Generate answer for the given question, with assigned session_id and project

/project/add: Add data to project (will create the project if not exist)

/project/drop: Drop project including delete data in both vector and memory storages.

For more details about how to use these APIs, refer to Online Operations.

Option 2: Gradio

You can also start a service with simple user interface powered by Gradio:

1. Start with Towhee or LangChain:

$ python gradio_demo.py --towhee
# $ python gradio_demo.py --langchain

2. Visit via browser:

Visit https://localhost:8900 (you can find your url from the output of step 1) in browser to access the demo.

For more details about Gradio demo, refer to Demo.