feat: added tokenizer params to the listing

[adubovik]

Added params related to tokenization to the listing as part of #54

DIAL core config:

• features.tokenizeEndpoint (URL optional) - the endpoint which allows to tokenize prompt
• features.truncatePromptEndpoint (URL optional) - the endpoint which serves context trimming API call. It trims the chat history up to the point when it fit requested number of tokens specified in request field max_prompt_tokens.
• tokenizerModel (string optional) - the reference model whose tokenization algorithm is matching the tokenization algorithm of the given model. It allows a user to make tokenization on their side. It's possible for models whose tokenization algorithm is publically known and implemented in a SDK (e.g. GPT and Anthropic). Models fall into families which share the same tokenization algorithm. tokenizer.referenceModel basically points to a representative model from a corresponding family. As a rule of thumb, choose the oldest representative from a family. See the families below.

Listing:

• features.tokenize (boolean optional) - true means that the core has <server host>/v1/deployments/<deployment name>/tokenize endpoint
• features.truncate_prompt (boolean optional) - true means that the core has <server host>/v1/deployments/<deployment name>/truncate_prompt endpoint
• tokenizer_model (string optional) - same value as in the core config

Model families

_tokenization_families:
  - gpt:
    - family_1:
      - gpt-3.5-turbo-0613
      - gpt-3.5-turbo-16k-0613
      - gpt-4-0314
      - gpt-4-32k-0314
      - gpt-4-0613
      - gpt-4-32k-0613
      # tokens_per_message = 3, tokens_per_name = 1
    - family_2:
      - gpt-3.5-turbo-0301
      # tokens_per_message=4, tokens_per_name=-1
    - family_3:
      - text-embedding-ada-002
      # just a string
  - gpt_encoding: cl100k_base
  - gpt_refs:
    - https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb
    - https://platform.openai.com/docs/models
    - https://tiktokenizer.vercel.app/
  - palm:
    - family_1:
      - chat-bison@001
      - codechat-bison@001
      - textembedding-gecko@001
  - anthropic:
    - family_1:
      - anthropic.claude-instant-v1
      - anthropic.claude-v1
      - anthropic.claude-v2
  - anthropic_refs:
    - https://github.com/anthropics/anthropic-sdk-python/blob/main/src/anthropic/_tokenizers.py