Disable special token restriction for claude ai

[xihajun]

<|endoftext|> is not the special token for claude ai

Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.

Traceback (most recent call last):
  File "/Users/xihajun/Library/Python/3.9/lib/python/site-packages/litellm/main.py", line 1028, in completion
    response = anthropic.completion(
  File "/Users/xihajun/Library/Python/3.9/lib/python/site-packages/litellm/llms/anthropic.py", line 203, in completion
    encoding.encode(prompt)
  File "/Users/xihajun/Library/Python/3.9/lib/python/site-packages/tiktoken/core.py", line 117, in encode
    raise_disallowed_special_token(match.group())
  File "/Users/xihajun/Library/Python/3.9/lib/python/site-packages/tiktoken/core.py", line 400, in raise_disallowed_special_token
    raise ValueError(
ValueError: Encountered text corresponding to disallowed special token '<|endoftext|>'.
If you want this text to be encoded as a special token, pass it to `allowed_special`, e.g. `allowed_special={'<|endoftext|>', ...}`.
If you want this text to be encoded as normal text, disable the check for this token by passing `disallowed_special=(enc.special_tokens_set - {'<|endoftext|>'})`.
To disable this check for all special tokens, pass `disallowed_special=()`.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
litellm	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Mar 5, 2024 3:51pm
litellm-dashboard	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Mar 5, 2024 3:51pm