Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(ai-plugins): streamline AI Proxy streaming system & add compatibilty for existing "client SDKs" #12903

Merged
merged 24 commits into from
Apr 26, 2024
Merged
Changes from 1 commit
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
2eb5c45
feat(ai-proxy): complete refactor of streaming subsystem
tysoekong Apr 22, 2024
2ce4587
feat(ai-proxy): added variable stencil mechanism for routing
tysoekong Apr 22, 2024
8ae0d51
feat(ai-proxy): folded in features from #12807
tysoekong Apr 22, 2024
6db54a2
fix(ai-proxy): azure missing url override
tysoekong Apr 22, 2024
4602277
feat(ai-proxy): changelog
tysoekong Apr 22, 2024
64f6a28
fix)(ai-proxy): #12903 fixes rollup
tysoekong Apr 23, 2024
bc251f9
feat(ai-proxy): make merge config defaults a shared feature
tysoekong Apr 24, 2024
ee642d1
fix(ai-proxy): function docs
tysoekong Apr 24, 2024
fc14768
fix(lint): ai-proxy lint
tysoekong Apr 24, 2024
a26dfb9
feat(ai-proxy): add 3.7 compatibility checkers
tysoekong Apr 24, 2024
8c0fba5
fix(ai-proxy): no longer require streaming conf option
tysoekong Apr 24, 2024
28dd7c4
fix(ai-proxy): latest fix rollup
tysoekong Apr 25, 2024
a1d432a
fix(ai-proxy): latest fix rollup
tysoekong Apr 25, 2024
35de2c7
Add calculate cost logic
AntoineJac Apr 25, 2024
bb845c2
Fix calculate cost function
AntoineJac Apr 25, 2024
47235e0
Fix error
AntoineJac Apr 25, 2024
be9c9b8
fix(ai-proxy): fix lint
AntoineJac Apr 25, 2024
33bc5a7
fix(ai-proxy): fix lint
AntoineJac Apr 25, 2024
2aef219
fix(ai-proxy): fix lint
AntoineJac Apr 25, 2024
45a062a
fix(ai-proxy): fix bad request case
AntoineJac Apr 25, 2024
ea61c0d
fix(ai-proxy): cohere analytics
tysoekong Apr 25, 2024
35c0f95
fix(ai-proxy): missing null checks from rebase; missing conf_m plugin…
tysoekong Apr 25, 2024
5d19ded
fix(ai-proxy): another null check
tysoekong Apr 25, 2024
748f361
fix(ai-proxy) fix missing __key__
AntoineJac Apr 25, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix(ai-proxy): fix lint
AntoineJac authored Apr 25, 2024

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
commit 2aef21982b11504799672dfa47d9776e6ddb966b
6 changes: 2 additions & 4 deletions kong/llm/drivers/shared.lua
Original file line number Diff line number Diff line change
@@ -416,8 +416,7 @@ function _M.pre_request(conf, request_table)
end

-- log tokens prompt for reports and billing
local prompt_tokens, err
prompt_tokens, err = _M.calculate_cost(request_table, {}, 1.0) or 0
local prompt_tokens, err = _M.calculate_cost(request_table, {}, 1.0)
if err then
kong.log.warn("failed calculating cost for prompt tokens: ", err)
end
@@ -509,8 +508,7 @@ function _M.post_request(conf, response_object)
end

-- log tokens response for reports and billing
local response_tokens, err
response_tokens, err = _M.calculate_cost(response_object, {}, 1.0) or 0
local response_tokens, err = _M.calculate_cost(response_object, {}, 1.0)
if err then
kong.log.warn("failed calculating cost for response tokens: ", err)
end