Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Chore: Sync dev to main #1696

Merged
merged 978 commits into from
Nov 18, 2024
Merged
Changes from 1 commit
Commits
Show all changes
978 commits
Select commit Hold shift + click to select a range
a043b9f
chore: refactor inference service (#1536)
vansangpfiev Oct 23, 2024
e394199
Merge pull request #1539 from janhq/j/update-openapi-reference-name
namchuai Oct 23, 2024
64b1d1e
ci: fix docusaurus (#1540)
hiento09 Oct 23, 2024
fd3f5cc
fix: incorrect downloadedBytes callback
namchuai Oct 23, 2024
f0f1e99
Update comment
namchuai Oct 23, 2024
4486408
Merge pull request #1542 from janhq/j/update-download-percentage-event
namchuai Oct 23, 2024
53e8f20
feat: add customize parameters to /v1/models/start (#1544)
vansangpfiev Oct 24, 2024
10b64ee
fix: remove embeddings from CLI help (#1543)
vansangpfiev Oct 24, 2024
22e344b
feat: Dockerfile (#1473)
hiento09 Oct 24, 2024
9fe0c6f
feat: add more parameters to /v1/models/start (#1545)
vansangpfiev Oct 24, 2024
a2a6550
fix: bypass model check for mmproj model (#1547)
vansangpfiev Oct 24, 2024
dd2fcfb
chore: copy edits
freelerobot Oct 25, 2024
8ca2012
chore: update docker readme (#1552)
hiento09 Oct 25, 2024
8c14e43
fix: crash while downloading model
namchuai Oct 25, 2024
5ec0b4f
chore: update docker readme (#1553)
hiento09 Oct 25, 2024
113ea98
Merge pull request #1554 from janhq/j/fix-crash
namchuai Oct 25, 2024
8a6387f
fix: only set dll search path if don't use ENGINE_PATH (#1557)
vansangpfiev Oct 28, 2024
58247a0
new: cortex enterprise page and form
Oct 28, 2024
3c4eebd
new: cortex enterprise and form
Oct 28, 2024
a7a7b16
chore: add enteprise menu navbar and button action shedule a call
urmauur Oct 28, 2024
c7915cf
update: Announcement copy and link
Oct 28, 2024
2380153
Merge pull request #1558 from janhq/feature/cortex-enterprise
urmauur Oct 28, 2024
5ef68b7
fix: broken api-reference page
urmauur Oct 29, 2024
04c5c40
Merge pull request #1561 from janhq/fix/api-reference
urmauur Oct 29, 2024
00af979
feat: cortex pull and cortex engines install CLI uses API server (#1550)
vansangpfiev Oct 29, 2024
d856ab4
Merge branch 'dev' into copyfix
freelerobot Oct 29, 2024
106ddd4
Merge pull request #1549 from janhq/copyfix
gabrielle-ong Oct 29, 2024
6cda3ee
fix: add n_parallel to model yaml config (#1571)
vansangpfiev Oct 30, 2024
ee41d48
fix: engines install pre-release (#1577)
vansangpfiev Oct 30, 2024
b5051ed
Update README.md
gabrielle-ong Oct 30, 2024
09434de
add nightly- pulls llama.cpp everynight
gabrielle-ong Oct 30, 2024
20c7579
update: enterprise calendar to get in touch form
Oct 28, 2024
c9f3512
fix: form get in touch
urmauur Oct 30, 2024
30cef62
update models
gabrielle-ong Oct 30, 2024
4d82faf
remove quicklinks
gabrielle-ong Oct 30, 2024
c6c8e03
Merge pull request #1560 from janhq/feature/cortex-enterprise
urmauur Oct 30, 2024
d54a7f8
fix: remove package json root
urmauur Oct 30, 2024
75cf152
Merge pull request #1585 from janhq/fix/remove-package-json
urmauur Oct 30, 2024
2149180
Merge branch 'dev' into docs-readme
gabrielle-ong Oct 30, 2024
fcf2fb3
Merge pull request #1581 from janhq/docs-readme
gabrielle-ong Oct 30, 2024
616fbe8
chore: bug report template
gabrielle-ong Oct 29, 2024
4c3803a
chore: bug report template
gabrielle-ong Oct 29, 2024
6c53282
chore: remove embeddings from sidebar
gabrielle-ong Oct 29, 2024
2e66496
models page: remove onnx trt toggle
gabrielle-ong Oct 30, 2024
b600af9
hero logo: remove onnx trt, add model variants
gabrielle-ong Oct 30, 2024
280c94b
Remove discord links
gabrielle-ong Oct 30, 2024
9b7eb44
Remove beta/nightly/onnx/trt from quickstart
gabrielle-ong Oct 30, 2024
e6a5548
Quickstart additional commands
gabrielle-ong Oct 30, 2024
0a56d50
Overview - models
gabrielle-ong Oct 30, 2024
2c44b1e
chore: readme
gabrielle-ong Oct 30, 2024
8091ade
update chat completion object
nguyenhoangthuan99 Oct 30, 2024
041f6c6
Merge pull request #1586 from janhq/docs
gabrielle-ong Oct 30, 2024
a6dae38
chore: update margin list avail download
urmauur Oct 30, 2024
42d416a
Merge pull request #1588 from janhq/chore/margin-list-download
urmauur Oct 30, 2024
53d5c80
Update chat completion response api
nguyenhoangthuan99 Oct 30, 2024
52786d9
Merge branch 'dev' into feat/api-docs
nguyenhoangthuan99 Oct 30, 2024
57ed629
chore: add download success e2e test (#1556)
namchuai Oct 30, 2024
6d553f9
chore: bump vcpkg to 2024.10.21 Release 10b7a17 (#1580)
vansangpfiev Oct 31, 2024
9a024fb
Merge branch 'dev' into feat/api-docs
nguyenhoangthuan99 Oct 31, 2024
6293903
fix: add notification for future plan
nguyenhoangthuan99 Oct 31, 2024
cc6a303
Merge branch 'feat/api-docs' of github.com:janhq/cortex.cpp into feat…
nguyenhoangthuan99 Oct 31, 2024
a529cd2
add api swagger for server check health and terminate process
nguyenhoangthuan99 Oct 31, 2024
26f3d6f
fix: download engine does not show progress (#1592)
vansangpfiev Oct 31, 2024
07caf8e
Merge branch 'dev' into feat/api-docs
nguyenhoangthuan99 Oct 31, 2024
0ebada2
fix: cache total bytes for download event (#1598)
vansangpfiev Oct 31, 2024
601437d
fix: cache by download item id (#1599)
vansangpfiev Oct 31, 2024
f80edba
Merge branch 'dev' into feat/api-docs
nguyenhoangthuan99 Oct 31, 2024
166cdb5
fix: use download event type to listen ws on client side (#1601)
vansangpfiev Nov 1, 2024
f5fbad6
chore: add model name as a parameter support during import via API (#…
louis-menlo Nov 1, 2024
f37ad6b
chore: beta version add discord notification (#1605)
hiento09 Nov 1, 2024
11cd87b
Feat e2e test cortexso hub (#1590)
hiento09 Nov 1, 2024
322a0cf
chore: notification discord add binary download url (#1607)
hiento09 Nov 1, 2024
f6978cd
chore: auto set pre-release for beta version
hiento09 Nov 1, 2024
43fbc52
chore: auto set pre-release for beta version (#1608)
hiento09 Nov 1, 2024
152b76f
fix: progress bar on CMD (#1609)
vansangpfiev Nov 1, 2024
f7e694a
Merge branch 'dev' into feat/api-docs
dan-menlo Nov 1, 2024
5802113
Merge pull request #1589 from janhq/feat/api-docs
dan-menlo Nov 1, 2024
02ae2c1
Refactor Cortex Docs structure
dan-menlo Nov 2, 2024
4d59e92
Update sidebars and OpenAPI spec
dan-menlo Nov 2, 2024
8c695c6
Merge branch 'dev' into d/cortex-docs-structure
dan-menlo Nov 2, 2024
6ca33f2
Merge branch 'd/cortex-docs-structure' of https://github.com/janhq/co…
dan-menlo Nov 2, 2024
d303c72
Scaffold key points for Hardware Awareness
dan-menlo Nov 2, 2024
5d8157b
Remove model.list
dan-menlo Nov 2, 2024
ca6ef40
Update links
dan-menlo Nov 2, 2024
58be664
Update broken links
dan-menlo Nov 2, 2024
267403e
Merge pull request #1612 from janhq/d/cortex-docs-structure
dan-menlo Nov 2, 2024
1b0e0f8
chore: add document for function calling
nguyenhoangthuan99 Nov 3, 2024
fd2a475
chore: add example on structure output
nguyenhoangthuan99 Nov 3, 2024
76d653f
fix: server uploads to temporary directory (#1610)
vansangpfiev Nov 4, 2024
5fde673
Merge branch 'dev' of github.com:janhq/cortex.cpp into dev
hiento09 Nov 4, 2024
1d53a8e
chore: add document for model vs engine
nguyenhoangthuan99 Nov 4, 2024
52f1aea
Merge branch 'dev' into chore/model-run-docs-update
nguyenhoangthuan99 Nov 4, 2024
bd38e71
Merge branch 'dev' of github.com:janhq/cortex.cpp into chore/model-ru…
nguyenhoangthuan99 Nov 4, 2024
6e11aa4
chore: model pull docs
namchuai Nov 4, 2024
7e83ab6
docs: docker installation instruction
hiento09 Nov 4, 2024
870fb47
update
namchuai Nov 4, 2024
e3eb215
chore: correct build steps (#1578)
vansangpfiev Nov 4, 2024
23c12e1
chore: change tag
vansangpfiev Nov 4, 2024
e72eec2
Update model.yml documentation
nguyenhoangthuan99 Nov 4, 2024
2559561
Merge branch 'dev' into chore/model-run-docs-update
nguyenhoangthuan99 Nov 4, 2024
d9b0856
update
namchuai Nov 4, 2024
df70397
chore: add Running Models tag
vansangpfiev Nov 4, 2024
815c077
fix: print error for models get non exists model (#1624)
vansangpfiev Nov 4, 2024
301dbe1
Merge branch 'dev' into chore/model-run-docs-update
nguyenhoangthuan99 Nov 4, 2024
5681ec6
chore: running models
vansangpfiev Nov 4, 2024
75d44ac
Merge branch 'chore/model-run-docs-update' of github.com:janhq/cortex…
nguyenhoangthuan99 Nov 4, 2024
a4df116
Update function calling note
nguyenhoangthuan99 Nov 4, 2024
4530042
chore: pulling models APIs
vansangpfiev Nov 4, 2024
d5d751a
chore: cortexrc docs update (#1620)
vansangpfiev Nov 4, 2024
f231e76
chore: add cortex.db architecture docs (#1619)
vansangpfiev Nov 4, 2024
b558009
chore: update structure output documentation
nguyenhoangthuan99 Nov 4, 2024
c483bcb
Merge branch 'dev' into chore/model-run-docs-update
nguyenhoangthuan99 Nov 4, 2024
82812a6
docs: installation linux instruction
hiento09 Nov 4, 2024
e3acf66
chore: updater docs (#1622)
vansangpfiev Nov 4, 2024
8961a0d
Merge pull request #1623 from janhq/j/update-model-pull-docs
namchuai Nov 4, 2024
5be4ec0
Cleanup CLI docs (remove chat, sidebar, nightly)
gabrielle-ong Nov 4, 2024
0ce34f8
chore: add more request body parameters for models start
vansangpfiev Nov 4, 2024
c9a2656
Update Chat completion api docs for llama-cpp supported params
nguyenhoangthuan99 Nov 4, 2024
5684fe6
chore: return model size after pulled (#1626)
louis-menlo Nov 4, 2024
42f6ae0
Merge branch 'dev' into chore/model-run-docs-update
nguyenhoangthuan99 Nov 4, 2024
f7aa5fe
chore: models update request body
vansangpfiev Nov 4, 2024
2691cfc
chore: models update
vansangpfiev Nov 4, 2024
4589ce7
Merge branch 'dev' into chore/models-api
vansangpfiev Nov 4, 2024
bd740a0
docs: installation windows, macos instruction
hiento09 Nov 4, 2024
16ba286
chore: server description
vansangpfiev Nov 4, 2024
38cde94
Merge branch 'chore/models-api' of github.com:janhq/nitro into chore/…
vansangpfiev Nov 4, 2024
98a0438
minor documentation corrections (#1611)
MikeBirdTech Nov 4, 2024
55bbe0d
chore: add model import option parameter (#1627)
louis-menlo Nov 4, 2024
afcbb87
typo remove nightly
gabrielle-ong Nov 4, 2024
6a36e90
docs: update cortex folder session
hiento09 Nov 4, 2024
8306b39
Merge branch 'dev' of github.com:janhq/cortex.cpp into dev
hiento09 Nov 4, 2024
ee97382
Merge branch 'dev' into docs/installation
hiento09 Nov 4, 2024
ae95f80
Merge branch 'dev' into chore/model-run-docs-update
nguyenhoangthuan99 Nov 4, 2024
611901a
Merge branch 'chore/model-run-docs-update' of github.com:janhq/cortex…
nguyenhoangthuan99 Nov 4, 2024
a5c0172
clean up models and engines, to refactor into separate pages
gabrielle-ong Nov 4, 2024
fe01a08
Merge branch 'dev' into feat/cli-docs-cleanup
gabrielle-ong Nov 4, 2024
7f52abe
Merge branch 'dev' into feat/cli-docs-models-engines
gabrielle-ong Nov 4, 2024
3c507fe
docs: different between nightly, beta and stablt
hiento09 Nov 4, 2024
a04a93a
docs: use tabs for multiple option in installation docs
hiento09 Nov 4, 2024
6b3795e
chore: adding engines docs
namchuai Nov 4, 2024
206650f
chore: update model's size on import (#1630)
louis-menlo Nov 4, 2024
c0a115f
Merge branch 'dev' into chore/models-api
vansangpfiev Nov 5, 2024
c44fb59
Merge pull request #1615 from janhq/chore/model-run-docs-update
gabrielle-ong Nov 5, 2024
0942fc1
Merge pull request #1621 from janhq/docs/installation
gabrielle-ong Nov 5, 2024
5338a78
Merge pull request #1631 from janhq/j/engines-docs
gabrielle-ong Nov 5, 2024
f8bf674
Merge branch 'dev' of github.com:janhq/nitro into chore/models-api
vansangpfiev Nov 5, 2024
9cc9e5d
Merge branch 'chore/models-api' of github.com:janhq/nitro into chore/…
vansangpfiev Nov 5, 2024
5ca8257
Merge pull request #1625 from janhq/chore/models-api
gabrielle-ong Nov 5, 2024
fd6333f
Merge branch 'dev' into feat/cli-docs-cleanup
gabrielle-ong Nov 5, 2024
c4b2759
Merge branch 'dev' into feat/cli-docs-models-engines
gabrielle-ong Nov 5, 2024
59d4a6f
feat: engine management
namchuai Oct 24, 2024
c6bce45
add update engine
namchuai Oct 30, 2024
35c7b6c
store default engine to cortex.rc
namchuai Oct 30, 2024
94060ce
finalizing
namchuai Oct 31, 2024
212749d
temporary remove cli
namchuai Oct 31, 2024
526312a
remove v2 commands
namchuai Oct 31, 2024
6240fc2
Add API references
namchuai Oct 31, 2024
fd961ce
fix ci
namchuai Oct 31, 2024
ae508f3
fix downloading engines with progress bar
namchuai Nov 1, 2024
dc43f6d
add -m flag
namchuai Nov 4, 2024
192a25d
fix ci
namchuai Nov 4, 2024
2848539
clean up
namchuai Nov 4, 2024
7b22b5a
add filter for variant list
namchuai Nov 4, 2024
835a654
update ld_library_path
namchuai Nov 5, 2024
25fc6fd
add dylib for windows
namchuai Nov 5, 2024
4b693f5
add dylib for linux
namchuai Nov 5, 2024
a322bac
remove /models-sources slug
gabrielle-ong Nov 5, 2024
dfcf355
replace /hub/ to /capabilities/models/sources broken links
gabrielle-ong Nov 5, 2024
c053cdd
yarn lock typo
gabrielle-ong Nov 5, 2024
ae7d3b5
broken link for deprecated /cli/chat
gabrielle-ong Nov 5, 2024
4793b3a
fix: network installer macos not install llama-cpp engine (#1633)
hiento09 Nov 5, 2024
4398417
chore: fix color scalar sidebar
urmauur Nov 5, 2024
f2de4b7
Merge pull request #1634 from janhq/fix/force-color-scalar-sidebar
urmauur Nov 5, 2024
39509bc
Merge pull request #1629 from janhq/feat/cli-docs-models-engines
gabrielle-ong Nov 5, 2024
1becaff
Merge pull request #1628 from janhq/feat/cli-docs-cleanup
gabrielle-ong Nov 5, 2024
3e04350
Merge pull request #1546 from janhq/j/engine-management
namchuai Nov 5, 2024
46caedf
Feat/function calling (#1572)
nguyenhoangthuan99 Nov 5, 2024
678bb3e
remove Architecture page, becomes generated index
gabrielle-ong Nov 5, 2024
56e900e
remove API server, combine into basic usage
gabrielle-ong Nov 5, 2024
42cef69
add Curl commands to Basic Usage
gabrielle-ong Nov 5, 2024
1bb23ca
Merge pull request #1637 from janhq/feat/docs-cleanup
gabrielle-ong Nov 5, 2024
90bcb79
fix: make cortex-nightly update -v works again (#1645)
vansangpfiev Nov 6, 2024
c48c7ee
v1.0.1 QA template (22 Oct 2024)
gabrielle-ong Nov 6, 2024
312e206
v1.0.2 QA checklist (6 Nov)
gabrielle-ong Nov 6, 2024
9e0834d
remove validation for bug report additional info
gabrielle-ong Nov 6, 2024
58fc107
Merge pull request #1647 from janhq/chore/github-issue-template
gabrielle-ong Nov 6, 2024
0b75004
fix: cors
namchuai Nov 6, 2024
c047ef8
feat: add api for configuration
namchuai Nov 6, 2024
aec5351
remove log
namchuai Nov 6, 2024
cdfda3a
chore: update API references
namchuai Nov 6, 2024
01a5d76
add: cli API
namchuai Nov 6, 2024
38233d8
fix build
namchuai Nov 6, 2024
229be62
fix build windows
namchuai Nov 7, 2024
086de70
Merge pull request #1640 from janhq/j/fix-cors
namchuai Nov 7, 2024
bc9544a
chore: support loglevel when start server (#1636)
nguyenhoangthuan99 Nov 7, 2024
481dfb3
fix: ci github ratelimit (#1649)
hiento09 Nov 7, 2024
6b4f9a5
update api
namchuai Nov 7, 2024
f738363
check latest llamacpp when start
namchuai Nov 7, 2024
ef9a966
remove some warnings
namchuai Nov 7, 2024
a674b9c
update
namchuai Nov 7, 2024
c3837c5
update
namchuai Nov 8, 2024
ceeaf51
fix build windows
namchuai Nov 8, 2024
1c516f0
Feat cicd docker (#1646)
hiento09 Nov 8, 2024
639b89f
update
namchuai Nov 8, 2024
c7765ec
Merge pull request #1650 from janhq/j/update-engine-mng
namchuai Nov 8, 2024
505cac8
fix: ci beta (#1651)
hiento09 Nov 8, 2024
ed39e16
docs: add cors docs
namchuai Nov 8, 2024
f648f63
chore/embeddings-docs
nguyenhoangthuan99 Nov 8, 2024
c8eab8a
chore: add embedding capabilities
nguyenhoangthuan99 Nov 8, 2024
71da65f
chore: remove un developed file
nguyenhoangthuan99 Nov 8, 2024
85f574a
chore: update download event
namchuai Nov 8, 2024
aa71b87
fix: download progress
vansangpfiev Nov 8, 2024
09599ec
Merge pull request #1658 from janhq/j/update-download-event
namchuai Nov 8, 2024
3a15db1
fix: update default cors to localhost
namchuai Nov 8, 2024
3cec0e0
feat: automation test docker (#1653)
hiento09 Nov 8, 2024
f354af6
Merge pull request #1659 from janhq/j/set-default-cors
namchuai Nov 8, 2024
5ec3a59
fix: ci qualitygate (#1664)
hiento09 Nov 11, 2024
7cf62d5
Fix/gemma2 chat template (#1657)
nguyenhoangthuan99 Nov 11, 2024
ac9c113
Merge pull request #1654 from janhq/chore/embeddings-docs
gabrielle-ong Nov 12, 2024
f700ba7
fix: correct remaining time (#1669)
vansangpfiev Nov 12, 2024
4434391
fix: incorrect mapper upload engines api
luke-nguyen990 Nov 13, 2024
187850a
chore: disable unit tests due to model got removed
luke-nguyen990 Nov 13, 2024
a221930
chore: disable test by using prefix
luke-nguyen990 Nov 13, 2024
a6d6be3
Merge pull request #1674 from janhq/fix/incorrect-mapper-unload-engines
luke-nguyen990 Nov 13, 2024
a055f69
feat: Hardware API (#1593)
vansangpfiev Nov 13, 2024
3d02299
fix: install local with nvidia-driver unavailable (#1676)
vansangpfiev Nov 13, 2024
9f9d294
feat: add load/unload engine cli
namchuai Nov 13, 2024
a41001f
feat: simultaneous download
namchuai Nov 11, 2024
6a7d288
handle error download
namchuai Nov 12, 2024
8d7c5fa
update
namchuai Nov 13, 2024
4dbca80
fix build windows
namchuai Nov 13, 2024
916e29f
Merge pull request #1678 from janhq/j/add-load-unload-cli
namchuai Nov 14, 2024
ccb9273
fix: bypass check if activate GPU list is empty (#1682)
vansangpfiev Nov 14, 2024
ce7af64
chore: correct discord message notification (#1686)
hiento09 Nov 14, 2024
9ae7e39
fix: download progress
sangjanai Nov 14, 2024
461601b
update
namchuai Nov 14, 2024
c01df03
fix ci
namchuai Nov 14, 2024
4c110bf
fix: abort download
namchuai Nov 14, 2024
11aaf19
Merge pull request #1665 from janhq/j/simultaneous-download
namchuai Nov 15, 2024
481a092
fix: support user defined engine path
namchuai Nov 14, 2024
3bf5f87
Merge pull request #1687 from janhq/j/update-user-defined-engine-path
namchuai Nov 15, 2024
65876fb
fix: prevent download event update after stopped
namchuai Nov 15, 2024
24bebed
Merge pull request #1689 from janhq/j/prevent-update-event-after-stopped
namchuai Nov 15, 2024
ba4c625
cleanup github readme + upcoming features
gabrielle-ong Nov 15, 2024
0075e5f
chore: CORS CLI doc
gabrielle-ong Nov 15, 2024
7e1dd01
Merge branch 'dev' into j/add-docs-cors
gabrielle-ong Nov 15, 2024
3c7022c
chore: update the wrong command for running built-in models
paopa Nov 16, 2024
7bf3f8f
Merge pull request #1693 from paopa/chore/update-readme-for-running-b…
namchuai Nov 17, 2024
2a8f187
Merge pull request #1691 from janhq/docs/v1.0.3-docs
gabrielle-ong Nov 18, 2024
6892823
Merge pull request #1652 from janhq/j/add-docs-cors
namchuai Nov 18, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions engine/cli/command_line_parser.cc
Original file line number Diff line number Diff line change
@@ -546,6 +546,7 @@ void CommandLineParser::ModelUpdate(CLI::App* parent) {
"stream",
"ngl",
"ctx_len",
"n_parallel",
"engine",
"prompt_template",
"system_template",
6 changes: 6 additions & 0 deletions engine/cli/commands/model_upd_cmd.cc
Original file line number Diff line number Diff line change
@@ -223,6 +223,12 @@ void ModelUpdCmd::UpdateConfig(Json::Value& data, const std::string& key,
data["ctx_len"] = static_cast<int>(f);
});
}},
{"n_parallel",
[this](Json::Value &data, const std::string& k, const std::string& v) {
UpdateNumericField(k, v, [&data](float f) {
data["n_parallel"] = static_cast<int>(f);
});
}},
{"tp",
[this](Json::Value &data, const std::string& k, const std::string& v) {
UpdateNumericField(k, v, [&data](float f) {
6 changes: 6 additions & 0 deletions engine/config/model_config.h
Original file line number Diff line number Diff line change
@@ -22,6 +22,7 @@ struct ModelConfig {
bool stream = std::numeric_limits<bool>::quiet_NaN();
int ngl = std::numeric_limits<int>::quiet_NaN();
int ctx_len = std::numeric_limits<int>::quiet_NaN();
int n_parallel = 1;
std::string engine;
std::string prompt_template;
std::string system_template;
@@ -125,6 +126,8 @@ struct ModelConfig {
ngl = json["ngl"].asInt();
if (json.isMember("ctx_len"))
ctx_len = json["ctx_len"].asInt();
if (json.isMember("n_parallel"))
n_parallel = json["n_parallel"].asInt();
if (json.isMember("engine"))
engine = json["engine"].asString();
if (json.isMember("prompt_template"))
@@ -204,6 +207,7 @@ struct ModelConfig {
obj["min_keep"] = min_keep;
obj["ngl"] = ngl;
obj["ctx_len"] = ctx_len;
obj["n_parallel"] = n_parallel;
obj["engine"] = engine;
obj["prompt_template"] = prompt_template;
obj["system_template"] = system_template;
@@ -313,6 +317,8 @@ struct ModelConfig {
if (ctx_len != std::numeric_limits<int>::quiet_NaN())
oss << format_utils::print_kv("ctx_len", std::to_string(ctx_len),
format_utils::MAGENTA);
oss << format_utils::print_kv("n_parallel", std::to_string(n_parallel),
format_utils::MAGENTA);
if (ngl != std::numeric_limits<int>::quiet_NaN())
oss << format_utils::print_kv("ngl", std::to_string(ngl),
format_utils::MAGENTA);
5 changes: 5 additions & 0 deletions engine/config/yaml_config.cc
Original file line number Diff line number Diff line change
@@ -113,6 +113,8 @@ void YamlHandler::ModelConfigFromYaml() {
tmp.ngl = yaml_node_["ngl"].as<int>();
if (yaml_node_["ctx_len"])
tmp.ctx_len = yaml_node_["ctx_len"].as<int>();
if (yaml_node_["n_parallel"])
tmp.n_parallel = yaml_node_["n_parallel"].as<int>();
if (yaml_node_["tp"])
tmp.tp = yaml_node_["tp"].as<int>();
if (yaml_node_["stream"])
@@ -216,6 +218,8 @@ void YamlHandler::UpdateModelConfig(ModelConfig new_model_config) {
yaml_node_["ngl"] = model_config_.ngl;
if (!std::isnan(static_cast<double>(model_config_.ctx_len)))
yaml_node_["ctx_len"] = model_config_.ctx_len;
if (!std::isnan(static_cast<double>(model_config_.n_parallel)))
yaml_node_["n_parallel"] = model_config_.n_parallel;
if (!std::isnan(static_cast<double>(model_config_.tp)))
yaml_node_["tp"] = model_config_.tp;
if (!std::isnan(static_cast<double>(model_config_.stream)))
@@ -368,6 +372,7 @@ void YamlHandler::WriteYamlFile(const std::string& file_path) const {
outFile << format_utils::writeKeyValue(
"ctx_len", yaml_node_["ctx_len"],
"llama.context_length | 0 or undefined = loaded from model");
outFile << format_utils::writeKeyValue("n_parallel", yaml_node_["n_parallel"]);
outFile << format_utils::writeKeyValue("ngl", yaml_node_["ngl"],
"Undefined = loaded from model");
outFile << "# END OPTIONAL\n";
6 changes: 6 additions & 0 deletions engine/test/components/test_yaml_handler.cc
Original file line number Diff line number Diff line change
@@ -62,6 +62,7 @@ top_p: 0.9
temperature: 0.7
max_tokens: 100
stream: true
n_parallel: 2
stop:
- "END"
files:
@@ -82,6 +83,7 @@ stream: true
EXPECT_FLOAT_EQ(config.temperature, 0.7f);
EXPECT_EQ(config.max_tokens, 100);
EXPECT_TRUE(config.stream);
EXPECT_EQ(config.n_parallel, 2);
EXPECT_EQ(config.stop.size(), 1);
EXPECT_EQ(config.stop[0], "END");
EXPECT_EQ(config.files.size(), 1);
@@ -101,6 +103,7 @@ TEST_F(YamlHandlerTest, UpdateModelConfig) {
new_config.temperature = 0.8f;
new_config.max_tokens = 200;
new_config.stream = false;
new_config.n_parallel = 2;
new_config.stop = {"STOP", "END"};
new_config.files = {"updated_file1.gguf", "updated_file2.gguf"};

@@ -116,6 +119,7 @@ TEST_F(YamlHandlerTest, UpdateModelConfig) {
EXPECT_FLOAT_EQ(config.temperature, 0.8f);
EXPECT_EQ(config.max_tokens, 200);
EXPECT_FALSE(config.stream);
EXPECT_EQ(config.n_parallel, 2);
EXPECT_EQ(config.stop.size(), 2);
EXPECT_EQ(config.stop[0], "STOP");
EXPECT_EQ(config.stop[1], "END");
@@ -135,6 +139,7 @@ TEST_F(YamlHandlerTest, WriteYamlFile) {
new_config.temperature = 0.6f;
new_config.max_tokens = 150;
new_config.stream = true;
new_config.n_parallel = 2;
new_config.stop = {"HALT"};
new_config.files = {"write_test_file.gguf"};

@@ -158,6 +163,7 @@ TEST_F(YamlHandlerTest, WriteYamlFile) {
EXPECT_FLOAT_EQ(read_config.temperature, 0.6f);
EXPECT_EQ(read_config.max_tokens, 150);
EXPECT_TRUE(read_config.stream);
EXPECT_EQ(read_config.n_parallel, 2);
EXPECT_EQ(read_config.stop.size(), 1);
EXPECT_EQ(read_config.stop[0], "HALT");
EXPECT_EQ(read_config.files.size(), 1);