-
Notifications
You must be signed in to change notification settings - Fork 884
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix issue when generating distros #755
Conversation
7186407
to
3d45a03
Compare
@dineshyv could you approve this PR? I am not sure if the changes to the |
@@ -95,62 +95,20 @@ metadata_store: | |||
db_path: ${env.SQLITE_STORE_DIR:~/.llama/distributions/fireworks}/registry.db | |||
models: | |||
- metadata: {} | |||
model_id: meta-llama/Llama-3.1-8B-Instruct | |||
model_id: ${env.INFERENCE_MODEL} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm, this run-with-safety template shouldn't be changed.
cc @vladimirivic Could you help take a look? I think this is introduced incorrectly in
llama-stack/llama_stack/templates/fireworks/fireworks.py
Lines 123 to 140 in 91907b7
"run-with-safety.yaml": RunConfigSettings( | |
provider_overrides={ | |
"inference": [ | |
inference_provider, | |
embedding_provider, | |
], | |
"memory": [memory_provider], | |
"safety": [ | |
Provider( | |
provider_id="llama-guard", | |
provider_type="inline::llama-guard", | |
config={}, | |
), | |
Provider( | |
provider_id="code-scanner", | |
provider_type="inline::code-scanner", | |
config={}, | |
), |
It should follow smt like together's run-with-safety template: https://github.com/meta-llama/llama-stack/blob/91907b714e825a1bfbca5271e0f403aab5f10752/llama_stack/templates/together/together.py#L120C32-L141
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I rebased and re-generated the files again. Not sure if it incorported the recent fixes correctly
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it pulled changes in #766.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The fireworks templates have been fixed in #766 . But I think the vllm run.yaml without tool_runtime
still suggest you might have an older version? Could you help double check?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks like there are indeed some issues with my local python env. I think I fixed them now. Just pushed an update
Signed-off-by: Yuan Tang <[email protected]>
3d45a03
to
7c72682
Compare
Signed-off-by: Yuan Tang <[email protected]>
LG! |
Addressed comment #723 (comment).
cc @yanxi0830
I am not 100% sure if the diff is correct though but this is the result of running
python llama_stack/scripts/distro_codegen.py
.