-
Notifications
You must be signed in to change notification settings - Fork 50
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add LMMSEvalInferenceEngine #1301
Conversation
Signed-off-by: elronbandel <[email protected]>
from unitxt.text_utils import print_dict | ||
|
||
with settings.context( | ||
disable_hf_datasets_cache=False, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should document this new flag in the documentation in a relatively prominent way. (e.g. in one of the first tutorials that loads data from HF).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Need to add this example to example main page, and disable the run of this examples by default in regressions.
streaming=True, | ||
) | ||
|
||
test_dataset = list(tqdm(dataset["test"], total=30)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That does this do? Why do you nee tqdm?
test_dataset = list(tqdm(dataset["test"], total=30)) | ||
|
||
predictions = inference_model.infer(test_dataset) | ||
evaluated_dataset = evaluate(predictions=predictions, data=test_dataset) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do you get same results as with the inference engine HFLlavaInferenceEngine?
Signed-off-by: Elron Bandel <[email protected]>
Signed-off-by: Elron Bandel <[email protected]>
Signed-off-by: Elron Bandel <[email protected]>
Signed-off-by: Elron Bandel <[email protected]>
Signed-off-by: elronbandel <[email protected]>
No description provided.