Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added watsonx.ai generator #1058

Open
wants to merge 9 commits into
base: main
Choose a base branch
from
Open

Conversation

iamnotcj
Copy link

@iamnotcj iamnotcj commented Jan 4, 2025

This is the generator for watsonx.ai shown a couple weeks ago. It passes all of the tests on my computer.
I updated the generator test to include a "project_id" variable as a requirement for watsonx too.

=============================================================================================== 2105 passed, 46 skipped in 566.53s (0:09:26) ===============================================================================================
(python3.12)  garak [main●] % 

I also tested it with the test probes and it was able to pass as well. It should be noted that I had to append a null byte to some of the probes, since it appears sending an empty string to leads to an api error in watsonx.

(python3.12)  garak [main●] % python -m garak -m watsonx -n google/flan-t5-xl -p test.Test
garak LLM vulnerability scanner v0.10.1.post1 ( https://github.com/NVIDIA/garak ) at 2025-01-04T15:09:08.723457
📜 logging to /Users/host/.local/share/garak/garak.log
/Users/host/venv/python3.12/lib/python3.12/site-packages/pydantic/_internal/_config.py:345: UserWarning: Valid config keys have changed in V2:
* 'fields' has been removed
  warnings.warn(message, UserWarning)
🦜 loading generator: watsonx: google/flan-t5-xl
⚠️  This run can be sped up 🥳 Generator 'watsonx:google/flan-t5-xl' supports parallelism! Consider using `--parallel_attempts 16` (or more) to greatly accelerate your run. 🐌
📜 reporting to /Users/host/.local/share/garak/garak_runs/garak.677c837c-eab1-4ab0-900d-0cd9fbbc9689.report.jsonl
🕵️  queue of probes: test.Test
probes.test.Test:   0%|                                                                                                                                                                                               | 0/8 [00:00<?, ?it/sWARNING: Empty prompt was found. Null byte character appended to prevent API failure. https://us-south.ml.cloud.ibm.com                                                                                                | 0/5 [00:00<?, ?it/s]
                                                                                                                                                                                                                                           WARNING: Empty prompt was found. Null byte character appended to prevent API failure. https://us-south.ml.cloud.ibm.com                                                                                        | 1/5 [00:02<00:08,  2.19s/it]
                                                                                                                                                                                                                                           WARNING: Empty prompt was found. Null byte character appended to prevent API failure. https://us-south.ml.cloud.ibm.com                                                                                        | 2/5 [00:04<00:06,  2.11s/it]
                                                                                                                                                                                                                                           WARNING: Empty prompt was found. Null byte character appended to prevent API failure. https://us-south.ml.cloud.ibm.com██████████████████▍                                                                     | 3/5 [00:06<00:04,  2.08s/it]
                                                                                                                                                                                                                                           WARNING: Empty prompt was found. Null byte character appended to prevent API failure. https://us-south.ml.cloud.ibm.com█████████████████████████████████████████████████████▏                                  | 4/5 [00:08<00:02,  2.15s/it]
test.Test                                                                                always.Pass: PASS  ok on   40/  40                                                                                                                 
📜 report closed :) /Users/host/.local/share/garak/garak_runs/garak.677c837c-eab1-4ab0-900d-0cd9fbbc9689.report.jsonl                                                                                                                       
📜 report html summary being written to /Users/host/.local/share/garak/garak_runs/garak.677c837c-eab1-4ab0-900d-0cd9fbbc9689.report.html
✔️  garak run complete in 86.16s

Please let me know if there are any issues with this request. :-)

Added watsonx generator
Black Format
Added more documentation.
Changed variable names and updated tests
Copy link
Contributor

github-actions bot commented Jan 4, 2025

DCO Assistant Lite bot All contributors have signed the DCO ✍️ ✅

@iamnotcj
Copy link
Author

iamnotcj commented Jan 4, 2025

I have read the DCO Document and I hereby sign the DCO

@iamnotcj
Copy link
Author

iamnotcj commented Jan 4, 2025

recheck

github-actions bot added a commit that referenced this pull request Jan 4, 2025
@iamnotcj iamnotcj changed the title Add watsonx.ai generator Added watsonx.ai generator Jan 4, 2025
Copy link
Collaborator

@jmartin-tech jmartin-tech left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@iamnotcj, thank you for the useful contribution this is a great first pass at adding support for a the IBM stack.

There are some newer patterns in gathering configuration values that I would like to see applied here, please take a look and offer thoughts or adjustments based on my comments.

The dependency concern may be the largest item to sort out. If there are strong reasons the dependencies are required the project can weight the value trade offs.

Note the current package requires the following which may support features garak does not really need per se:

    "pandas<2.2.0,>=0.24.2",
    "certifi",
    "tabulate",
    "packaging",
    "ibm-cos-sdk<2.14.0,>=2.12.0",
    "importlib-metadata",

Published code would help with understanding the needs for these dependencies. It is possible my search simply missed some public location.

garak/generators/watsonx.py Show resolved Hide resolved
garak/generators/watsonx.py Outdated Show resolved Hide resolved
garak/generators/watsonx.py Outdated Show resolved Hide resolved
pyproject.toml Outdated Show resolved Hide resolved
@iamnotcj
Copy link
Author

Hey @jmartin-tech, thank you for the feedback! For the dependency problem, I think I will be able to just use the "request" library instead of the SDK. I will go ahead and make all of the necessary edits you highlighted in the other comments as well and update the request when its ready 👍

Dropped the SDK for requests library,
Added some tests
@iamnotcj iamnotcj requested a review from jmartin-tech January 25, 2025 07:33
erickgalinkin pushed a commit to erickgalinkin/garak that referenced this pull request Jan 29, 2025
Copy link
Collaborator

@jmartin-tech jmartin-tech left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great, thanks @iamnotcj for adjusting the dependencies here. I have noted a minor flexibility enhancement I think will impact sustainability.

Test failure are due to a dependency change addressed in #1092, new commits to this PR will bring in that update during test runs.

garak/generators/watsonx.py Outdated Show resolved Hide resolved
garak/generators/watsonx.py Outdated Show resolved Hide resolved
iamnotcj and others added 3 commits January 30, 2025 15:20
Added version parameter

Co-authored-by: Jeffrey Martin <[email protected]>
Signed-off-by: iamnotcj <[email protected]>
Added version parameter

Co-authored-by: Jeffrey Martin <[email protected]>
Signed-off-by: iamnotcj <[email protected]>
Updated the default version variable and updated tests.
@iamnotcj iamnotcj requested a review from jmartin-tech January 30, 2025 22:17
Copy link
Collaborator

@jmartin-tech jmartin-tech left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome, this looks great!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants