FEAT Adding retry functionality to PromptSendingOrchestrator

[blakebullwinkel]

Description

This PR adds the option in PromptSendingOrchestrator to re-send prompts based on the result of a scorer. For now, this is implemented using a SelfAskRefusalScorer, but in the future we may want to replace this with a generic Scorer and implement more general retry logic. We expose this functionality by adding one new arguments to the send_normalizer_requests_async method: max_retries. The example below shows how to retry until all responses are scored as False (i.e., non-refusal), up to a maximum of 5 retries.

send_prompts_async( 
        prompt_list=prompt_list,
        max_retries=5,
)

Thanks @rlundeen2 for the design and @romanlutz for the help with implementation!

Tests and Documentation

This change is documented in the GCG suffix demo in 1_auxiliary_attacks.ipynb but I have not written tests for it yet.

[rlundeen2]

Needs unit tests added

[rlundeen2]

I just re-worked PromptSendingOrchestrator so this may need some changes.

I think we should probably make this a subclass of PromptSendingOrchestrator, or make retrying a separate function. I think that would simplify things. So this could be PromptSendingWithRetriesOrchestrator.

The logic can stay the same for the most part, but you'd overwright/call PromptSendingOrchestrator.send_prompts_async and just do the retries in the parent function

[corporate87]

...¿∆ΠΩμ*?.

[corporate87]

...

[corporate87]

...

[corporate87]

...

[corporate87]

...¿∆ΠΩμ*?.

[corporate87]

...¿∆ΠΩμ*?.

[corporate87]

...¿∆ΠΩμ*?.

[corporate87]

...¿∆ΠΩμ*?.