Optional fields #28

aelaguiz · 2024-07-10T20:48:30Z

Adds a FakeLLM pattern (See test_fake_anthropic) for tests
Adds support for any number of optional output fields:

class OptionalOutputPromptSignature(PromptSignature):
    input1 = InputField(name="input1", desc="Input field 1")
    input2 = InputField(name="input2", desc="Input field 2")
    output1 = OutputField(name="output1", desc="Output field 1")
    output2 = OutputField(name="output2", desc="Output field 2", optional=True)

This is useful for handling degraded responses from models cleanly.

…lues don't match the expected fields.

…FailedPromptStrategy classes to address pytest collection warnings.

Implemented tests for failed prompts, including missing inputs, extra inputs, and missing outputs, and suppressed pytest collection warnings.

…LLM from langchain_contrib.llms.testing.

…asses to `FailedPromptSignature` and `FailedPromptStrategy`, respectively, and added an import statement for `langchain_contrib`.

…fail to True by default.

…ass to improve readability and maintainability.

…ropic' LLM type in prompt formatting and output parsing.

…mpt_strategies.py`.

…ass.

…the expected and received input keys in the correct order.

…tailed information about missing or unexpected keys.

…le before loading the langdspy module.

rblakemesser · 2024-07-10T21:29:48Z

langdspy/prompt_runners.py

-
-        llm_model = self._determine_llm_model(config['llm']) 
-
+        hard_fail = config.get('hard_fail', True)


oh yeah this file is a lot cleaner now

aelaguiz added 20 commits July 10, 2024 14:11

Added new unit tests to simulate failed prompts where the returned va…

fb42a84

…lues don't match the expected fields.

Removed __init__ constructors from TestFailedPromptSignature and Test…

3075808

…FailedPromptStrategy classes to address pytest collection warnings.

The commit message for the changes made in the provided diffs is:

14e6333

Implemented tests for failed prompts, including missing inputs, extra inputs, and missing outputs, and suppressed pytest collection warnings.

Added support for FakeLLM in PromptRunner.

397f0f5

Added support for a new 'fake_anthropic' LLM type that wraps the Fake…

b54d783

…LLM from langchain_contrib.llms.testing.

Added unit tests for the FakeLLM from langchain_contrib.

170d276

Renamed TestFailedPromptSignature and TestFailedPromptStrategy cl…

b58d41f

…asses to `FailedPromptSignature` and `FailedPromptStrategy`, respectively, and added an import statement for `langchain_contrib`.

Rewritten test_failed_prompts to use the fake Anthropic LLM pattern.

2cc14c7

Improved error message for failed prompt output validation.

35680de

Added a new test for optional output field.

d6513c0

Configured the PromptRunner to use the correct LLM type and set hard_…

5c0d0b4

…fail to True by default.

Refactored the _invoke_with_retries function in the PromptRunner cl…

10c37d8

…ass to improve readability and maintainability.

Removed unnecessary print statements and added support for 'fake_anth…

dd432e6

…ropic' LLM type in prompt formatting and output parsing.

Removed the print_prompt variable from prompt_runners.py and `pro…

f626efb

…mpt_strategies.py`.

Implemented handling of optional output fields in the PromptRunner cl…

38ff31e

…ass.

Corrected the error message in the PromptStrategy class to display …

79c2909

…the expected and received input keys in the correct order.

Improved the exception raised in validate_inputs to provide more de…

b01677c

…tailed information about missing or unexpected keys.

Added logging to the tests/test_model_train.py file.

d979b3d

Initialized the langdspy logger and configured it to log to the conso…

cf3cea0

…le before loading the langdspy module.

All tests pass

c92959c

aelaguiz merged commit b7bd813 into main Jul 10, 2024
1 check passed

rblakemesser reviewed Jul 10, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Optional fields #28

Optional fields #28

aelaguiz commented Jul 10, 2024

rblakemesser Jul 10, 2024


		llm_model = self._determine_llm_model(config['llm'])

		hard_fail = config.get('hard_fail', True)

Optional fields #28

Optional fields #28

Conversation

aelaguiz commented Jul 10, 2024

rblakemesser Jul 10, 2024

Choose a reason for hiding this comment