Add LLMCheckerChain

[andychenmagrathea-06e0a82f-34fc-48ca]

Implementation of https://github.com/jagilley/fact-checker. Works pretty well.

Verifying this manually:

1. "Only two kinds of egg-laying mammals are left on the planet today—the duck-billed platypus and the echidna, or spiny anteater." https://www.scientificamerican.com/article/extreme-monotremes/
2. "An [Echidna] egg weighs 1.5 to 2 grams (0.05 to 0.07 oz)[19] and is about 1.4 centimetres (0.55 in) long." https://en.wikipedia.org/wiki/Echidna#:~:text=sleep%20is%20suppressed.-,Reproduction,a%20reptile%2Dlike%20egg%20tooth.
3. "A [platypus] lays one to three (usually two) small, leathery eggs (similar to those of reptiles), about 11 mm (7⁄16 in) in diameter and slightly rounder than bird eggs." https://en.wikipedia.org/wiki/Platypus#:~:text=It%20lays%20one%20to%20three,slightly%20rounder%20than%20bird%20eggs.
4. Therefore, an Echidna is the mammal that lays the biggest eggs.

cc @hwchase17

[hwchase17]

so im down to check this in as is because i think its great, but one thing id like to think about before checking this in - how general could we make this? eg what is the most general prompt/chain that i could have that would work with this?

eg right now the use case here is you ask a generic question in the first prompt. are there other good use cases for this?

[andychenmagrathea-06e0a82f-34fc-48ca]

so im down to check this in as is because i think its great, but one thing id like to think about before checking this in - how general could we make this? eg what is the most general prompt/chain that i could have that would work with this?

In the most general case, I think this verification becomes an agent, not a chain? You probably want to have an agent list its assumptions and then verify them with its tools, e.g. go to Wikipedia to look up a list of mammals.

This actually looks a lot like a parallel version of self-ask?

@hwchase17

[hwchase17]

so im down to check this in as is because i think its great, but one thing id like to think about before checking this in - how general could we make this? eg what is the most general prompt/chain that i could have that would work with this?

In the most general case, I think this verification becomes an agent, not a chain? You probably want to have an agent list its assumptions and then verify them with its tools, e.g. go to Wikipedia to look up a list of mammals.

This actually looks a lot like a parallel version of self-ask?

@hwchase17

oh i like that idea. in that framing the checkers are the tools then right? so what part of this chain is that checker/tool? not the whole part, because it also contains the question. maybe the next two parts at the end?

[andychenmagrathea-06e0a82f-34fc-48ca]

Right so I guess there's two options:

1. If we turn LLMCheckerChain into an agent, the verification of each individual assumption is routed to the appropriate tool by the agent. I can see this as potentially being more robust, at the cost of increased compute (especially if you allow the agent to call itself; self-ask doesn't have as much of a problem because there's no fanout)

2. On the other hand, I do see benefits of keeping things as-is and having a fast-running verification chain as just a tool — seems like a cheap step to throw in if an agent is unsure of an answer.

Maybe we do both? #1 used when compute / latency isn't as much an issue, #2 more widely used to make things a bit more correct?

[hwchase17]

Right so I guess there's two options:

1. If we turn LLMCheckerChain into an agent, the verification of each individual assumption is routed to the appropriate tool by the agent. I can see this as potentially being more robust, at the cost of increased compute (especially if you allow the agent to call itself; self-ask doesn't have as much of a problem because there's no fanout)
2. On the other hand, I do see benefits of keeping things as-is and having a fast-running verification chain as just a tool — seems like a cheap step to throw in if an agent is unsure of an answer.

Maybe we do both? #1 used when compute / latency isn't as much an issue, #2 more widely used to make things a bit more correct?

what would (1) look like exactly? would we change the prompt of the agent, to tell it to answer the question and then use a tool to fact check? what would that tool look like? would htat tool basically be the prompts here?

is there a common prereq for both (1) and (2)? namely the tool/chain that does the verfication? and in (2) its just a simple sequence of original LLM -> verificaton chain, and in (1) its a tool the agent has access to?

[andychenmagrathea-06e0a82f-34fc-48ca]

Scoping this PR to a chain for now. Ready for review!