feat: extractor api improvements

[0xMochan]

• I have looked for existing issues (including closed) about this

Feature Request

• Extractors exist as a wrapper on top of agents with a pre-made tool. Since it constructs and abstracts the agent away, this means you can't customize the agent. This is a problem for anthropic as anthropic requires max_tokens to be set atm. This is also quite restrictive in other regards.

• Extractors are also missing the prompt trait in general making them less composible (as per the feat/agentic-chains branch).

• Extractors only work with full types and not with any serde_json::Value which means things like Vec<MyType>, bool, etc won't work.

• Lastly, extractors are built with the use of tools to produce a json output but with the release of json-mode and similar in many providers (like OpenAI, etc), it might be suitable for provider specific or customizable extractors that make use of that feature rather than a custom tool. This customizability isn't really hit within our extractor implementation.

Proposal

Rather than Extractor being a whole struct, I wonder if .extract::<Type>(...) would be a better api. Or some other Extractor trait based on the provider that can be written (otherwise using a default tool implementation) to make this a bit more seemless. It seems like a general extractor improvements branch is in order.

Alternatives

Keeping the Extractor struct but making the agent exposable or even configurable. All the bugs could be fixed in the current implementation.

[Lur1an]

I might have a proposal:

• Is it possible to model this functionality as an extension trait on Agent? If someone still wants a single instance of a struct just for data extraction one can create a dedicated Agent, otherwise if one is using a quite generic Agent it can be reused.
• Would it make sense to create two variations of this functionality:
  1. As before, use a tool with a submit function
  2. Append to the end of the user prompt an instruction to respond in structured json only, then try extracting structured json from the choice? I have already built a similar extension trait in one of my projects due to ollama limitations with nested tool call properties

If these ideas make sense I can look into drafting up a PR for the implementation.

[0xMochan]

I might have a proposal:

• Is it possible to model this functionality as an extension trait on Agent? If someone still wants a single instance of a struct just for data extraction one can create a dedicated Agent, otherwise if one is using a quite generic Agent it can be reused.

• Would it make sense to create two variations of this functionality:

  1. As before, use a tool with a submit function
  2. Append to the end of the user prompt an instruction to respond in structured json only, then try extracting structured json from the choice? I have already built a similar extension trait in one of my projects due to ollama limitations with nested tool call properties

If these ideas make sense I can look into drafting up a PR for the implementation.

Hey, great ideas! I actually have a partial working implementations of a refactor w/ extractors on my local. I decided to halt on in for a different Messages refactor which should make things cleaner for multi-modal. There's also other refactors (#149) which should make it easier to build clients easier and hopefully a better opt in system to enable structured outputs support that something like extractors can hook into.

There's two paths forward:

• A new StructuredPrompt trait which accepts a type as a generic and agents built with said type will output said generic type.

// Roughly
agent = Client::from_env()
  .agent("model-name")
  .cast::<MyType>()
  .build();

output: MyType = agent.prompt("give me something in MyType")

• A refactor of the current Prompt trait to accept any type as a generic. A trait alias would be set as SimplePrompt that would use String as "reasonable" default.

Both of these are tricky since Prompt is sorta universally handled throughout the codebase. StructuredPrompt sorta bypasses some of complexity of Prompts usage, but it feels nicer to just nail the Prompt upgrade and let the rust compiler OK the refactor!

Atm, Extractor is set as a wrapper on agent which is quite awkward to extend, you are right. My idea was to make the Prompt or StructuredPrompt work, like a trait on the client on whether extraction should use a tool (works universally) or structured prompts (limited support, client specific, perhaps a ClientBuilder option). Then, prompt defers to said trait (Castable or something) when actually building the final type. We would also have a base Extractor that more or less acts as a pre-built agent builder (so you can add onto it) which will be suitable for easy tutorials and guides (+ helpful to look under the curtain to learn more!) A large part of my inspiration for this design comes from pydantic-ai actually!

All in all, it's a bit of a delicate refactor and I want to work it out further with the team. Atm, the team is taking off for the holidays but I might put together some small PR demos so you can experiment if you like.

---

We've discussed this extensively internally, so it's more inclined we are going to want to design this refactor a bit more in-house, but we'll always welcome external PRs — just no guarantees! Please let me know if you have any questions or thoughts, I'll incorporate it into my mental model for this issue!