LLM Change Agent

Overview

LLM Change Agent is a command-line tool designed to generate knowledge graph change language (KGCL) commands that request changes in a knowledge graph such as an ontology. It generates KGCL commands using specified large language models and providers via prompts. The prompts can be GitHub issue description/comments or natural language text.

Features

• Given a prompt relevant to making ontology resource changes, the agent responds with KGCL change statements.
• Supports OpenAI, Ollama, Anthropic and CBORG (LBNL-hosted) models.

⚠️ OpenAI, Anthropic and CBORG model use require availability of corresponding keys as environment variables locally.

Installation

To install the dependencies, run:

pip install llm-change-agent

Usage

The CLI provides several commands to interact with the language models.

List Available Models

To list all available models from supported providers:

llm-change list-models

OpenAI models:
  gpt-4o-2024-08-06
  gpt-4o-mini
  gpt-4o-mini-2024-07-18
  gpt-4o-2024-05-13
  gpt-4o
  gpt-4-turbo-2024-04-09
  gpt-4-turbo
  gpt-4-turbo-preview
Anthropic models:
  claude-3-5-sonnet-20240620
  claude-3-opus-20240229
  claude-3-sonnet-20240229
  claude-3-haiku-20240307
Ollama models:
  llama3.1
LBL-CBORG models:
  lbl/cborg-chat:latest
  lbl/cborg-chat-nano:latest
  openai/gpt-3.5-turbo
  openai/gpt-4o
  openai/gpt-4o-mini
  anthropic/claude-haiku
  anthropic/claude-sonnet
  anthropic/claude-opus
  google/gemini-pro
  google/gemini-flash

Generate Text

To generate text using a specified model and provider:

llm-change execute --model <MODEL_NAME> --provider <PROVIDER_NAME> --prompt "<YOUR_PROMPT>"

Replace <MODEL_NAME>, <PROVIDER_NAME>, and <YOUR_PROMPT> with your desired values.

Examples

• Generate text using CBORG's lbl/llama-3 model with a custom prompt:

llm-change execute --model lbl/llama-3 --prompt "I want to change the definition of class ABC:123 to 'foo bar' and also create a new class labelled 'bar foo' with the curie DEF:123."

OR

llm-change execute --provider cborg --prompt "I want to change the definition of class ABC:123 to 'foo bar' and also create a new class labelled 'bar foo' with the curie DEF:123."

generates

Final Answer: 

[
  "change definition of ABC:123 to 'foo bar'",
  "create class DEF:123 'bar foo'"
]

Evaluations

Input

The project also contains input data for evaluations in the form of YAML files for the following ontologies:

• Envo
• GO
• MONDO
• Cell Ontology
• Uberon
• PATO

Expected changes

The expected changes for specific pull requests are listed ontologywise below:

• Envo
• GO
• MONDO
• Cell Ontology
• Uberon
• PATO

Actual results

The actual results for some of the LLM models evaluated are here and corresponding metrics here

Development

To run the project locally, clone the repository and navigate to the project directory:

git clone https://github.com/yourusername/llm-change-agent.git
cd llm-change-agent
poetry install

Make sure you have poetry installed in your system.

Contributing

Contributions are welcome! Please submit a pull request or open an issue to discuss any changes.

License

This project is licensed under the MIT License. See the LICENSE file for details.

---

Acknowledgements

This cookiecutter project was developed from the monarch-project-template template and will be kept up-to-date using cruft.