Seeking Guidance: Is It Possible to Implement a Model Context Protocol Server for Open-Source LLMs?

[avinashkurup]

Pre-submission Checklist

• I have checked that this question would not be more appropriate as an issue in a specific repository
• I have searched existing discussions and documentation for answers

Question Category

• Protocol Specification
• SDK Usage
• Server Implementation
• General Implementation
• Documentation
• Other

Your Question

Hi all,

We’re exploring the idea of creating a server that adheres to the Model Context Protocol and integrates with open-source LLMs (like Hugging Face Transformers, Grok, Mistral etc.). Before diving in, we’d love some guidance from the community to better understand if this is feasible and how to approach it.

We would like to know more about,

• How suitable is the Model Context Protocol for interacting with open-source LLMs? Any potential challenges or considerations?

• What kind of architecture would you recommend for scalability and modularity?

• Are there any existing frameworks or tools that could make this easier?

• Have you or anyone you know integrated the protocol with open-source LLMs?

• Any suggestions for handling model-specific quirks?

Understanding the feasibility and potential hurdles will help us decide whether to proceed and how to design this project in the best possible way. Your expertise could save us (and others) a lot of time and effort!

Thank you,

[evalstate]

Hi Avinash, you might be interested in https://github.com/evalstate/mcp-hfspace.

There are some videos of it in operation with Claude Desktop here including integration with Mistral and Qwen chat models.

It's definitely possible - there are plenty of open source chat servers that have multiple back-end integrations that could have an MCP front-end bolted on. It would probably be helpful to understand more about the Client and intended usage?

[gl2007]

I am in the same boat as OP. I want to use mcp/servers into my own App which can use any LLM model I configure it with. From the documentation, it certainly seems possible but documentation seems to cover only usage with Claude desktop. Examples show a generic python app using these servers would be great, unless I a have missed them.

[gl2007]

@avinashkurup check this