Implement Groq Gateway Service

[AtlantisPleb]

This issue outlines the implementation of a Groq Gateway service that implements the Gateway trait similar to the existing Deepseek service.

Overview

The Groq service will provide:

• OpenAI-compatible API interface
• Support for chat completions
• Streaming support
• Proper error handling and rate limiting
• Configuration via environment variables

Implementation Details

1. Create Service Structure

// backend/src/server/services/groq/service.rs
use reqwest::{Client, ClientBuilder};
use std::time::Duration;

#[derive(Debug, Clone)]
pub struct GroqService {
    client: Client,
    api_key: String,
    base_url: String,
}

impl GroqService {
    pub fn new(api_key: String) -> Self {
        let client = ClientBuilder::new()
            .timeout(Duration::from_secs(180))
            .build()
            .expect("Failed to create HTTP client");

        let base_url = std::env::var("GROQ_API_URL")
            .unwrap_or_else(|_| "https://api.groq.com/v1".to_string());

        Self {
            client,
            api_key,
            base_url,
        }
    }

    pub fn with_base_url(api_key: String, base_url: String) -> Self {
        let client = ClientBuilder::new()
            .timeout(Duration::from_secs(180))
            .build()
            .expect("Failed to create HTTP client");

        Self {
            client,
            api_key,
            base_url,
        }
    }
}

2. Implement Gateway Trait

use crate::server::services::gateway::{Gateway, GatewayMetadata};
use anyhow::Result;
use std::pin::Pin;
use tokio_stream::Stream;

#[async_trait::async_trait]
impl Gateway for GroqService {
    fn metadata(&self) -> GatewayMetadata {
        GatewayMetadata {
            name: "Groq".to_string(),
            openai_compatible: true,
            supported_features: vec![
                "chat".to_string(),
                "streaming".to_string(),
            ],
            default_model: "mixtral-8x7b-32768".to_string(),
            available_models: vec![
                "llama-3.1-8b-instant".to_string(),
                "llama-3.3-70b-versatile".to_string(),
                "mixtral-8x7b-32768".to_string(),
            ],
        }
    }

    async fn chat(&self, prompt: String, use_reasoner: bool) -> Result<(String, Option<String>)> {
        // Implementation similar to transcribe.rs example
        let response = self.client
            .post(format!("{}/chat/completions", self.base_url))
            .header("Authorization", format!("Bearer {}", self.api_key))
            .json(&serde_json::json!({
                "model": self.metadata().default_model,
                "messages": [{
                    "role": "user",
                    "content": prompt
                }],
                "temperature": if use_reasoner { 0.0 } else { 0.7 },
                "stream": false
            }))
            .send()
            .await?;

        // Handle response and extract content
        let json: serde_json::Value = response.json().await?;
        let content = json["choices"][0]["message"]["content"]
            .as_str()
            .unwrap_or("")
            .to_string();

        Ok((content, None))
    }

    async fn chat_stream(
        &self,
        prompt: String,
        use_reasoner: bool,
    ) -> Result<Pin<Box<dyn Stream<Item = Result<String>> + Send>>> {
        // Implementation using SSE streaming
        // Similar to transcribe.rs but with stream=true
        todo!("Implement streaming")
    }
}

3. Add Types Module

// backend/src/server/services/groq/types.rs
use serde::{Deserialize, Serialize};

#[derive(Debug, Serialize, Deserialize)]
pub struct ChatCompletion {
    pub id: String,
    pub object: String,
    pub created: i64,
    pub model: String,
    pub choices: Vec<Choice>,
    pub usage: Usage,
}

#[derive(Debug, Serialize, Deserialize)]
pub struct Choice {
    pub index: i32,
    pub message: Message,
    pub finish_reason: Option<String>,
}

#[derive(Debug, Serialize, Deserialize)]
pub struct Message {
    pub role: String,
    pub content: String,
}

#[derive(Debug, Serialize, Deserialize)]
pub struct Usage {
    pub prompt_tokens: i32,
    pub completion_tokens: i32,
    pub total_tokens: i32,
}

4. Add Error Handling

// backend/src/server/services/groq/error.rs
use thiserror::Error;

#[derive(Error, Debug)]
pub enum GroqError {
    #[error("API request failed: {0}")]
    RequestFailed(String),

    #[error("Failed to parse response: {0}")]
    ParseError(String),

    #[error("Rate limit exceeded")]
    RateLimitExceeded,

    #[error("Invalid API key")]
    InvalidApiKey,
}

5. Add Configuration

// backend/src/server/services/groq/config.rs
use serde::{Deserialize, Serialize};

#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct GroqConfig {
    pub api_key: String,
    pub base_url: Option<String>,
    pub default_model: Option<String>,
    pub timeout_secs: Option<u64>,
}

impl Default for GroqConfig {
    fn default() -> Self {
        Self {
            api_key: std::env::var("GROQ_API_KEY").expect("GROQ_API_KEY must be set"),
            base_url: None,
            default_model: None,
            timeout_secs: Some(180),
        }
    }
}

6. Add Tests

// backend/tests/groq.rs
use crate::server::services::groq::GroqService;
use crate::server::services::gateway::Gateway;

#[tokio::test]
async fn test_groq_metadata() {
    let service = GroqService::new("test-key".to_string());
    let metadata = service.metadata();
    assert_eq!(metadata.name, "Groq");
    assert!(metadata.openai_compatible);
}

#[tokio::test]
async fn test_groq_chat() {
    let service = GroqService::new("test-key".to_string());
    let (response, _) = service.chat("Hello".to_string(), false).await.unwrap();
    assert!(!response.is_empty());
}

#[tokio::test]
async fn test_groq_chat_stream() {
    let service = GroqService::new("test-key".to_string());
    let stream = service.chat_stream("Hello".to_string(), false).await.unwrap();
    // Test streaming functionality
}

Tasks

1. Create basic service structure
2. Implement Gateway trait
3. Add types and error handling
4. Add configuration support
5. Implement streaming support
6. Add tests
7. Add documentation
8. Update environment variables documentation

Environment Variables

Add to .env.example:

GROQ_API_KEY=your-api-key-here
GROQ_API_URL=https://api.groq.com/v1

Related Issues

• Design and Implement Real-Time Sync Engine for OpenAgents #730 - Design and Implement Real-Time Sync Engine
• Implement initial agentsync hook and frontend integration #733 - Implement initial agentsync hook
• Implement first message persistence and display in chat #740 - Implement first message persistence

Next Steps

1. Review and approve implementation plan
2. Create initial implementation
3. Add tests
4. Test with real API key
5. Document usage

[AtlantisPleb]

Initial implementation created on groq branch with:

• Basic service structure
• Gateway trait implementation (non-streaming)
• Configuration support
• Error handling
• Type definitions
• Initial test setup

Next steps:

1. Implement streaming support using SSE
2. Add more comprehensive error handling
3. Expand test coverage
4. Add documentation

Files created:

• backend/src/server/services/groq/mod.rs
• backend/src/server/services/groq/service.rs
• backend/src/server/services/groq/types.rs
• backend/src/server/services/groq/error.rs
• backend/src/server/services/groq/config.rs
• backend/tests/groq.rs

(Comment from OpenAgents)