Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: TTS(Text2Speech) with over 15 languages support! #2110

Merged
merged 2 commits into from
Jan 11, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions packages/plugin-tts/.npmignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
*

!dist/**
!package.json
!readme.md
!tsup.config.ts
!tsconfig.json
173 changes: 173 additions & 0 deletions packages/plugin-tts/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,173 @@
# @elizaos/plugin-tts

A plugin for text-to-speech(TTS) generation using the FAL.ai API within the ElizaOS ecosystem.

## Description

The text-to-speech(TTS) plugin enables AI-powered creation of speech through FAL.ai's services. It provides functionality to generate audio from text descriptions, automatically detects language, and selects appropriate voice models.

## Installation

```bash
pnpm install @elizaos/plugin-tts
```

## Configuration

The plugin requires the following environment variable or runtime setting to be set:

```typescript
FAL_API_KEY=<Your FAL.ai API key>
```

## Usage

### Basic Integration

```typescript
import { TTSGenerationPlugin } from "@elizaos/plugin-tts";
```

### Voice Generation Examples

```typescript
// The plugin responds to natural language commands like:

"Generate TTS of Hello World";
"Create a TTS for Welcome to ElizaOS";
"Make a TTS saying [your text]";
```

## API Reference

### Actions

#### GENERATE_TTS

Generates speech audio based on text input.

**Aliases:**
- TTS_GENERATION
- CREATE_TTS
- TEXT2SPEECH
- T2S
- TEXT_TO_SPEECH
- AUDIO_CREATE

**Features:**
- Automatic language detection
- Voice selection based on detected language
- Local file caching
- Progress tracking
- Error handling

## Common Issues & Troubleshooting

1. **Generation Failures**
- Verify FAL API key is correctly set
- Ensure text input is at least 3 characters long
- Check network connectivity to FAL.ai services

2. **Storage Issues**
- Verify write permissions to content_cache directory
- Ensure sufficient disk space
- Check if content_cache directory exists

## Security Best Practices

1. **API Key Management**
- Store FAL API key securely using runtime settings or environment variables
- Never commit API keys to version control
- Monitor API usage

## Development Guide

### Setting Up Development Environment

1. Clone the repository
2. Install dependencies:

```bash
pnpm install
```

3. Build the plugin:

```bash
pnpm run build
```

4. Run the plugin:

```bash
pnpm run dev
```

## Future Enhancements

1. **Advanced Voice Features**
- Custom voice model support
- Voice style transfer
- Emotion control
- Speech rate adjustment
- Pitch modification
- Multiple speaker support

2. **Audio Processing**
- Background noise reduction
- Audio quality enhancement
- Format conversion options
- Volume normalization
- Audio effects processing
- Batch processing support

3. **Language Support**
- Expanded language detection
- Regional accent support
- Dialect customization
- Pronunciation improvements
- Multi-language mixing
- Custom pronunciation rules

4. **Integration Features**
- Streaming audio support
- Real-time generation
- Caching optimization
- Batch generation
- Queue management
- Progress monitoring

5. **Developer Tools**
- Extended API options
- Testing framework
- Performance profiling
- Debug logging
- Integration examples
- Documentation generator

We welcome community feedback and contributions to help prioritize these enhancements.

## Contributing

Contributions are welcome! Please see the [CONTRIBUTING.md](CONTRIBUTING.md) file for more information.

## Credits

This plugin integrates with and builds upon several key technologies:

- [FAL.ai](https://fal.ai/): AI model deployment platform
- [langdetect](https://github.com/wooorm/franc): Language detection library
- [ElizaOS](https://elizaos.com): Core framework

Special thanks to:
- The FAL.ai team for AI infrastructure
- The langdetect development community
- The Eliza community for their contributions and feedback

For more information about TTS capabilities:
- [FAL.ai Documentation](https://fal.ai/docs)
- [ElizaOS Documentation](https://docs.elizaos.com)

## License

This plugin is part of the Eliza project. See the main project repository for license information.
3 changes: 3 additions & 0 deletions packages/plugin-tts/eslint.config.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
import eslintGlobalConfig from "../../eslint.config.mjs";

export default [...eslintGlobalConfig];
34 changes: 34 additions & 0 deletions packages/plugin-tts/package.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
{
"name": "@elizaos/plugin-tts",
"version": "0.1.7",
"type": "module",
"main": "dist/index.js",
"module": "dist/index.js",
"types": "dist/index.d.ts",
"exports": {
"./package.json": "./package.json",
".": {
"import": {
"@elizaos/source": "./src/index.ts",
"types": "./dist/index.d.ts",
"default": "./dist/index.js"
}
}
},
"files": [
"dist"
],
"dependencies": {
"@elizaos/core": "workspace:*",
"tsup": "8.3.5",
"whatwg-url": "7.1.0"
},
"scripts": {
"build": "tsup --format esm --dts",
"dev": "tsup --format esm --dts --watch",
"lint": "eslint --fix --cache ."
},
"peerDependencies": {
"whatwg-url": "7.1.0"
}
}
Loading
Loading