This project is aimed at delivering a highly customizable, and easily configurable, Twitch AI chatbot that evolves and learns from individual user interactions. Powered by the Google Gemini 1.5 Flash model, the chatbot generates contextually aware text responses. It leverages SQLite for storing user-submitted prompts, allowing the chatbot to enhance its performance and relevance over time by referencing historical data.
3
- Generative AI: Utilizes the Gemini 1.5 Flash model for generating context-aware text responses.
- User-Specific Learning: The bot stores and references historical user prompts to refine responses over time.
- Customizable Instructions: Instructions for the AI are defined in the
chatbot_instructions.txtfile, allowing for tailored behavior and specific responses. - Text-to-Speech (TTS) Integration: The bot can generate spoken responses, enhancing user interaction with voice communication.
- Speech-to-Text (STT) Integration: The bot can transcribe spoken input, enabling voice-driven interactions.
- Dynamic Response Buffering: Both text and audio responses are managed with a buffering system, ensuring smooth and timely delivery.
- Robust Logging: Multiple levels of logging are implemented, including DEBUG mode for tracing and troubleshooting, along with INFO level logging for monitoring AI functions.
- API Key Management: API keys are securely stored in a local
.envfile, with enhanced feedback on API key failures. - Safety Controls: Configurable safety settings to ensure appropriate content moderation.
- Wikipedia Integration: Accesses Wikipedia API for more knowledgeable responses.
- Reinforcement Learning: Utilizes reinforcement learning to fine-tune responses based on interactions.
- Emotion Detection: Detects and adjusts responses based on user sentiment analysis.
- Mood-Based Emotional Range: Expresses a wide array of emotions based on user interactions, enhancing dynamic engagement. Users can customize emotional responses using the
emotional_states.txtfile. - Real-Time Feature Management: Allows users to change feature flags while the bot is running, enabling or disabling functions on the fly.
- Twitch Account Setup: Create a Twitch account for your chatbot and obtain the Client ID and OAuth token with read/write access:
- Client ID: Twitch Token Generator
- OAuth Token: Twitch Developer Console
- Gemini API Key: Get an API key for the Google Gemini LLM from Google AI Studio.
- TTS/SST API: To get your API keys for SST and TTS, go to Google Cloud Console
- Environment Configuration: Store your API keys and setup variables in a
.envfile. Ensure the file is correctly loaded usingload_dotenv()in your script. - AI Instructions: Write the AI’s behavioral guidelines and specific response rules in the
chatbot_instructions.txtfile. - Launch the Bot: Run the script to start your chatbot. If you encounter any issues, set the logging level to DEBUG and check the logs for detailed error information.
This Twitch AI chatbot offers a powerful and flexible platform for engaging with users in a personalized way. With its ability to learn from past interactions, adapt its responses, and manage emotional states, it provides a dynamic and interactive experience for Twitch communities. The addition of TTS and STT features, along with enhanced logging and real-time feature management, makes it a comprehensive solution for modern Twitch streaming.
Possibly redo standard-user set-up. Put all user variables in the .env?
This will make the CLI only for debugging and make it simpler to use.
- Added all necessary configurables to .env for easy of use. Added a .bat to run the scriptAlso, redo the instructions. Now that it is fully released, make it simpler for new users to implement.
Will write a detailed instructional manual- Possibly improve or replace LLM, may try a custom trained Gemini first.
- Can we go 100% open-source?
- Ongoing testing of WikipediaAPI, Emotion Detection, Moods, and AI Memory
- Android APP for individual use?
The model functions well, but is very limited on knowledge.
- TTS Feature: Implemented a Text-to-Speech (TTS) feature, allowing the bot to respond with synthesized speech.
- TTS Buffer/Queue: Added a buffer/queue system for managing TTS responses, ensuring they are processed in order.
- STT Integration: Integrated Speech-to-Text (STT) functionality, enabling the bot to transcribe spoken prompts and respond appropriately.
- Simultaneous Message and Audio Responses: Enabled the bot to queue both message and audio responses at the same time.
- Dynamic Flag Changes: Made feature flags changeable while the bot is running, offering greater flexibility.
- TTS Redemptions: Fixed the requirement for "Chester" at the front of TTS redemptions, streamlining user interactions.
- Memory Handling: Fixed an error where the bot improperly processed historical data in the full_prompt.
- AI User Reference: Resolved an issue where the AI consistently believed the user was referencing it in questions.
- Wikipedia Queries: Fixed an error in handling Wikipedia queries, improving search accuracy.
- .env Handling: Addressed an error related to handling empty or non-existent .env files.
- Chatbot Instructions: Fixed an issue with creating and loading new chatbot_instructions when one was not present.
- STT Audio: Introduced an adaptive noise gate that continuously improves its response to background noise over time, ensuring clearer audio capture.
- Additionally, a silence filter was implemented to accurately detect the start and end of recordings, optimizing resource usage and enhancing transcription accuracy. - API Key Feedback: Added more feedback on specific API key failures to assist users in troubleshooting.
- INFO Level Logging: Enhanced INFO level logging to not only debug issues but also to observe the AI's functions in real-time.
- Authorized Users List: Added an authorized users list to manage access to administrative commands.
- Emotional States Customization: Added an emotional_states.txt file to allow users to easily customize AI emotions.
- Audio File Handling: Switched the Read/Write operations of audio files to a buffer for improved performance.
- Async and Threads: Cleaned up and improved async and threaded operations
---------------------------------------------------------------------------------------------------------------------------------
- Implemented a feedback spam filter to ensure only the user who submitted a prompt can provide feedback once per prompt.
- Introduced a feature flag section, allowing users to easily enable or disable specific chatbot functions.
- Enhanced the feedback tracker to utilize a list-based approach for storing user IDs, ensuring feedback is processed correctly and efficiently.
- Fixed emotion detection by changing the model, so the bot no longer constantly detects the user as angry or in fear.
- Resolved an issue where the bot's emotional state could not be adjusted.
- Refactored code for improved maintainability and readability.
- Created an easy-to-use setup section for users unfamiliar with Python or the APIs.
- Updated the
adjust_emotional_statefunction to handle edge cases where the emotional state could exceed predefined limits, ensuring consistent behavior. - Added more logging and error checking to improve debugging and stability.
- Removed DuckDuckGo API for websearch
---------------------------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------------------------------------
- Integrated DuckDuckGo Instant Answer API for quick and relevant search results in chatbot responses.
- Implemented a mood-based system allowing the chatbot to exhibit a range of emotional states: Happy, Sad, Angry, Excited, Confused, Bored, Curious, Calm, Nervous, and Motivated.
- Developed slider functionality for gradual changes in emotional state, enabling smooth transitions based on user interactions.
- Integrated Wikipedia API to query keywords in user prompts to increase accuracy and depth of responses.
- Developed an Emotion Detection model to enhance the understanding of user prompts.
- Enhanced emotional state management by integrating mood variables into the
chatbox_instructionalprompt for more nuanced interactions. - Replaced
chatbot_memory.jsonwith SQLite for persistent memory storage. - Optimized memory handling to prioritize current conversations over historical data for improved relevance and accuracy.
- Resolved issues with emotional state transitions for appropriate mood adjustments.
- Improved handling of mood-based responses with updated
chatbox_instructionalprompt structure. - Enhanced error handling and logging for better debugging and monitoring of emotional state changes and memory interactions.
- Integrated DuckDuckGo Instant Answer API for improved search result integration.
- Revised SQLite library usage to support updated database management features.
---------------------------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------------------------------------
- Switched to environment variables for configuration using a
.envfile. - Added support for persistent memory storage in
chatbot_memory.jsonfor user-specific interactions. - Implemented user-specific memory in AI responses to retain context across messages.
- Integrated
dotenvfor managing environment variables securely.
- Updated AI model's system instruction and safety settings for better performance and content moderation.
- Revised message handling to include user-specific context and handle bot commands.
- Improved logging to include detailed information about memory interactions and API calls.
- Adjusted AI response temperature for a balance between creativity and coherence.
- Refined automated response logic to use the updated bot name and nickname.
- Resolved issues with handling environment variables and file loading errors.
- Fixed problems with saving and loading persistent memory.
- Addressed issues with message content filtering and response accuracy.
- Added support for external configuration files and environment variables for improved security and flexibility.
- Introduced a more robust system for managing and utilizing persistent memory in AI interactions.
- Enhanced the automated response system for more engaging interactions with viewers.
- Added
dotenvfor environment variable management. - Revised dependencies related to AI model configuration and memory handling.
---------------------------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------------------------------------
- Integrated Google Gemini API for advanced AI responses.
- Added automated response functionality to engage viewers after a set number of messages.
- Updated logging configuration to include timestamps and log levels for better debugging.
- Replaced Hugging Face GPT-2 model with Google Gemini for more dynamic and creative responses.
- Enhanced safety settings to block harmful content categories from the Google Gemini API.
- Implemented automated responses that trigger after a specific number of messages.
- Fixed handling of invalid responses from Hugging Face API.
- Improved accuracy of AI responses by correcting message prompt handling.
- Resolved issues with bot message filtering and message counting.
- Added detailed logging for API interactions and message processing.
- Improved error handling for API request failures and message sending issues.
- Updated prompt processing to handle message content variations more effectively.
- Switched from Hugging Face API to Google Gemini API for natural language generation.
---------------------------------------------------------------------------------------------------------------------------------