[Feature] Grading and Feedback System for Questions and Study Performance

[miguelcsx]

Description

Develop a comprehensive grading and feedback system with two primary components:

1. Question Grading Workflow: Implement a user-driven system where users can rate responses, helping refine future interactions. When a user asks a question, they will first see a list of top 5 related FAQs. If none are satisfactory, they will receive a response from an LLM. If the LLM answer is unsatisfactory, the query will be escalated to an assistant, and finally, to a teacher expert.
2. Study Performance Grading: Implement a spaced repetition-based grading system for evaluating user studies by topics. Users will be asked questions related to their study topic, and their performance will be graded to determine when and what themes to evaluate next.

Goals

• Bidirectional Question Grading:

  • Allow users to rate answers during the question-handling process, improving future question-answer interactions.
  • Implement a multi-step escalation process for questions where users can escalate their question from FAQs, to LLM responses, to an assistant, and finally to a teacher expert.
  • Log the ratings from each step for future improvement of the question-answer system.

• Study Performance Grading:

  • Classify user studies by topics and track performance through a spaced repetition system.
  • Regularly evaluate users on their study topics and grade their responses to determine when to reintroduce certain topics for review.
  • Provide users with performance feedback on their study progress and track learning gaps.

Tasks

1. Question Grading System:

   • Step 1: Retrieve FAQs:
     • When a user submits a question, retrieve the top 5 related questions from the database.
     • Allow the user to select one of the FAQs as a satisfying answer.
   • Step 2: LLM Response:
     • If no FAQ is satisfactory, provide an LLM-generated response.
     • Allow the user to rate the LLM answer (e.g., 👍 or 👎).
   • Step 3: Escalation to Assistant:
     • If the LLM response is unsatisfactory, escalate the question to an assistant for review.
     • The user can rate the assistant’s response.
   • Step 4: Escalation to Teacher Expert:
     • If the assistant’s response is not helpful, escalate the question to a teacher expert for a final review and answer.
     • Track the ratings at each step for future improvements.

2. Study Performance Grading System:

   • Step 1: Topic Classification:
     • Classify the user’s studies by topics and store their performance data based on those topics.
   • Step 2: Spaced Repetition Evaluation:
     • Ask users questions related to their study topic, based on the spaced repetition algorithm, to evaluate retention and understanding.
     • Grade their performance on these questions to adjust the frequency of topic reviews (e.g., more frequent review of weaker topics).
   • Step 3: Performance Feedback:
     • Provide users with performance feedback based on their answers, including suggestions for improvement and topic areas that need more attention. [Feature] Generate Role-Based Statistical Reports and Insights on User Performance #6
   • Step 4: Data Collection:
     • Store user performance data to track progress over time and adjust learning paths based on past performance.

3. User Interface and Experience:

   • Design a clear and user-friendly interface for rating questions and viewing performance feedback.
   • Provide intuitive grading options for users at each step of the question escalation process.

4. API and Database Integration:

   • Use the API to store and retrieve user ratings and study performance data.
   • Integrate the database with the spaced repetition system to fetch questions related to study topics and track performance metrics. [Feature] Implement Spaced Repetition Learning Events via API Integration #7
   • Implement endpoints to handle question grading workflow, from FAQs to teacher expert escalation.

5. Error Handling and Feedback:

   • Implement error handling to ensure smooth transitions between each step of the grading process.
   • Provide clear feedback to the user if any part of the system fails (e.g., FAQ retrieval issues, LLM failure).

6. Testing and Optimization:

   • Test the system across different scenarios to ensure accurate grading of both questions and study performance.
   • Gather user feedback to fine-tune the rating system, LLM responses, and study performance evaluations.

Acceptance Criteria

• Users can rate responses during the question-handling process, with the system escalating questions as needed, from FAQs to LLM, assistant, and teacher expert.
• The system tracks ratings for LLM, assistant, and teacher answers and stores them for future improvement of question handling.
• Users’ study performance is graded based on a spaced repetition system, which adjusts the frequency of topic reviews based on performance.
• Users receive feedback on their study progress and have access to performance insights across topics.
• The system handles data retrieval and updates seamlessly using the API and is robust against errors.

Priority

High

Type

Feature

Notes

Ensure that user satisfaction data collected during the question workflow is used to refine the FAQ, LLM, and assistant responses over time. Provide easy access to past performance reports for both students and teachers to track progress over time.