Adaptive learning: Improve progress, mastery and confidence

[JohannesStoehr]

Feature Proposal

Spec Version 0.2.0

Context

Problem

Artemis uses metrics like exercise scores and completed lecture units to mea- sure the performance of a student and suggest the next exercise. However, these calculations are currently simple and inaccurate, leading to problems. If values are too high students do not get enough exercises to practice; if too low students are stuck solving similar exercises about a topic they already understand. Additionally, displaying these inaccurate values can mislead students about their proficiency.

Motivation

Improving accurate measurement of students’ progress in topics is essential to enhancing educational outcomes. By refining them, Artemis can provide students with clear, personalized guidance and suggest suitable exercises in the individual learning path. Accurate assessments of progress help to understand strengths and areas for improvement. This not only helps students stay motivated and focused. Ultimately, optimising the selection of exercises in learning paths and progress measurement contributes to the development of more competent and confident learners.

Requirements Engineering

Existing (Problematic) Solution / System

Currently the mastery of a competency is simply a weighted sum of the progress and confidence.
The progress is the percent of completed learning objects. For exercises it is the case whenever the student submitted anything in that exercise.
The confidence is the average score of the student in the exercises related to the competency.

Proposed System

The progress should reflect how far along the student is to completing the competency. Lectures units should be completed when they are accessed by the student. For exercises this should be the score weighted by the points in the exercise.
Formula: P = |E|/(|E|+|L|) * (Sum for each Exercise: Latest_Score * MaxPoints) / TotalPointsInExercises + |L|/(|E|+|L|) * |L_c| / |L|

The confidence value should reflect if the progress corresponds to the actual understanding of the student of the competency. Instead of a value between 0 and 100, this should be a factor, with 1 meaning that the progress reflects the understanding correctly. It should be influenced by:

• The variance of the exercise scores within a difficulty. A higher variance should decrease the confidence. Calculation: (Average coefficient of variation per difficulty level)
• The recency of scores. Newer scores better reflect the current understanding. If newer scores are higher the confidence should also be higher and vice versa. Calculation: (Average Score) - (Sum for each Exercise: Latest_Score * (1+progress between first and last result)) / (Sum for each Exercise: 1+progress between first and last result)
• The percent of easy vs. hard points. If the portion of points gained in hard exercises is higher than the portion of hard points in the competency, the confidence should be higher. For easy exercises it should be the other way round. Calculation: ((Average of hard points achieved) - (Average of hard points in competency)) - ((Average of easy points achieved) - (Average of easy points in competency))
• Learning Goals: Use additional metrics for confidence score #6084 suggests to take the time or number of submissions into account. While a high submission count or long time are not a good indicator for not having achieved the competency (see comment in issue), few submissions and solving the exercise in a short time can be a indicator. Calculation: (Number of exercises solved quickly) / (Number of expected exercises)
  A possible formula could look like: C = 1 + Sum of all indicators, weighted by some factors

The mastery is the multiplication of the progress and confidence, clamped to at most 100%

The progress and mastery should be understandable to the student as they are.
The confidence might not be understandable right away. To ensure the understandability, Artemis should show short and simple explanations in a tooltip why the confidence is above or below 1.

Requirements

1. NFR: Performance, speed: Fast Calculation: The values should be computable within a second for a student in a course with 20 competencies and 100 exercises
2. NFR: Usability, Consistency: No Mastery Decline by Participation: Participating in exercises should never decrease the mastery of a student as long as the score does not decrease
3. NFR: Usability, Consistency: No Decline by Time: The value of any metric may not decline over time, e.g. when other students solve the exercises and are used for comparison.
4. NFR: Reliability, Robustness: Students have to be able to achieve 100% mastery regardless of their prior score history

Analysis

Analysis Object Model

What are the involved Analysis Objects?

System Architecture

Subsystem Decomposition

Show the involved subsystems and their interfaces. Make sure to describe the APIs that you add/change in detail. Model the DTOs you intend to (re)use or change!
The rest endpoints will not change their purposes but their return values will be replaced by DTOs only containing the necessary data.

Persistent Data Management

Describe the Database changes you intend to make.
Outline new configuration options you plan to introduce
Describe all other data persistence mechanisms you may use.
The current values are stored in the CompetencyProgress objects in the database. There are a few possibilities to handle these old values:

• Write a Java migration that updates all competency progress after the startup (drawback, slow)
• Let the old values be in the database. Whenever a student interacts with an exercise or lecture unit connected to a competency, the corresponding competency progress is updated fixing the values on demand.
• Drop all entries of the competency progress table from the database. This way no misleading data is shown to students. After that every competency progress has to be recalculated when the student tries to access their competency progress.

Additionally it would be beneficial to store the reason for a high/low confidence in the CompetencyProgress, so users can see a tooltip explaining their mastery in the ui without the need for reevaluations on the server.

Other Design Decisions

Potential topics to discuss here include: WebSockets, testing strategies.

UI/UX Design

Screenshots of the final UI mockups (mandatory): Please include screenshots to provide a clear and persistent visual reference of the design.
Link to the design mockup (optional): Additionally, you may include a link to the live design mockup (e.g., Figma, Sketch) for a more interactive view. Note that this link is supplementary and should not replace the required screenshots.

@JohannesWt
Only show progress and mastery
Show tooltip explaining why the mastery is higher/lower than the progress