Development: Provide documentation on how to use automatic assessment suggestions for modeling exercises
#8704
Conversation
github-actions
bot
added
documentation
config-change
…//github.com/ls1intum/Artemis into feature/athena-modeling-module-documentation
WalkthroughThe recent updates introduce the Athena service integration into Artemis for automated feedback in modeling exercises. The documentation now includes details on setup, usage, and UI interactions for assessment suggestions. No changes were made to the declarations of exported or public entities. Changes
Sequence Diagram(s) (Beta)sequenceDiagram
participant User
participant Artemis
participant Athena
User->>Artemis: Submit modeling exercise
Artemis->>Athena: Request assessment suggestions
Athena-->>Artemis: Return assessment suggestions
Artemis->>User: Display suggestions in grading interface
User->>Artemis: Accept or discard suggestions
Artemis->>Artemis: Update exercise assessment
Recent review detailsConfiguration used: CodeRabbit UI Files selected for processing (2)
Additional comments not posted (4)
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (invoked as PR comments)
Additionally, you can add CodeRabbit Configration File (
|
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | ||
| This section intends to provide insights into how automated feedback suggestions are generated for modeling exercises | ||
| using Athena. While Athena does support having various evaluation modules per exercise type, at the moment | ||
| only one module exists for the evaluation of modeling exercises (``module_modeling_llm``). The following section |
There was a problem hiding this comment.
While this is true that we have only modeling_llm for modeling exercises, would the process actually differ for another module types?
There was a problem hiding this comment.
Good question - it is kind of hard to tell as we might also build Athena modules that are using other ML approaches than LLMs in the future 🤔
| instructions. | ||
|
|
||
| 4. **Token Limit Check:** Feedback generation is aborted if the prompt is still too long after removing omittable | ||
| features. Otherwise, the prompt is executed on the connected language model. |
There was a problem hiding this comment.
Will the request still be considered successful? What is the error message the user will see on the UI?
There was a problem hiding this comment.
The request is still considered successful as there is still suggestions that can be made even with less information present. At the moment Athena does not indicate if information has been omitted due to exceeded token counts to at the moment we do also not indicate that in the UI. 🤔
|
|
||
| Evaluate the following 10 criteria: | ||
|
|
||
| 1. Give 1 point if all elements described in the problem statement are present in the submission, 0 otherwise. |
There was a problem hiding this comment.
Shouldn't the precise number of points depend on the exercise?
There was a problem hiding this comment.
It does still depend on the exercise - but matching the number of points assigned for the various grading instructions with the number of points achievable in the exercise helps to decrease the variance in the number of points assigned :)
…//github.com/ls1intum/Artemis into feature/athena-modeling-module-documentation
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
I think it would be good to have a background to this model. If dark mode users are reading the documentation, the model would be very hard to read.
| 2. Give 1 point if the outgoing flows from an exclusive gateway are also labeled if there is more than one outgoing flow from the exclusive gateway, 0 otherwise. | ||
| 3. Give 1 point if a start-event is present in the student's submission, 0 otherwise. | ||
| 4. Give 1 point if an end-event is present in the student's submission, 0 otherwise. | ||
| 5. Give 0 points if the activities in the diagram are not in the correct order according to the problem statement, 1 otherwise. |
There was a problem hiding this comment.
| 5. Give 0 points if the activities in the diagram are not in the correct order according to the problem statement, 1 otherwise. | |
| 5. Give 1 point if the activities in the diagram are in the correct order according to the problem statement, 0 otherwise. |
|
|
||
| One of the strategies for optimizing the prompt results of an LLM is instructing the model as clearly as possible about the expected output of the task at hand. | ||
| The following listing shows grading instructions for an exemplary BPMN process modeling exercise optimized for automatic assessment. | ||
| The instructions explicitly list all aspects Athena should assess of and how credits should be assigned accordingly ensuring consistent suggestions across all submissions. |
There was a problem hiding this comment.
| The instructions explicitly list all aspects Athena should assess of and how credits should be assigned accordingly ensuring consistent suggestions across all submissions. | |
| The instructions explicitly list all aspects Athena should assess and how credits should be assigned accordingly, ensuring consistent suggestions across all submissions. |
krusche
changed the title
Development: Provide documentation on how to use automatic assessment suggestions for modeling exercises
Checklist
General
Description
This PR adds documentation on how to use the automated feedback suggestions provided by Athena for modeling exercises and how the generation works on a technical level. In addition to that, a new documentation page is added that describes the process of how feedback suggestions are generated for modeling exercises and how exercises can be optimized for automated evaluation.
Testserver States
Note
These badges show the state of the test servers.
Green = Currently available, Red = Currently locked
Screenshots
Documentation on how to use automated feedback suggestions for modeling exercises
Documentation how automated feedback suggestions for modeling exercises are generated