diff --git a/cEP-0020.md b/cEP-0020.md new file mode 100644 index 000000000..c7ca0a4fb --- /dev/null +++ b/cEP-0020.md @@ -0,0 +1,589 @@ +# Newcomer metrics and Gamification + +| Metadata | | +| -------- | ------------------------------------------------------ | +| cEP | 20 | +| Version | 1.0 | +| Title | Newcomer metrics and Gamification system | +| Authors | Shrikrishna Singh | +| Status | Proposed | +| Type | Process | + +## Abstract + +This cEP describes a gamification system to automate the +recognition of activities performed by people who join the +coala community, and encourage them to progress towards +obtaining the role of "Developer" in the community. + +This cEP is a part of the +[GSoC project](https://summerofcode.withgoogle.com/projects/#5892040252981248). + +For the implementation of this process, this project will +incorporate a gamification system which will allow us to +track the progress of newcomers and assign some points to +each of the activities they do. There will be different +levels based on point thresholds and a newcomer will also +be eligible for specific privileges after the completion of +each level. There will also be different badges on the basis of +performed activities by the newcomers. + +So, every newcomer at coala will have to complete the processes +involved in the gamification system in order to get the +**coala all-rounder** badge and then they will be eligible to apply +for the developers role at coala. + +## Background + +### Newcomer Experience + +A newcomer at coala have to follow the +[newcomer guides](https://api.coala.io/en/latest/Developers/Newcomers_Guide.html) +and perform certain tasks mentioned there in order to be eligible +to apply for the "Developers" role at coala. + +[Here](https://github.com/coala/coala/blob/4f2b3c101b8d67a17078dfb2603083c9c00537a6/docs/Developers/Newcomers_Guide.rst) +is a permanent link of the newcomer guides at the moment. + +The "Developers" role at coala means that the recipient is now a part +of more trusted team, which is the organization's developers team +on GitHub. + +**The problems:** + +A good way to keep an open source project alive is to get new +contributors as the old ones will be busy in their own work +someday and couldn't be able to contribute on the daily basis. +So, the first and foremost problem is, how to attract +more newcomers to contribute to the community? + +But, with increasing amount of newcomers, chaos comes. And it will +be hard to focus on each and every new contributor. So, there has +to be a way to manage the same. Also, to keep the newcomers +contributing to the community, a good way is needed to teach and +help them so that the work on coala progress at a fast pace. + +Also there has to be a way to get more effective feedback from +the newcomers who have just started using the tools, organization +is building. And a way to use those feedback to improve coala and +its processes. + +## Why Gamification? + +As coala tries to be a welcoming organization for newcomers, +it gives them clear pathways and as much direct assistance as +possible. However, it is already known that newcomers face many +barriers while attempting to contribute to open source for the +first time, and additional barriers while attempting to +contribute to projects run by the coala organization. + +Some barriers they face include orientation issues that can +potentially demotivate newcomers from placing their first +contribution. On the other hand, gamification is widely used +to engage and motivate people to accomplish tasks and improve +their performance. + +Therefore, the goal of this project is to use gamification to +orient and motivate newcomers to overcome onboarding barriers +to contribute and engage with coala and its community. + +## Gamification Introduction + +There will be four game design elements to help newcomers: +Activities, Points, Badges and Levels. For each of these +elements, there will be a set of rules to describe the +operation of the gamified environment. + +### Activities + +Following are the activities which could tracked in this system: + +- Create bug report +- Create newcomer difficulty issue +- Create external linter bear proposal issue +- Create native bear proposal issue +- Create generic bear proposal issue +- Create documentation issue +- Create low difficulty issue +- Create medium difficulty issue +- Create high difficulty issue +- Create high importance issue +- Create medium importance issue +- Create low importance issue +- Create trivial importance issue +- Merge pull request that fixes a bug +- Merge pull request for external linter bear proposal issue +- Merge pull request for native bear proposal issue +- Merge pull request for generic bear proposal issue +- Merge pull request for a documentation issue +- Merge pull request for newcomer issue +- Merge pull request for low difficulty issue +- Merge pull request for medium difficulty issue +- Merge pull request for high difficulty issue +- Merge pull request for high importance issue +- Merge pull request for medium importance issue +- Merge pull request for low importance issue +- Merge pull request for trivial importance issue +- Gitmate errors on pull request +- CI status of the pull request +- Review comments on pull request +- Number of questions asked in `coala/coala` room +- Number of answers given in `coala/coala` room +- Number of questions asked in `coala-bears` room +- Number of answers given in `coala-bears` room +- Number of questions asked in `coala/corobo` room +- Number of answers given in `coala/corobo` room +- Number of questions asked in `coala/community` room +- Number of answers given in `coala/community` room +- Number of messages in coala other rooms +- Make use of `corobo` on Gitter +- Introduce `.coafile` in user own repository +- Introduce `.coafile` in an organisation with 50 or more stars + for its repositories +- Introduce CI in the users own repository to run coala using + Travis CI +- Setup the Circle CI to the fork of coala repository +- Setup Travis CI to the fork of coala repository +- Setup Appveyor CI to the fork of coala repository + +### Points + +Points are a simple tool to reward positive contributions to coala, +and they provide an excellent base upon which to build a gamified +design. A good way to start designing this system is to assign +points to some of the tracked activities above: + +- Create Issue + - 5 points will be deduced in any type of duplicate issues + - 5 points for importance trivial + - 5 points for difficulty newcomer + - 5 points for external linter bear proposal + - 7 points for native bear proposal + - 8 points for difficulty low + - 10 points for type/bug + - 10 points for importance low + - 10 points for generic bear proposal + - 12 points for difficulty medium + - 15 points for difficulty high + - 15 points for importance medium + - 20 points for importance high +- Merge Pull Request + - 5 points will be deduced if the pull request is duplicate + - 5 points will be deduced if the pull request is closed without merge + - If there are no Gitmate errors + - 10 points for difficulty newcomer + - 10 points for importance low + - 12 points for writing external bear + - 16 points for writing native bear + - 20 points for bug fixes + - 20 points for difficulty low + - 20 points for importance medium + - 20 points for writing generic bear + - 30 points for importance high + - 30 points for difficulty medium + - 40 points for difficulty high + - If Gitmate errors are b/w 1 to 10 + - 5 points for difficulty newcomer + - 8 points for importance low + - 8 points for writing external bear + - 12 points for writing native bear + - 15 points for bug fixes + - 15 points for difficulty low + - 15 points for importance medium + - 15 points for writing generic bear + - 20 points for difficulty medium + - 25 points for importance high + - 30 points for difficulty high + - If Gitmate errors are b/w 10 and beyond + - 5 points for importance low + - 5 points for difficulty newcomer + - 5 points for writing external bear + - 7 points for writing native bear + - 10 points for bug fixes + - 10 points for difficulty low + - 10 points for importance medium + - 10 points for writing generic bear + - 15 points for difficulty medium + - 20 points for difficulty high + - 20 points for importance high +- Activities related to introducing `.coafile` + - 5 points +- Activities related to setting up CIs to the fork + - 5 points + +Note that these points allocations to the activities can be +change during implementation, it is just an example to have +a better understanding of this system. + +This cEP relies on labels being up to date before a PR is +merged. This can be achieved by developers checking labels +on new issues, and maintainer checking the labels are up to +date before merging, but systems to improve the accuracy of +labelling of issues and PRs are beyond scope of this cEP. + +In case if a newcomer creates a medium level generic bear proposal +issue, he has met the criteria for the activities of: + +- Create medium difficulty issue +- Create a generic bear proposal issue + +Note that in this case points would not add up. There will be +a method to check that which activity is assigned with the +highest number of points and the newcomer will be awarded +with those points. Also, there will be more method for scoring in +the cases when assigining points based on the activity with higher +points would not be good. E.g.: + +Creating an issue which has importance/high(20 points) and +difficulty/newcomer(5 points) labels. + +### Levels + +To increase a user's competitive instinct, a series of levels can be +implemented that confer rank as newcomers become more active. A good +and straightforward way to award levels in a new gamified design is +to base them on point thresholds. As users earn points, they move up +an incremental series of levels. To infuse the level with meaning i.e.: +a ranking within the system, the levels should be named in a way that +indicates status. + +But the levels wouldn't be enough motivation to go until newcomers +have some special privileges after completing each level: + +Note that the following levels are an example of possible levels +with suggested privileges based on the example points allocations +in the previous section. + +- Level-I: + - Name: Fresher + - Points_required = 0 + - Privileges: + - No privileges for freshers +- Level-II: + - Name: Newbie + - Points_required = 10 + - Privileges: + - Get assigned to newcomers issue +- Level-III: + - Name: Beginner + - Points_required = 20 + - Privileges: + - Get assigned to a difficulty low issue +- Level-IV: + - Name: Intermediate + - Points_required = 50 + - Privileges: + - Mark your own PR as wip +- Level-V: + - Name: Cool + - Points_required = 100 + - Privileges: + - Mark a PR by another newcomer as wip + - Get assigned to difficulty medium issue +- Level-VI: + - Name: Expert + - Points_required = 200 + - Privileges + - Assign newcomers to difficulty newcomer issue + - Assign newcomers to difficulty low issue +- Level-VII: + - Name: Legend + - Points_required = 350 + - Privileges + - Invite newcomers to org + - Assign newcomers to difficulty medium issue + - Permission to `ack` and `unack` on the pull requests + +Implementation of privileges is beyond scope of the +gamification system. The gamification system will merely +expose a list of privileges that each person has, and other +systems are expected to adapt to use these privileges. +E.g.: [corobo](https://github.com/coala/corobo) + +While implementing privileges, a first manual check should +be done for testing purpose, and then automated only if there +is no serious risk or there is a good risk reaction mechanism +in place. + +### Badges + +Another way of explicitly nudging a newcomer to action is to +award badges for completing tasks. Badges enable newcomers to +follow their performance and compare to other newcomers. +Badges will be awarded after performing certain activities by +the newcomers: + +- The bug finder: + - Details: The one who find bugs in the existing codebase + - Activities: + - Create at least one issue which labelled as bug +- The bear hunter: + - Details: The one who create issues about new bears + - Activities: + - Create at least one native bear proposal issue + - Create at least one generic bear proposal issue +- The bear writer: + - Details: The one who write new bears + - Activities: + - Merge at least one external linter bear proposal pull request + - Merge at least one native bear proposal pull request +- The helper: + - Details: The one who help other newcomers + - Activities: + - At least 20 answers given on coala Gitter rooms + - Reviewed at least 1 high importance pull request + - Merge at least 1 high importance pull request +- The reviewer: + - Details: The one who review others' pull requests + - Activities: + - Review at least 3 difficulty newcomer pull requests + - Review at least 2 difficulty low pull requests +- The super reviewer: + - Details: The one who is the legend of reviewing + - Activities: + - Review at least 10 newcomer difficulty pull requests + - Review at least 7 low difficulty pull requests + - Review at least 3 medium difficulty pull requests +- The coala all-rounder: + - Details: Has met the prerequisites to apply for the coala developer role + - Activities: + - Introduce `.coafile` in user own repository + - Merge a difficulty/newcomer Pull Request + - Review at least a difficulty/newcomer Pull Request + - Merge a difficulty/low Pull Request + - Review at least a difficulty/low or higher Pull Request + +So, to be eligible to be a developer at coala, a newcomer must +complete all the activities mentioned in the `coala all-rounder` +badge. But in case if some of the newcomers are not able to be +eligible to apply for coala developer role, they could take away +the other badges. + +Names and metrics of tracked activities will also be shown +on the leaderboard along with the number of times they +have been done. That will help maintainers to varify if an +automatically awarded badge is correct or not. + +Note that in case of an issue or a pull request has two labels which +are counted in this system, the performed activities by the user will +be two, and both the activities will be counted while awarding badges. +So in case if there is a badge for creating a medium difficulty issue +and there is a badge for creating a bear proposal issue, both badges +will be awarded to the newcomer who has created a medium difficulty +bear proposal issue. + +## Implementation + +### Getting the data + +#### GitHub/GitLab + +For getting the data related to newcomers activity on +GitHub/GitLab to the community, this system will mainly +use [IGitt](https://gitlab.com/gitmate/open-source/IGitt). + +#### Gitter + +For getting the stats of newcomers' messages, this system +will use +[Gitter API](https://developer.gitter.im/docs/messages-resource) +and import all the messages by newcomers in webservices. And +then those messages will be imported to the community repo +into two Django models, i.e.: `Question` and `Answer`. +The `Question` model will be responsible for storing all +the questions asked by the newcomers in coala Gitter rooms. +And the `Answer` model will be responsible for storing all +the answers given by the newcomers in coala Gitter rooms. +And these numbers will be shown on the user's dashboard along +with each completed milestones of the newcomer. This will help +us to figure out how much a newcomer has been noisy, or how +much they have asked questions and how much they have helped +others while completing certain tasks and not promote them +to developers team until they have learned to learn things +by themselves. + +E.g.: [Here](https://gist.github.com/sks444/66c5d8e7704708fda50569fd3ce3a4db) +is the log of some frequent questions asked by a newcomer +while solving an issue, which clearly shows that the +newcomer is noisy and has not been through newcomer docs. +So, this system will count the number of questions asked by +that newcomer, and show it to the leaderboard along with +the issue they have completed. + +### A System for Tracking Activities, Points and Levels + +Now that the rules for our gamification design are ready and +the relevant data is imported in the community repo, a good + +way to start the implementation of gamification system is to +design some models and methods to track the user's activity, +points and levels: + +#### Creating the activity model + +An Activity model will give a way to keep a log of each time +the user earns or loses points so that there will be a record +of his points history on a dashboard. + +A mockup of a basic `Activity` Django model would look something +like: + +```python +class Actvity(models.Model): + name = models.CharField(max_length=500) + details = models.TextField() + points = models.IntegerField() + number_of_times = model.IntegerField(default=0) +``` + +`number_of_times` field store the value of number of times +an activity has been done by the same user. + +The `User` model will have a `ManyToMany` relationship with +the `Activity` model with a method named +`count_similar_activity` which will check if the action +by the user has been done previously? If Yes, it will +increase `number_of_times` field with one. + +#### Creating the Level Model + +A mockup of basic `Level` Django model would look something like: + +```python +class Level(models.Model): + name = models.CharField() + points_required = model.IntegerField() + user = models.ManytoManyField(User) + privileges = models.ForeignKey(Privilege) + + +class Privilege(models.Model): + name = models.CharField(max_length=500) +``` + +And a query to create a level would be: + +```python +privilege = Privilege.objects.create( + name='Get assigned to a newcomer issue', + ) +level1 = Level.obejects.create( + name='Newbie', + points_required=5, + privileges=privilege, + ) +``` + +#### Extending the User Model to add points and Levels + +Whenever a user perform an activity his points will be increased. +So, there should be some methods to check if the user has achieved +a new level: + +```python +class User(models.Model): + ... + ... + points = models.IntegerField() + level = models.CharField() + ... + + def add_points() + a_method_to_update_points_and_level() + a_method_for_the_log_of_the_activities() +``` + +#### Creating the Badges model + +```python +class Badge(models.Model): + name = models.CharField() + description = model.TextField() +``` + +The Badge model much like our Level model, defines a range of +possible awards that a user can receive. Unlike levels, users +can have many badges at one time. This sets up a many-to-many +relationship between badges and users, so there is a need of an +intermediate model to track which badges a user has earned. +The model will be called `Achievement`: + +```python +class Achievement(models.Model): + user_id = ManyToManyField(User) + bage_id = ManyToManyField(Badge) +``` + +Now, to make awarding badges easier, let’s also add a +convenience method to the User model: + +```python +def award_badge(name): + badge = Badge.objects.filter(name=name) + badges.award(name) +``` + +### Deployment and The leaderboard + +There will be Django management commands for each of the +needed processes which will be triggered on each build of +netlify and show the useful updated user metrics and stats +on the leaderboard. + +The leaderboard will have the following columns: + +| Username | Activities Performed | Points | Levels | Badges | + +`Username` column will show the usernames of each newcomer. + +`Activities Performed` column will show all the activities perforemd +by the newcomer along with the number of questions aksed and answers +given on coala Gitter channels while performing those activities. + +`Points` column will show the total points earned by the newcomer. + +`Levels` column will show the current level of the +newcomer with enabled privileges. + +`Badges` column will show the badges earned by the user based on +their activity. + +The leaderboard will be sorted based on the total points earned +by newcomers, which means that the newcomer with highest points +will be on top and the points are directly connected to the levels, +So, the newcomer at the top will also be on the highest level. + +## Conclusion + +This project addresses the most of the problems a +newcomer at coala face during the newcomer to +developer process and tries to provide a simplified and +automated solution: + +- A maintainer doesn't have to check manually if the + newcomer has done all the activities involved + in newcomer->developer process, they can just look at + the points and badges earned by the newcomer through + the gamification system. However, this system doesn't yet + provide maintainers with the assurance of the quality of the + work, so some manual verification will still be required. + +- This project provides a fun way to work in the community, + newcomers will be motivated to work when they will earn + points and badges even for the tiniest bit of their + contributions. + +- Having a gamification system will be a good resource to + attract more newcomers and the automation process could + easily deal with an increase in the number of new contributors. + +- In case if some of the newcomers are not able to complete + the newcomer process they can get other available badges + based on their activities in the community, which will be a + kind of symbol of their talent and work experience in OSS + projects. + +- Keeping track of newcomers' messages on Gitter channels will + help us to know how much noisy a newcomer is during performing + certain tasks and not promote them to developers until they + have started to find the solution of their problems on their own.