OpenDigger Biweekly Report 2024-09-18

[Peng99999]

Description

Opendigger：

In the past two weeks, Opendigger has completed the following work:

Added label data：

• [Chinese Developer] I wish to submit 378 Chinese Developers recognized as Top 10K gobal developers of 2024 Jan-July #1603 and label: add chinese users #1606： Added 378 Chinese Developers recognized as Top 10K gobal developers of 2024 Jan-July

• [Chinese Developer] Add more Chinese developers from Ant Group #1608 and label: update label for CN and bot #1609：Add more Chinese developers from Ant Group

• [Batch Label Data] Add more label data for Database technical area labled on dbdb.io and DB-Engines Ranking up to August 29, 2024. #1610 and Update data for Database technical area labled on dbdb.io and DB-Engines Ranking up to August 29, 2024 and update database/index.yml. #1611： Add more label data for Database technical area labled on dbdb.io and DB-Engines Ranking up to August 29, 2024.

• [help wanted] Submitting Application for Open Leaderboard #1617：Added a company label with the name: Kylin Soft. label: add kylin soft and openkylin #1618

• label: add more robots account #1612

• label: add kwdb #1620：Added Inspur and KaiwuDB label data.

• label: add a Chinese user #1616

Refeactor：

• refactor: add order to export list #1604: Sort the export repo and user list with OpenRank in desc order.This will help with some advance search functions.

• refactor: use PushEvent in global openrank #1613 : Modify the global OpenRank algorithm for a little bit. Use PushEvent instead of PullRequest merge event, because actually the pull request merge event will lead to a corresponding push event, but direct push will not contain a pull request event. Although we would like to encourage issue and pull request collaboration, direct pushes are also widely occurs in the open source world even in many really important projects. To fit the different collaboration model in open source world, use PushEvent with a relatively low weight will do. Set the monthly activity limit for a single developer to the 75th percentile of the overall active users, which is approximately achieving 2 PRs per month to reach the limit for that month. Output the users who active in more 300 repos within a month to help us to find out robot accounts. If calculation for a month fails, we should restart the process because graph database may not stable.

• refactor: optimize the openrank sql #1622: 1. Optimize the custom repo names, org names and user logins transfer to ids, only use global_openrank table rather than original events table which is much faster. 2. Optimize the user community OpenRank API, filter the original data with user ids which helps a lot in memory consumption.

Feat:

• feat: add user community openrank interface #1614: Add user community OpenRank interface, this is the first interface which will both use repo filter and user filter. Add yyx to China label.

• feat: export commumity openrank to files #1605: Export the community OpenRank while calculate the metrics. The data will be exported to （https://oss.x-lab.info/open_digger/github/X-lab2017/open-digger/community_openrank.json) file.

• feat: change open leaderboard user list to community openrank #1619 : 1. Add a few more bot account. 2. Use user community OpenRank instead of user global OpenRank in OpenLeaderboard. 3. To accomplish the second one, add withoutDetail option to user community OpenRank interface to reduce the memory use. 4. Still the monthly data for global user OpenRank is too memory consuming for ClickHouse database so split the query to batches and merge in local memory.

FIx:

• fix: fix milvus label to has a single source #1621 : if a project label has more than 1 source, the project result will be wrong and contain more than 1 same item. So fix the milvus label, the AI tech label will directly use the company's project label.