Automatically get GitHub contributors

[RichardLitt]

@eminence:

Once we get a repo, we should check-in a tool that will get the list of weekly contributors, and map their emails to github usernames.

To Do:

• Get GitHub commits and all contributors who made them in a given week, including merged commits
• Get GitHub comments for all repos in the past week, get all contribs
• Parse into one list

[jbenet]

Doable with GitHub api-- there's some npm modules to facilitate
On Thu, Dec 17, 2015 at 18:26 Richard Littauer [email protected]
wrote:

@eminence ipfs/team-mgmt#73 (comment):

Once we get a repo, we should check-in a tool that will get the list of
weekly contributors, and map their emails to github usernames.

—
Reply to this email directly or view it on GitHub
#1.

[harlantwood]

For using the github API via JS, I recommend https://github.com/philschatz/octokat.js/ -- supports entire API, allows either callbacks or promises.

If you want to do this via webhooks (probably you don't) I recommend https://www.npmjs.com/package/githubhook

[RichardLitt]

@harlantwood I love Octokat, worked great for https://github.com/RichardLitt/open-github-notifications. Thanks for the rec.

[RichardLitt]

Changed requirements, after seeing this comment in #2:

write/find a better tool to get list of committers, and write/find a tool to get a list of all comments.

[RichardLitt]

So, I have a working method for everything not covered already by @eminence. As it turns out, getting commits is really hard to do - this is largely because of the amount of calls I need to do for IPFS, which is essentially one for each branch, and then one for each page. That turns into something like 400 calls to the API for each time I try to get committers. I've got this working mostly with get-committers, but I want to hold off before I integrate it into the main module I am using to get contributors, name-your-contributors.

Currently, in name-your-contributors, I am getting these:

• get-code-reviewers: Get everyone who commented on a commit or on a Pull Request.
• get-issue-commenters: Get everyone who commented on an issue.
• get-github-issue-creators:Get everyone who created an issue.
• get-github-user:Get user objects from usernames (used for getting names to sort the above list).

I had a lot of fun with this project (perhaps too much, he says, as he is coding at 5:00am on four hours of sleep), and had to create depaginate to depaginate the GitHub api, and sort-alphabetic to sort ignoring case, as well as generator-nms, a yeoman scaffold for standard and ava node modules. If anyone wants to code review those, that would be awesome.

More work to do:

• Review of this
• combine this with @eminence's script to get committers and refactor https://github.com/RichardLitt/get-committers to do it in js
• Publish! :D

Here is an example result of the above.

$ 🐕name-your-contributors ipfs 2016-01-10T04:29:11.301Z
[@amstocker](//github.com/amstocker) (Andrew Stocker)
[@anarcat](//github.com/anarcat) (anarcat)
[@ansuz](//github.com/ansuz) (ansuz)
[@Ape](//github.com/Ape) (Lauri Niskanen)
[@bdunlay](//github.com/bdunlay) (Brian Dunlay)
[@benjaminbollen](//github.com/benjaminbollen) (Benjamin Bollen)
[@brailateo](//github.com/brailateo) (Constantin Teodorescu)
[@btrask](//github.com/btrask) (Ben Trask)
[@CaioAlonso](//github.com/CaioAlonso) (Caio Alonso)
[@chriscool](//github.com/chriscool) (Christian Couder)
[@ConsciousCode](//github.com/ConsciousCode) (Conscious Code)
[@david415](//github.com/david415) (David Stainton)
[@davidar](//github.com/davidar) (David A Roberts)
[@diasdavid](//github.com/diasdavid) (David Dias)
[@Dignifiedquire](//github.com/Dignifiedquire) (Friedel Ziegelmayer)
[@dylanPowers](//github.com/dylanPowers) (Dylan Powers)
[@eminence](//github.com/eminence) (Andrew Chin)
[@Faleidel](//github.com/Faleidel)
[@fazo96](//github.com/fazo96) (Enrico Fasoli)
[@findkiko](//github.com/findkiko)
[@GitCop](//github.com/GitCop)
[@greenkeeperio-bot](//github.com/greenkeeperio-bot) (Greenkeeper)
[@harlantwood](//github.com/harlantwood) (Harlan T Wood)
[@ikreymer](//github.com/ikreymer) (Ilya Kreymer)
[@jbenet](//github.com/jbenet) (Juan Benet)
[@jbshirk](//github.com/jbshirk) (Joe)
[@JesseWeinstein](//github.com/JesseWeinstein)
[@jhamfler](//github.com/jhamfler) (Hamfler)
[@Kubuxu](//github.com/Kubuxu) (Jakub Sztandera)
[@kyledrake](//github.com/kyledrake) (Kyle Drake)
[@lgierth](//github.com/lgierth) (Lars Gierth)
[@lidel](//github.com/lidel) (Marcin Rataj)
[@luigiplr](//github.com/luigiplr) (Luigi Poole)
[@Luzifer](//github.com/Luzifer) (Knut Ahlers)
[@matshenricson](//github.com/matshenricson) (Mats Henricson)
[@MChabez](//github.com/MChabez)
[@Mec-iS](//github.com/Mec-iS) (Lorenzo)
[@MichaelMure](//github.com/MichaelMure) (Michael Muré)
[@mildred](//github.com/mildred) (Mildred Ki'Lya)
[@Mithgol](//github.com/Mithgol)
[@NeoTeo](//github.com/NeoTeo) (Teo Sartori)
[@NightRa](//github.com/NightRa) (Ilan Godik)
[@noffle](//github.com/noffle) (Stephen Whitmore)
[@peerchemist](//github.com/peerchemist)
[@pra85](//github.com/pra85) (Prayag Verma )
[@prusnak](//github.com/prusnak) (Pavol Rusnak)
[@ralphtheninja](//github.com/ralphtheninja) (Lars-Magnus Skog)
[@Red5d](//github.com/Red5d)
[@reit-c](//github.com/reit-c)
[@rht](//github.com/rht)
[@RichardLitt](//github.com/RichardLitt) (Richard Littauer)
[@rugk](//github.com/rugk) (rugk)
[@SCBuergel](//github.com/SCBuergel) (Sebastian C. Bürgel)
[@seclorum](//github.com/seclorum) (seclorum)
[@sivachandran](//github.com/sivachandran) (Sivachandran)
[@thelinuxkid](//github.com/thelinuxkid) (Andres Buritica)
[@travisperson](//github.com/travisperson) (Travis Person)
[@whyrusleeping](//github.com/whyrusleeping) (Jeromy Johnson)
[@wking](//github.com/wking) (W. Trevor King)
[@xicombd](//github.com/xicombd) (Francisco Baio Dias)
[@zignig](//github.com/zignig) (Simon Kirkby)

[eminence]

yayy ⭐ will have a closer look later, but this is great stuff!

[darwin]

Nice job Richard! Data mining with GitHub APIs is fun (I did it myself several times with octokit ruby library). But I think in this case there could be another solution worth consideration.

What about having a simple script which would fetch all relevant branches from all relevant repos into one repo. This repo can be then processed by existing git tools (e.g. git-extras comes to mind).

Also you would get GitHub's pulse and contributors pages for free (including nice graphs and possible weekly/monthly/range views). Unfortunately this would not include issues/PR activity (without some non-trivial git/github-api work).

Just my .02c

[jbenet]

@RichardLitt this is perfect!! 👍 super cool modules

[eminence]

we already have a tool that can get data from clones of the ipfs repos. but it suffers from two problems:

1. It can't look up github usernames
2. It can't tell us who has commented on github issues

@RichardLitt work here addresses both of these issues

[RichardLitt]

@darwin Good to see you here! And good idea about grabbing everything into one repo. The issue is that comments are really quite important, in some cases more than commits. Comments show that people came and joined in the discussion, which often is longer, more broad-ranging, and more impactful than small bug fixes and the like. I want to include both. A hybrid model might work best, given the issues we have with getting commits.

For now, I've just got it working with ranges, which is sweet.

[Kubuxu]

Other solution would be to install organisation hook for those events. Even though they are not visible in Settings there are hooks for comments, reviews and PRs and some more it just has to be enabled straight via GitHubAPI. Hook can be installed organisation wise, only requirement is to have something receive and collect those events.

Ref: https://developer.github.com/v3/orgs/hooks/ https://developer.github.com/v3/activity/events/types/#event-types--payloads

[harlantwood]

Again, a good node js webhook module is https://www.npmjs.com/package/githubhook -- I set up a test on a heroku server in minutes, receiving all events for a github org.

[RichardLitt]

@harlantwood @Kubuxu That's a good idea. I am not sure how to do that on Heroku; I know that it can listen, but I don't know where to access the files, where the logs are stored, and how to parse the logs usefully. Would you be able to help me out with understanding this? Are there any good tutorials you know about?

[dignifiedquire]

@RichardLitt why not use one of the ipfs servers for this?

[RichardLitt]

@dignifiedquire Maybe I'm not being clear enough. I don't have the knowledge on how logs and constant processes work. I'll need some help learning how to set these up. I'm also worried about downtime and not catching things when our webhook service goes down (it will. They always do).

[harlantwood]

@RichardLitt I was playing with webhooks a while ago, see:

https://github.com/core-network/ci-status/blob/9bf894905c2c9b6cb4606724599e24124e4b4066/src/RepoWatcher.coffee

https://github.com/core-network/ci-status/blob/9bf894905c2c9b6cb4606724599e24124e4b4066/package.json#L13

I'd suggest creating a repo with a similar script and a similar npm start command and push it to a heroku app (good for testing, could later be one of our servers).

Last step is to go to ipfs org settings >> webhooks, and add the server URL there.

Then you should have a working webhook server -- in this case it will just log every githook event. It gives you all the info you need tho to trigger any action from the events.

[harlantwood]

Might be out of scope here, but would be cool to run https://github.com/joeyh/github-backup or similar upon githook triggers, and mirror all of our repos/comments/PRs to IPFS/IPNS... discussed some in ipfs-inactive/archives#22

[daviddias]

@renrutnnej @ipfs/community-wg can you get this back on the newsletter? It was a big part of the old times newsletter :)

[RichardLitt]

Happy to help with this effort as needed.

[jennwrites]

iirc @mikeal was already looking into this for next quarter.

[mikeal]

Yup, it's probably going to be handled by @pkafei but we need to do something more than just a "list of contributors." We want to find a way to highly new contributors and significant contributions by people outside of PL. Just listing every contributor produces, mostly, the same names each week that are employed to work on IPFS. The goal should be to encourage new contributors and retain them and that will be lost if we just include a massive list of full time contributors each week.

[RichardLitt]

It's easy enough to make a list of contributors you want to filter out, and to do so over a time period, too (filter all contributors who have contributed more than once in the past). That should solve that issue.

[daviddias]

Just listing every contributor produces, mostly, the same names each week that are employed to work on IPFS.

Do a diff on the first weeklys, the list kept changing and it was great :)

• https://blog.ipfs.io/4-ipfs-weekly-1/
• https://blog.ipfs.io/5-ipfs-weekly-2/
• https://blog.ipfs.io/6-ipfs-weekly-3/
• https://blog.ipfs.io/7-ipfs-weekly-4/

[andrew]

There's now a "First time contributors" page in the Ecosystem dashboard that could be used to enable this functionality if it's still something we want to do? https://ipfs.ecosystem-dashboard.com/contributors/new?range=7

[RichardLitt]

Still sounds like a good idea to me.

[jennwrites]

Given the volume of contributors now, I think something like this would be more appropriate in a quarterly/yearly recap. Thanks for the rec @andrew!

Closing for now.