Override probot/stale defaults, if necessary

[kytrinyx]

Per the discussion in exercism/discussions#128 we
will be installing the probot/stale integration on the Exercism organization on
April 10th, 2017.

By default, probot will comment on issues that are older than 60 days, warning
that they are stale. If there is no movement in 7 days, the bot will close the issue.
By default, anything with the labels security or pinned will not be closed by
probot.

If you wish to override these settings, create a .github/stale.yml file as described
in https://github.com/probot/stale#usage, and make sure that it is merged
before April 10th.

If the defaults are fine for this repository, then there is nothing further to do.
You may close this issue.

[robphoenix]

I've added a pinned label to our long running issues.

[petertseng]

What would y'all say about adding "good first patch" to the list of labels that will not be closed? See how many previous good first patch issues did not see activity for >60 days, but they still remained relevant (thus useful to stay open), until they were finally done in Hacktoberfest?

[petertseng]

Or, we could choose not to create .github/stale.yml in this repo, which will cause probot-stale to not act on this repo (https://github.com/probot/stale/blob/master/index.js#L57), if that's the option y'all want to go with.

[kytrinyx]

Aaaah, I missed that. Dang it. I read the README wrong, and thought that defaults apply if no .github/stale.yml file exists.

I'm going to have to post an update to all the issues I created.

[kytrinyx]

I made a mistake.

This was not explicitly documented in the README, but probot/stale requires the presence of a .github/stale.yml file.

If you want the default behavior, you need to add an empty .github/stale.yml file to this repo, otherwise the bot will not be active.

[petertseng]

It is at this point that I wish I had the time to use https://developer.github.com/v3/issues/comments/ to get some data on this: Which issues would have been auto-closed by probot, with a 60-day strategy? Should they have been closed?

What do I mean by "should", even? Well, it seems to me that there are various kinds of issues:

• Issues where we've decided on a clear plan of action
  • Good examples: the good first patch issues.
  • These issues should probably stay open until the action is taken, no matter how stale they get.
  • Only exception probably is if they get so stale that the action they recommend is no longer relevant to the repo, but the bot does not have the ability to decide that, so I say the bot should never close these issues.
  • For these issues, the bot forces extra work on us, by forcing us to apply a label to exempt it.
• Issues where there isn't a clear plan of action, perhaps discussions or questions.
  • Good examples are Alphametics #159 or Paasio issues #143 which should have been closed long before they were actually closed.
  • For these it seems being stale implies not enough consensus to change the status quo and/or the querent is no longer interested in the question.
  • For these issues, the bot saves us work.

If the bot creates more work than it saves, then I would obviously oppose having it. Since I don't have that data right now, I'll defer my vote. Feel free to proceed and make a decision without having my vote.

[petertseng]

I'm finally ready to look at the data.

I used https://gist.github.com/petertseng/04edd07d4b35cda2cbc1d8142b6fcc13 to get a rough idea of the issues that would have been closed by this integration. (Limitations: It doesn't look at pull requests, and it may have false positives for issues that are edited after they are closed)

In this track's history, 52/157 issues went stale (I deliberately do not include a # in front of each since I don't mean to reference those issues from this one):

420 Consider using Example tests for some exercises
414 secret-handshake, and others: use %q to make difference between [""] and [] clear
357 test generators should use post-re-organisation x-common JSON file paths
345 Update config.json to match new specification
340 sum-of-multiples README is confusing, test code (unnecessarily?) expects higher-order function
328 Updated tests for the Custom Set problem
317 accumulate: test failure message prints unhelpful function pointer, go vet complains
311 word-search: Improve error-handling behavior
310 trinary: Improve error-handling behavior
309 tree-building: Improve error-handling behavior
308 tournament: Improve error-handling behavior
307 saddle-points: Improve error-handling behavior
306 roman-numerals: Improve error-handling behavior
305 rna-transcription: Improve error-handling behavior
304 queen-attack: Improve error-handling behavior
303 poker: Improve error-handling behavior
302 phone-number: Improve error-handling behavior
301 palindrome-products: Improve error-handling behavior
300 paasio: Improve error-handling behavior
299 octal: Improve error-handling behavior
298 nucleotide-count: Improve error-handling behavior
297 minesweeper: Improve error-handling behavior
296 matrix: Improve error-handling behavior
295 ledger: Improve error-handling behavior
294 largest-series-product: Improve error-handling behavior
293 kindergarten-garden: Improve error-handling behavior
292 house: Improve error-handling behavior
291 hexadecimal: Improve error-handling behavior
290 hamming: Improve error-handling behavior
289 grains: Improve error-handling behavior
288 gigasecond: Improve error-handling behavior
287 error-handling: Improve error-handling behavior
286 connect: Improve error-handling behavior
285 circular-buffer: Improve error-handling behavior
284 bracket-push: Improve error-handling behavior
283 binary: Improve error-handling behavior
282 beer-song: Improve error-handling behavior
281 bank-account: Improve error-handling behavior
242 Consider re-implementing house to not guide the solution
203 Consider moving excercises to subdirectory
200 Consider creating new Unicode exercise
197 Add broad range of inputs for leap benchmark?
177 Should we swap testVersion and TestVersion?
157 Int type used in largest-series-product may be too small
151 Meetup - 5th Monday
148 Add stub files to problems that have a testVersion
143 Paasio issues
138 Weird order of tests in config.json
116 Custom Set:  Use of DeepEqual excludes valid slice-based solutions.
110 gigasecond_test.go should be improved
 97 provide implementation file
 88 Test fail messages

Looking at these 52 issues, I see that 31 of them are the "improve error-handling" issues, which had clear enough calls to action that it seems like it doesn't make sense to close them even if they go stale. I haven't even looked at any of the others. But it seems based on this that in at least 31 cases the bot would cause extra work for the maintainers of this track and thus in no more than 21 cases could the bot have saved work.

As I promised, if the bot would create more work than it saves, I would oppose having it. So here is my vote in opposition.

[tleen]

Would it make more sense to bundle the improve error-handling under a single issue? One of those slow-burn issues that is handled an exercise at a time?

[kytrinyx]

@tleen I think it makes sense to keep them separate—that makes it easier for individual people to tackle just one of them when they have a few minutes, and the commit/PR will close the related issue (provided that they reference it). It's a lot harder to keep a checklist up to date (I learned this the hard way).

[robphoenix]

tbh I'm easy either way, I don't have a particularly strong feeling on this. I assume it should be easy enough to just remove my commit adding in the stale.yml file. So I'm happy to support the majority decision either way.

In light of @kytrinyx comment, should we split #604 & #605 into seperate exercise specific issues? And perhaps even #345?

[kytrinyx]

I think for #345 it doesn't make sense. Since we don't currently use the topics for anything I think we could safely put off adding them. For the difficulty level, that's about comparing exercises to each other, so I don't think it makes sense to have stand-alone issues for them. We also don't use the difficulty levels (other than to manually adjust the ordering of tracks), so we could also decide to close that issue and open a new one when we think it would be useful to do the work.

[robphoenix]

Ok, that's fair enough. I think it makes sense to continue on with #345 for the moment, if only to be able to consider any re-ordering of exercises in the future.

I think also we should consider splitting #604 & #605 into separate issues, for the reasons stated by @kytrinyx, even though it does mean there may be a lot of long running issues for a while.

@tleen @ferhatelmas @leenipper where do you stand on the stale bot? As the strongest opinion so far is in opposition, I think we remove my commit adding the .github/stale.yml file unless you all feel otherwise.

[petertseng]

splitting #604 & #605 into separate issues

Y'all are right about the difficulty of keeping checklists up to date. We've seen it done both ways before in this repo too!

For #275 we split up into many different issues. I think this was when Blazon was new and I was very much still in that mindset.

In #414 and in #470 we had a checklist. Of these two, I was managing the checklist in #414 and not #470, but #414 is much smaller so I don't have much experience with that. I can say that when #414 was filed there were only eight exercises affected and I didn't want to set up filing the per-exercise issue for just eight of them. The recommendation is to use https://github.com/github/hub , I simply didn't have that set up (I used a modified version of Blazon for #275 ).

Which makes contributors' lives easier (Don't think only of ourselves)? It can be that the individual issues seem more tractable (harder for someone to know they need only take one item in a checklist). The disadvantage is if there's a chance of contributors seeing "Oh this project has 60 open issues they're obviously not very active" but perhaps that is not serious.

If agree on splitting them up, feel free to repurpose the #604 and #605 as the progress issue that the per-exercise issues link back to. I can't get to splitting them up until at least this weekend if not longer, so feel free to beat me to it. I'll say something if I start so that we don't duplicate work.

strongest opinion so far

I'll give the caveat that it's still only a 6/10 in strength, since I wouldn't know firsthand any benefits or detriments of using such a bot. So other voices are going to be really helpful here.

[tleen]

I'm officially neutral on the stale bot. Have not been around long enough to see how long-term issues are resolved so unsure how it would impact workload. I do think the workload-on-maintainers measurement is the proper metric. We can always reverse the decision now that we know the bot is a possibility if we end up wanting to go the other way.

For the big-issue vs many small issues its a rather unique thing to Exercism because of the track exercise model. Some issues impact them all. We did #470 as a monolithic exercise and it was not too troublesome. But looking back I think multiple small issues can encourage "Good first patch" contributions and that alone may be enough reason to go that way.

[kytrinyx]

Which makes contributors' lives easier (Don't think only of ourselves)? It can be that the individual issues seem more tractable (harder for someone to know they need only take one item in a checklist). The disadvantage is if there's a chance of contributors seeing "Oh this project has 60 open issues they're obviously not very active" but perhaps that is not serious.

That's a good point.

I wonder if we could have labels (and a bit of documentation in CONTRIBUTING.md, maybe) that help signal the intent of these issues: "good first patch" is a good one, perhaps though, we could label everything related to one single thing with a label for that series, so that it's clear that it's related.

[leenipper]

adding "good first patch" to the list of labels that will not be closed

I like this suggestion, if we do allow the bot to run, perhaps in a customized .github/stale.yml.
Consider using "daysUntilStale: 90" instead of 60 ?

[robphoenix]

we could label everything related to one single thing with a label for that series, so that it's clear that it's related.

This is a great idea.

If agree on splitting them up, feel free to repurpose the #604 and #605 as the progress issue that the per-exercise issues link back to. I can't get to splitting them up until at least this weekend if not longer, so feel free to beat me to it. I'll say something if I start so that we don't duplicate work.

I'm going to split up #604 today. And move the discussion regarding this to #605 with a note of these intentions, so as not to confuse this original issue.

Regarding stale bot, I agree with @kytrinyx comment here exercism/problem-specifications#746 (comment) and we remove it until such time as the repo's maintainership requires it. Apologies for jumping the gun a bit quickly and adding it before the discussion had properly played out.