-
Notifications
You must be signed in to change notification settings - Fork 1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
A proposal for bug and issue handling, and roadmap management #902
Changes from 2 commits
951a4e1
ecde79b
021e100
578263c
de4f4b2
a3eca0f
11b450e
b726684
b71de25
30d7d2f
efb0661
c38da9d
0fde68a
86d0db5
fe984c5
f7a97e3
05c5ecd
a4e69f4
614ec9d
0218391
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,171 @@ | ||
Galaxy Issue Management | ||
======================= | ||
|
||
The purpose of this document is to formalize how we manage the full cycle | ||
(reporting, tracking progress, resolution) of both feature/enhancement ideas | ||
and bugs, as well as providing a few general guidelines for longer term | ||
planning for various Galaxy related projects. Significant inspiration taken | ||
from the way the [Docker](https://github.com/docker/docker) project manages | ||
issues. | ||
|
||
We've tried several different approaches over the course of the project and two | ||
common problems we've tried to address in this approach are: | ||
|
||
* The issue graveyard -- where, once off the first 'page' of issues or below | ||
the 'fold' in Trello, things sometimes do not resurface and are lost to | ||
history. | ||
|
||
* Difficulty clearly presenting and maintaining a high level project roadmap | ||
and associated meta-issues. | ||
|
||
|
||
Milestones | ||
========== | ||
|
||
Every pull request should, prior to merge, be assigned to the milestone | ||
corresponding to the Galaxy release it will first appear in (15.10, 16.01, and | ||
so on). This, along with the tags applied, will be used to generate the | ||
release notes. | ||
|
||
Any non-PR issue assigned to a milestone *must* be resolved or reassigned prior | ||
to publishing that release -- that is, this is the new way to create and track | ||
"Issues Blocking Release". This will be a primary way to report bugs and force | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I assume that once merged this document becomes active. Thus I would use 'This is the primary way to force bugs reconciliation' rather than 'This will be.' There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yep, good catch. This was more editorialization on my part that I've formalized as it would be once active. |
||
them to be reconciled (fixed, closed, or intentionally postponed) prior to | ||
release. In practice, bugs should almost always be tagged with a milestone | ||
which forces the reconciliation date. Issues *may* be, but they don't | ||
necessarily have to be -- this is subjective and it depends on if the submitter | ||
(or any other contributor) wants to force the issue to be revisited at the next | ||
release. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This creates a lot of drama and doesn't really set about how to resolve any of it. Lets adapt a non-fictitious example from Trello where Developer A is assigned some issue and Developer B decides it is an "Issue Blocking the Release" but doesn't want to work on it his or her self. Can Developer A just push it back a release, and can Developer B then re-push it forward? And what if it has no assignee, can any developer just delay a release indefinitely this way? I'm not sure what the solution is, but my preference it is probably to be conservative and explicitly just say that any committer can push back the milestone on an issue and no other committer is allowed to move it forward again except through issuing a pull request that resolves the issue in the older release. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @jmchilton Hrmm, didn't really anticipate this as a drama-causing rule at all. In the context of non-bug issues I was mainly thinking 'Ok, this is a TODO that someone (any contributor) can add to indicate that we don't want to forget to at least think about (enough to fix, close, or punt) before the next release'. |
||
|
||
Effective use of milestones should prevent bugs from falling through the | ||
cracks, and will provide a mechanism for forcing the revisitation (and thus | ||
progress or even potential discard) of ideas for enhancements or features. | ||
|
||
|
||
Labeling Structure | ||
================== | ||
|
||
To allow for easy search, filtering, and general issue management every issue | ||
or PR (not tagged `Procedures` or `Planning`) is *required* to have three | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. .lower() |
||
labels which indicate the type, status, and focus area of the issue. Any issue | ||
without these three tags will be automatically have a `triage` label applied | ||
indicating that it needs human intervention to be correctly tagged. These | ||
`triage` tagged issues will be regularly reviewed and tagged as appropriate. | ||
|
||
Type Labels | ||
----------- | ||
|
||
The 'class' label set is used for classifying the type of contribution or | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I really like "kind" better than class here. |
||
request/report to separate enhancements and new features from bugs, etc. | ||
|
||
* `class/bug` - something is broken, and it needs fixing | ||
* `class/documentation` - documentation is unclear or can be improved | ||
* `class/enhancement` - polish to an existing feature or interface | ||
* `class/feature` - something brand new | ||
|
||
Status Labels | ||
------------- | ||
|
||
The `status` of an issue should be tracked using the following stages: | ||
|
||
* `status/triage` - brand new issue/pr that doesn't offer a concrete plan or | ||
solution | ||
* `status/planning` - issue reviewed and has a sufficiently detailed primary | ||
message and/or commentary | ||
* `status/WIP` - this issue or PR is currently being worked on. It should not | ||
be merged (or closed) without pinging the owner/submitter. | ||
* `status/review` - issue is resolved or PR is complete and needs review | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Here "issue is resolved" is not clear to me, do you mean that the issue has been addressed with a merged PR and we ask the submitter to check if the issue is resolved for him? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yeah, my intent was that it indicated the issue needed some extra verification prior to close. That might not be a useful indicator for issues, though, and I'd expect many issues to go directly from WIP to closed (especially when using the GitHub auto-close like "resolves #ISSUENUMBER". |
||
|
||
Note that there are no `status/complete`, `status/wontfix`, `status/duplicate`, | ||
or other terminal status indicators. This is intentional to keep the tail end | ||
of bookkeeping from getting onerous. These sorts of terminal states and their | ||
justifications should be indicated in the closing comment by the issue closer. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I would say a little more here. Basically, if you are saying something is wontfix you need to say why. For duplicate, you must reference the other issue. |
||
|
||
Focus Labels | ||
------------ | ||
|
||
The 'focus' label is used for tagging issues and pull requests to a particular | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. ...and I really prefer "area" here. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Done and done. I'd changed these last night thinking that classification and focus sounded better, but the more I think about it I definitely like 'area' better than focus. class vs kind is less clear, but I don't feel that strongly about it. |
||
focus area. This allows for easy searching within that particular domain, as | ||
well as more organized release notes. Some examples, not-exhaustive, are here: | ||
|
||
* `focus/API` | ||
* `focus/cleanup` | ||
* `focus/jobs` | ||
* `focus/tests` | ||
* `focus/GIEs` | ||
* `focus/toolshed` | ||
* `focus/UI-UX` | ||
* `focus/workflows` | ||
|
||
This list will definitely grow over time. | ||
|
||
Other Useful Labels | ||
------------------- | ||
|
||
While the three labels sets indicating status, focus, and type are required | ||
there are several other labels that are be useful and/or have special purpose. | ||
|
||
* `Procedures` is a special tag that indicates that the issue is related to | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Lowercase these? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yeah, I went with what we currently use as a label, but since this is an overhaul we should standardize now. |
||
project governance, and it overrides the need for the trio of | ||
class/status/focus tags, and these are never auto-flagged for triage. | ||
|
||
* `Planning` is also a special tag that indicates the issue is related to | ||
larger-scale issue planning. These issues are typically meta-issues | ||
containing checklists and references to other issues which are subcomponents | ||
and stepping stones necessary for issue resolution. These *can* utilize the | ||
`focus/*` tags but are not required to. Status and type make little sense | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. focus -> There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yep, missed one when I swapped them back, good catch. |
||
here. | ||
|
||
* `Roadmap` is a reserved tag for the primary project roadmap. This is a | ||
meta-issue that is not expected to be completed, but rather serves as an | ||
entrypoint to the high level development of the project. | ||
|
||
* `beginner-friendly` can be used to indicate a nice entry-level issue that | ||
only requires limited understanding of the larger Galaxy framework and | ||
ecosystem. This is useful for encouraging new contributors. | ||
|
||
|
||
The Roadmap | ||
=========== | ||
|
||
We will maintain a single `Roadmap` tagged meta-issue which will describe (at a | ||
very high level) the *current* major areas of focus for the project. This is | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I would amend this to say NIH grant funded team members of the project or maybe the committers group. Obviously the projects grows in completely unexpected directions and the focus of the project as a whole is very much affected by non-committers and the priorities they pursue. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm not sure NIH funded or not is the right distinction. However this is one of the most challenging things to figure out. Is this just for the Taylor and Nekrutenko labs? Is it for the committers group? Do we need to solve this now or wait until there are other organizations with more than a few people contributing regularly? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. You can say it is just a couple people, but we have people contributing to the project from the "outside" that are forces of nature. I can think of a few that have managed to add whole new dimensions to the project and really refine and set priorities for the project as a whole. I would say roadmap should be for the committers group or shouldn't exist. If the Nekrutenko and Taylor labs want to maintain their own independent priorities (not something I am poo-pooing at all, I like to eat and you guys feed people and are responsible for feeding people so you need to have the freedom to set priorities and ensure things get done) then it should be in Trello or another repo or somewhere else. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @jmchilton My goal is to find a process that works for all contributors. I was simply posing the question. I strongly agree it should be for the committers group at this stage, and would add that members of that group should be clear about their own motivations and priorities. My point regarding organizations has nothing to do with amount of contribution as much has process. As more organizations with more people get involved, how do we have a unified roadmap while also capturing organizational priorities? I'm not diminishing anyones contributions, I'm trying to be flexible in how different contributors contribute to the overall roadmap. (Is this a problem anyone has solved well?) |
||
similar to our PRIORITIES 2014/15 cards on Trello. Using [Task | ||
Lists](https://github.com/blog/1375-task-lists-in-gfm-issues-pulls-comments), | ||
this issue will link to sub-issues which will go into much more detail, might | ||
have its own checklists to even more subcomponent cards, and so on. | ||
|
||
This `Roadmap` issue will be assigned to every release milestone, forcing | ||
periodic review of the roadmap. | ||
|
||
To prevent the roadmap from being tied completely to github, and to facilitate | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This I'm less sure about because I don't want updating roadmap to be an incremental change. We should go through the exercise of remaking roadmap every release, forcing us to think about each task and decide whether it is worth migrating forward. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Basically what I had in mind was every time the roadmap was updated (the live GitHub issue, which we will handle exactly like you're suggesting) we copy/paste that into roadmap.md to have a historical record of it regardless of where the codebase and/or issue tracking ends up moving next. Maybe this isn't worth the effort though. |
||
portable change tracking over time, we will also maintain the file | ||
project/ROADMAP.md within the repository. Whenever the ROADMAP issue text is | ||
changed, ROADMAP.md should be updated correspondingly. | ||
|
||
This document won't have the organizational integration that a live github | ||
issue does, but this way we're be able to have a ROADMAP.md permanently | ||
attached to the code regardless of what issue tracking or organizational | ||
software we use in the future. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm -0.5 on maintaining this file in the repository - we won't maintain it at all, so in order to encourage us to maintain it at all we should make it as easy as possible to edit. An issue is less onerous to edit given our project procedures. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'll go further with "-1 but you can try to change my mind". All redundancy must die. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think the advantage of having it in the repository would be that it is kept clean and trimmed. If it's an issue, either we're all editing the main issue text (which GH doesn't track yet), or we're commenting on a mile long thread of which only a portion may be relevant to the current roadmap. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @jmchilton I wasn't expecting any actual 'maintenance' of the file via the repository, just keeping it as a historical record -- all edits will be in the issue itself, which is (maybe even automatically, instead of copy/paste?) replicated in ROADMAP.md to preserve the roadmap. It's just markdown, whether in the issue or file. If people don't care about having the roadmap in the repository and want the only authoritative copy (and history of it) on GitHub, then this can definitely go. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @erasche What about commenting urself with what you updated in the main issue text? Or having P4 post diffs of main text periodically. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @jxtx ah, wasn't clear that the "overall project roadmap" would be tossed out every few months. That's a very different beast. In that case, then yes, I'm fine with a mutable GH issue for a roadmap, maybe automatically mirrored into docs. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @jxtx Let me merge two threads here - I feel like the problem with redoing the roadmap every few months is that we never seem to cut it down. I feel like every time we tried to adjust the one on Trello we would just add new stuff to it (planemo, IEs, etc...) and it never got smaller. Also to your comment about organizations, the organizational structure for better or worse for this repository is an individual focused one and not an organizational focused one - which I think was an interesting choice and very different than what most projects do. So what if we leverage the fact we are different and use it to ensure the roadmap stays trim. We have a roadmap issue and every committer gets one line to discuss what they intend to work on for the next three months. We meet as a committer group once every three months to discuss and update. Obviously, Anton is going to have a tremendous amount of say in what I and Martin work on for instance, and James will have a tremendous amount of say in what Dannon and Carl work on but ultimately that is reflected through the individual - not through an organization. I think this solves both problems nicely and leverages the unique structure of our project. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
This is the motivation beyond redoing the roadmap and not updating it. There has to be an accompanying social piece here in which we actually consider each item on the roadmap and figure our whether it is still alive and where it should go. Roadmap is a bit more than just priorities, we should also put timelines on things (even if the granualarity, is "next release", "this year", "soon", and "the distant future"). Will this work? I don't know. Is it worth trying? I think so.
Is this a feature or a bug?
I think what you are proposing just reinforces what we currently have. I think the goal here is to push for a little bit of change. We routinely hear complaints that we lack a transparent high-level roadmap, and that people outside (and sometimes inside) the team aren't clear on the big picture priorities.
Is this really where we want to be? Even now, @nekrut and @jxtx make prioritization decisions together, with input from everyone on their teams. Here we have the potential to open this up, and make it more transparent and inclusive. Sure, we can just have a list of what each person thinks they are working on, but that loses all of the planning behind things. And it is inherently isolating. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @jxtx I don't disagree with anything you said per se. This team has too many optimists and too much ambition - its reach exceeds its grasp - so I don't feel like a list of priorities without an assignment of resources to them is useful - even if that is what people request. Assigning people seems like it would make things much more realistic at the release level. Also, it is isolating but we are trying to do 45 things with 15 people, these tasks will be isolated for the most part and the practice reflects the theory on that I think. I'll drop the idea, it is... odd for sure - I just thought it meshed well with the reality of how things will proceed. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
As you know, I don't consider these to be bad things ;)
I totally agree that people should be assigned to anything on the roadmap with a concrete timeframe (though I don't want to lost sight of "the distant future"). I just want to get there through a group discussion. I agree we are pushed toward isolation in practice based on the nature of the work, but I'd prefer to actively push back rather than just accept it. |
||
|
||
|
||
Voting | ||
====== | ||
|
||
Users can vote for issues by commenting with a +1. It's possible to sort the | ||
issue list by 'most commented' which would be a good indicator of what issues | ||
are 'hot', though this doesn't necessarily indicate a high vote. It's possible | ||
that that this is good enough good enough and in some ways potentially more | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Good catch, thanks. |
||
useful to find 'hot' issues than a flat vote count. | ||
|
||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We could automate a There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yep, I could definitely add this feature. Would be nice to raising priorities. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Oh my... I actually like this. But please not "hotness". There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We could do reddit style and have a There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 100% do not do hotness - that was a joke. Maybe There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. ... in retrospect that should've been obvious. |
||
|
||
Automation | ||
========== | ||
|
||
For now, we will rely on a few simple automation rules: | ||
|
||
* All issues, unless tagged `Procedures` or `Planning` will automatically be | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. .lower() There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Good catch, thanks! |
||
tagged `triage`, indicating that they require attention. | ||
|
||
* All PRs that are not assigned to a milestone will be tagged `triage` to | ||
indicate that they require attention prior to merge. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Will add to p4. Update: galaxyproject/p4#2 |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
Galaxy Project Roadmap | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The contents of this file I would rather see as a combination of GH Issues and Milestones than an extra file we need to keep updating. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @martenson Not sure what you mean, can you say a little more about how it'd look? As mentioned in the other discussion, I mainly just wanted it as a permanent historical record of the primary roadmap that was portable outside of github. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. So this file would disappear and its contents would be recorded similar to nodejs/node#2522 (using issues and milestones hyperlinking). This is similar to what @jmchilton and @jxtx mentioned above, just more poorly worded. Also see another node example of meta-planning: nodejs/node#3000 There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @martenson Now I'm really confused. Yes, what I was proposing was that the contents of this file mirror exactly what is in the github roadmap issue. It's just a stub for now. |
||
|
||
This is a document that outlines the Galaxy Roadmap. It should cover, at a | ||
high level, current development interests and plans. Including links to active | ||
GitHub meta-issues is highly encouraged. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The following three paragraphs are not necessary in this document. We stated what, and we are about to say how. I do not see a need to mention history or Trello UI.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1. They are helpful in framing this discussion, but should be removed for the final document.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yep, I definitely started to editorialize a bit in here, fully expecting it to be cleaned up.