Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

A proposal for bug and issue handling, and roadmap management #902

Merged
merged 20 commits into from
Oct 22, 2015
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
171 changes: 171 additions & 0 deletions project/ISSUES.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,171 @@
Galaxy Issue Management
=======================

The purpose of this document is to formalize how we manage the full cycle
(reporting, tracking progress, resolution) of both feature/enhancement ideas
and bugs, as well as providing a few general guidelines for longer term
planning for various Galaxy related projects. Significant inspiration taken
from the way the [Docker](https://github.com/docker/docker) project manages
issues.

We've tried several different approaches over the course of the project and two
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The following three paragraphs are not necessary in this document. We stated what, and we are about to say how. I do not see a need to mention history or Trello UI.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. They are helpful in framing this discussion, but should be removed for the final document.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, I definitely started to editorialize a bit in here, fully expecting it to be cleaned up.

common problems we've tried to address in this approach are:

* The issue graveyard -- where, once off the first 'page' of issues or below
the 'fold' in Trello, things sometimes do not resurface and are lost to
history.

* Difficulty clearly presenting and maintaining a high level project roadmap
and associated meta-issues.


Milestones
==========

Every pull request should, prior to merge, be assigned to the milestone
corresponding to the Galaxy release it will first appear in (15.10, 16.01, and
so on). This, along with the tags applied, will be used to generate the
release notes.

Any non-PR issue assigned to a milestone *must* be resolved or reassigned prior
to publishing that release -- that is, this is the new way to create and track
"Issues Blocking Release". This will be a primary way to report bugs and force
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume that once merged this document becomes active. Thus I would use 'This is the primary way to force bugs reconciliation' rather than 'This will be.'
Also this is not a "way to report bugs".

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, good catch. This was more editorialization on my part that I've formalized as it would be once active.

them to be reconciled (fixed, closed, or intentionally postponed) prior to
release. In practice, bugs should almost always be tagged with a milestone
which forces the reconciliation date. Issues *may* be, but they don't
necessarily have to be -- this is subjective and it depends on if the submitter
(or any other contributor) wants to force the issue to be revisited at the next
release.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This creates a lot of drama and doesn't really set about how to resolve any of it. Lets adapt a non-fictitious example from Trello where Developer A is assigned some issue and Developer B decides it is an "Issue Blocking the Release" but doesn't want to work on it his or her self. Can Developer A just push it back a release, and can Developer B then re-push it forward? And what if it has no assignee, can any developer just delay a release indefinitely this way?

I'm not sure what the solution is, but my preference it is probably to be conservative and explicitly just say that any committer can push back the milestone on an issue and no other committer is allowed to move it forward again except through issuing a pull request that resolves the issue in the older release.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jmchilton Hrmm, didn't really anticipate this as a drama-causing rule at all. In the context of non-bug issues I was mainly thinking 'Ok, this is a TODO that someone (any contributor) can add to indicate that we don't want to forget to at least think about (enough to fix, close, or punt) before the next release'.


Effective use of milestones should prevent bugs from falling through the
cracks, and will provide a mechanism for forcing the revisitation (and thus
progress or even potential discard) of ideas for enhancements or features.


Labeling Structure
==================

To allow for easy search, filtering, and general issue management every issue
or PR (not tagged `Procedures` or `Planning`) is *required* to have three
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.lower()

labels which indicate the type, status, and focus area of the issue. Any issue
without these three tags will be automatically have a `triage` label applied
indicating that it needs human intervention to be correctly tagged. These
`triage` tagged issues will be regularly reviewed and tagged as appropriate.

Type Labels
-----------

The 'class' label set is used for classifying the type of contribution or
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I really like "kind" better than class here.

request/report to separate enhancements and new features from bugs, etc.

* `class/bug` - something is broken, and it needs fixing
* `class/documentation` - documentation is unclear or can be improved
* `class/enhancement` - polish to an existing feature or interface
* `class/feature` - something brand new

Status Labels
-------------

The `status` of an issue should be tracked using the following stages:

* `status/triage` - brand new issue/pr that doesn't offer a concrete plan or
solution
* `status/planning` - issue reviewed and has a sufficiently detailed primary
message and/or commentary
* `status/WIP` - this issue or PR is currently being worked on. It should not
be merged (or closed) without pinging the owner/submitter.
* `status/review` - issue is resolved or PR is complete and needs review
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here "issue is resolved" is not clear to me, do you mean that the issue has been addressed with a merged PR and we ask the submitter to check if the issue is resolved for him?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, my intent was that it indicated the issue needed some extra verification prior to close. That might not be a useful indicator for issues, though, and I'd expect many issues to go directly from WIP to closed (especially when using the GitHub auto-close like "resolves #ISSUENUMBER".


Note that there are no `status/complete`, `status/wontfix`, `status/duplicate`,
or other terminal status indicators. This is intentional to keep the tail end
of bookkeeping from getting onerous. These sorts of terminal states and their
justifications should be indicated in the closing comment by the issue closer.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would say a little more here. Basically, if you are saying something is wontfix you need to say why. For duplicate, you must reference the other issue.


Focus Labels
------------

The 'focus' label is used for tagging issues and pull requests to a particular
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

...and I really prefer "area" here.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done and done. I'd changed these last night thinking that classification and focus sounded better, but the more I think about it I definitely like 'area' better than focus. class vs kind is less clear, but I don't feel that strongly about it.

focus area. This allows for easy searching within that particular domain, as
well as more organized release notes. Some examples, not-exhaustive, are here:

* `focus/API`
* `focus/cleanup`
* `focus/jobs`
* `focus/tests`
* `focus/GIEs`
* `focus/toolshed`
* `focus/UI-UX`
* `focus/workflows`

This list will definitely grow over time.

Other Useful Labels
-------------------

While the three labels sets indicating status, focus, and type are required
there are several other labels that are be useful and/or have special purpose.

* `Procedures` is a special tag that indicates that the issue is related to
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lowercase these?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I went with what we currently use as a label, but since this is an overhaul we should standardize now.

project governance, and it overrides the need for the trio of
class/status/focus tags, and these are never auto-flagged for triage.

* `Planning` is also a special tag that indicates the issue is related to
larger-scale issue planning. These issues are typically meta-issues
containing checklists and references to other issues which are subcomponents
and stepping stones necessary for issue resolution. These *can* utilize the
`focus/*` tags but are not required to. Status and type make little sense
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

focus -> area/* I think you mean

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, missed one when I swapped them back, good catch.

here.

* `Roadmap` is a reserved tag for the primary project roadmap. This is a
meta-issue that is not expected to be completed, but rather serves as an
entrypoint to the high level development of the project.

* `beginner-friendly` can be used to indicate a nice entry-level issue that
only requires limited understanding of the larger Galaxy framework and
ecosystem. This is useful for encouraging new contributors.


The Roadmap
===========

We will maintain a single `Roadmap` tagged meta-issue which will describe (at a
very high level) the *current* major areas of focus for the project. This is
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would amend this to say NIH grant funded team members of the project or maybe the committers group. Obviously the projects grows in completely unexpected directions and the focus of the project as a whole is very much affected by non-committers and the priorities they pursue.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure NIH funded or not is the right distinction. However this is one of the most challenging things to figure out. Is this just for the Taylor and Nekrutenko labs? Is it for the committers group?

Do we need to solve this now or wait until there are other organizations with more than a few people contributing regularly?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can say it is just a couple people, but we have people contributing to the project from the "outside" that are forces of nature. I can think of a few that have managed to add whole new dimensions to the project and really refine and set priorities for the project as a whole. I would say roadmap should be for the committers group or shouldn't exist. If the Nekrutenko and Taylor labs want to maintain their own independent priorities (not something I am poo-pooing at all, I like to eat and you guys feed people and are responsible for feeding people so you need to have the freedom to set priorities and ensure things get done) then it should be in Trello or another repo or somewhere else.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jmchilton My goal is to find a process that works for all contributors. I was simply posing the question. I strongly agree it should be for the committers group at this stage, and would add that members of that group should be clear about their own motivations and priorities. My point regarding organizations has nothing to do with amount of contribution as much has process. As more organizations with more people get involved, how do we have a unified roadmap while also capturing organizational priorities? I'm not diminishing anyones contributions, I'm trying to be flexible in how different contributors contribute to the overall roadmap. (Is this a problem anyone has solved well?)

similar to our PRIORITIES 2014/15 cards on Trello. Using [Task
Lists](https://github.com/blog/1375-task-lists-in-gfm-issues-pulls-comments),
this issue will link to sub-issues which will go into much more detail, might
have its own checklists to even more subcomponent cards, and so on.

This `Roadmap` issue will be assigned to every release milestone, forcing
periodic review of the roadmap.

To prevent the roadmap from being tied completely to github, and to facilitate
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This I'm less sure about because I don't want updating roadmap to be an incremental change. We should go through the exercise of remaking roadmap every release, forcing us to think about each task and decide whether it is worth migrating forward.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Basically what I had in mind was every time the roadmap was updated (the live GitHub issue, which we will handle exactly like you're suggesting) we copy/paste that into roadmap.md to have a historical record of it regardless of where the codebase and/or issue tracking ends up moving next. Maybe this isn't worth the effort though.

portable change tracking over time, we will also maintain the file
project/ROADMAP.md within the repository. Whenever the ROADMAP issue text is
changed, ROADMAP.md should be updated correspondingly.

This document won't have the organizational integration that a live github
issue does, but this way we're be able to have a ROADMAP.md permanently
attached to the code regardless of what issue tracking or organizational
software we use in the future.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm -0.5 on maintaining this file in the repository - we won't maintain it at all, so in order to encourage us to maintain it at all we should make it as easy as possible to edit. An issue is less onerous to edit given our project procedures.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll go further with "-1 but you can try to change my mind". All redundancy must die.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the advantage of having it in the repository would be that it is kept clean and trimmed. If it's an issue, either we're all editing the main issue text (which GH doesn't track yet), or we're commenting on a mile long thread of which only a portion may be relevant to the current roadmap.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jmchilton I wasn't expecting any actual 'maintenance' of the file via the repository, just keeping it as a historical record -- all edits will be in the issue itself, which is (maybe even automatically, instead of copy/paste?) replicated in ROADMAP.md to preserve the roadmap. It's just markdown, whether in the issue or file.

If people don't care about having the roadmap in the repository and want the only authoritative copy (and history of it) on GitHub, then this can definitely go.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@erasche What about commenting urself with what you updated in the main issue text? Or having P4 post diffs of main text periodically.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jxtx ah, wasn't clear that the "overall project roadmap" would be tossed out every few months. That's a very different beast. In that case, then yes, I'm fine with a mutable GH issue for a roadmap, maybe automatically mirrored into docs.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jxtx Let me merge two threads here - I feel like the problem with redoing the roadmap every few months is that we never seem to cut it down. I feel like every time we tried to adjust the one on Trello we would just add new stuff to it (planemo, IEs, etc...) and it never got smaller.

Also to your comment about organizations, the organizational structure for better or worse for this repository is an individual focused one and not an organizational focused one - which I think was an interesting choice and very different than what most projects do.

So what if we leverage the fact we are different and use it to ensure the roadmap stays trim. We have a roadmap issue and every committer gets one line to discuss what they intend to work on for the next three months. We meet as a committer group once every three months to discuss and update. Obviously, Anton is going to have a tremendous amount of say in what I and Martin work on for instance, and James will have a tremendous amount of say in what Dannon and Carl work on but ultimately that is reflected through the individual - not through an organization.

I think this solves both problems nicely and leverages the unique structure of our project.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like the problem with redoing the roadmap every few months is that we never seem to cut it down. I feel like every time we tried to adjust the one on Trello we would just add new stuff to it (planemo, IEs, etc...) and it never got smaller.

This is the motivation beyond redoing the roadmap and not updating it. There has to be an accompanying social piece here in which we actually consider each item on the roadmap and figure our whether it is still alive and where it should go. Roadmap is a bit more than just priorities, we should also put timelines on things (even if the granualarity, is "next release", "this year", "soon", and "the distant future"). Will this work? I don't know. Is it worth trying? I think so.

individual focused one and not an organizational focused one

Is this a feature or a bug?

So what if we leverage the fact we are different and use it to ensure the roadmap stays trim. We have a roadmap issue and every committer gets one line to discuss what they intend to work on for the next three months.

I think what you are proposing just reinforces what we currently have. I think the goal here is to push for a little bit of change. We routinely hear complaints that we lack a transparent high-level roadmap, and that people outside (and sometimes inside) the team aren't clear on the big picture priorities.

Anton is going to have a tremendous amount of say in what I and Martin work on for instance, and James will have a tremendous amount of say in what Dannon and Carl work on but ultimately that is reflected through the individual

Is this really where we want to be? Even now, @nekrut and @jxtx make prioritization decisions together, with input from everyone on their teams. Here we have the potential to open this up, and make it more transparent and inclusive.

Sure, we can just have a list of what each person thinks they are working on, but that loses all of the planning behind things. And it is inherently isolating.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jxtx I don't disagree with anything you said per se. This team has too many optimists and too much ambition - its reach exceeds its grasp - so I don't feel like a list of priorities without an assignment of resources to them is useful - even if that is what people request. Assigning people seems like it would make things much more realistic at the release level.

Also, it is isolating but we are trying to do 45 things with 15 people, these tasks will be isolated for the most part and the practice reflects the theory on that I think.

I'll drop the idea, it is... odd for sure - I just thought it meshed well with the reality of how things will proceed.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

too many optimists and too much ambition - its reach exceeds its grasp

As you know, I don't consider these to be bad things ;)

Assigning people seems like it would make things much more realistic at the release level.

I totally agree that people should be assigned to anything on the roadmap with a concrete timeframe (though I don't want to lost sight of "the distant future"). I just want to get there through a group discussion.

I agree we are pushed toward isolation in practice based on the nature of the work, but I'd prefer to actively push back rather than just accept it.



Voting
======

Users can vote for issues by commenting with a +1. It's possible to sort the
issue list by 'most commented' which would be a good indicator of what issues
are 'hot', though this doesn't necessarily indicate a high vote. It's possible
that that this is good enough good enough and in some ways potentially more
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good enough good enough -> good enough

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, thanks.

useful to find 'hot' issues than a flat vote count.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could automate a hotness style tag, more than 5 👍s and it is warm, more than 10 👍s and it is steamy, more than 15 and it is hot. Give the way the PR reviewer works this seems like it would be easy to implement and it would be a common thing to filter on, at least as common as area of focus in my mind.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, I could definitely add this feature. Would be nice to raising priorities.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh my... I actually like this. But please not "hotness".

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could do reddit style and have a hotness/warm hotness/steamy hotness/controversial, where the last relates to long discussions with less respect to number of 👍s

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

100% do not do hotness - that was a joke. Maybe popular/5, popular/10, popular/15, popular/controversial.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

... in retrospect that should've been obvious.


Automation
==========

For now, we will rely on a few simple automation rules:

* All issues, unless tagged `Procedures` or `Planning` will automatically be
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.lower()

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, thanks!

tagged `triage`, indicating that they require attention.

* All PRs that are not assigned to a milestone will be tagged `triage` to
indicate that they require attention prior to merge.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will add to p4. Update: galaxyproject/p4#2

5 changes: 5 additions & 0 deletions project/ROADMAP.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
Galaxy Project Roadmap
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The contents of this file I would rather see as a combination of GH Issues and Milestones than an extra file we need to keep updating.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@martenson Not sure what you mean, can you say a little more about how it'd look? As mentioned in the other discussion, I mainly just wanted it as a permanent historical record of the primary roadmap that was portable outside of github.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So this file would disappear and its contents would be recorded similar to nodejs/node#2522 (using issues and milestones hyperlinking). This is similar to what @jmchilton and @jxtx mentioned above, just more poorly worded.

Also see another node example of meta-planning: nodejs/node#3000

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@martenson Now I'm really confused. Yes, what I was proposing was that the contents of this file mirror exactly what is in the github roadmap issue. It's just a stub for now.


This is a document that outlines the Galaxy Roadmap. It should cover, at a
high level, current development interests and plans. Including links to active
GitHub meta-issues is highly encouraged.