Home Page latest issues empty because nothing in the top of issues list

[karlcow]

This morning I noticed:

Ah? Let's see.
https://webcompat.com/issues?page=1&per_page=50&state=open&stage=new&sort=created&direction=desc

ok there are still new issues. Are we filtering on time?

When requesting the home page we call

http GET https://webcompat.com/api/issues/category/new

Through

var diagnose = diagnose || {
},
issues = issues || {
};
diagnose.NewCollection = Backbone.Collection.extend({
  model: issues.Issue,
  url: '/api/issues/category/new'
}),
diagnose.NewView = Backbone.View.extend({
  el: $('#js-lastIssue'),
  initialize: function () {
    var self = this,
    headersBag = {
      headers: {
        Accept: 'application/json'
      }
    };
    this.issues = new diagnose.NewCollection,
    this.issues.fetch(headersBag).success(function () {
      self.render()
    }).error(function () {
    })
  },
  template: _.template($('#new-tmpl').html()),
  render: function () {
    return this.$el.html(this.template({
      newIssues: this.issues.toJSON().slice(0, 10)
    })),
    this
  }
}),
$(function () {
  new diagnose.NewView
});

The request indeed returns nothing:

HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 2
Content-Type: application/json
Date: Tue, 22 Mar 2016 02:44:19 GMT
Server: nginx/1.1.19
Strict-Transport-Security: max-age=31536000
cache-control: private, max-age=60, s-maxage=60
etag: W/"111688516d5d49d35a332964daa85bf8"
link: </api/issues?page=2>; rel="next", </api/issues?page=15>; rel="last"

[]

OK let's go to the python request:
https://github.com/webcompat/webcompat.com/blob/master/webcompat/api/endpoints.py#L98

@api.route('/issues/category/<issue_category>')

which

    elif issue_category == 'new':
        issues = api_request('get', issues_path, params=params)
        # api_request returns a tuple of format:
        #       (content, status_code, response_headers)
        # So we make a dict here for improved readability
        content, status_code, response_headers = issues
        if status_code != 304:
            content = filter_new(json.loads(content))
        return (content, status_code, response_headers)

Here we are basically making a generic request to the issues list instead of all the open new issues.

In our realm, the new issues have no status-* label and are open. Making a query on everything but not status-* labels is a bit expensive I guess. Maybe one solution is to automatically for every new bugs to add a label status-needstriage Then it becomes a lot easier to extract those. (Probably related to #886 on using Milestones for status.)

[miketaylr]

I do like the idea of "needstriage", which gets moved into "needsdiagnosis" or closed.

I know it's kind of far in the future, but maybe milestones might be a good mini-project for London (or a focused work-week thing after).

[miketaylr]

Hmm... but what changed here to break this? Was it GitHub, or was it us?

[miketaylr]

Yeah, I'm confused as to why this is returning empty.

webcompat/web-bugs#1949, for example, should be showing up because it doesn't have any status labels on it.

Is the problem that we're not requesting enough issues? Or rather, the latest issues have all been triaged, and don't qualify for filter_new?

[miketaylr]

Is the problem that we're not requesting enough issues? Or rather, the latest issues have all been triaged, and don't qualify for filter_new?

@karlcow is that what you meant by "nothing in the top of issues list"?

[karlcow]

The thing which is happening is that we request the list of all issues. which is sending paginated json. Once the new issues are old enough that there are more triaged issues than non-triaged, aka there are no non-triaged issues in the first JSON, the filter will return an empty json. It's normal but not very useful for the home page.

So my proposal to automatically tag the new bug with a flag.

There are three other possibilities:

• reuse the search for new issues https://webcompat.com/issues?page=1&per_page=50&state=open&stage=new&sort=created&direction=desc which I think is more expensive in CPU for GitHub, but maybe it doesn't matter.
• create a label needstriage used for filtering.
• follow the pagination until we find all new issues. Not effective in terms of HTTP requests

[miketaylr]

I'm fine with "create a label needstriage", use that label by default, and change the header "Latest Untriaged Bugs".

It's more of an implicit call to action, I think.

We just need to make sure that selecting any other status will unselect needstriage (it may be that way not I don't recall).

[miketaylr]

reuse the search for new issues https://webcompat.com/issues?page=1&per_page=50&state=open&stage=new&sort=created&direction=desc which I think is more expensive in CPU for GitHub, but maybe it doesn't matter.

This seems like a good "quick" fix while we move to your other suggestion. I'm not so worried about GitHub's CPU (we pay them a few bucks a month for our org membership, I think it will be OK).

[miketaylr]

⚡ 💡 Oh, I think I know what the change was (on GitHub's end).

See the comment here: https://github.com/webcompat/webcompat.com/blob/master/webcompat/api/endpoints.py#L127

Note that 'new' here is primarily used on the homepage.

For paginated results on the /issues page, see /issues/search/new.

That endpoint used to not be paginated (I think). I recall some other bugfix from GitHub where they fixed a pagination inconsistency. It doesn't matter for what you're suggesting @karlcow, this is mostly for historical context of why it changed.

[miketaylr]

reuse the search for new issues https://webcompat.com/issues?page=1&per_page=50&state=open&stage=new&sort=created&direction=desc which I think is more expensive in CPU for GitHub, but maybe it doesn't matter.

I'm not so worried about GitHub's CPU (we pay them a few bucks a month for our org membership, I think it will be OK).

Ah, it's not a CPU constraint. It's an API rate limit constraint. Clicking on "New" uses the Search API (probably because the Issues API wasn't paginated as state above). So if we stick that on the homepage we're gonna see 403s.

(You can even get that if you mash on the New button like 30 times in a minute when not logged in, which probably nobody but me will ever do:
)

[miketaylr]

So a few things from this bug report:

• change the "New" filter to use the Issues API, and not the Search API
• create a "needstriage" label (or milestone!)
• have label bot add that for all newly created issues
• investigate status mutual exclusivity (or, don't bother and just write the milestone code instead)
• update header on homepage to say "Latest Untriaged Issues"

[deepthivenkat]

A new label "status-needstriage" has been added.

Currently, when a bug is reported from github without any tag , we see a "New Issue" label for that bug in webcompat.

Then, when a bug is reported from github with a "needstriage" tag, we see a "Needs Triage" label for that bug in webcompat. Here the "New Issue" label doesn't show any more in the webcompat and only "Needs Triage" label will only be shown.

If we are going to automatically assign a tag "status-needstriage" for all the new bugs as soon as it is posted, then we will not be getting "New label" in the webcompat, for all the bugs reported here on.

Is this an expected behaviour?

Also in "All issues" tab, if we click on "New Issues" milestone, all the new bugs will default have the tag "Needs Triage" label here on. So will there be no difference clicking on a "New Issues" label/milestone and "Needs Triage" label? (As we are automatically adding the new label "status-needstriage" for all the new bugs here on)

Or we have some difference for the "New Issues" and "Needs Triage"?

[deepthivenkat]

Remaining tasks

1. Automatically add a label "needstriage" to all the new issues
2. Automatically add a label "needstriage" for all the existing untriaged issues
3. Will raise a separate issue and handle the failing functional tests.

[miketaylr]

If we are going to automatically assign a tag "status-needstriage" for all the new bugs as soon as it is posted, then we will not be getting "New label" in the webcompat, for all the bugs reported here on.
Is this an expected behaviour?

I think so. I think we want all issues to start with "status-needstriage" which indicates someone needs to look at it, try to reproduce, dupe it, close it, move to "status-needsdiagnosis", etc. So really the notion of "New" is going away.

Also in "All issues" tab, if we click on "New Issues" milestone, all the new bugs will default have the tag "Needs Triage" label here on.

I think we should rename "New Issues" here to "Needs Triage". Then we're consistent.

@karlcow do you agree?

[miketaylr]

Automatically add a label "needstriage" for all the existing untriaged issues

We usually don't have too many of these, so it might be better done manually (and might be way simpler).

[miketaylr]

Automatically add a label "needstriage" to all the new issues

That could happen here: https://github.com/webcompat/webcompat.com/blob/master/webcompat/webhooks/helpers.py#L48. Either hard-coding payload to look something like payload = [label, 'status-needstriage'], or refactoring things a bit.

[miketaylr]

• Make /new API endpoints return 301 to needstriage category endpoints

(see 2314850#commitcomment-17799390)