Testing offline - Milestones initialization

[karlcow]

Steps to reproduce

1. Switch off your network connection
2. Activate the project environment
3. Enter nosetests.

Actual:

🐝 10:02:31 ~/code/webcompat.com
→ nosetests
Milestones are not initialized. Check logs.

This should not happen. The tests should be able to run independently of any milestones initialization.
We probably need at least for testing a milestones.json ready to be used so that we do not have to fetch them.

[karlcow]

In #2527 (comment)

@miketaylr said:

Hmm... @karlcow I guess a Travis IP will hit the rate limit. I wonder if we need to hard-code milestones.json for tests?

[karlcow]

I will work on this today, given the failures on Travis and the hurdles for people testing locally with/without network.

[karlcow]

Some context for myself.

The milestones need to be understood in the context of #1836

Just to repeat what was said there. When

• executing tests with nosetest
• running the app online (prod, staging)

run.py is never used.

What is used is webcompat/*. Each time in a module, we get a import webcompat, we kick the __init__.py which starts the milestones initialization. We probably do not want this or not in that way at least. So it would be probably good to kick it differently.

At least #2519 reduces the network access bit.

[laghee]

So after kicking Travis three times today to get tests to pass or trying (#2553 ), I've become obsessed ... and confused. For one thing, I'm pretty sure this:

When ... executing tests with nosetest ... run.py is never used.

...isn't true when running either on localhost or Travis. (Not sure if we're running nosetests anywhere else?)

As far as I can tell, to get notestests to run, we're always calling run.py or run.py -t either directly or by using run start or run start:test.

Couldn't we just stick a dictionary constant in run.py like the one we use to compare against in test_config? Since staging and production are using some mysterious fabfile, I assume they wouldn't be affected.

Going forward, it seems like it might create less confusion to have a more parallel process in dev/staging/prod that just pivots on environment variables or something (which I guess it probably how it works between staging and production?)...

...but for now at least we could stop Travis from crashing all the time ❓

[karlcow]

@laghee are you sure, sure and sure ;)

How can we demonstrate this?

For example, let's add in run.py

if __name__ == '__main__':
    print('We are starting run.py')

then let's type

nosetests -v --nocapture

I get

Statuses Initialization…
Milestones in memory
Pagination in comments Link response header. ... ok
API issue for a non existent number returns JSON 404. ... ok
API access to labels without auth returns JSON 200. ... ok
Patching the issue is working only with certain circumstances. ... ok
API with wrong parameter returns JSON 404. ... ok
API setting labels without auth returns JSON 403 error code. ... ok
API access to user activity without auth returns JSON 401. ... ok
API with wrong category returns JSON 404. ... ok
API with wrong route returns JSON 404. ... ok
Update statuses with real milestones id. ... A milestone is missing or has been added: [u'needstriage']
A milestone is missing or has been added: [u'bogus']
ok
Check we receive the right list of browsers. ... ok
Assess the filtering is correct. ... ok
Check if a list of labels contains the 'status-needsinfo'. ... ok
Check if an issue is older than a certain time gap. ... ok
Expected HTML is returned for a json object or a string. ... ok
The data body sent to GitHub API. ... ok
Check that domain name is extracted. ... ok
Assert we handle valid JSON and other values. ... ok
Checks we return the right form with the appropriate data. ... ok
HTML comments need the right values depending on the keys. ... ok
Test that we can handle a browser-foo inside of EXTRA_LABELS. ... ok
Assert domains validity in issue reporting. ... ok
Assert that we get the form parameters we want. ... ok
Check that metadata is processed and wrapped. ... ok
Avoid some type of strings. ... ok
Check that URL is normalized. ... ok
Check that appropriate radio button label is returned. ... ok
Test HTTP Links formating. ... ok
Test browser parsing via get_browser helper method. ... ok
Test browser name parsing via get_browser_name helper method. ... ok
Test name extraction from Dict via get_name helper method. ... ok
Test OS parsing via get_os helper method. ... ok
Test getting an expected string, including Python to js-style ... ok
Test version string composition from Dict ... ok
Test that API params are correctly converted to Search API. ... ok
normalize_api_params shouldn't transform unknown params. ... ok
Test HTTP Links parsing for GitHub only. ... ok
test_rewrite_and_sanitize_link (unit.test_helpers.TestHelpers) ... ok
Test we're correctly rewriting the passed in link. ... ok
Test that we're removing access_token parameters. ... ok
Check Cache-Control for issues. ... ok
Check ETAG for issues. ... ok
Checks if we receive a 304 Not Modified. ... ok
An addon link should be in the top navigation bar. ... ok
Page title format for different URIs. ... ok
Testing user activity page title. ... ok
Final changelog formatting. ... ok
Normalize the title. ... ok
Build a Query String. ... ok
Build a request. ... ok
Extract text from a node. ... ok
Test that Base64 screenshots return the expected status codes. ... ok
Test that /upload/ doesn't let you GET. ... ok
Test that uploaded files return the expected status code. ... ok
Test that /about exists. ... ok
Test that asks user to log in before displaying activity. ... ok
Test that /contact exists. ... ok
Test that /contributors exists. ... ok
Test that /contributors/build-tools exists. ... ok
Test that /contributors/diagnose-bug exists. ... ok
Test that /contributors/organize-webcompat-events exists. ... ok
Test that /contributors/report-bug exists. ... ok
Test that /contributors/reproduce-bug exists. ... ok
Test that /contributors/site-outreach exists. ... ok
Test that /contributors/web-platform-research exists. ... ok
Test POST to /csp-report w/ correct content-type returns 204. ... ok
Test POST w/ wrong content-type to /csp-report returns 400. ... ok
Request to /dashboard should be 404. ... ok
Request to /dashboard/triage should be 200. ... ok
Test that post is not working on /issues/new. ... ok
Test that the home page exists. ... ok
Test if issues are really integer. ... ok
Test that the /issues/<number> exists, and does not redirect. ... ok
Test that the /issues route gets 200 and does not redirect. ... ok
Sends 400 to POST on /issues/new with missing parameters. ... ok
Test that /issues/new exists. ... ok
/issues/new POST exit with 400 if missing parameters. ... ok
Test that /privacy exists. ... ok
Rate Limit URI sends 410 Gone. ... ok
Test that post is working on /issues/new. ... ok
Test that the /tools/cssfixme route gets 200. ... ok
Extract browser label name. ... ok
Extract 'extra' label. ... ok
Extract dictionary of metadata for an issue body. ... ok
Extract priority label. ... ok
POST without bogus signature on labeler webhook is forbidden. ... ok
POST with event not being 'issues' or 'ping' fails. ... ok
POST without signature on labeler webhook is forbidden. ... ok
POST with an unknown action fails. ... ok
GET is forbidden on labeler webhook. ... ok
Extract the right information from an issue. ... ok
Extract list of labels from an issue body. ... ok
Validation tests for GitHub Webhooks. ... ok
POST with PING events just return a 200 and contains pong. ... ok

----------------------------------------------------------------------
Ran 94 tests in 1.588s

OK

aka run.py didn't execute.

run.py -t is necessary only for functional tests (not unit tests)
and if we start run.py -t for nosetests in travis configuration, that's a bug! :)

As for

...but for now at least we could stop Travis from crashing all the time ❓

Yeah! that's the goal.
For Travis it is important to know when this is due to unittest or to functional tests. :)

[laghee]

@karlcow Sorry, my bad! Not sure if I wrote that wrong or had actually misunderstood the process. Sometimes things really do make more sense when you come back to them later! (#1 lesson I keep learning but not fully incorporating so far.)

At any rate, you're right, of course -- it's not that nosetests kicks run.py directly... it's that in Travis we always execute run.py -t before we run nosetests (or, rather, all the test scripts). (At least digging into all this, I learned a bunch of stuff about our configuration! (-; )

[karlcow]

Trying to resolve this

At any rate, you're right, of course -- it's not that nosetests kicks run.py directly... it's that in Travis we always execute run.py -t before we run nosetests (or, rather, all the test scripts). (At least digging into all this, I learned a bunch of stuff about our configuration! (-; )

in #2567

This current issue still needs to be solved though.

[karlcow]

@miketaylr hmm something is rotten…
https://travis-ci.org/webcompat/webcompat.com/builds/431489833

next week. Fix the milestone story.

The command "nosetests" exited with 1.
0.00s$ python run.py -t &
The command "python run.py -t &" exited with 0.
5.00s$ sleep 5
Writing logs to: /home/travis/build/webcompat/webcompat.com/tmp
Statuses Initialization…
Oooops.
We can't find /home/travis/build/webcompat/webcompat.com/data/milestones.json
Double check that everything is configured properly
in config/secrets.py and try again. Good luck!
Fetching milestones from Github…
It failed with 403 Client Error: Forbidden for url: https://api.github.com/repos/webcompat/webcompat-tests/milestones!
We will read from data/milestones.json.
Milestones are not initialized. Check logs.