Some of our tests are testing our backend code infrastructure and the others are testing the frontend logic of the application.
Our tests are written in Python. We assume you have already installed the development Python modules with
pip install -r config/requirements-dev.txtWe are using the pytest framework for running our Python unit tests. Use the following commands:
pip install -e .
pytestThis will display an output similar to this when all tests are passing.
======================== test session starts =========================
platform darwin -- Python 3.9.1, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
rootdir: /Users/karl/code/webcompat.com, configfile: pytest.ini, testpaths: tests/unit
plugins: mock-3.5.1
collected 190 items
tests/unit/test_api_helpers.py .. [ 1%]
tests/unit/test_api_urls.py ........... [ 6%]
tests/unit/test_config.py .. [ 7%]
tests/unit/test_console_logs.py ..... [ 10%]
tests/unit/test_form.py ..................... [ 21%]
tests/unit/test_helpers.py ............................ [ 36%]
tests/unit/test_http_caching.py ... [ 37%]
tests/unit/test_issues.py ......... [ 42%]
tests/unit/test_rendering.py ..... [ 45%]
tests/unit/test_template.py ............. [ 52%]
tests/unit/test_tools_archive.py ............s. [ 59%]
tests/unit/test_tools_changelog.py .. [ 60%]
tests/unit/test_topsites.py ... [ 62%]
tests/unit/test_uploads.py ... [ 63%]
tests/unit/test_urls.py ................................ [ 80%]
tests/unit/test_webhook.py .................... [ 91%]
tests/unit/test_webhook_model.py ................. [100%]
=================== 189 passed, 1 skipped in 1.50s ===================
You can also run them with following:
npm run test:python
Running functional tests is a bit more involved. You can also run both test suites at once (see the next section).
Tests are also run automatically on Circle CI for each commit. If you would like to skip running tests for a given commit, you can use use the magical [ci skip] string in your commit message. See the Circle CI docs for more info.
We use Intern to run JavaScript unit tests.
A series of unit tests have been created for the JavaScript code managing the frontend of our application.
You can run them by using the command.
npm run test:unit-js
The output will be something similar to:
> webcompat@ test:unit-js /Users/karl/code/webcompat.com
> intern config=./webcompat/static/js/lib/wizard/tests/intern.json
✓ node - notify - subscribe throws an error if no channel or callback provided (0.001s)
✓ node - notify - subscribe returns and object with unsubscribe property (0s)
✓ node - notify - publishing to a channel calls callback fn (0.001s)
✓ node - notify - publishing to a channel with no listeners doesn't call callback fn (0s)
✓ node - notify - publishing data to callback makes it to callback fn (0s)
✓ node - notify - publishing no data to callback results in an empty object being passed to callback fn (0.001s)
✓ node - notify - callback fn is not called after unsubscribe (0s)
✓ node - utils - extractPrettyUrl extracts url without protocol and slashes (0s)
✓ node - utils - charsPercent returns percent of entered chars limiting percent to 100 (0s)
✓ node - utils - isSelfReport determines if report was filed for our own site (0s)
✓ node - utils - getDataURIFromPreview gets the data URI portion inside of a serialized data URI (0s)
✓ node - validation - isUrlValid detects if url is valid (0s)
✓ node - validation - isEmpty detects if string is empty trimming spaces (0s)
✓ node - validation - isImageTypeValid detects if uploaded image type is valid (0s)
✓ node - validation - blobOrFileTypeValid detects if uploaded file type is valid (0s)
✓ node - validation - isImageDataURIValid detects is data URI is valid image (0s)
✓ node - validation - isGithubUserNameValid detects if github username is valid (0s)
TOTAL: tested 1 platforms, 17 passed, 0 failed
We use Intern to run functional tests.
Note: This version is known to work with Firefox 74.0. If things aren't working with the current stable version of Firefox, please file a bug!.
Be sure that you have installed local npm dependencies and run the build before trying to run functional tests - if not, you will notice problems with the css. See dev env setup for details.
Java is used to run Selenium functional tests. Version 1.8.0+ is required.
To test if your version of Java is recent enough, type the java -version into your terminal.
> java -version
java version "1.8.0_51"
Java(TM) SE Runtime Environment (build 1.8.0_51-b16)
Java HotSpot(TM) 64-Bit Server VM (build 25.51-b03, mixed mode)
Download from java.com/en/download/.
sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get install oracle-java8-installer
The firefox binary will also need to be in your PATH. Here's how this can be done on OS X:
export PATH="/Applications/Firefox.app/Contents/MacOS/:$PATH"If you are using your own test repo, your tests will fail. Change it back to the default, at least for running tests:
ISSUES_REPO_URI = 'webcompat/webcompat-tests/issues'.
Start the application server in test mode:
source env/bin/activate && python run.py -tor with the short form
npm run start:testWe start the server in test mode to mock the communications with GitHub API servers using local fixture data. The files in
/tests/fixtures/directory will be served as responses.
In a separate terminal window or tab, run the tests:
npm run test:jsYou can also run the functional tests as well as the Python tests in a separate tab, after starting the server, with:
npm test
Shortly after running this command, a progress indicator should appear in the terminal window and spin through each functional test in Firefox and then in Chrome. When all the tests complete, you'll see the final report of passes and failures in the terminal window that you ran the intern-runner command in.
To run a single test suite, where foo.js is the file found in the tests/functional directory:
npm run test:js functionalSuites=tests/functional/foo.jsTo filter which tests within a single test suite you run, you can use the grep argument:
npm run test:js grep=tacos functionalSuites=tests/functional/foo.jsThis will run any test within the foo.js suite that has "tacos" in its name.
Right now the tests are running in Firefox and Chrome as a default. You can specify which browsers you want to test by modifying intern.json in the project root.
By default, Chrome and Firefox will run in headless mode. To display the browser UI when running tests, remove the headless arguments for each browser from the environments key.
For a list of the recognized browser names, just refer to Browser enum
Contributions that add or modify major functionality to the project should typically come with tests to ensure we're not breaking things (or won't in the future!). There's always room for more testing, so general contributions in this form are always welcome.
- Open an issue (if it doesn't exist yet)
- Write the tests
- Implement the feature
- Send a pull request
- The discussion around the code can start
Our Python unit tests are written using the pytest framework.
Historically, we had been writing our tests with the Python standard unittest module. We are slowly transitioning to pytest for some of the tests when we have to modify them. So you will find some test files with a mix of both unittest and pytest. If you feel converting the tests from unittest to pytest, please pick up one test file, open an issue and send a pull request with your modifications. See below for an example of pytest test skeleton.
Unit tests placed in the tests/unit directory will be automatically detected by pytest.
Unit tests are preferred for features or functionality that are independent of the browser front-end, i.e., API responses, application routes, etc.
pytest has far less boilerplate requirements than the unittest module.
#!/usr/bin/env python
# -*- coding: utf-8 -*-
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
"""Tests for @@THIS_MODULE@@ methods."""
import pytest
import webcompat
# import the module you need to test
# example: from webcompat.api.helpers import get_html_comments
@pytest.fixture
def client():
webcompat.app.config['TESTING'] = True
with webcompat.app.test_client() as client:
with webcompat.app.app_context():
# intialize something
pass
yield client
def test_my_new_test(client):
"""Test docstring."""
rv = client.get('/')
assert b'yes! expected data' in rv.dataCheck, for example, the tests for the API helpers
Functional tests are written in JavaScript, using Intern. There's a nice guide on the Intern wiki that should explain enough to get you started.
- Leadfoot: the library that drives the browser (via Selenium).
- ChaiJS: the library used for assertions.
- Intern docs: contains useful examples.
It's also recommended to look at the other test files in the tests/functional directory to see how things are commonly done.
To avoid hitting the GitHub API during the tests, we need to make sure to have local representations of the data. When running the tests, the webcompat Flask app will then use these data for answering instead of asking data.
🚨You need to make sure that your data are representative of what GitHub is sending. This is the danger with any fixtures is when the fixtures data are drifting from the reality of what the third party API is sending.
To indicate that the app should send fixture data, use the Python @mockable_response decorator for a Python API endpoint.
If the endpoint you are trying to mock has GET parameters, you will need to create a file that has the GET parameters encoded in the filename. The source of @mockable_repsonse explains how this is done:
if get_args:
# if there are GET args, encode them as a hash so we can
# have different fixture files for different response states
checksum = hashlib.md5(json.dumps(get_args)).hexdigest()
file_path = FIXTURES_PATH + request.path + "." + checksum
print('Expected fixture file: ' + file_path + '.json')You can look at the server console's Expected fixture file: message to know what file it is expecting.
The data fixtures are in the tests directory.
tests/fixtures/api/follow the structure of the API hierarchy. For exampletests/fixtures/api/contains the requests done forhttps://webcompat.com/apitests/fixtures/api/issues/categorycontains the requests done forhttps://webcompa.com/api/issues/category
tests/fixtures/webhookscontains the fixture data for simulating the webhook requests.tests/fixtures/configcontains the fixture which are related to configuration initializationtests/fixturescontains some example of images (This should probably go to another directory).