Accommodate User Accounts functional tests

[seav]

Note: This PR is branched off from PR #1774. To see the specific changes for this PR, do a branch comparison: functest/loadfixtures...functest/accounts

Proposed changes in this pull request

This PR incorporates changes to accommodate automated User Accounts tests in the cadasta-test repo.

• Update the tag of cadasta-test repo to v0.2.1.
• Update settings for Travis to use a dummy email backend and set the max file upload size (for the future).
• Update the runtests-functional bash script to run non-file-upload tests using BrowserStack and the remaining tests using the local ChromeDriver.
• Update the runtests.py Python script to pass on extra command-line arguments to pytest when running under --functional.

When should this PR be merged

The following conditions need to be met:

• PR Add management command to load functional test fixtures #1774 is merged. This is because this PR is currently branched off from that PR.
• PR #39 in the cadasta-test repo is merged and tagged with v0.2.1.

Risks

No risks foreseen.

Follow-up actions

Continue functional test coding.

Checklist (for reviewing)

General

• Is this PR explained thoroughly? All code changes must be accounted for in the PR description.
  • Review 1
  • Review 2
• Is the PR labeled correctly? It should have the migration label if a new migration is added.
  • Review 1
  • Review 2
• Is the risk level assessment sufficient? The risks section should contain all risks that might be introduced with the PR and which actions we need to take to mitigate these risks. Possible risks are database migrations, new libraries that need to be installed or changes to deployment scripts.
  • Review 1
  • Review 2

Functionality

• Are all requirements met? Compare implemented functionality with the requirements specification.
  • Review 1
  • Review 2
• Does the UI work as expected? There should be no Javascript errors in the console; all resources should load. There should be no unexpected errors. Deliberately try to break the feature to find out if there are corner cases that are not handled.
  • Review 1
  • Review 2

Code

• Do you fully understand the introduced changes to the code? If not ask for clarification, it might uncover ways to solve a problem in a more elegant and efficient way.
  • Review 1
  • Review 2
• Does the PR introduce any inefficient database requests? Use the debug server to check for duplicate requests.
  • Review 1
  • Review 2
• Are all necessary strings marked for translation? All strings that are exposed to users via the UI must be marked for translation.
  • Review 1
  • Review 2
• Is the code documented sufficiently? Large and complex classes, functions or methods must be annotated with comments following our code-style guidelines.
  • Review 1
  • Review 2
• Has the scalability of this change been evaluated?
  • Review 1
  • Review 2
• Is there a maintenance plan in place?
  • Review 1
  • Review 2

Tests

• Are there sufficient test cases? Ensure that all components are tested individually; models, forms, and serializers should be tested in isolation even if a test for a view covers these components.
  • Review 1
  • Review 2
• If this is a bug fix, are tests for the issue in place? There must be a test case for the bug to ensure the issue won’t regress. Make sure that the tests break without the new code to fix the issue.
  • Review 1
  • Review 2
• If this is a new feature or a significant change to an existing feature? has the manual testing spreadsheet been updated with instructions for manual testing?
  • Review 1
  • Review 2

Security

• Confirm this PR doesn't commit any keys, passwords, tokens, usernames, or other secrets.
  • Review 1
  • Review 2
• Are all UI and API inputs run through forms or serializers?
  • Review 1
  • Review 2
• Are all external inputs validated and sanitized appropriately?
  • Review 1
  • Review 2
• Does all branching logic have a default case?
  • Review 1
  • Review 2
• Does this solution handle outliers and edge cases gracefully?
  • Review 1
  • Review 2
• Are all external communications secured and restricted to SSL?
  • Review 1
  • Review 2

Documentation

• Are changes to the UI documented in the platform docs? If this PR introduces new platform site functionality or changes existing ones, the changes must be documented in the Cadasta Platform Documentation.
  • Review 1
  • Review 2
• Are changes to the API documented in the API docs? If this PR introduces new API functionality or changes existing ones, the changes must be documented in the API docs.
  • Review 1
  • Review 2
• Are reusable components documented? If this PR introduces components that are relevant to other developers (for instance a mixin for a view or a generic form) they should be documented in the Wiki.
  • Review 1
  • Review 2

[seav]

Note: Test PR using the accounts branch of the cadasta-test repo has passed the Travis build.

[seav]

I added a commit a couple of days ago. Builds were encountering errors (the test that uses the local ChromeDriver, not BrowserStack) as well as when running tests in the VM. This error started occurring when Chromium was updated to version 61 late last week. Other people have also encountered the same error. Please refer to this Ubuntu bug report.

For the meantime, and in order not to hold up this PR, I have switched to using Firefox+GeckoDriver. For the Travis build, this should only affect just 1 functional test (covering 4 test cases). The rest of the tests still succeed using BrowserStack.

[alukach]

Required a complete rebuild of my dev VM but this worked. Nice job!

As an aside, I do wonder if we could speed up these tests by altering the test class so that it uses the same browser rather than closing and re-opening it after every test.

[seav]

As an aside, I do wonder if we could speed up these tests by altering the test class so that it uses the same browser rather than closing and re-opening it after every test.

I also thought of this possible improvement. This does need some small modifications to the test teardown such as logging out any logged in user. Also, I still need to see if there are any time limits to a single BrowserStack session which would become long if we don't shut down the remote browser driver; I haven't found any documentation regarding this yet. But anyway, a good compromise would be to start and stop the browser session on a module basis instead of the current class method basis.

[oliverroick]

@seav That's definitely something worth looking into. Running the functional that we have so far already takes ~35 min, so anything that makes it faster is more than welcome.

[seav]

@oliverroick, @alukach: I did a couple of experiments on how to make functional tests run faster and here are the results:

• Avoiding restarting the browser after every test shaves off around 10 minutes. This makes the current set of tests run in around 27 minutes instead of 35–36 minutes: https://travis-ci.org/Cadasta/cadasta-platform/jobs/279996890
• Avoiding restarting the browser, and avoiding BrowserStack itself by running all tests locally in the Travis VM makes the whole process more than 3× faster. The current set of tests would now run in under 10 minutes: https://travis-ci.org/Cadasta/cadasta-platform/jobs/280017948

Based on the results above, maybe we should only reserve using BrowserStack for commits to the master branch itself and use the local webdriver otherwise.

[alukach]

@seav Can you remind me what BrowserStack is offering? Just a richer set of browsers to test against?

[seav]

@alukach, yes. They also include a lot of mobile browsers for Android and iOS, something that we would want to eventually leverage as well. I also like that apart from automated tests, you yourself can use any desktop or mobile browser to view any website via BrowserStack.