Add Cypress tests for built site & gatsby-plugin-offline

[KyleAMathews]

We should automatically verify Gatsby's production run-time with a Cypress test suite, similar to what we've added for Gatsbygram & the develop runtime.

The tests should build the site and then verify various things like 404 support, visiting pages, clicking on links, various types of offline support etc.

@davidbailey00 has created a great list of manual checks @ #7355 (comment)

We should develop tests to automatically run the same tests on PRs.

First step is to setup a basic test that builds the site and runs a few basic tests. Then we extend that to run more and more tests.

[vtenfys]

Here's an updated version of the checklist, with some clearer descriptions:

checklist:

- 1) online without gatsby-plugin-offline
- 2) online with gatsby-plugin-offline
- 3) offline with gatsby-plugin-offline
- 4) online in develop


sub-checklist 1:

- 1) with a custom 404 page
- 2) without a custom 404 page


sub-checklist 2:

- 1) 404s are handled correctly from internal links
  (custom 404 or server 404 if custom is unavailable)

  requires: 1.1, 1.2, 2.1, 2.2, 3.1, 3.2, 4.1, 4.2
  passed:

- 2) 404s are handled correctly when entering a URL directly (as above)

  requires: 1.1, 1.2, 2.1, 2.2, 3.1, 3.2, 4.1, 4.2
  passed:

- 3) pages which fail to load display a native offline error (no blank pages)

  requires: 3.1, 3.2
  passed:

- 4) the service worker remains installed after a 404/offline error

  requires: 3.1, 3.2
  passed:

- 5) Netlify CMS /admin/ loads or displays an offline error (no 404/blank page)

  requires: 1.1, 1.2, 2.1, 2.2, 3.1, 3.2, 4.1, 4.2
  passed:

- 6) going to a page by entering the URL directly works

  requires: 1.1, 1.2, 2.1, 2.2, 3.1, 3.2, 4.1, 4.2
  passed:

Regarding the PR, currently tests 4.1.1 and 4.1.2 (i.e. custom 404s with gatsby develop) are failing but everything else passes.

[shobhitchittora]

@KyleAMathews @davidbailey00 On it! 👍

[vtenfys]

@shobhitchittora Awesome, thanks! Also since I made the last comment, all tests are now passing in the PR but various ones will fail with the current latest version of Gatsby.

[KyleAMathews]

We should explore this more. Get some basic tests for gatsby-plugin-offline to see if they're reliable and then continue to test smaller edge cases.

Perhaps helpful: cypress-io/cypress#702

/cc @davidbailey00

[vtenfys]

@KyleAMathews I've still got a branch in progress where I've added some tests - still haven't opened a PR because other bugs have popped up which I've shifted my focus onto. Will try to continue work on this branch next week.

(Link: https://github.com/davidbailey00/gatsby/tree/add-offline-tests)

[KyleAMathews]

Perfect! Thanks for the update!

[vtenfys]

Closed by #10940