Upgrade to Gatsby 3 and fix Heroku caching (#2851)

* Add tbd comment to pages/404.tsx * Add comment to 404 page * Update gatsby 2 to 3 * update needed dependencies to get gatsby 3 working * add gatsby-plugin-image * Update gatsby-source-github-api * Take off tbd comment in 404.tsx * Delete unneeded file in .github/workflows * Fix attempted imported css warnings * Add tbd comment to 404.tsx * Rollback [email protected] to 3.6.0 * Remove comment in 404 page * Move gatsby pkg back to 3.10.2 * Add tbd comment in 404 page * Update outdated packages * Delete tbd comment in 404 page * Place console.log in DefaultSEO * Run 'yarn upgrade' * Delete comment in 404.tsx * Upgrade @reach/portal * Delete console.log in DefaultSeo * Update @react/portal * Upgrade some packages * Update some packages * Add tbd comment in 404.tsx * Delete tbd comment * Test static query * Refactor useGlossary hook * Move gatsby to 3.9.0 * Add tbd comment * Update gatsby to 3.8.0 * Delete comment in 404 page * Move gatsby to 3.6.0 * Add tbd comment * Move gatsby to 3.4.0 * Delete comment in 404 page * Move gatsby to 3.2.0 * Add tbd comment in 404 page * Delete comment in 404 page * Move gatsby to 3.0.0 * Add tbd comment in 404 page * Update gatsby to 2.32.12 * Add tbd comment in 404 page * Update gatsby to 3.14.0 * Move gatsby to 3.10.2 * Delete and recreate yarn.lock * Delete tbd comment * Update typescript to 4.4.2 * Update package.json and yarn.lock to match 1be35ce * Add tbd comment * Delete test query from siteMeta * Delete console.log in siteMeta.ts * Remove resolutions from package.json * Delete tbd comment * Upgrade gatsby * Revert "Test static query" This reverts commit 51a6012. * Add PQR flag * Upgrade gatsby-*-sharp and add lmdb-store * Remove picture.small * Update to node 16 * Add tbd comment * Move gatsby to ^3.11.0 * Delete tbd comment * Move gatsby to ^3.12.0 * Delete tbd comment * Move gatsby to ^3.13.0 * Update some more packages * Add dummy post * Add ability to disable build retry in DEPLOY_OPTIONS * Change content * Change content * Remove deprecated gatsby-image * Try consecutive builds * Undo consecutive builds experiment * Log env * Change content even more * Remove dummy page * Remove env display * Add dummy page again * Make an interim deploy script * chmod +x * Use vars and add test echos * Remove echos and debug with which * More debugging * Even more debugging * Remove debug stuff and mkdir that caused issues * update content * Remove dummy post * Remove dummy blog post * DVCLive User Guide updates (#2814) * Updated top-level index. Refactored user-guide to top level * Updated links * Added dvclive-html.gif * Explicit mention to automatic html update * dvc 2.7.2 (#2813) * dvc 2.7.2 * gha: update: delay by 2 hours Co-authored-by: efiop <[email protected]> Co-authored-by: Ruslan Kuprieiev <[email protected]> * Fix broken dvc downlaod link check (#2820) * config: sync local/CI Prettier and Restyled (#2774) * test: change content/docs/user-guide/contributing/blog.md for #2641 * test: bad change to .../contributing/blog.md * test: try reverting 2dd2e91#diff-eb41540235b5687201bb19ed968c284453eea68563f5908a6e15ed8cbfa959de * format .restyled.yaml * config: sort prettier settings by abc * test: expanded Restyled config file per #2774 (comment) * Restyled by prettier (#2775) Co-authored-by: Restyled.io <[email protected]> * guide: roll back dummy changes (fixes formatting) * resyled: add comment on matching path pattern per #2774 (review) Co-authored-by: restyled-io[bot] <32688539+restyled-io[bot]@users.noreply.github.com> Co-authored-by: Restyled.io <[email protected]> * Added `Invalid authentication for Git Remote` to troubleshooting (#2731) * Added note about [SSH Git URLs] in dvc exp commands * Applied suggestions * Updated warning * Removed SSH from synopsis * Added Invalid authentication for Git Remote to troubleshooting * Added shorter title * Applied P.R. suggestions * Update content/docs/user-guide/troubleshooting.md * Update content/docs/user-guide/troubleshooting.md * Apply suggestions from code review Co-authored-by: Jorge Orpinel <[email protected]> * valid -> authenticated * Format * Update troubleshooting.md * Removed authenticated * Added note about git credentials Co-authored-by: Jorge Orpinel <[email protected]> * dvc 2.7.3 (#2829) Co-authored-by: efiop <[email protected]> * gha: update: run every day at 13:00 UTC * adding sept heartbeat (#2825) * adding sept heartbeat * punctuation, links, code revisions * Update content/blog/2021-09-14-september-21-dvc-heartbeat.md * adding cover image * corrections to description * fix the uploaded file link Co-authored-by: David de la Iglesia Castro <[email protected]> Co-authored-by: rogermparent <[email protected]> Co-authored-by: Ivan Shcheklein <[email protected]> * Fix broken link to Azure authentication examples (#2830) This link should point to: https://dvc.org/doc/command-reference/remote/modify#example-some-azure-authentication-methods * gha: update download links at 18:00 * dvc 2.7.4 (#2837) Co-authored-by: efiop <[email protected]> * fix: correct argument order (#2842) The order of arguments for `dvc remote rename` is reversed in the old version. According to the page @ `dvc remote rename -h`, the old remote `name` comes first followed by the `new` remote name argument. * Rename all references to `get-started-experiments` to `example-dvc-experiments` (#2817) Fixes #2784 * updating events page (#2847) * Refactor post by Batuhan (#2835) Co-authored-by: Batuhan Taskaya <[email protected]> Co-authored-by: Casper da Costa-Luis <[email protected]> * Add --show-csv flag to dvc exp show (#2740) * Add --show-csv flag to dvc exp show * Apply suggestions from code review * Format problem Co-authored-by: Jorge Orpinel <[email protected]> * use folded block scalar type (#2871) * Revert some unnecessary changes * Some minor/patch package upgrades * Add bug mitigation * Disable PQR and bug mitigation * Add USE_PRODUCTION_CACHE to deploy script for debugging * Revert style import change and undo malformed comment * Allow LayoutAlert to be false in typedef * Remove nonstandard space and use width for spacing for lint-css * Remove commented-out classnames * Upgrade gatsby and some other minor/patch upgrades * Add aptfile * Revert "Remove commented-out classnames" This reverts commit edae234. * Gatsby 4 again * Remove explicit typescript plugin and upgrade some packages * Remove unused wrapper class * Upgrade a bunch of packages * upgrade plugin-feed, downgrade transformer-remark * Update some more packages and try container stack * Update some other packages to trigger a cache clear * Revert stack change attempt, async setPageContext, remove Aptfile * Upgrade CircleCI node and un-refactor glossary query * Resolve ts-lint issues * Prettier on content/docs/user-guide/contributing/blog.md * Upgrade packages * Remove composes from * Remove source-github-api and other uses of deprecated runQuery * Back to Gatsby 3 * Remove explicit webpack * Use blur placeholder by default * Fix misconfig on blur * Upgrade some packages and remove unused lmdb-store * yarn upgrade (lockfile only) * Re-add typescript plugin definition (but not package.json) * Re-add node_modules cacheDirectories Co-authored-by: julieg18 <[email protected]> Co-authored-by: Julie <[email protected]> Co-authored-by: David de la Iglesia Castro <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: efiop <[email protected]> Co-authored-by: Ruslan Kuprieiev <[email protected]> Co-authored-by: Jorge Orpinel <[email protected]> Co-authored-by: restyled-io[bot] <32688539+restyled-io[bot]@users.noreply.github.com> Co-authored-by: Restyled.io <[email protected]> Co-authored-by: Jenifer De Figueiredo <[email protected]> Co-authored-by: Ivan Shcheklein <[email protected]> Co-authored-by: Jose Celano <[email protected]> Co-authored-by: Gavin Masterson <[email protected]> Co-authored-by: Emre Sahin <[email protected]> Co-authored-by: Batuhan Taskaya <[email protected]> Co-authored-by: Casper da Costa-Luis <[email protected]> Co-authored-by: Gao <[email protected]>
iterative · Oct 19, 2021 · 3b3b52d · 3b3b52d
1 parent 66003af
commit 3b3b52d
Show file tree

Hide file tree

Showing 89 changed files with 8,125 additions and 10,047 deletions.
diff --git a/.circleci/config.yml b/.circleci/config.yml
@@ -8,7 +8,7 @@ defaults: &defaults
   working_directory: ~/repo
   docker:
     # Specify the version you desire here.
-    - image: circleci/node:12
+    - image: circleci/node:16
 
     # Specify service dependencies here if necessary.
     # CircleCI maintains a library of pre-built images,

diff --git a/content/docs/user-guide/contributing/blog.md b/content/docs/user-guide/contributing/blog.md
@@ -41,7 +41,6 @@ tags:
   - Version Control
   - AI
 ---
-
 ```
 
 - `title` (**required**) - title of the post.

diff --git a/gatsby-config.js b/gatsby-config.js
@@ -57,6 +57,7 @@ const plugins = [
       path: path.join(__dirname, 'static')
     }
   },
+  'gatsby-plugin-image',
   'community-page',
   {
     resolve: 'gatsby-transformer-remark',
@@ -122,7 +123,14 @@ const plugins = [
     }
   },
   'gatsby-transformer-sharp',
-  'gatsby-plugin-sharp',
+  {
+    resolve: 'gatsby-plugin-sharp',
+    options: {
+      defaults: {
+        placeholder: 'blurred'
+      }
+    }
+  },
   {
     resolve: 'gatsby-plugin-catch-links',
     options: {
@@ -224,28 +232,6 @@ const plugins = [
   }
 ]
 
-if (process.env.GITHUB_TOKEN) {
-  plugins.push({
-    resolve: `gatsby-source-github-api`,
-    options: {
-      // token: required by the GitHub API
-      token: process.env.GITHUB_TOKEN,
-
-      // GraphQLquery: defaults to a search query
-      graphQLQuery: `
-          {
-            repository(owner: "iterative", name: "dvc") {
-              stargazers {
-                totalCount
-              }
-            }
-          }
-        `,
-      variables: {}
-    }
-  })
-}
-
 if (process.env.CONTEXT === 'production') {
   plugins.push({
     resolve: 'gatsby-plugin-google-analytics',
@@ -258,11 +244,7 @@ if (process.env.CONTEXT === 'production') {
 
 if (process.env.ANALYZE) {
   plugins.push({
-    resolve: 'gatsby-plugin-webpack-bundle-analyzer',
-    options: {
-      analyzerPort: 4000,
-      production: process.env.NODE_ENV === 'production'
-    }
+    resolve: 'gatsby-plugin-webpack-bundle-analyser-v2'
   })
 }
 

diff --git a/package.json b/package.json
@@ -9,7 +9,7 @@
     "author": "LIMIT_BLOG_PAGES=1 SKIP_DOCS=true gatsby develop",
     "build": "gatsby build",
     "start": "node ./src/server/index.js",
-    "heroku-postbuild": "./scripts/deploy-with-s3.js",
+    "heroku-postbuild": "./scripts/heroku-deploy.sh",
     "test": "jest",
     "format-staged": "pretty-quick --staged --no-restage --bail",
     "format-check": "prettier --check '**/*.{js,jsx,md,tsx,ts,json}'",
@@ -33,50 +33,51 @@
   },
   "homepage": "https://github.com/iterative/dvc.org#readme",
   "engines": {
-    "node": "<=15.x"
+    "node": "<=16.x"
   },
   "dependencies": {
     "@hapi/wreck": "^17.0.0",
-    "@octokit/graphql": "^4.3.1",
-    "@reach/portal": "^0.10.0",
-    "@reach/router": "^1.3.3",
-    "@reach/tooltip": "^0.10.0",
+    "@octokit/graphql": "^4.8.0",
+    "@reach/portal": "^0.16.2",
+    "@reach/router": "^1.3.4",
+    "@reach/tooltip": "^0.16.2",
     "classnames": "^2.2.6",
-    "color": "^3.1.2",
+    "color": "^4.0.1",
     "compression": "^1.7.4",
-    "date-fns": "^2.11.1",
+    "date-fns": "^2.25.0",
     "docsearch.js": "^2.6.3",
     "ease-component": "^1.0.0",
     "express": "^4.17.1",
-    "fs-extra": "^9.0.0",
-    "gatsby": "^2.20.13",
-    "gatsby-image": "^2.3.1",
-    "gatsby-link": "^2.3.2",
+    "fs-extra": "^10.0.0",
+    "gatsby": "^3.14.3",
+    "gatsby-link": "^3.14.0",
+    "gatsby-plugin-image": "^1.14.1",
     "gatsby-plugin-parent-resolvers": "^1.0.1",
-    "gatsby-source-github-api": "^0.2.1",
     "github-markdown-css": "^4.0.0",
-    "iso-url": "^0.4.7",
-    "isomorphic-fetch": "^2.2.1",
+    "graphql": "^15.6.1",
+    "iso-url": "^1.1.5",
+    "isomorphic-fetch": "^3.0.0",
     "lodash": "^4.17.21",
     "moment": "^2.25.3",
-    "nanoid": "^3.0.2",
+    "nanoid": "^3.1.30",
     "node-cache": "^5.1.0",
     "perfect-scrollbar": "^1.5.0",
-    "pretty-quick": "^2.0.1",
+    "postcss": "^8.3.9",
+    "pretty-quick": "^3.1.1",
     "prismjs": "^1.25.0",
     "promise-polyfill": "^8.1.3",
     "prop-types": "^15.7.2",
     "raf-polyfill": "^1.0.0",
-    "react": "^16.13.1",
+    "react": "^17.0.2",
     "react-collapse": "^5.0.1",
     "react-collapsible": "^2.7.0",
-    "react-dom": "^16.13.1",
-    "react-ga": "^2.7.0",
-    "react-helmet": "^5.2.1",
+    "react-dom": "^17.0.2",
+    "react-ga": "^3.3.0",
+    "react-helmet": "^6.1.0",
     "react-popover": "^0.5.10",
-    "react-slick": "^0.25.2",
+    "react-slick": "^0.28.1",
     "react-use": "^14.0.0",
-    "rehype-react": "^5.0.1",
+    "rehype-react": "^6.2.1",
     "remark-preset-lint-recommended": "^5.0.0",
     "repo-link-check": "^0.7.1",
     "reset-css": "^5.0.1",
@@ -86,10 +87,10 @@
     "slick-carousel": "^1.8.1",
     "title-case": "^3.0.2",
     "unist-util-visit": "2.0.2",
-    "upath": "^1.2.0"
+    "upath": "^2.0.1"
   },
   "devDependencies": {
-    "@babel/core": "^7.9.0",
+    "@babel/core": "^7.15.8",
     "@svgr/webpack": "^5.3.1",
     "@types/classnames": "^2.2.10",
     "@types/isomorphic-fetch": "^0.0.35",
@@ -99,66 +100,65 @@
     "@types/react-dom": "^16.9.6",
     "@types/react-helmet": "^5.0.15",
     "@types/react-popover": "^0.5.3",
-    "@types/react-slick": "^0.23.4",
+    "@types/react-slick": "^0.23.6",
     "@types/rehype-react": "^4.0.0",
     "@typescript-eslint/eslint-plugin": "^2.27.0",
     "@typescript-eslint/parser": "^2.27.0",
-    "autoprefixer": "^9.7.6",
+    "autoprefixer": "^10.3.7",
     "babel-eslint": "^10.1.0",
     "babel-jest": "^26.0.1",
-    "babel-plugin-transform-define": "^2.0.0",
+    "babel-plugin-transform-define": "^2.0.1",
     "babel-plugin-transform-object-assign": "^6.22.0",
     "eslint": "^6.8.0",
     "eslint-config-prettier": "^6.10.1",
-    "eslint-plugin-json": "^2.1.1",
+    "eslint-plugin-json": "^3.1.0",
     "eslint-plugin-jsx-a11y": "^6.2.3",
     "eslint-plugin-prettier": "^3.1.2",
-    "eslint-plugin-react": "^7.19.0",
-    "gatsby-plugin-catch-links": "^2.2.1",
-    "gatsby-plugin-feed": "^2.4.1",
-    "gatsby-plugin-google-analytics": "^2.2.2",
-    "gatsby-plugin-manifest": "2.2.23",
-    "gatsby-plugin-postcss": "^2.2.1",
-    "gatsby-plugin-react-helmet": "^3.2.1",
+    "eslint-plugin-react": "^7.26.1",
+    "gatsby-plugin-catch-links": "^3.14.0",
+    "gatsby-plugin-feed": "^3.14.0",
+    "gatsby-plugin-google-analytics": "^3.14.0",
+    "gatsby-plugin-manifest": "^3.14.0",
+    "gatsby-plugin-postcss": "^4.14.0",
+    "gatsby-plugin-react-helmet": "^4.14.0",
     "gatsby-plugin-sentry": "^1.0.1",
-    "gatsby-plugin-sharp": "2.2.32",
-    "gatsby-plugin-sitemap": "^2.3.1",
-    "gatsby-plugin-svgr": "^2.0.2",
-    "gatsby-plugin-twitter": "^2.2.2",
-    "gatsby-plugin-typescript": "^2.3.1",
-    "gatsby-plugin-webpack-bundle-analyzer": "^1.0.5",
-    "gatsby-remark-autolink-headers": "^2.2.1",
-    "gatsby-remark-copy-linked-files": "^2.2.1",
+    "gatsby-plugin-sharp": "^3.14.1",
+    "gatsby-plugin-sitemap": "^4.10.0",
+    "gatsby-plugin-svgr": "^3.0.0-beta.0",
+    "gatsby-plugin-twitter": "^3.14.0",
+    "gatsby-plugin-webpack-bundle-analyser-v2": "^1.1.25",
+    "gatsby-remark-autolink-headers": "^4.11.0",
+    "gatsby-remark-copy-linked-files": "^4.11.0",
     "gatsby-remark-embed-gist": "^1.1.9",
-    "gatsby-remark-embedder": "^2.0.0",
+    "gatsby-remark-embedder": "^5.0.0",
     "gatsby-remark-external-links": "^0.0.4",
-    "gatsby-remark-images": "^3.2.2",
-    "gatsby-remark-prismjs": "^3.4.1",
+    "gatsby-remark-images": "^5.11.0",
+    "gatsby-remark-prismjs": "^5.11.0",
     "gatsby-remark-relative-images": "0.2.3",
-    "gatsby-remark-responsive-iframe": "^2.3.1",
-    "gatsby-remark-smartypants": "^2.2.1",
-    "gatsby-source-filesystem": "^2.2.2",
-    "gatsby-transformer-remark": "^2.7.1",
-    "gatsby-transformer-sharp": "2.2.23",
+    "gatsby-remark-responsive-iframe": "^4.11.0",
+    "gatsby-remark-smartypants": "^4.11.0",
+    "gatsby-source-filesystem": "^3.14.0",
+    "gatsby-transformer-remark": "^4.11.0",
+    "gatsby-transformer-sharp": "^3.14.0",
     "hast-util-select": "^4.0.0",
     "husky": "^4.2.3",
-    "jest": "^26.0.1",
-    "lint-staged": "^10.1.2",
+    "jest": "^27.3.0",
+    "lint-staged": "^11.2.3",
     "postcss-color-mod-function": "^3.0.3",
-    "postcss-custom-media": "^7.0.8",
-    "postcss-custom-properties": "^9.1.1",
-    "postcss-mixins": "^6.2.3",
-    "postcss-nested": "^4.2.1",
-    "prettier": "^2.2.1",
-    "rehype-parse": "^6.0.2",
-    "rehype-stringify": "^7.0.0",
-    "remark": "^12.0.0",
-    "remark-html": "^13.0.2",
+    "postcss-custom-media": "^8.0.0",
+    "postcss-custom-properties": "^12.0.0",
+    "postcss-mixins": "^8.1.0",
+    "postcss-nested": "^5.0.6",
+    "prettier": "^2.4.1",
+    "rehype-parse": "^7.0.1",
+    "rehype-stringify": "^8.0.0",
+    "remark": "^13.0.0",
+    "remark-html": "^13.0.1",
     "remark-parse": "^8.0.2",
     "stylelint": "^13.3.0",
-    "stylelint-config-standard": "^20.0.0",
+    "stylelint-config-standard": "^22.0.0",
     "typescript": "^3.8.3",
-    "unist-util-remove-position": "^2.0.1"
+    "unist-util-remove-position": "^3.0.0"
   },
   "husky": {
     "hooks": {

diff --git a/scripts/deploy-with-s3.js b/scripts/deploy-with-s3.js
@@ -3,8 +3,9 @@
 require('dotenv').config()
 const path = require('path')
 const PRODUCTION_PREFIX = 'dvc-org-prod'
+const { mkdirSync } = require('fs')
 
-const { DEPLOY_OPTIONS } = process.env
+const { DEPLOY_OPTIONS, USE_PRODUCTION_CACHE } = process.env
 const clearCloudflareCache = require('./clear-cloudflare-cache')
 
 // Generate deploy options from a comma separated string in the DEPLOY_OPTIONS
@@ -20,6 +21,7 @@ const deployOptions = DEPLOY_OPTIONS
   : {
       download: true,
       build: true,
+      retry: true,
       upload: true,
       clean: true,
       clearCloudflareCache: true
@@ -69,9 +71,8 @@ const {
   cleanEntry
 } = require('./s3-utils')
 const { move } = require('fs-extra')
-const { downloadAllFromS3, uploadAllToS3, cleanAllLocal } = withEntries(
-  cacheDirs
-)
+const { downloadAllFromS3, uploadAllToS3, cleanAllLocal } =
+  withEntries(cacheDirs)
 
 function run(command) {
   execSync(command, {
@@ -86,32 +87,42 @@ async function main() {
   // This greatly speeds up PR initial build time.
 
   if (deployOptions.download) {
-    if (emptyPrefix) {
+    if (USE_PRODUCTION_CACHE) {
+      console.warn('USE_PRODUCTION_CACHE is set, downloading from production')
+      await downloadAllFromS3(PRODUCTION_PREFIX)
+    } else if (emptyPrefix) {
       console.warn(
         `The current prefix "${s3Prefix}" is empty! Attempting to fall back on production cache.`
       )
       await downloadAllFromS3(PRODUCTION_PREFIX)
     } else {
       await downloadAllFromS3(s3Prefix)
     }
+    mkdirSync('.cache/json', { recursive: true })
   }
 
   if (deployOptions.build) {
     try {
       run('yarn build')
     } catch (buildError) {
-      // Sometimes gatsby build fails because of bad cache.
-      // Clear it and try again.
-
-      console.error('------------------------\n\n')
-      console.error('The first Gatsby build attempt failed!\n')
-      console.error(buildError)
-      console.error('\nRetrying with a cleared cache:\n')
-
-      // Clear only .cache so we re-use images
-      await cleanEntry(cacheDirs[1])
-
-      run('yarn build')
+      if (deployOptions.retry) {
+        // Sometimes gatsby build fails because of bad cache.
+        // Clear it and try again.
+
+        console.error('------------------------\n\n')
+        console.error('The first Gatsby build attempt failed!\n')
+        console.error(buildError)
+        console.error('\nRetrying with a cleared cache:\n')
+
+        // Clear only .cache so we re-use images
+        await cleanEntry(cacheDirs[1])
+
+        run('yarn build')
+      } else {
+        throw new Error(
+          'The first Gatsby build attempt failed, and DEPLOY_OPTIONS does not include "retry"'
+        )
+      }
     }
   }
 

diff --git a/scripts/heroku-deploy.sh b/scripts/heroku-deploy.sh
@@ -0,0 +1,6 @@
+#!/usr/bin/env bash
+NEWPWD="/tmp/gatsby-build"
+mv $OLDPWD $NEWPWD
+ln -s $NEWPWD $OLDPWD
+cd $NEWPWD
+./scripts/deploy-with-s3.js
-Original file line number
+Diff line change
@@ Expand Up / @@ -41,7 +41,6 @@ tags: @@
       - Version Control
       - AI
     ---
     ```
     - `title` (**required**) - title of the post.
@@ Expand Down @@