Weekly Digest (26 April, 2020 - 3 May, 2020) #1211

weekly-digest · 2020-05-03T01:02:50Z

Here's the Weekly Digest for iterative/dvc.org:

ISSUES

This week, 28 issues were created. Of these, 23 issues have been closed and 5 issues are still open.

OPEN ISSUES

💚 #1210 GSOD 2020 blog post, et al., by jorgeorpinel
💚 #1198 doc engine: fix scroll bar for the content sidebar, by shcheklein
💚 #1195 Regular updates (Apr 27), by jorgeorpinel
💚 #1186 Plots, by dmpetrov
💚 #1175 metrics: update docs per recent changes, by jorgeorpinel

CLOSED ISSUES

LIKED ISSUE

The issue most liked this week has been:
👍 #1200 Remove reference to core.loglevel in docs, by woodshop
It received 👍 x1, 😄 x0, 🎉 x0 and ❤️ x1.

NOISY ISSUE

The issue most discussed this week has been:
🔈 #1186 Plots, by dmpetrov
It received 7 comments.

PULL REQUESTS

This week, 18 pull requests were proposed. Of these, 0 pull requests have been merged and 3 are still open.

OPEN PRs

💚 #1210 GSOD 2020 blog post, et al., by jorgeorpinel
💚 #1195 Regular updates (Apr 27), by jorgeorpinel
💚 #1186 Plots, by dmpetrov

CONTRIBUTORS

This week, 7 users have contributed to this repository.
They are jorgeorpinel, rogermparent, woodshop, shcheklein, casperdcl, shizacat, and dmpetrov.

STARGAZERS

This week, no user has starred this repository.

COMMITS

This week, there have been 29 commits in the repository.
These are:
🛠️ Use upath in Gatsby foreign-file resolvers (#1205) by rogermparent
🛠️ [Fix and improve cache pruning (#1209)

I had some testing behavior in the cache file that logged file removals without
actually doing them. This would be fine if I had marked it in a TODO or any
comment whatsoever.

Anyway, this commit fixes that.

I also changed the logic to use regular fs and Promise.new instead of
fs.promises to avoid the experimental warning. upath is used because
I believe that without it cache pruning won't work on Windows.](083a343) by rogermparent
🛠️ Make Link component recognize mailto links (#1208) by rogermparent
🛠️ [Explicitly add node_modules to cacheDirectories for Heroku (#1203)

This seems to be a more "full" cache than the default behavior, despite
the Heroku node buildpack docs stating node_modules is cached by default.](1a615ed) by rogermparent
🛠️ Add a script in package.json enabling "yarn run dev" (#1207) by rogermparent
🛠️ Remove source for outdated doc at doc/user-guide/api-reference/open (#1206) by rogermparent
🛠️ blog: May heartbeat update a broken link by shcheklein
🛠️ [Rebuilds caching (#1182)

Store .cache and public fully on S3
Do not cache public and .cache folders on Heroku
Clean up outdated page-data on rebuilds
Get rootDir with process.cwd instead of __dirname

This way the code doesn't depend on the source file's relative directory.

Parallelize uploadAllToS3, downloadAllFromS3, and clean.

I'm not sure if there's any rate limits in the stack that will cause this to break, but on first blush this looks like a quick and easy improvement.

The cached folders are also now accessed by a shared array that is hard-coded for now. This both makes them easier to work with uniformly and open to sourcing directories from elsewhere in the future.

Fix a broken path.join
Remove unreachable return
Improve grammar in log messages
Further grammar changes in logging
Move cache pruning into a gatsby hook and fix prefixes.

I believe the crux of what keeps this PR from working is that public and
.cache the folders were being uploaded to public and ,cache the S3
prefixes when the site is deployed from the root prefix. This change alters how
the cache directories are stored: now as tuple pairs with the local directory in
slot 0 and a string that's appended to the base prefix in slot 2. public work
with the root prefix and the .cache folder works with the root prefix with
-cache appended to it such that it's another "folder" that doesn't get
downloaded with the root prefix.

I find the tuple syntax more concise, but I'm open to changing it if we prefer
objects for better grokking at a glance.

This change also moves the responsibility of cache culling into an onPostBuild
Gatsby hook. This means we don't have to parse the sitemap XML after the fact,
and as such the XML parser used to do it is removed. Since this replaces
cleanUpPageData, it also invalidates the whole page-data-utils file which
has been removed.

Remove left over console.log statement
Refactor S3 calls from deploy-with-s3 to s3-utils
Re-add functionality for moving index.html to project root.
Add a clarifying comment on the 404 HTML file moving.
Initialize dotenv in the deploy script and check it to determine if we want to clean local cache post-build.
Check cwd specifically for heroku
List the public images cache for heroku
Make down/uploads synchronous again
Limit page builds
Fix re-syncifying downloads/uploads
Add root-level 404.html to gitignore
Fix prune-cache to work post-Models

It was firing falsely because some pages still have slashes.
This commit makes the path set that cache is run against remove all
trailing slashes, matching the outputof crawlPageData.

Start logging
Improve cache dir debug tools
Consolidate deploy options into one env var.

The only deploy options env var is DEPLOY_OPTIONS, and it takes a
comma-separated string of build steps to run on the deploy.

A setting of all possible steps would be:
DEPLOY_OPTIONS="download,logTrees,logHashes,bailAfterLogs,build,upload,clean"

Having DEPLOY_OPTIONS unset gives the following default:
DEPLOY_OPTIONS="download,build,upload,clean"

If DEPLOY_OPTIONS is specified, it will only perform the provided steps. The
default options are disregarded.

logHashes makes a small console log of a hash describing the structure of the
directory trees of .cache and public. If any file is added, removed, or
renamed, this hash will change. File content is disregarded, however cache
busted files with content hashes in their filenames will effectively have
content-detection behavior.

Change heroku hook to run on bash and save output
Make deploy logging script log both to file and console
Remove NM cache heroku test
log pages
log branch env var
Revert sharp-related gatsby plugins to pre-Jobsv2 behavior to help with nonstandard caching
Remove page logging from build script and re-enable prod cache warming.
Re-async downloads and uploads
Fix {} typo in downloadAll function
Remove branch logging from onPostBuild
Remove deploy-with-s3.sh and direct call the js variant again.
Re-add node_modules cacheDirectories entry

I don't know if it's redundant with the buildpack, but I don't think it can hurt.

Remove temporary deploy logs.

They can be re-added later if needed, possibly in another branch.

Remove S3 testing utils import

oops!

Test Heroku cache
Don't specify cache directories
Bump CircleCI cache version (?)

Co-authored-by: Pavel Grinchenko [email protected]
Co-authored-by: RMP [email protected]](822b51a) by shcheklein
🛠️ blog: April heartbeat fix broken link by shcheklein
🛠️ [Merge pull request #1200 from woodshop/patch-1

Remove reference to core.loglevel in docs](8cebde9) by jorgeorpinel
🛠️ [Fixes #1199

core.loglevel was removed from the available config options in iterative/dvc#3321. This PR updates the documentation to remove any references to the deprecated configuration setting.](d05d4b3) by woodshop
🛠️ Update add.md by shcheklein
🛠️ blog: september heartbeat fix broken links by shcheklein
🛠️ minor: make titles shorter to avoid too long content sidebar by shcheklein
🛠️ [gdrive: add gdrive_user_credentials_file description (#1192)

gdrive: add gdrive_user_credentials_file description
gdrive remote modify: simplify a bit gdrive_user_credentials_file description
gdrive remote: apply suggestions from code review

Co-Authored-By: Jorge Orpinel [email protected]

Update content/docs/user-guide/setup-google-drive-remote.md
gdrive: update text after testing the gdrive_user_credentials_file option

Co-authored-by: Jorge Orpinel [email protected]](cba33a3) by shcheklein
🛠️ blog: fix broken link by shcheklein
🛠️ [SEO Improvements (#1183)

add meta tags for SEO
Add aria-label props to improve SEO

This adds 10+ points to Lighthouse accessibility score.

Lighthouse sources:

https://web.dev/button-name
https://web.dev/link-name

Make meta image sources absolute and re-add old sizes

Some external readers and parsers dislike root-relative links,
so this commit adds the site URL before all of them.

Also, I re-added the meta icons previously removed. I didn't find explicit
reasons for or against their inclusion, but considering the extra resource usage
is minimal I'm going to invoke chesterton's fence here and leave it as it was
before.

Change SEO image prop from string to fluid image

This is what the TS def and the component itself expects.

Revert favicons to original state

Co-authored-by: Fábio Santos [email protected]](1dd6226) by rogermparent
🛠️ test: abstract all helpers to utils.sh (#1194) by casperdcl
🛠️ [Merge pull request #1174 from iterative/2020-04-21

Regular updates (Apr 21)](d10763c) by jorgeorpinel
🛠️ tutorials: fix deep/prep page format
per https://github.com/iterative/dvc.org/pull/1174#pullrequestreview-401183016 by jorgeorpinel
🛠️ [link-check: fix CI daily (#1193)

Fixes #1191](aaf04f0) by casperdcl
🛠️ cmd ref: avoid using term "output" in the context of simple DVC-files (dvc run)
per https://github.com/iterative/dvc.org/pull/1174#discussion_r415168286 by jorgeorpinel
🛠️ cmd ref: review term "output" in lit, gets, imports cmds
per https://github.com/iterative/dvc.org/pull/1174#pullrequestreview-400440864 by jorgeorpinel
🛠️ [link-check: fix exclusions (#1190)

link-check: fix exclusions
link-check: add a couple of excludes](0b709c6) by casperdcl
🛠️ cmd ref: fix metrics index synopsis by jorgeorpinel
🛠️ fix anchors by shcheklein
🛠️ [test: link-check: trim and manage exclusions (test: link-check: trim and manage exclusions #1189)
link-check: abstract away finder
link-check: add exclude-links-check.sh
link-check: purge unused exclusions
test: add link-check-exclude
link-check: exclude *.test.js

also ensure link-check-git-all follows exclusions

link-check: exclude more missing
link-check: final exclude purge](f5bb7ea) by casperdcl
🛠️ [test: link-check: misc fixes (test: link-check: misc fixes #1169)
test: fix link-check diffs

Fixes #1148

test: link-check: fix file paths, add comments
test: link-check: use git pathspec exclusions
test: link-check: more path safety and comments
test: link-check: misc tidy
test: link-check: fix git diff multi-errors
test: link-check: fix mac sed
test: link-check: diff: whitelist rather than blacklist

As with link-check-git-all.sh, only include md & js
rather than include all except specified files

link-check: add more file extensions
link-check: add and sort exclusions
link-check: add *.css
link-check: re-exclude redirects-list.json](4b2868c) by casperdcl
🛠️ Accessibility: have focus styles without having them too for those who don't use keyboards. (#1153) by nisarhassan12

RELEASES

This week, no releases were published.

That's all for this week, please watch 👀 and star ⭐ iterative/dvc.org to receive next weekly updates. 😃

The text was updated successfully, but these errors were encountered:

weekly-digest bot added the weekly-digest label May 3, 2020

shcheklein closed this as completed May 3, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Weekly Digest (26 April, 2020 - 3 May, 2020) #1211

Weekly Digest (26 April, 2020 - 3 May, 2020) #1211

weekly-digest bot commented May 3, 2020

Weekly Digest (26 April, 2020 - 3 May, 2020) #1211

Weekly Digest (26 April, 2020 - 3 May, 2020) #1211

Comments

weekly-digest bot commented May 3, 2020

ISSUES

OPEN ISSUES

CLOSED ISSUES

LIKED ISSUE

NOISY ISSUE

PULL REQUESTS

OPEN PRs

CONTRIBUTORS

STARGAZERS

COMMITS

RELEASES