Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs(core): database cache concept page #28935

Merged
merged 6 commits into from
Nov 18, 2024
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/blog/2024-09-25-introducing-nx-powerpack.md
Original file line number Diff line number Diff line change
Expand Up @@ -125,7 +125,7 @@ Read all about how to [configure Codeowners for your project in our docs](/nx-en

A continuous effort on our Nx core is to improve speed. Last year, we began **rewriting performance-critical parts of Nx into Rust**, and more core components are being rewritten. As part of this effort, we also changed how we manage local cache, moving from a **file-based to a database-based approach**. In addition to small performance gains from reduced I/O, this opens up many opportunities for improving local cache handling, such as keeping only relevant cache based on usage, more easily controlling maximum cache size, and optimizing task orchestration by running failed tasks earlier.

As part of this new approach we're also going to [deprecate custom task runners](/deprecated/custom-task-runners) in Nx 20. I bring this up because it might affect users that relied on 3rd party tools that hooked into the task runners API.
As part of this new approach we're also going to [deprecate custom task runners](/deprecated/legacy-cache) in Nx 20. I bring this up because it might affect users that relied on 3rd party tools that hooked into the task runners API.

To fill in on the custom task runner API we're providing a new Powerpack plugin that allows you to use S3 or a network drive as your storing mechanism for your Nx cache.

Expand Down
12 changes: 6 additions & 6 deletions docs/generated/manifests/menus.json
Original file line number Diff line number Diff line change
Expand Up @@ -4796,9 +4796,9 @@
"disableCollapsible": false
},
{
"name": "Custom Task Runners",
"path": "/deprecated/custom-task-runners",
"id": "custom-task-runners",
"name": "Legacy Cache",
"path": "/deprecated/legacy-cache",
"id": "legacy-cache",
"isExternal": false,
"children": [],
"disableCollapsible": false
Expand Down Expand Up @@ -4919,9 +4919,9 @@
"disableCollapsible": false
},
{
"name": "Custom Task Runners",
"path": "/deprecated/custom-task-runners",
"id": "custom-task-runners",
"name": "Legacy Cache",
"path": "/deprecated/legacy-cache",
"id": "legacy-cache",
"isExternal": false,
"children": [],
"disableCollapsible": false
Expand Down
18 changes: 9 additions & 9 deletions docs/generated/manifests/nx.json
Original file line number Diff line number Diff line change
Expand Up @@ -6572,14 +6572,14 @@
"tags": []
},
{
"id": "custom-task-runners",
"name": "Custom Task Runners",
"id": "legacy-cache",
"name": "Legacy Cache",
"description": "",
"mediaImage": "",
"file": "shared/deprecated/custom-task-runners",
"file": "shared/deprecated/legacy-cache",
"itemList": [],
"isExternal": false,
"path": "/deprecated/custom-task-runners",
"path": "/deprecated/legacy-cache",
"tags": []
},
{
Expand Down Expand Up @@ -6741,15 +6741,15 @@
"path": "/deprecated/workspace-generators",
"tags": []
},
"/deprecated/custom-task-runners": {
"id": "custom-task-runners",
"name": "Custom Task Runners",
"/deprecated/legacy-cache": {
"id": "legacy-cache",
"name": "Legacy Cache",
"description": "",
"mediaImage": "",
"file": "shared/deprecated/custom-task-runners",
"file": "shared/deprecated/legacy-cache",
"itemList": [],
"isExternal": false,
"path": "/deprecated/custom-task-runners",
"path": "/deprecated/legacy-cache",
"tags": []
},
"/deprecated/workspace-executors": {
Expand Down
6 changes: 3 additions & 3 deletions docs/map.json
Original file line number Diff line number Diff line change
Expand Up @@ -1388,9 +1388,9 @@
"file": "shared/deprecated/workspace-generators"
},
{
"name": "Custom Task Runners",
"id": "custom-task-runners",
"file": "shared/deprecated/custom-task-runners"
"name": "Legacy Cache",
"id": "legacy-cache",
"file": "shared/deprecated/legacy-cache"
},
{
"name": "Workspace Executors",
Expand Down
12 changes: 0 additions & 12 deletions docs/shared/deprecated/custom-task-runners.md

This file was deleted.

22 changes: 22 additions & 0 deletions docs/shared/deprecated/legacy-cache.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# Legacy Cache

## tasksRunnerOptions

As of Nx 20, the `tasksRunnerOptions` property in `nx.json` is deprecated. This property was used to register custom task runners. `tasksRunnerOptions` and custom task runners only work with the legacy file system cache.

## NX_REJECT_UNKNOWN_LOCAL_CACHE

The `NX_REJECT_UNKNOWN_LOCAL_CACHE` environment variable only works with the legacy file system local cache.
isaacplmann marked this conversation as resolved.
Show resolved Hide resolved

## Next Steps
isaacplmann marked this conversation as resolved.
Show resolved Hide resolved

In Nx 20, the local cache metadata and project graph are stored in a database, rather than using the file system. (Cache artifacts are still stored on the file system.) In Nx 21 the legacy file system cache will be removed and the database cache will be the only option. The database cache has two benefits:

1. Cache reads and writes are faster.
2. The local cache is more secure since other processes with access to the file system can no longer read or modify the cache.

For most organizations, this feature is a net positive. If you are currently using a custom task runner or the `NX_REJECT_UNKNOWN_LOCAL_CACHE` environment variable, you are most likely providing your own custom [remote cache](/ci/features/remote-cache) storage location. You have several options moving forward:
isaacplmann marked this conversation as resolved.
Show resolved Hide resolved

1. Use [Nx Cloud](/nx-cloud) for your remote cache
2. Use an [Nx Powerpack](/powerpack) plugin to store your remote cache on [Amazon S3](/nx-api/powerpack-s3-cache), [Google Cloud](/nx-api/powerpack-gcs-cache), [Azure](/nx-api/powerpack-azure-cache) or a [network drive](/nx-api/powerpack-shared-fs-cache)
3. Use the legacy file system cache until Nx 21 by setting `useLegacyCache: true` in your `nx.json` file
10 changes: 2 additions & 8 deletions docs/shared/guides/unknown-local-cache.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,14 +44,8 @@ error.

## You Share Cache with Another Machine Using a Network Drive

You can prefix any Nx command with `NX_REJECT_UNKNOWN_LOCAL_CACHE=0` to ignore the errors (
e.g., `NX_REJECT_UNKNOWN_LOCAL_CACHE=0 nx run-many -t build test`). This is similar to
setting `NODE_TLS_REJECT_UNAUTHORIZED=0` to ignore any errors stemming form self-signed certificates. Even though it
will make it work, this approach is discouraged.

Storing Nx's local cache on a network drive can present security risks. When a network drive is shared, every CI run has
access to all the previously created Nx cache artifacts. Hence, it is plausible for every single artifact - for every
single task hash - to be accessed without leaving any trace. This is feasible due to the network drive's capability to
Storing Nx's local cache on a network drive presents security risks, so we prevent you from doing so. When a network drive is shared, every CI run has access to all the previously created Nx cache artifacts. Hence, it is plausible for every single artifact - for every
single task hash - to be accessed without leaving any trace. This is possible due to the network drive's capability to
allow overwrites.

## How Nx Replay Makes Sure Sharing Cache is Safe
Expand Down
2 changes: 1 addition & 1 deletion docs/shared/reference/sitemap.md
Original file line number Diff line number Diff line change
Expand Up @@ -221,7 +221,7 @@
- [workspace.json](/deprecated/workspace-json)
- [As Provided vs. Derived](/deprecated/as-provided-vs-derived)
- [Workspace Generators](/deprecated/workspace-generators)
- [Custom Task Runners](/deprecated/custom-task-runners)
- [Legacy Cache](/deprecated/legacy-cache)
- [Workspace Executors](/deprecated/workspace-executors)
- [runtimeCacheInputs](/deprecated/runtime-cache-inputs)
- [cacheableOperations](/deprecated/cacheable-operations)
Expand Down
1 change: 1 addition & 0 deletions nx-dev/nx-dev/redirect-rules.js
Original file line number Diff line number Diff line change
Expand Up @@ -1099,6 +1099,7 @@ const removedDeprecatedUrls = {
'/recipes/storybook/overview-react', // 417 views: mostly people searching "React Storybook" is matching this outdated page that was for Nx 12.7
'/deprecated/storybook/upgrade-storybook-v6-react':
'/recipes/storybook/overview-react', // 80 views
'/deprecated/custom-task-runners': '/deprecated/legacy-cache',
};

const decisionsSection = {
Expand Down