Memory-friendly per-query disk caching API

[scarlac]

The current persistor API and experimental implementations for both web and React Native will work for minor use cases, however the general approach isn't easy to make performant as it assumes the entire state should be persisted on every change and is always available after initial restoration.

The current API looks like this:

export interface Persistor {
    persistClient(persistClient: PersistedClient): Promisable<void>;
    restoreClient(): Promisable<PersistedClient | undefined>;
    removeClient(): Promisable<void>;
}

Even when throttling the persistClient() function, the entire app state can easily be several megabytes of data. If any part of your app changes this causes several megabytes worth of JSON serialization which is unfortunately very costly in both terms of CPU, battery use, and as a result the UX will degrade.

The issue isn't unique to React Query though. Redux Persist tries to mitigate this issue by allowing setting a custom persistor for each reducer. This approach is also not sustainable when 1 payload can be 1 MB, and you potentially need to store several.

On web (let's face it: Desktop machines) apps typically are short lived, counted in seconds and perhaps a minute or two. In React Native, apps can live on for hours, and with a persistor, days. In these situations it's favorable that we have a better API for persisting according to the cache key of the data.

E.g. (pseudo-code) instead of caching as such:

await AsyncStorage.setItem('REACT_QUERY_OFFLINE_CACHE', JSON.stringify(client));

You'd store each piece of data as such:

for(const query of clientState.queries) {
  await AsyncStorage.setItem('RC_${query.queryHash}', JSON.stringify(query.data));
}

This can be implemented using the current API, but it would cause re-writing of the entire cache, for every query data entry. It makes the app more responsive because every item is awaited but overall performance is roughly the same (or slightly worse because of await having a slight overhead).

This general challenge is not unique to React Query. Redux Persist has it too. They allow breaking reducers into separate storage configurations such that you can store data separately and avoid having to re-serialize everything even for small changes. However, this approach also doesn't address storage of individual items - you have to persist the entire reducer so if your data is stored as such:

myReducer: {
  byId: {
    'id-42': { id: 42, username: 'scarlac' },
    'id-21': { id: 21, username: 'example' },
    // ... [and 1000 more items]
  }
}

Then you need to re-serialize myReducer in it's entirety even if only id-42 changes.
Ideally this would also allow the cache to be filled on-demand, such that byId could start out empty (byId: {}) and then get filled in by calling the persistor API to check if there is a disk cache item with that particular key before triggering a refetch.

I understand that the current API is experimental. I think now is a fantastic time to consider this use case for mobile apps where memory is severely constrained but disk usage isn't.

[henrymoulton]

@scarlac I was also thinking along these lines and noticed #2864 was merged to potentially allow for custom functions for serialisation/deserialisation. Would that help?

[scarlac]

@henrymoulton Unfortunately it wouldn't work well because the serialize/deserialize is a store-everything approach. With a big cache this becomes a performance issue/it doesn't scale. The real underlying problem is that the queryCache is synchronous and assumed to always be in memory. If you really truly use RQ on a big application you'll find that you end up storing too much in memory and that becomes the problem: There is no way to "off-load" to a permanent storage, let alone have the permanent storage be garbage collected (because the cacheTime GC assumes it's in memory as well).

I actually discussed this with Tanner and he suggested to hook into the query events, which allows me to selectively create, inject, and store to disk. However, it breaks anything relating to cacheTime, and there are other downsides as well (e.g. you have to manually delete query data while still keeping metadata, otherwise some queries may break).

[gabrielbull]

I'm having the same issue as well. I cannot use react-query's persistor on a native app because it will use too much memory and the OS will terminate the app. Since most native apps need to be offline-first, this makes react-query unusable in my case. It's unfortunate as I really like the library but for now I cannot use it and directly use axios with realm.js.

[henrymoulton]

@gabrielbull how much memory is this app that you're using?

[gabrielbull]

The data itself is not that big, sometimes only a few MBs. It seems to get exponential when used with react-query though, a few MBs of data = hundreds of MBs of memory usage.

[TkDodo]

@scarlac I'm open for suggestions on how to improve this for v4. Just so that I understand what we'd want here:

• still store the whole cache / whichever queries are chosen via dehydrateOptions, but store them on a per-query basis
• we'd still want to restore all entries on page load? Note that I have some changes for that api here: feat(persistQueryClient): PersistQueryClientProvider #3248
• when one query updates, we don't write everything, but only the affected entries

I guess this wouldn't alleviate the memory pain much, as we would still have the cache in memory. I guess unused queries could be cleared out by setting cacheTime - but I assume this would also delete it from the persisted cache?

Maybe we would need to find a way to keep queries persisted while having them cleared from memory. I'm open to suggestions here :)

---

@gabrielbull react-query only keeps one copy per query of your data in memory, so I'm not sure how a few MB of data can lead to hundreds of MB memory usage 🤔

[TkDodo]

RQ uses undefined itself already, so I'm not sure how it could be removed. If a query doesn't exist, what does RQ v4 then return on the first render?

I specifically said for successful queries - so queries that are in the _success_state. Of course loading queries will return undefined. You can read here for the reasoning behind it: #2925

Reg cacheTime:0, yes this is because of the way cacheTime works: it's not the total cache time, it's relative to whenever a query becomes unused. I had to read the source to fully understand this but if set to 0, the cache remains in memory until all observers are gone, and all outstanding promises are resolved. Once resolved, it's garbage collected using a simply setTimeout and cacheTime. At least, this is how I understand it.

Your understanding is correct, which is precisely why I don't understand how this works, especially this part about the observerRemoved event: https://gist.github.com/scarlac/34f5bd183d849086654400367817f09b#file-memory-efficient-rq-client-tsx-L106-L111

If you remove the last observer, cacheTime: 0 will completely remove the query for you from the cache.

---

I generally like this idea, just have to get my head around how it exactly works and how we could make it a bit "less hacky" :) But it has potential to become a separate plugin, like a LazyPersisterPlugin or so :)

[scarlac]

Sorry, my comment on cacheTime:0 was made mostly as a theory. In practice I have supplied random values so this has mostly been the default. I have not thoroughly tested the 0-value. It's entirely possible it doesn't work reliably. If I recall correctly, the cacheTime & GC only comes into play with 0 observers and a non-loading query. So if a query was in a loading state, the GC is postponed and re-tried later.

Yeah - the idea was actually to abstract the "RNFS" (React Native File System) stuff and make a "plugin" out of this, allowing any type of per-query persistence and memory efficient use of RQ.
Perhaps all we do here is end up agreeing that "empty queries" are okay, and then figure out how we can deal with these lingering "empty" & slow queries so we get full efficiency.

I'm generally OKAY with not making RQ's cache async if we get more control over garbage collection and somehow figure out how to deal with removing queries that are technically still in flight (e.g. so they get re-created temporarily once resolved, or any method that allows the plugin to deal with it). That's really the limitation that this "plugin" can't deal with as it stands.

[henrymoulton]

There isn't a trend to move to synchronous code in RN. MMKV uses an unofficial API (JSI) and also uses an in memory cache as well, as well as writing to disk. I would not advise a sync persistence API.

Trying to stay on-topic but my opinion is that there is a move towards more synchronous code for persistence in RN:
https://github.com/greentriangle/react-native-leveldb
https://medium.com/@Sendbird/extreme-optimization-of-asyncstorage-in-react-native-b2a1e0107b34
https://github.com/ospfranco/react-native-quick-sqlite

and in Node too:
https://github.com/JoshuaWise/better-sqlite3

Not sure what you mean by JSI being "unofficial", but overall sounds like you might have some interesting thoughts and would love to hear them.

[scarlac]

@henrymoulton Would love to talk about it but this isn't the forum. To quote Tk for a second, it's easy to go sync -> async, but not the other way around. However right now the main issue (for this discussion) is that the arch just doesn't lend itself to be persisted at all.

@TkDodo Do you think there is a chance v4 would see changes that allowed us deeper integration?

[KannarFr]

@TkDodo here is a little reminder.

Also, if it won't be implemented in v4. I think we should warn users that offlineFirst mode can be very tricky without enough place to store data on mobile storage.

We should choose/define the way to implement it. I'll be glad to work on it if needed. TBH, I discovered RQv4 and rewrote all my application using offlineFirst and it's now "stuck" because I can't store enough required data on mobile storage (yet!).

Maybe a quick win could be storing queryData (only) on the mobile's filesystem when offlineFirst is enabled, or when a threshold is reached. Just dirty suggestions which can drive us to nicer ideas.

[jonathanstanley]

To try and narrow towards a proposal:

Memory management

To actually reduce memory, RQ would probably need to introduce additional timers in useQuery.

memoryTime

If this expired, it would trigger the item to be dropped from memory. It could remain in the persisted state, so long as cacheTime has not expired. However, this is tricky:

1. Somehow the Persister needs to be able to keep the item
2. There has to be an async lookup to the persisted data... which could be tricky for state. Maybe its state must be treated just like a new query for RQ with a network call (but instead uses the local persisted cache first / offline mode).
3. Default probably should be Infinity for backwards compatibility

persistTime

In some cases, a user may want to keep some data forever while in an memory. However, once persisted, the app may prefer that the persisted data isn't used after some limit (ex: 30 days). To be clear about security: I'm not aware of any solution that can guarantee a local cache is deleted at a certain time.

Individual persistence

By persisting each item separately instead of bundling in PersistedClient, we solve the memoryTime issue 1 above. dropping an item from memory could be independent of whatever is kept in memory. FWIW, Google also advises this strategy:

Instead of storing the entire state tree in a single record, you should break it up into individual records and only update the records that actually change.

However, this could introduce a bunch of transactions into overhead; perhaps slowing performance in some cases. Without knowing the use and more specific test, we won't know which is most efficient. One way to handle would be to add an option on useQuery so devs can choose for themselves:

individualPersistence:boolean

false all bundles into the main PersistClient (as currently done)
true persisted independently and not included in the main PersistClient

IndexedDB

If any part of your app changes this causes several megabytes worth of JSON serialization which is unfortunately very costly in both terms of CPU, battery use, and as a result the UX will degrade.

Persisting items individually would definitely help. Furthermore, by using the IndexedDB persister (see #3198 ), the stringify penalty is avoided. It still requires a structured clone, but this should be much faster.

Summary

In short, add three options to useQuery:

1. memoryTime:number
2. persistTime:number
3. individualPersistence:boolean

[teaguestockwell]

Instead of rehydrating queries with the persister, would it be possible to create a query cache with a persistent synchronous storage client?

What if there was a query cache that replaced it's references to queries and queriesMap with something that:

• could read from the disk, then rehydrate a query
• could dehydrate a query and save it to the disk

Looking at the query cache, there are a few methods that expect the queries to be hydrated:

1. remove(): calls query.destroy
2. get(): This may be able to be replaced by a lookup to disk followed by rehydrating the query?
3. getAll(): This looks like it's primarily used to show queries in the dev tools, and to determine if the query should be dehydrated. For the dev tools, we may be able to get away with only returning only the active queries as these should be small enough to fit into memory. And using the query cache as the persistence layer we would not need to persist any queries through the persister.

I'm not certain how having to rehydrate queries would effect onFocus and onOnline, or if there is a way to only have to call these methods on only active queries?

[scarlac]

@TkDodo Any major changes in v4 that could potentially carve a path for a lazy loaded query cache, and/or persisting/restoring data per-query?
I read the 'migrating to v4' article and your blog post but I don't see the offline mode as a step towards this but I did see some changes to persistence/persistor that I am hoping makes some of these things easier?

[TkDodo]

no, we didn't do this in v4 because I don't know how the apis should be yet. The persisters are designed to just sync the current QueryCache somewhere. The QueryCache currently contains all the queries in-memory.

My gut feeling says to decouple the requirements here from persisters. The persisters should stay as they are (write the whole cache). However, we can think about implementing a different cache. Like, one that doesn't store everything in-memory, but only keeps a bit in-memory and writes the rest somewhere else.

Then you could do:

const queryClient = new QueryClient({
-  queryCache: new QueryCache()
+  queryCache: new AsyncQueryCache({ some options })
})

that cache could e.g. only keep all keys in memory, and maybe a bunch of values with a max-size; if data is not there, it would read it from an external source; the drawback would be that the query would be in hard loading state while data would be read, but I think this is a wanted trade-off here. What do you think?

[scarlac]

Generally speaking that sounds like the right approach @TkDodo
The only thing I could fault, but I might be misunderstanding you:

[...] write the whole cache [...]

I think we need a per-item cache write functionality. This could in theory be implemented by the AsyncQueryCache class but the challenge is the before/after comparisons you'd have to do. This would cause a constant overhead where one has to iterate over all cache items, restore them from disk, compare semi-deeply (because references can't be compared), and keep doing that over and over. If each disk read+comparison takes e.g. 10ms per disk item, that quickly becomes too slow to use in practice.
So: Having AsyncQueryCache being informed of which items change would be important to achieve any kind of performance.

Anyway, I think we're on the right path then! I can't quite figure out how to structure an API that would make everyone happy here, so perhaps we can get more concrete now with your idea in mind?

[TkDodo]

I think we need a per-item cache write functionality

Not sure about that. The idea was to keep the in-memory cache small right? like:

queryCache: new AsyncQueryCache({ threshold: 7 })

or something. Then, writing the whole cache would only persist those 7 entries that are currently in memory. All other items are offloaded to the async cache storage anyways

[TkDodo]

We've created an issue to track this for v5 here:

• Investigation: fine-grained async control over caching behaviour #4693

[TkDodo]

We've shipped this in v5.0.0-beta.28:

https://tanstack.com/query/v5/docs/react/plugins/createPersister

PLEASE try it out and give feedback, because there's still time to change things now :)

[shashankitsoft]

@TkDodo
It is good that v5 considered for such API. There I see the filters option to narrow down which Queries should be persisted. But the problem with this is we have to mention all query keys at one place ex App.js where we wrap root in PersistQueryClientProvider.
There can be many queries used in react app in different components where we call like useQuery({ queryKey, queryFn, ...options }).
And using 'filters' to define all queryKeys at one place in top root component to include is weird. It would be better and more practical if we can pass 'persist' flag true or false in useQuery options, so that we can individually define which query data we want to be in persistent storage and which not.
This will be useful and easier if we only want to save few query in local storage and not others (or vice-versa) specifying there itself while api/useQuery call, without need to put all those names in root component.

Is this possible?

[TkDodo]

I'm not sure what you're talking about exactly 😅 . With the new persister plugin, you can pass a persister directly to useQuery if you want, so it will only be for those queries. It doesn't use or need the PersistQueryClientProvider at all.

If you use the provider, you can use the meta flag to opt queries in or out of persistence

[scarlac]

@TkDodo I'm late to the after-party but I've reviewed the APIs and it's a step in the right direction! It gets us to a point where I don't have to override onFetch() but it gets us no further than that. Since we last discussed this last time, I had modified my implementation from using the internal events to using onFetch on the queryClient like so:

getQueryClient().setDefaultOptions({
      ...defaults,
      queries: {
        ...defaults.queries,
        behavior: {
          ...defaults.queries?.behavior,
          onFetch: (context) => {
            const fetchFn = context.fetchFn as () => unknown;
            context.fetchFn = async (...args) => {
            // ...code ala "await AsyncStorage.setItem(...)" goes here

This is less than 100 lines of code to get to the same functionality as there is in the experimental APIs, I think.

The complaints I see around setQueryData not persisting either seem quite legitimate. I reviewed a bunch of existing code in our code bases, and simply using the experimental API would yield the same sort of bugs as people are reporting.
To avoid rehashing that entire discussion here, I'd like to just thank you and the team for how far we've come, but just let you know that I don't think we're found "the" solution yet. I do understand the problems around making everything async, so I'll leave it at that.

[cah4a]

I have the same issue.

The more cache I want to store the more app performance drops.
Only way to have both is to have serialization and saving per query.
I throw react-async-storage-persister away and wrote my own implementation.

Which is working pretty flawlessly:

export const queriesStorage = new MMKV({ id: "queries" });
const queriesTimestamps = new Map<string, number>();
const xj = new XtraJson();

export const queryClientPersister: Persister = {
   persistClient: throttle((client) => {
      const time = Date.now();
      let updated = 0;
      let deleted = 0;

      queriesStorage.set("BUSTER", client.buster);
      queriesStorage.set("TIMESTAMP", client.timestamp);
      const keysToLive = new Set(["BUSTER", "TIMESTAMP"]);

      for (const query of client.clientState.queries) {
         if (query.state.status === "success" && !query.state.isInvalidated) {
            keysToLive.add(`QUERY_${query.queryHash}`);

            const updatedAt = queriesTimestamps.get(query.queryHash) ?? 0;

            if (updatedAt < query.state.dataUpdatedAt) {
               updated++;
               queriesTimestamps.set(
                  query.queryHash,
                  query.state.dataUpdatedAt,
               );
               const data = xj.stringify({
                  queryKey: query.queryKey,
                  state: query.state,
               });
               queriesStorage.set(`QUERY_${query.queryHash}`, data);
            }
         }
      }

      for (const key of queriesStorage.getAllKeys()) {
         if (!keysToLive.has(key)) {
            deleted++;
            queriesStorage.delete(key);
         }
      }

      console.log(
         `[queries] Saved in ${Date.now() - time}ms. Items count: ${keysToLive.size}. Updated: ${updated}. Deleted ${deleted}`,
      );
   }),
   restoreClient() {
      const time = Date.now();

      const result = {
         buster: queriesStorage.getString("BUSTER") ?? "",
         timestamp: queriesStorage.getNumber("TIMESTAMP") ?? 0,
         clientState: {
            queries: queriesStorage
               .getAllKeys()
               .filter((key) => key.startsWith("QUERY_"))
               .map((key) => {
                  const queryHash = key.replace("QUERY_", "");
                  const { queryKey, state } = xj.parse(
                     queriesStorage.getString(key) ?? "{}",
                  ) as {
                     queryKey: string[];
                     state: QueryState;
                  };

                  queriesTimestamps.set(queryHash, state.dataUpdatedAt);

                  return {
                     queryHash,
                     queryKey,
                     state,
                  };
               }),
            mutations: [],
         },
      };
      console.log(
         `[queries] Restored in ${Date.now() - time}ms. Items count: ${result.clientState.queries.length}`,
      );
      return result;
   },
   removeClient() {
      for (const key of queriesStorage.getAllKeys()) {
         queriesStorage.delete(key);
      }
      queriesTimestamps.clear();
      console.log("[queries] purged");
   },
};

function throttle<TArgs extends Array<unknown>>(
   func: (...args: TArgs) => void,
   wait = 100,
) {
   let timer: ReturnType<typeof setTimeout> | null = null;
   let params: TArgs;
   return function (...args: TArgs) {
      params = args;
      if (timer === null) {
         timer = setTimeout(() => {
            func(...params);
            timer = null;
         }, wait);
      }
   };
}

[frederikhors]

Did you tried https://tanstack.com/query/latest/docs/framework/react/plugins/createPersister?

[cah4a]

It's definitely a step forward, however it doesn't solve all issues for me.

It doesn't know about all queries and can't effectively purge stale data.
Storage size will never goes down in size as maxAged data is immediately replaced with fresh data from server.

[frederikhors]

Yeah, we have #7049 for this.