BOT Caching

At the moment, a cache concept appears in numerous ways across the BOT, with both user-facing aspects & developer-facing aspects

User Facing

The Cache folder is (and increasingly needs to be) user-facing, to allow configuration of its location & sometimes manual operations.

Developer Facing

File Based Caching

The bot.internal.cache object manages file-based caching including:

• Item Manifest caching
• Data Table Caching
• Data File Caching

Item Table ("Manifest") Caching

Tables of BOT items (aka "item manifests") are cached locally as the "object cache" as follows:

• The object cache is a single .MAT file whose variables map each dataset to its own MAT file
• Each dataset MAT files stores the dataset tables as variables

This caching is implemented in a general way via the bot.internal.ObjectCacher; but appears only to be used for item table caching.

Data Table Caching

Tables returned by the API (other than the item tables) are cached locally in the "cloud cache" as follows:

• The cloud cache is a single .MAT file whose variables map API calls to files
• Retrieved table contents are stored in MAT files as structures

Data File Caching

Retrieved data files (NWB and otherwise) are cached locally in the "cloud cache" as follows:

• The cloud cache is a single .MAT file whose variables map API calls to files
• Retrieved files are stored directly within the "external" folder hierarchy

In-Memory Caching

Item Table Caching

Item tables are further cached in memory (i.e., within a MATLAB session) using memoize within the respective bot.item.internal.XXXManifest classes

This includes in-memory memoization of "annotated" & otherwise updated item tables for display

On Demand Property Caching

Lazy-load property values* are stored in the property_cache property of the OnDemandProps mixin (bot.item.mixin.OnDemandProps)

(*) At moment specifically for linked file properties

Resetting Cache

There are two internal methods for clearing the cache (cloud and object cache):

1. bot.internal.cache.clearInMemoryCache() Clears the cache from memory by resetting singleton instances, persistent variables and memoized functions.

2. bot.internal.cache.reset() This will completely reset the cache, by first clearing the cache from memory using the function above AND ALSO delete all the cached content from disk.

It is also possible to use the method bot.internal.reset. This, in addition to resetting the cache (point 2), also resets the preferences.

Future Possibilities

Property Value File Caching (Issue #79]

In principle it could make sense to cache retrieved 'on demand' property values for use in future BOT sessions.

On-Demand Property Cache Display (Issue #75]

Maybe file size can be used for coarse estimate of property retrieval time? Issue note suggests possible ambiguity in how LFP/CSD are displayed?

Open Questions

• Why are experiments/cells/sessions not handled in a uniform way?
• Does "Manifest' uniquely refer to item tables? If so, why is there a SaveManifest in both CloudCacher & ObjectCacher classes?

Answered Questions

• Confirm fetch_ABO_query() is defunct? ** Confirmed, it appears that way
• Why are retrieved tables stored as structures? Are these created by BOT or how webread reads the RESTful response directly? ** The latter, driven by webread
• Clarify the original & annotated tables for Ephys ** Annotated tables are joined tables with some other updates (renamed columns, added columns w computed values)