Skip to content

Commit

Permalink
Updates to match the latest interfaces in tatami's core. (#10)
Browse files Browse the repository at this point in the history
This mostly involves transforming the Custom*ChunkedMatrix classes to comply
with the new interfaces, though the oracle-aware caches also need some updates
to use a random-access oracle instead of expecting a stream of predictions.
The test suite has also been upgraded to use the latest tatami_test functions.

We create dedicated Mock*Chunk classes to better document the chunk
requirements for the Custom*ChunkedMatrix classes. Chunk expectations have also
been simplified to remove the need for an explicit single-element extract()
function, and to better distinguish between simple and subsettable chunks.
  • Loading branch information
LTLA authored Apr 5, 2024
1 parent 56b2c87 commit 761aa6c
Show file tree
Hide file tree
Showing 26 changed files with 5,025 additions and 2,527 deletions.
12 changes: 10 additions & 2 deletions .github/workflows/run-tests.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -43,8 +43,16 @@ jobs:

- name: Run the tests
run: |
cd build
ctest
cd build/tests
# Avoid using ctest because it's so slow; it starts a
# new process for each individual test, which is crazy.
for exe in $(ctest --show-only=json-v1 | jq -r ".tests[] | .command | .[0]" | sort | uniq)
do
echo "#### RUNNING ${exe} ####"
echo
${exe} --gtest_brief=1
echo
done
- name: Generate code coverage
if: ${{ matrix.config.cov }}
Expand Down
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -2,3 +2,4 @@ build/
*.swp
docs/html
docs/latex
docs/*.tag
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,14 +21,14 @@ We typically want to load and cache an entire slab at once, ensuring that future
The **tatami_chunked** library provides the `LruSlabCache` and `OracleSlabCache` classes to facilitate caching of the slabs in `tatami::Matrix` extractors.
The `TypicalSlabCacheWorkspace` class allows developers to easily switch between caching strategies, depending on whether an oracle is provided to predict the future access pattern.

The `CustomChunkedDenseMatrix` and `CustomChunkedSparseMatrix` classes implement the `tatami::Matrix` interface on top of a matrix of custom chunks.
These classes automatically perform slab caching given a set of options including the maximum cache size (see the `CustomChunkedOptions` class).
The `CustomDenseChunkedMatrix` and `CustomSparseChunkedMatrix` classes implement the `tatami::Matrix` interface on top of a matrix of custom chunks.
These classes automatically perform slab caching given a set of options including the maximum cache size (see the `CustomDenseChunkedOptions` and `CustomSparseChunkedOptions` classes).
Developers can use this to quickly create matrix representations with arbitrary chunk compression schemes that can reduce the memory footprint, e.g., DEFLATE, run length encodings.
Obviously, this comes at the cost of speed whereby the chunks must be unpacked to extract the relevant data -
developers are expected to define an appropriate extraction method for dense/sparse chunks.

In simple cases, chunk extraction is "atomic", i.e., the entire chunk must be unpacked to extract a subset of data.
Developers can then use the `SimpleDenseChunkWrapper` and `SimpleSparseChunkWrapper` to wrap these simple chunks for use in the `CustomChunked*Matrix` classes.
Developers can then use the `SimpleDenseChunkWrapper` and `SimpleSparseChunkWrapper` to wrap these simple chunks for use in the `Custom*ChunkedMatrix` classes.
These wrappers only need a method to inflate the entire chunk; they will automatically handle the extraction of the desired block/subset from each chunk.
(More advanced developers may prefer to write their own extraction methods that avoid inflating the entire chunk, in which case these wrappers are not necessary.)

Expand Down
898 changes: 0 additions & 898 deletions include/tatami_chunked/CustomChunkedMatrix.hpp

This file was deleted.

Loading

0 comments on commit 761aa6c

Please sign in to comment.