v0.3.0

This release contains the Memo class for better memory efficiency when benchmarking multiple models or datasets with large memory footprint in succession.

It also introduces Transforms as a way to encapsulate processing of records for multiple purposes, like generating statistics or serialization/deserialization.

What's Changed

• Example Guide: Benchmark on Saved Models by @maxmynter in #92
• Add a lightning demo to the README.md by @nicholasjng in #98
• Hotfix: Adjust snippet paths by @nicholasjng in #102
• Add Parameters to Benchmarks by @maxmynter in #103
• Run pre-commit autoupdate, bump dependencies by @nicholasjng in #104
• Implement a FSspec based ArtifactLoader by @maxmynter in #100
• Add Artifacts guide by @maxmynter in #107
• Remove the ArtifactCollection class by @maxmynter in #109
• Inline the _cleanup function for the FilePathArtifactLoader. by @maxmynter in #111
• Unique keys by @maxmynter in #112
• functools.partialize the nnbench.parametrize decorators. by @maxmynter in #116
• Add transform proposal by @nicholasjng in #113
• Make Transform an abstract base class, delete iapply method ... by @nicholasjng in #118
• Check loaded module names before importing a new one from a file by @nicholasjng in #119
• Rollforward of lakefs-spec doc changes by @nicholasjng in #123
• Add transform submodule, parameter compression transform by @nicholasjng in #124
• Feature: Thunks as an alternative to artifacts by @nicholasjng in #120
• Add new memoization guide by @nicholasjng in #129
• State injection into setup and teardown by @maxmynter in #127
• Add global memo cache and integrate with the setUp and teardown injection by @maxmynter in #130
• Update dev requirements, add Memray artifacts to gitignore by @nicholasjng in #136
• Organize nnbench.types.types into different files by @maxmynter in #135
• Implement memo garbage collection by @nicholasjng in #137
• Update HuggingFace example to use memoization by @nicholasjng in #133

Full Changelog: v0.2.0...v0.3.0

v0.2.0

Summary

This release makes a wide variety of benchmarking functionality available: Contexts for easier metadata organization, providers for basic information, file IO for record persistence on disk, and more.

It also adds new guides and code examples on how to benchmark code easily in a variety of scenarios, such as in orchestrator workflows and with duckDB.

What's Changed

• Add git info and Python package info providers by @nicholasjng in #64
• Group context structs under top-level semantic keys by @nicholasjng in #65
• Simplify nullcols helper, flatten nested context by @nicholasjng in #68
• Fix runner collection by checking file path first by @nicholasjng in #70
• Change nnbench.reporter file to submodule, add base reporter by @nicholasjng in #71
• Inline console reporter into BenchmarkReporter.display() by @nicholasjng in #76
• Orchestrator examples by @maxmynter in #73
• Improve some aspects on benchmark reporters by @nicholasjng in #77
• Add barebones duckDB reporter by @nicholasjng in #75
• Add BenchmarkContext Class by @maxmynter in #79
• Improve file drivers, add dict roundtrip methods to BenchmarkRecord by @nicholasjng in #82
• Unify batching in file drivers, remove read/write stubs on base reporter by @nicholasjng in #83
• Add compression algorithms facility to file IO by @nicholasjng in #84
• Streamlit Integration by @maxmynter in #81
• Add BQ example to showcase cloud streaming of benchmark records by @nicholasjng in #85
• Add parquet, ndjson drivers, change registration mechanisms by @nicholasjng in #87
• Revert file extension coercion, add raise statement for missing driver by @nicholasjng in #88
• Switch author and maintainer description in example .toml's by @maxmynter in #89
• Restructure test suite and add file IO roundtrip tests by @nicholasjng in #90
• Add duckDB example by @nicholasjng in #91
• Run pre-commit autoupdate, upgrade requirement lockfiles by @nicholasjng in #93

Full Changelog: v0.1.0...v0.2.0

v0.1.0

This marks the first stable release of nnbench. From now on, the package is available for installation on PyPI using standard Python dependency management tools like pip and poetry.

Contents

• Definition and parametrization of benchmarks via decorators
• Collection and running in a benchmark runner class
• Result reporting via a benchmark reporter class (currently, only tabular console output is supported.)
• Documentation including quickstart, a more advanced example, and an API reference.

Breaking changes

None.

Deprecations

None.

v0.1.0-rc1

This is a test PyPI release.