You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
evaluation.json should be made if some ENV variable is set, otherwise pprint something
for each benchmark:
benchmark()
default transform
the dataset
default parameters
documentation:
dataset examples
default transform example
input fed to model, and expected output
link to examples of benchmarked models
a library of transforms (maybe)
And additional requests:
BenchmarkResult return value should also contain: 1) the dataset used, 2) the transform used, 3) input parameters used when invoking the function, 4) anything else - so it's a self-contained record of results
The text was updated successfully, but these errors were encountered:
TODO sotabench lib:
benchmark.py
And additional requests:
BenchmarkResult
return value should also contain: 1) the dataset used, 2) the transform used, 3) input parameters used when invoking the function, 4) anything else - so it's a self-contained record of resultsThe text was updated successfully, but these errors were encountered: