Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add versioning docs #495

Merged
merged 60 commits into from
Oct 19, 2020
Merged
Changes from 1 commit
Commits
Show all changes
60 commits
Select commit Hold shift + click to select a range
a516733
add time and perf benchmark for es
brandenchan Aug 27, 2020
36fed3b
Add retriever benchmarking
Aug 31, 2020
a7cab15
Add Reader benchmarking
Aug 31, 2020
62a571f
Merge branch 'benchmarks' of https://github.com/deepset-ai/haystack i…
brandenchan Sep 4, 2020
7b6e237
add nq to squad conversion
brandenchan Sep 9, 2020
12f10e3
add conversion stats
brandenchan Sep 15, 2020
11a013b
clean benchmarks
brandenchan Sep 16, 2020
b224812
Add link to dataset
brandenchan Sep 16, 2020
aefe431
Merge branch 'master' into benchmarks
brandenchan Sep 18, 2020
a4f7ca2
Update imports
brandenchan Sep 18, 2020
baeae8c
Merge branch 'master' into benchmarks
brandenchan Sep 21, 2020
f7ca141
add first support for neg psgs
brandenchan Sep 21, 2020
3663871
Refactor test
brandenchan Sep 23, 2020
e48b314
set max_seq_len
brandenchan Sep 24, 2020
0ca92e5
cleanup benchmark
brandenchan Sep 24, 2020
abdbed1
Merge branch 'master' into benchmarks
brandenchan Sep 24, 2020
7e31588
begin retriever speed benchmarking
brandenchan Sep 24, 2020
768ad2b
Merge branch 'master' into benchmarks
brandenchan Sep 28, 2020
7611a31
Add support for retriever query index benchmarking
brandenchan Sep 28, 2020
22543be
improve reader eval, retriever speed benchmarking
brandenchan Sep 29, 2020
8f2c3a6
improve retriever speed benchmarking
brandenchan Oct 1, 2020
8b45c3e
Add retriever accuracy benchmark
brandenchan Oct 1, 2020
c238b4f
merge latest master
tholor Oct 2, 2020
03b80f6
Add neg doc shuffling
brandenchan Oct 2, 2020
acaab3c
Merge branch 'benchmarks' of https://github.com/deepset-ai/haystack i…
brandenchan Oct 2, 2020
8b0dc46
Add top_n
brandenchan Oct 5, 2020
5b5f44a
3x speedup of SQL. add postgres docker run. make shuffle neg a param.…
tholor Oct 5, 2020
b90b3c6
Add models to sweep
brandenchan Oct 6, 2020
5eb7aa5
Merge branch 'benchmarks' of https://github.com/deepset-ai/haystack i…
brandenchan Oct 6, 2020
7f70cdb
merge latest master
tholor Oct 6, 2020
331ddf8
Merge branch 'benchmarks' of github.com:deepset-ai/haystack into benc…
tholor Oct 6, 2020
cc44d0d
add option for faiss index type
tholor Oct 6, 2020
009eeb9
remove unneeded line
brandenchan Oct 6, 2020
c2862a9
change faiss to faiss_flat
brandenchan Oct 6, 2020
93090ee
begin automatic benchmark script
brandenchan Oct 6, 2020
7088b57
remove existing postgres docker for benchmarking
tholor Oct 6, 2020
ce055bd
Merge branch 'benchmarks' of github.com:deepset-ai/haystack into benc…
tholor Oct 6, 2020
44376cb
Add data processing scripts
brandenchan Oct 7, 2020
f20e226
Remove shuffle in script bc data already shuffled
brandenchan Oct 7, 2020
dcc79e3
switch hnsw setup from 256 to 128
tholor Oct 7, 2020
0f4a11d
Merge branch 'benchmarks' of github.com:deepset-ai/haystack into benc…
tholor Oct 7, 2020
cabf0e1
change es similarity to dot product by default
brandenchan Oct 7, 2020
af127db
Error includes stack trace
brandenchan Oct 7, 2020
ce2da2a
Change ES default timeout
brandenchan Oct 7, 2020
4ca0cf3
remove delete_docs() from timing for indexing
tholor Oct 8, 2020
3f78eba
Merge branch 'benchmarks' of github.com:deepset-ai/haystack into benc…
tholor Oct 8, 2020
37f3c92
Add support for website export
brandenchan Oct 8, 2020
51c2df7
update website on push to benchmarks
PiffPaffM Oct 8, 2020
8b57f0e
add complete benchmarks results
brandenchan Oct 8, 2020
1e3ee45
Merge branch 'benchmarks' of https://github.com/deepset-ai/haystack i…
brandenchan Oct 8, 2020
b9dc09f
new json format
PiffPaffM Oct 8, 2020
4f272c5
Merge branch 'benchmarks' of https://github.com/deepset-ai/haystack i…
PiffPaffM Oct 8, 2020
716cf7b
removed NaN as is not a valid json token
PiffPaffM Oct 8, 2020
a06f032
versioning for docs
PiffPaffM Oct 8, 2020
ddea3d3
merged master
PiffPaffM Oct 13, 2020
080739e
unsaved changes
PiffPaffM Oct 13, 2020
26b5fc6
cleaning
PiffPaffM Oct 13, 2020
80b61cb
cleaning
PiffPaffM Oct 13, 2020
2cb0295
Edit format of benchmarks data
brandenchan Oct 16, 2020
a6f65b0
update also jsons in v0.4.0
PiffPaffM Oct 16, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add top_n
brandenchan committed Oct 5, 2020

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
commit 8b0dc4638b66509c5ad720f989bc8ebce44b3692
1 change: 1 addition & 0 deletions haystack/reader/farm.py
Original file line number Diff line number Diff line change
@@ -471,6 +471,7 @@ def eval(
"EM": eval_results[0]["EM"],
"f1": eval_results[0]["f1"],
"top_n_accuracy": eval_results[0]["top_n_accuracy"],
"top_n": self.inferencer.model.prediction_heads[0].n_best,
"reader_time": reader_time,
"seconds_per_query": reader_time / n_queries
}
1 change: 1 addition & 0 deletions test/benchmarks/reader.py
Original file line number Diff line number Diff line change
@@ -37,6 +37,7 @@ def benchmark():
results = {'EM': 0.,
'f1': 0.,
'top_n_accuracy': 0.,
'top_n': 0,
'reader_time': 0.,
"passages_per_second": 0.,
"seconds_per_query": 0.,