Skip to content

Commit

Permalink
[Benchmark] Remove ingest results collection (#272)
Browse files Browse the repository at this point in the history
Removes collecting ingest results from the benchmarking tool. On big
data sets, this will prevent the process from going out of memory.

Signed-off-by: John Mazanec <[email protected]>
  • Loading branch information
jmazanec15 authored Jan 28, 2022
1 parent 7e9d4c5 commit c46b3de
Showing 1 changed file with 3 additions and 3 deletions.
6 changes: 3 additions & 3 deletions benchmarks/perf-tool/okpt/test/steps/steps.py
Original file line number Diff line number Diff line change
Expand Up @@ -287,14 +287,14 @@ def _action(self):
def action(doc_id):
return {'index': {'_index': self.index_name, '_id': doc_id}}

index_responses = []
# Maintain minimal state outside of this loop. For large data sets, too
# much state may cause out of memory failure
for i in range(0, self.doc_count, self.bulk_size):
partition = self.dataset.read(self.bulk_size)
if partition is None:
break
body = bulk_transform(partition, self.field_name, action, i)
result = bulk_index(self.opensearch, self.index_name, body)
index_responses.append(result)
bulk_index(self.opensearch, self.index_name, body)

self.dataset.reset()

Expand Down

0 comments on commit c46b3de

Please sign in to comment.