Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Performance boost #72

Merged
merged 2 commits into from
Mar 17, 2021
Merged

Conversation

dennispo
Copy link
Contributor

@dennispo dennispo commented Mar 14, 2021

Addresses issue #67 .

The main idea in this change is to push the "unpacking" of instances to a later stage. It will happen after the initial filtering of the observables, but before the high level operators, like REPEATS or FOLLOWEDBY.

Based on experiments on real data, created by stix-shifter of low level monitoring events, such as EDRs or Sysmon, the boost in performance can be enormous. As an example, there are 15,754 observables generated out of 100 original observables. In another example, there are 189,023 instances generated out of 300 original observables.

When comparing the two above example with a version without instances duplication, the timing is as follows:

Improvement Time measured for 100 observed_data % of improvement Time measured for 300 observed_data % of improvement
basic 0:04:59.641   1:00:40.532  
events deduplication 0:00:06.043 97.98% 0:00:15.945 99.56%

For the above 2 examples, the performance boost will be x50 and x200 respectively.

@CLAassistant
Copy link

CLAassistant commented Mar 14, 2021

CLA assistant check
All committers have signed the CLA.

@codecov-io
Copy link

codecov-io commented Mar 14, 2021

Codecov Report

Merging #72 (654591f) into master (20b7d86) will increase coverage by 0.11%.
The diff coverage is 94.11%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master      #72      +/-   ##
==========================================
+ Coverage   90.00%   90.11%   +0.11%     
==========================================
  Files          13       13              
  Lines        1070     1082      +12     
==========================================
+ Hits          963      975      +12     
  Misses        107      107              
Impacted Files Coverage Δ
stix2matcher/test/test_complex.py 100.00% <ø> (+4.54%) ⬆️
stix2matcher/matcher.py 87.23% <94.11%> (+0.07%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 20b7d86...654591f. Read the comment docs.

@mdazam1942 mdazam1942 merged commit 59d13c7 into oasis-open:master Mar 17, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants