Skip to content
This repository has been archived by the owner on Sep 2, 2020. It is now read-only.

engine: simplify in-memory aggregation logic. add instrumentation. #134

Merged
merged 1 commit into from
Oct 6, 2015

Conversation

pyr
Copy link
Owner

@pyr pyr commented Oct 6, 2015

This commit brings in a number of changes:

  • Handle input drift globally. A new component is introduced.
    to facilitate testing, the wall-clock is abstrated into
    its own component as well which the drift component depends
    on.
  • Simplify the way the engine holds on to values in memory.
    Values are stored in an atom and updates trigger a
    swap! on the atom, yielding snapshots.
  • Move queueing mechanism into a proper component
  • Allow per-queue capacity and pool-size parameters. A good
    rule of thumb is to make room for twice your average input
    size, which gives the system some time to cope with bursts.
  • Instrument critical operations, such as taking snapshots
    these are available as JMX by default. Configuration support
    will be provided to handle reporting destinations.

This commit brings in a number of changes:

- [X] Handle input drift globally. A new component is introduced.
  to facilitate testing, the wall-clock is abstrated into
  its own component as well which the drift component depends
  on.
- [X] Simplify the way the engine holds on to values in memory.
  Values are stored in an atom and updates trigger a
  `swap!` on the atom, yielding snapshots.
- [X] Move queueing mechanism into a proper component
- [X] Allow per-queue capacity and pool-size parameters. A good
  rule of thumb is to make room for twice your average input
  size, which gives the system some time to cope with bursts.
- [X] Instrument critical operations, such as taking snapshots
  these are available as JMX by default. Configuration support
  will be provided to handle reporting destinations.
@pyr
Copy link
Owner Author

pyr commented Oct 6, 2015

This takes care of #121 which is able to handle large loads without suffering at -Xmx512m additional tuning will make it possible to go further.

It should be noted that the default in-memory store doesn't cope too well with large payloads and will be adapted. In the mean-time large clusters are encouraged to rely on the elasticsearch index.

pyr added a commit that referenced this pull request Oct 6, 2015
engine: simplify in-memory aggregation logic. add instrumentation.
@pyr pyr merged commit cf51549 into master Oct 6, 2015
@pyr pyr deleted the wip/instrumented branch October 6, 2015 13:53
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant