Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add option elasticsearch-start-es-after-block to es plugin #1458

Merged
merged 2 commits into from
Dec 21, 2018

Conversation

oxarbitrage
Copy link
Member

#1455

Will start inserting after specified block has passed, example usage:

./programs/witness_node/witness_node --plugins "elasticsearch" --elasticsearch-start-es-after-block 100000

In the first commit(5909b13) i add the check for the option right after getting a block.

In the second one(ee4cc4b) i modify that and place it much more inside the plugin logic. This is because i realized we were going to lose ID numeration, undos, total_ops, etc.

So at the end this basically don't add anything to the bulk buffer and by that will not send any data to elasticsearch.
I hope this alone can speed up the process when the node already have elasticsearch data up to a block but need testing. Any help with that is appreciated.

@pmconrad
Copy link
Contributor

Code looks good, but I'm unsure if this is a desirable feature. It's probably a good way to shoot yourself in the foot, by supplying the wrong block number.
How about an automatism? Write database version and current LIB into a special index entry, read that on startup and use the info to decide whether to do a full ES replay or start at the previous LIB.

@clockworkgr
Copy link
Member

Most ES node operators are savvy enough to do this properly imho.

If unsure what you're doing you should always sync from scratch.

The setup doesn't allow duplicates anyway so starting further back is not an issue.

I'm gonna give this patch a go to restart my ES node when the proposal patch is done.

@clockworkgr
Copy link
Member

Appears to be working fine for me btw

Copy link
Contributor

@pmconrad pmconrad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I'll not be standing in the way of a happy user. :-)

@oxarbitrage
Copy link
Member Author

In conversations with @clockworkgr there are several use cases that you want to start adding data to ES after a specific block number such as:

  • accidentally deleted some data.
  • applied extra index or data and want to roll back.
  • remove old data, save space.

Plus the main purpose of bringing a crashed ES node back confirmed by clockwork that was able to bring his es node back in a few mins after mainnet patch was released(#1479)

LIB is a good idea but the specific option should be also available so i am merging this as it is.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants