elasticsearch history api #1682 #1725
Conversation
libraries/app/api.cpp
Outdated
| if(_app.is_plugin_enabled("elasticsearch")) { | ||
| auto es = _app.get_plugin<elasticsearch::elasticsearch_plugin>("elasticsearch"); | ||
| auto _thread = std::make_shared<fc::thread>("elasticsearch"); | ||
| return _thread->async([&](){ return es->get_account_history(account, stop, limit, start); }, |
There was a problem hiding this comment.
Please capture only what's necessary.
There was a problem hiding this comment.
Also, don't use this if mode is only_save.
There was a problem hiding this comment.
Does this actually work as intended? That would be cool.
We should load-test this before putting it into production.
| { | ||
| "range": { | ||
| "block_data.block_time": { | ||
| "gte": "now-20y", |
There was a problem hiding this comment.
Put a TODO in your calendar for the year 2034: bump this. ;-)
Can't you leave out this part of the query?
There was a problem hiding this comment.
I agree that the range could be omitted here.
Also I am wondering why you do query_string and not
"query": {
"match" : {
"account_history.operation_id" : "operation_id_string"
}
}
There was a problem hiding this comment.
my apologies, this was not a production ready query but just a proof of concept probably pasted from kibana. i had changed this at 1f6ac14
There was a problem hiding this comment.
for the second query i removed the range but query_string needs to stay as we are making a lucene query there and not just matching a field. 63f7aff
| @@ -59,6 +58,8 @@ class elasticsearch_plugin_impl | |||
| std::string _elasticsearch_index_prefix = "bitshares-"; | |||
| bool _elasticsearch_operation_object = false; | |||
| uint32_t _elasticsearch_start_es_after_block = 0; | |||
| bool _elasticsearch_operation_string = true; | |||
| std::string _elasticsearch_mode = "only_save"; | |||
There was a problem hiding this comment.
For more efficient comparison, make this an enum type.
There was a problem hiding this comment.
It is a bit tricky to do this because the boost::program_options will not accept the enum but basically just a string, number or boolean.
I picked the number option to do a static cast to the enum by checking the upper boundary to avoid inserting invalid options.
Implemented at de76301
There was a problem hiding this comment.
changed the exception introduced here to graphene::chain::plugin_exception in the context of 5838a38
|
|
||
| if(my->_elasticsearch_mode != "only_query") { | ||
| if (my->_elasticsearch_mode == "all") | ||
| my->_elasticsearch_operation_string = true; |
There was a problem hiding this comment.
Silently overriding this will cause confusion.
IMO add log output if user has configured this to false. Or perhaps fail to make this very explicit.
There was a problem hiding this comment.
Agree, i am now not allowing the mode to be all if operation_string is false so no silent changes are made.
| ("elasticsearch-mode", boost::program_options::value<std::string>(), | ||
| "Mode of operation: only_save, only_query, all(only_save)") | ||
| ("elasticsearch-mode", boost::program_options::value<uint16_t>(), | ||
| "Mode of operation: only_save(0), only_query(1), all(2) - Default: 0") |
There was a problem hiding this comment.
I'd use values 1,2,3 so the option behaves like a bitset. But that's just me.
| "query": { | ||
| "match": | ||
| { | ||
| "account_history.operation_id": )" + operation_id_string + R"(" |
There was a problem hiding this comment.
A " is missing before )" + operation_id_string.
It seems this code is not used anywhere though.
@sschiessl-bcp I found that you were in the conversation, do you remember anything about this?
#1682
Added
get_operation_by_idas requested at ES Plugin: Get operation by ID #1682 (comment) to elasticsearch plugin.Added
get_account_historyto elasticsearch and use it when plugin is available to serve get_account_history api calls.Test cases added are the same as
history_api_tests/get_account_history_additionalto make sure results are consistent.Considerations:
elasticsearch-operation-stringwas added and is needed to serve the history api calls. Theop_objectfield is modified to fit elasticsearch when indexing, we need the original object in a string to transmit it back on request. This will increase the size of the resources needed for ES but as an optional option users can use the string if they only want the node to serve history api calls or save the object for kibana/others. Can also store the 2 things if he haves the space.Questions/TODO:
get_account_history_operations- should be easy, it is almost the same query asget_account_history.get_relative_account_history- this is a bit trickier to do in ES - do we need it ?get_operation_by_idas api call?