get_key_references performance issue?

[abitmore]

Bug Description
If a key is in authorities of many accounts, get_key_references API will return all of them and will subscribe to all of them, this could lead to potential performance issues. This API is used by GUI to get related accounts when importing a key.

bitshares/bitshares-ui#2367 is related.

Update:
due to a bug, (I believe) the API is not subscribing to the returned accounts, so we can

• completely remove the code about subscribing to accounts

The result can still be too large, E.G. containing thousands of accounts, thus can be problematic.

• cap size of returned data?

Impacts
Describe which portion(s) of BitShares Core may be impacted by this bug. Please tick at least one box.

• API (the application programming interface)
• Build (the build process or something prior to compiled code)
• CLI (the command line wallet)
• Deployment (the deployment process after building such as Docker, Travis, etc.)
• DEX (the Decentralized EXchange, market engine, etc.)
• P2P (the peer-to-peer network for transaction/block propagation)
• Performance (system or user efficiency, etc.)
• Protocol (the blockchain logic, consensus, validation, etc.)
• Security (the security of system or user data, etc.)
• UX (the User Experience)
• Other (please add below)

CORE TEAM TASK LIST

• Evaluate / Prioritize Bug Report
• Refine User Stories / Requirements
• Define Test Cases
• Design / Develop Solution
• Perform QA/Testing
• Update Documentation

[xqcool]

IMO，it is api design issue，unless you can limit the searching depth and get the right result.

[clockworkgr]

wasn't this always standard behaviour? has something changed?

[jmjatlanta]

Are you worried about the performance of querying for all of them? Or are you worried about the performance of all those subscriptions? Or both?

If we limit the subscriptions, how would we communicate to the user that may be expecting us to subscribe to all of them?

[abitmore]

This issue is related to UI issue bitshares/bitshares-ui#2367. Please check there for more info.

[abitmore]

get_full_accounts API has a limitation when subscribing to accounts, but get_key_references doesn't. IMHO we should apply same limitation to the two APIs.

• get_full_accounts API:

  bitshares-core/libraries/app/database_api.cpp

  Lines 767 to 773 in cca5fd8

  	if( subscribe )
  	{
  	if(_subscribed_accounts.size() < 100) {
  	_subscribed_accounts.insert( account->get_id() );
  	subscribe_to_item( account->id );
  	}
  	}

• get_key_references API:

  bitshares-core/libraries/app/database_api.cpp

  Lines 597 to 598 in cca5fd8

  	for( auto i : final_result )
  	subscribe_to_item(i);

[jmjatlanta]

i is a vector<graphene::chain::account_id_type>, which according to subscribe_to_item gets fc::raw::pack'd, then inserted into the _subscribe_filter bloom filter.

I would be surprised if that works, as you are subscribing to a vector, and not individual accounts. Is there some magic happening here, or is my surprise warranted?

[abitmore]

... you are subscribing to a vector ...

LOL. I don't think it will work. So, due to this, we effectively didn't subscribe to anything? As discussed in the pull request, let's remove the code completely.

[abitmore]

I think it's still problematic if the returned nested vector is too large. OP updated.

[clockworkgr]

thing is, UI still does a get_full_accounts on them after received...so it effectively/eventually does subscribe to all of them.

we've had a discussion on UI group about this and it seems this will need some major refactoring (to fix other performance issues as well)

[abitmore]

Every account ID in returned vector is around 16 bytes, so if the JSON-RPC interface is able to return data packets greater than 1M bytes, that means 64K accounts, so it is a bit hard to attack although possible.

[jmjatlanta]

This has been fixed by PR #1499

Note: There are related outstanding PRs at #782 #1513

[manikey123]

@jmjatlanta @pmconrad is this hard coded part to be changed and added to the config file?
53feeff

[jmjatlanta]

@jmjatlanta @pmconrad is this hard coded part to be changed and added to the config file?
53feeff

Yes. We hard-coded that the user should be limited to passing no more than 100 keys. We would like that to be configurable, so that a node can easily customize the quantity their client can request.