Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add cache to native contract executions V2 #1766

Merged
merged 11 commits into from
Jul 17, 2020

Conversation

shargon
Copy link
Member

@shargon shargon commented Jul 14, 2020

Close #1764
Other version for share the snapshot

Based on #1764 (comment)

This cache avoid the second deserialization process

Benchmarks:

[TestMethod]
        public void TestGetValidators1()
        {
            var snapshot = Blockchain.Singleton.GetSnapshot();

            using (ScriptBuilder sb = new ScriptBuilder())
            {
                sb.EmitAppCall(NativeContract.NEO.Hash, "getNextBlockValidators", new object[0]);

                // Fake validators

                StorageItem storage = snapshot.Storages.GetAndChange(CreateStorageKey(14), () => new StorageItem());
                storage.Value = NativeContract.NEO.GetValidators(snapshot).ToByteArray();

                // Start benchmark

                Stopwatch sw = Stopwatch.StartNew();

                for (int x = 0; x < 1_000; x++)
                {
                    using var engine = ApplicationEngine.Run(sb.ToArray(), snapshot, testMode: true);

                    var result = engine.ResultStack.Peek();
                    result.GetType().Should().Be(typeof(VM.Types.Array));
                    ((VM.Types.Array)result).Count.Should().Be(7);
                    ((VM.Types.Array)result)[0].GetSpan().ToHexString().Should().Be("02486fd15702c4490a26703112a5cc1d0923fd697a33406bd5a1c00e0013b09a70");
                    ((VM.Types.Array)result)[1].GetSpan().ToHexString().Should().Be("024c7b7fb6c310fccf1ba33b082519d82964ea93868d676662d4a59ad548df0e7d");
                    ((VM.Types.Array)result)[2].GetSpan().ToHexString().Should().Be("02aaec38470f6aad0042c6e877cfd8087d2676b0f516fddd362801b9bd3936399e");
                    ((VM.Types.Array)result)[3].GetSpan().ToHexString().Should().Be("03b209fd4f53a7170ea4444e0cb0a6bb6a53c2bd016926989cf85f9b0fba17a70c");
                    ((VM.Types.Array)result)[4].GetSpan().ToHexString().Should().Be("03b8d9d5771d8f513aa0869b9cc8d50986403b78c6da36890638c3d46a5adce04a");
                    ((VM.Types.Array)result)[5].GetSpan().ToHexString().Should().Be("02ca0e27697b9c248f6f16e085fd0061e26f44da85b58ee835c110caa5ec3ba554");
                    ((VM.Types.Array)result)[6].GetSpan().ToHexString().Should().Be("02df48f60e8f3e01c48ff40b9b7f1310d7a8b2a193188befe1c2e3df740e895093");
                }

                sw.Stop();
                Console.WriteLine(sw.Elapsed);
            }
        }
Without this PR: 00:00:02.1273432
With this PR: 00:00:00.1614797

@shargon shargon requested a review from erikzhang July 14, 2020 17:13
@shargon
Copy link
Member Author

shargon commented Jul 14, 2020

Also it was detected that in policy we used ToByteArray for store BigInteger instead of ToByteArrayStandard

@Tommo-L
Copy link
Contributor

Tommo-L commented Jul 15, 2020

This cache avoid the second deserialization process

Transaction also have this problem, when nodes start to relay a transaction after verification.

@@ -99,7 +99,7 @@ protected override void OnPersist(ApplicationEngine engine)
{
base.OnPersist(engine);
StorageItem storage = engine.Snapshot.Storages.GetAndChange(CreateStorageKey(Prefix_NextValidators), () => new StorageItem());
storage.Value = GetValidators(engine.Snapshot).ToByteArray();
storage.Set(GetValidators(engine.Snapshot));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it'd be nice to optimize this one also while we're here.

If you're to look at preview2 state change dump, you'll see something like

 {
  "block": 3,
  "size": 1,
  "storage": [
   {
    "state": "Changed",
    "key": "ffffffff0e00000000000000000000000000000001",
    "value": "e807023e9b32ea89b94d066e649b124fd50e396ee91369e8e2a6ae1b11c170d022256d03009b7540e10f2562e5fd8fac9eaec25166a58b26e412348ff5a86927bfac22a202ba2c70f5996f357a43198705859fae2cfea13e1172962800772b3d588a9d4abd03408dcd416396f64783ac587ea1e1593c57d9fea880c8a6a1920e92a25947780602a7834be9b32e2981d157cb5bbd3acb42cfd11ea5c3b10224d7a44e98c5910f1b0214baf0ceea3a66f17e7e1e839ea25fd8bed6cd82e6bb6e68250189065f44ff01030205e9cefaea5a1dfc580af20c8d5aa2468bb0148f1a5e4605fc622c80e604ba00"
   }
  ]
 },

for every block and this value obviously doesn't change often. I guess it would be same even with an updated (post-preview2) governance system. But we shouldn't update storage at all if this value doesn't change, it's not hard to do, actually the only reason we didn't do it immediately in neo-go is that it would break state change dumps compatibility for us (even though technically storage contents would be the same for each block).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem is that TrackeableState is outside StorageItem, I know what you say, I will try to improve it in other PR.

@shargon
Copy link
Member Author

shargon commented Jul 17, 2020

I don't know why coveralls still failing

@shargon
Copy link
Member Author

shargon commented Jul 17, 2020

Wait for #1775

@shargon shargon merged commit da6be43 into neo-project:master Jul 17, 2020
@shargon shargon deleted the native-cache-v2 branch July 17, 2020 14:36
shargon added a commit that referenced this pull request Jul 23, 2020
* Allow call from native contract (#1700)

* Fix MethodCallback (#1723)

* Fix #1714 (#1715)

* Add base64 SYSCALLs (#1717)

* Neo.VM.3.0.0-CI00230 (#1725)

* Allow to iterate buffer (#1726)

* Buffer iterator

* Rename

* Remove using

Co-authored-by: Erik Zhang <[email protected]>

* Update StorageContext.cs (#1728)

StorageContext needs to be public so it can be accessed by Neo Debugger

Co-authored-by: Erik Zhang <[email protected]>

* Fix return value check (#1730)

Co-authored-by: Shargon <[email protected]>

* Fix ContractEventDescriptor (#1733)

* fix enumerator (#1744)

Co-authored-by: Tommo-L <[email protected]>

* Change json fields to all lower case for consistency (#1736)

* Replace DataCache.Find by DataCache.Seek (#1740)

* replace DataCache.Find by DataCache.Seek

* fix order

* optimize

* Update ByteArrayComparer.cs

* Update ByteArrayComparer.cs

* Update DataCache.cs

* fix comments

* fix comments

* fix comments

* Update DataCache.cs

* Update DataCache.cs

* Reorder methods

Co-authored-by: Tommo-L <[email protected]>
Co-authored-by: erikzhang <[email protected]>
Co-authored-by: Shargon <[email protected]>

* Add MaxVerificationGas (#1745)

Co-authored-by: Shargon <[email protected]>

* Create KeyBuilder (#1748)

* Speed up the initialization of ApplicationEngine. (#1749)

* Change nef checksum to double SHA256 (#1751)

* Change to double SHA256

* Clean code

* Revert change

* Clean code

* Fix checksum

* Update NefFile.cs

* Fix compile

* Clean code

Clean code
Fix checksum
Update NefFile.cs
Fix compile

* Optimize

* Optimize

* Fix

Co-authored-by: erikzhang <[email protected]>

* Check witnesses on isStandard (#1754)

* Check witness on isStandard

* Add IsDeployed

* Add IsPayable

* Fix UT

* format

* Add coverage

* Remove IsPayable, IsDeployed

* Move NEP10 to manifest (#1729)

* NEP10 abi

* To lower case

* Move to Manifest

* Clean code

* Update ContractManifest.cs

* Update ContractManifest.cs

* Fix native contracts

Co-authored-by: Erik Zhang <[email protected]>

* Capture fault Exception (#1761)

* Capture faultException

* Update Wallet error

* Remove flag check

* Clean code

* Sender from signers (#1752)

* Sender from signers

* Remove co-

* Move signers outside attributes

* Fix UT and remove Signers class

* Fix UT

* Add FeeOnly scope

* Remove orderBy

* Remove _signersCache

* Fix Signers

* Fix WitnessScope

* Fix Sender

* Update TransactionAttributeType.cs

* Update Wallet.cs

* Fix Wallet

* Rename

* Update Wallet.cs

* Update Wallet.cs

* Partial UT fix

* More UT fixes

* Fix Sender's WitnessScope

* Fix Wallet

* Fix UT

* Explicit FeeOnly for DeployNativeContracts

* Same order as serialization

* Test FeeOnly

* dotnet format

* format

Co-authored-by: Erik Zhang <[email protected]>

* IApplicationEngineProvider (#1758)

* Remove the lock from SendersFeeMonitor and rename it to TransactionVerificationContext (#1756)

* Add EffectiveVoterTurnout (#1762)

* Remove AllowedTriggers from SYSCALLs (#1755)

* fix validatorscount (#1770)

Co-authored-by: Tommo-L <[email protected]>

* workflows: use checkout action v2 (#1775)

* Update git workflow

* Update .github/workflows/main.yml

Co-authored-by: Erik Zhang <[email protected]>

Co-authored-by: Erik Zhang <[email protected]>

* Add cache to native contract executions V2 (#1766)

* cache v2

* More optimizations

* Policy contract optimization

* Remove interporable variable

* Two more

* Add Set

* Optimizations

* Optimize

Co-authored-by: erikzhang <[email protected]>

* Optimize attributes (#1774)

* Add length before compression data (#1768)

* Add length before compression

* Optimize

* Fix UT

* Add UT

* Add UT

Co-authored-by: erikzhang <[email protected]>

* Fix VerifyWitnesses (#1776)

* Ensure non predictable peers (#1739)

* Plugins from List to array

* Move false to init

* Fix UT

* Refactor

* Add Exception Message For CreateContract (#1787)

* Add Exception Message When Create Contract

* Add Exception Message

* Code optimization

* add UT

Co-authored-by: Shargon <[email protected]>
Co-authored-by: Qiao Jin <[email protected]>

Co-authored-by: Erik Zhang <[email protected]>
Co-authored-by: Shargon <[email protected]>
Co-authored-by: Harry Pierson <[email protected]>
Co-authored-by: Luchuan <[email protected]>
Co-authored-by: Tommo-L <[email protected]>
Co-authored-by: joeqian <[email protected]>
Co-authored-by: cloud8little <[email protected]>
Co-authored-by: Qiao Jin <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants