beta channel ./net.sh sanity fails frequently

[mvines]

With #2278 fixed ./net.sh sanity succeeds sometimes but is still not stable (observable by the "testnet-edge Sanity" notifications in #ci-status).

Examining recent failures most of them seem to be related to failing to successfully execute/confirm an airdrop transaction. Need further debug on a local setup.

Blocked by:

• Cluster stalls after some time and nodes can't find each other #3171
• solana-ledger-tool verify command is too slow #3155
• Gossip ContactInfo messages are incorrectly signed #3191

[mvines]

Note: it's all good with leader rotation disabled

[mvines]

Simple STR for at least one of the N restart/sanity problems:

$ cd net/
$ ./ec2.sh create -n 3 -c 0
$ ./net.sh start
$ while sleep 60; do ./net.sh restart -r; done

[mvines]

Mostly waiting on #2317 to be fixed now

[mvines]

#2484 contains a couple fixes that ought to help with ./net.sh sanity

[mvines]

Recently there's been a noticeable improvement in the ./net.sh sanity pass rate. Needs continued monitoring but we appear to be on the right path

[mvines]

We've moved backwards here significantly with all the new features coming in over the last couple weeks. Putting this on hold for a while until more of forking lands

[mvines]

No longer blocked by anything. This still manifests on testnet-beta. Here's one example from today, https://buildkite.com/solana-labs/testnet-management/builds/26334#01916cd9-c153-4ef7-99b8-987818aea73a, where an airdrop failed during the sanity test:

[2019-03-23T02:13:36.204647707Z INFO  solana_drone::drone] request_airdrop_transaction: drone_addr=13.57.20.27:9900 id=Bh8xara7wSc2LekDF7ML8CSEzedRRYQCXLzgZK252p1M lamports=60 blockhash=CnkLbhs9xGuXCBoNu1yJWRK3WNHRajqfoJ2KuxSDW18W
Error: RpcRequestError("Unable to fetch new blockhash, blockhash stuck at CnkLbhs9xGuXCBoNu1yJWRK3WNHRajqfoJ2KuxSDW18W")

[pgarg66]

@CriesofCarrots , is this the same problem you triaged yesterday? If so, could you add your analysis here?

[CriesofCarrots]

Looking at the logs for these failures, it appears the the poh_recorder on the leader is no longer progressing, which results in the stuck blockhash and no transactions being processed.
That is as far as I dug, as my focus was on testnet boot errors.

[mvines]

#3749 hopefully helps here...

[mvines]

Tip of master looks pretty stable recently, let's see how beta performs when we branch off v0.13 later today...

[mvines]

065cf51

[mvines]

Looking pretty solid now. No sanity failures overnight. If we're in a similar state tomorrow then this issue can be closed