[cherry-pick v1.10] BlockSTM error propagation and handling

[igor-aptos]

straight up cherry pick of the following PRs from main:
#12230
#12468

[trunk-io]

⏱️ 4h 31m total CI duration on this PR

Job	Cumulative Duration	Recent Runs
rust-unit-tests	1h 8m	🟩 🟩
windows-build	52m	🟩 🟩
rust-smoke-tests	34m	🟩
execution-performance / single-node-performance	21m	🟩
rust-lints	16m	🟩 🟩
rust-images / rust-all	15m	🟩
forge-compat-test / forge	15m	🟩
forge-e2e-test / forge	14m	🟩
check	9m	🟩 🟩
run-tests-main-branch	8m	🟥 🟥
cli-e2e-tests / run-cli-tests	7m	🟩
check-dynamic-deps	5m	🟩 🟩
general-lints	5m	🟩 🟩
node-api-compatibility-tests / node-api-compatibility-tests	53s	🟩
semgrep/ci	47s	🟩 🟩
file_change_determinator	22s	🟩 🟩
file_change_determinator	22s	🟩 🟩
file_change_determinator	10s	🟩
execution-performance / file_change_determinator	9s	🟩
permission-check	9s	🟩 🟩
permission-check	8s	🟩 🟩
permission-check	5s	🟩 🟩
permission-check	5s	🟩 🟩
determine-docker-build-metadata	4s	🟩
permission-check	2s	🟩

🚨 1 job on the last run was significantly faster/slower than expected

Job	Duration	vs 7d avg	Delta
rust-unit-tests	37m	27m

_{settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io}

[github-actions]

✅ Forge suite realistic_env_max_load success on d63700cc9df38ba866b28750b218afd3adce304c

two traffics test: inner traffic : committed: 6002 txn/s, latency: 6507 ms, (p50: 6000 ms, p90: 8700 ms, p99: 13800 ms), latency samples: 2611200
two traffics test : committed: 100 txn/s, latency: 2040 ms, (p50: 2000 ms, p90: 2400 ms, p99: 2900 ms), latency samples: 1860
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.227, avg: 0.200", "QsPosToProposal: max: 0.288, avg: 0.231", "ConsensusProposalToOrdered: max: 0.628, avg: 0.580", "ConsensusOrderedToCommit: max: 0.501, avg: 0.464", "ConsensusProposalToCommit: max: 1.129, avg: 1.044"]
Max round gap was 1 [limit 4] at version 1222663. Max no progress secs was 4.22078 [limit 15] at version 1222663.
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Test runner output
• Test run is land-blocking

[github-actions]

✅ Forge suite compat success on aptos-node-v1.9.5 ==> d63700cc9df38ba866b28750b218afd3adce304c

Compatibility test results for aptos-node-v1.9.5 ==> d63700cc9df38ba866b28750b218afd3adce304c (PR)
1. Check liveness of validators at old version: aptos-node-v1.9.5
compatibility::simple-validator-upgrade::liveness-check : committed: 5977 txn/s, latency: 4864 ms, (p50: 4800 ms, p90: 8000 ms, p99: 9000 ms), latency samples: 239080
2. Upgrading first Validator to new version: d63700cc9df38ba866b28750b218afd3adce304c
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 697 txn/s, latency: 33918 ms, (p50: 35400 ms, p90: 52100 ms, p99: 55200 ms), latency samples: 57220
3. Upgrading rest of first batch to new version: d63700cc9df38ba866b28750b218afd3adce304c
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 284 txn/s, submitted: 650 txn/s, expired: 365 txn/s, latency: 27321 ms, (p50: 27100 ms, p90: 49700 ms, p99: 54300 ms), latency samples: 22195
4. upgrading second batch to new version: d63700cc9df38ba866b28750b218afd3adce304c
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 2098 txn/s, latency: 13146 ms, (p50: 13800 ms, p90: 15600 ms, p99: 19600 ms), latency samples: 102820
5. check swarm health
Compatibility test for aptos-node-v1.9.5 ==> d63700cc9df38ba866b28750b218afd3adce304c passed
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Test runner output
• Test run is land-blocking