Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merge branch feature/abci++ppp into main #9301

Merged
merged 20 commits into from
Aug 22, 2022
Merged

Merge branch feature/abci++ppp into main #9301

merged 20 commits into from
Aug 22, 2022

Conversation

sergio-mena
Copy link
Contributor

@sergio-mena sergio-mena commented Aug 22, 2022 •
edited
Loading

Closes #9300

This PR is a merging branch feature/abci++ppp into main

After this PR is in, branch feature/abci++ppp will be removed (as its history is now merged into the main branch).

The contents of this PR represent the whole set of changes backported from v0.35.x/v0.36.x to deliver PrepareProposal/ProcessProposal in the next release.

(the CHANGELOG_PENDING.md update is coming in a PR to be created shortly)

PR checklist

  • Tests written/updated, or no tests needed
  • CHANGELOG_PENDING.md updated, or no changelog entry needed
  • Updated relevant documentation (docs/) and code comments, or no
    documentation updates needed

sergio-mena and others added 20 commits July 27, 2022 15:09
@sergio-mena @tac0turtle
Old PrepareProposal on feature/abci++ppp (#9106)
58effdd 
* [Rebased to v0.34.x] abci: PrepareProposal (#6544)

* fixed cherry-pick

* proto changes

* make proto-gen

* UT fixes

* generate Client directive

* mockery

* App fixes

* Disable 'modified tx' hack

* mockery

* Make format

* Fix lint

Co-authored-by: Marko <[email protected]>
@sergio-mena @thanethomson
@tac0turtle @tychoish @williambanfield
Update feature/abci++ppp branch with latest infra changes in main (…
61619ab 
…#9157)

* Update CODEOWNERS to use teams (#9129)

* Update CODEOWNERS to use teams

Update the `CODEOWNERS` file to use the
@tendermint/tendermint-engineering and @tendermint/tendermint-research
teams as opposed to adding people one by one. This makes repository
administration somewhat easier to manage, especially when
onboarding/offboarding people.

Signed-off-by: Thane Thomson <[email protected]>

* Add Ethan as superuser

Signed-off-by: Thane Thomson <[email protected]>

* Prepare `main` to become new default branch (#9095)

* Update Makefile with changes from #7372

Signed-off-by: Thane Thomson <[email protected]>

* Sync main GitHub config with master and update

Signed-off-by: Thane Thomson <[email protected]>

* Remove unnecesary dot folders

Signed-off-by: Thane Thomson <[email protected]>

* Sync dotfiles

Signed-off-by: Thane Thomson <[email protected]>

* Remove unused Jepsen tests for now

Signed-off-by: Thane Thomson <[email protected]>

* tools: remove k8s (#6625)

Remove mintnet as discussed on team call.

closes #1941

* Restore nightly fuzz testing of P2P addrbook and pex

Signed-off-by: Thane Thomson <[email protected]>

* Fix YAML lints

Signed-off-by: Thane Thomson <[email protected]>

* Fix YAML formatting nits

Signed-off-by: Thane Thomson <[email protected]>

* More YAML nits

Signed-off-by: Thane Thomson <[email protected]>

* github: fix linter configuration errors and occluded errors (#6400)

* Minor fixes to OpenAPI spec to sync with structs on main

Signed-off-by: Thane Thomson <[email protected]>

* Remove .github/auto-comment.yml - does not appear to be used

Signed-off-by: Thane Thomson <[email protected]>

* Add issue config with link to discussions

Signed-off-by: Thane Thomson <[email protected]>

* Adjust issue/PR templates to suit current process

Signed-off-by: Thane Thomson <[email protected]>

* Remove unused RC branch config from release workflow

Signed-off-by: Thane Thomson <[email protected]>

* Fix wildcard matching in build jobs config

Signed-off-by: Thane Thomson <[email protected]>

* Document markdownlint config

Signed-off-by: Thane Thomson <[email protected]>

* Restore manual E2E test group config

Signed-off-by: Thane Thomson <[email protected]>

* Document linter workflow with local execution instructions

Signed-off-by: Thane Thomson <[email protected]>

* Document and fix minor nit in Super-Linter markdownlint config

Signed-off-by: Thane Thomson <[email protected]>

* Update .github/ISSUE_TEMPLATE/bug-report.md

Co-authored-by: William Banfield <[email protected]>

* Apply suggestions from code review

Co-authored-by: William Banfield <[email protected]>

* Update pull request template to add language around discussions/issues

Signed-off-by: Thane Thomson <[email protected]>

* .golangci.yml: Deleted commented-out lines

Signed-off-by: Thane Thomson <[email protected]>

* ci: Drop "-2" from e2e-nightly-fail workflow

Signed-off-by: Thane Thomson <[email protected]>

* Address triviality concern in PR template

Signed-off-by: Thane Thomson <[email protected]>

Co-authored-by: Marko <[email protected]>
Co-authored-by: Sam Kleinman <[email protected]>
Co-authored-by: William Banfield <[email protected]>

Co-authored-by: Thane Thomson <[email protected]>
Co-authored-by: Marko <[email protected]>
Co-authored-by: Sam Kleinman <[email protected]>
Co-authored-by: William Banfield <[email protected]>
@sergio-mena @mconcat
Old PrepareProposal and vote extension integration on feature/abci++p…
2164883 
…pp (#9117)

* abci: PrepareProposal-VoteExtension integration [2nd try] (#7821)

* PrepareProposal-VoteExtension integration (#6915)

* make proto-gen

* Fix protobuf crash in e2e nightly tests

* Update types/vote.go

Co-authored-by: M. J. Fromberger <[email protected]>

* Addressed @creachadair's comments

Co-authored-by: mconcat <[email protected]>
Co-authored-by: M. J. Fromberger <[email protected]>

* Proto changes

* make proto-gen

* Fixed UTs

* bump

* lint

* lint2

* lint3

* lint4

* lint5

* lint6

* no_lint

Co-authored-by: mconcat <[email protected]>
Co-authored-by: M. J. Fromberger <[email protected]>
@sergio-mena @mconcat
@JayT106 @tychoish @williambanfield
Sync PrepareProposal with Spec. Main part (#9158)
d268e56 
* ----start----

* [PARTIAL cherry-pick] ABCI Vote Extension 2 (#6885)

* Cherry-picked #6567: state/types: refactor makeBlock, makeBlocks and makeTxs (#6567)

* [Cherrypicked] types: remove panic from block methods (#7501)

* [cherrypicked] abci++: synchronize PrepareProposal with the newest version of the spec (#8094)

This change implements the logic for the PrepareProposal ABCI++ method call. The main logic for creating and issuing the PrepareProposal request lives in execution.go and is tested in a set of new tests in execution_test.go. This change also updates the mempool mock to use a mockery generated version and removes much of the plumbing for the no longer used ABCIResponses.

* make proto-gen

* Backported EvidenceList's method ToABCI from #7961

* make build

* Fix mockery for Mempool

* mockery

* Backported abci Application mocks from #7961

* mockery2

* Fixed new PrepareProposal test cases in state/execution_test.go

* Fixed returned errors in consensus/state.go

* lint

* Addressed @cmwaters' comment

Co-authored-by: mconcat <[email protected]>
Co-authored-by: JayT106 <[email protected]>
Co-authored-by: Sam Kleinman <[email protected]>
Co-authored-by: William Banfield <[email protected]>
@sergio-mena @williambanfield
@tychoish @mergify
Follow-up fixes to main PrepareProposal implementation (#9162)
b2409b3 
* -----start------

* [cherrypicked] state: panic on ResponsePrepareProposal validation error (#8145)

* state: panic on ResponsePrepareProposal validation error

* lint++

Co-authored-by: Sam Kleinman <[email protected]>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* [cherrypicked] abci++: remove CheckTx call from PrepareProposal flow (#8176)

* [cherrypicked] abci++: correct max-size check to only operate on added and unmodified (#8242)

* [cherrypicked] Remove `ModifiedTxStatus` from the spec and the code (#8210)

* Outstanding abci-gen changes to 'pb.go' files

* Removed modified_tx_status from spec and protobufs

* Fix sed for OSX

* Regenerated abci protobufs with 'abci-proto-gen'

* Code changes. UTs e2e tests passing

* Recovered UT: TestPrepareProposalModifiedTxStatusFalse

* Adapted UT

* Fixed UT

* Revert "Fix sed for OSX"

This reverts commit e576708.

* Update internal/state/execution_test.go

Co-authored-by: William Banfield <[email protected]>

* Update abci/example/kvstore/kvstore.go

Co-authored-by: M. J. Fromberger <[email protected]>

* Update internal/state/execution_test.go

Co-authored-by: William Banfield <[email protected]>

* Update spec/abci++/abci++_tmint_expected_behavior_002_draft.md

Co-authored-by: William Banfield <[email protected]>

* Addressed some comments

* Added one test that tests error at the ABCI client + Fixed some mock calls

* Addressed remaining comments

* Update abci/example/kvstore/kvstore.go

Co-authored-by: William Banfield <[email protected]>

* Update abci/example/kvstore/kvstore.go

Co-authored-by: William Banfield <[email protected]>

* Update abci/example/kvstore/kvstore.go

Co-authored-by: William Banfield <[email protected]>

* Update spec/abci++/abci++_tmint_expected_behavior_002_draft.md

Co-authored-by: William Banfield <[email protected]>

* Addressed William's latest comments

* Adressed Michael's comment

* Fixed UT

* Some md fixes

* More md fixes

* gofmt

Co-authored-by: William Banfield <[email protected]>
Co-authored-by: M. J. Fromberger <[email protected]>

* make proto-gen

* Fixed testcase on PrepareProposal error

* mockery

Co-authored-by: William Banfield <[email protected]>
Co-authored-by: Sam Kleinman <[email protected]>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: M. J. Fromberger <[email protected]>
@cmwaters
abci: implement process proposal to spec (#9122)
f861062
@sergio-mena @williambanfield @mergify
PrepareProposal/ProcessProposal: Dealing with block data exposed …
25b0c7c 
…to App (#9219)

* [cherrypicked] abci++: only include meaningful header fields in data passed-through to application (#8216)

* make proto-gen

* Fix kvstore tests

* Small changes to abci protobufs taken from v0.36.x

* make proto-gen (again)

* [Partial cherrypick] Restore `Commit` to the ABCI++ spec, and other late modifications (backport #8796) (#8936)

* Restore `Commit` to the ABCI++ spec, and other late modifications (#8796)

* Added VoteExtensionsEnableHeight

* Fix reference to `modified`

* Removed old pseudo-code, now included in spec

* Removed markdown warnings in abci++_basic_concepts_002_draft.md

* Restored `Commit` in the Methods section

* Addressed remaining markdown warnings

* Revisited intro and basic concepts section

* Extra pass at all spec sections to recover Commit, and other ABCI++ spec modifications

* Fixed links

* make proto-gen

* Remove _primes_ from spec notation

* Update proto/tendermint/abci/types.proto

Co-authored-by: Callum Waters <[email protected]>

* Update spec/abci++/abci++_tmint_expected_behavior_002_draft.md

Co-authored-by: Callum Waters <[email protected]>

* Addressed @cmwaters' comments

* Addressed @angbrav's and @mpoke's comments on spec

* make proto-gen

* Fix MD anchor reference

* Clarify throughout the spec when `ProcessProposal` and `VerifyVoteExtension` are called

* Update spec/abci++/abci++_app_requirements_002_draft.md

Co-authored-by: M. J. Fromberger <[email protected]>

* Update spec/abci++/abci++_app_requirements_002_draft.md

Co-authored-by: M. J. Fromberger <[email protected]>

* Update spec/abci++/abci++_app_requirements_002_draft.md

Co-authored-by: William Banfield <[email protected]>

* Update spec/abci++/abci++_basic_concepts_002_draft.md

Co-authored-by: William Banfield <[email protected]>

* Update spec/abci++/abci++_basic_concepts_002_draft.md

Co-authored-by: M. J. Fromberger <[email protected]>

* Update spec/abci++/abci++_basic_concepts_002_draft.md

Co-authored-by: William Banfield <[email protected]>

* Update spec/abci++/abci++_methods_002_draft.md

Co-authored-by: M. J. Fromberger <[email protected]>

* Update spec/abci++/abci++_tmint_expected_behavior_002_draft.md

Co-authored-by: William Banfield <[email protected]>

* Addresed comments

* Renamed 'draft' files

* Adatped links to new filenames

* Fixed links and minor cosmetic changes

* Renamed 'byzantine_validators' to 'misbehavior' in ABCI++ spec and protobufs

* make proto-gen

* Renamed 'byzantine_validators' to 'misbehavior' in the code

* Fixed link

* Update spec/abci++/abci++_basic_concepts.md

Co-authored-by: Daniel <[email protected]>

* Update spec/abci++/abci++_basic_concepts.md

Co-authored-by: Daniel <[email protected]>

* Update spec/abci++/abci++_basic_concepts.md

Co-authored-by: Daniel <[email protected]>

* Update spec/abci++/abci++_basic_concepts.md

Co-authored-by: Daniel <[email protected]>

* Update spec/abci++/abci++_basic_concepts.md

Co-authored-by: Daniel <[email protected]>

* Update spec/abci++/abci++_basic_concepts.md

Co-authored-by: Daniel <[email protected]>

* Update spec/abci++/abci++_basic_concepts.md

Co-authored-by: Daniel <[email protected]>

* Update spec/abci++/abci++_basic_concepts.md

Co-authored-by: Daniel <[email protected]>

* Update spec/abci++/abci++_basic_concepts.md

Co-authored-by: Daniel <[email protected]>

* Update spec/abci++/abci++_basic_concepts.md

Co-authored-by: Daniel <[email protected]>

* Update spec/abci++/abci++_basic_concepts.md

Co-authored-by: Daniel <[email protected]>

* Update spec/abci++/abci++_basic_concepts.md

Co-authored-by: Daniel <[email protected]>

* Update spec/abci++/abci++_basic_concepts.md

Co-authored-by: Daniel <[email protected]>

* Update spec/abci++/abci++_basic_concepts.md

Co-authored-by: Daniel <[email protected]>

* Update spec/abci++/abci++_basic_concepts.md

Co-authored-by: Daniel <[email protected]>

* Update spec/abci++/abci++_basic_concepts.md

Co-authored-by: Daniel <[email protected]>

* Update spec/abci++/abci++_basic_concepts.md

Co-authored-by: Daniel <[email protected]>

* Update spec/abci++/abci++_basic_concepts.md

Co-authored-by: Daniel <[email protected]>

* Update spec/abci++/abci++_basic_concepts.md

Co-authored-by: Daniel <[email protected]>

* Update spec/abci++/abci++_basic_concepts.md

Co-authored-by: Daniel <[email protected]>

* Update spec/abci++/abci++_basic_concepts.md

Co-authored-by: Daniel <[email protected]>

* Update spec/abci++/abci++_basic_concepts.md

Co-authored-by: Daniel <[email protected]>

* Update spec/abci++/abci++_methods.md

Co-authored-by: Daniel <[email protected]>

* Update spec/abci++/abci++_methods.md

Co-authored-by: Daniel <[email protected]>

* Update spec/abci++/abci++_methods.md

Co-authored-by: Daniel <[email protected]>

* Update spec/abci++/abci++_methods.md

Co-authored-by: Daniel <[email protected]>

* Update spec/abci++/abci++_methods.md

Co-authored-by: Daniel <[email protected]>

* Update spec/abci++/abci++_methods.md

Co-authored-by: Daniel <[email protected]>

* Update spec/abci++/abci++_methods.md

Co-authored-by: Daniel <[email protected]>

* Update spec/abci++/abci++_methods.md

Co-authored-by: Daniel <[email protected]>

* Update spec/abci++/abci++_methods.md

Co-authored-by: Daniel <[email protected]>

* Update spec/abci++/abci++_methods.md

Co-authored-by: Daniel <[email protected]>

* Update spec/abci++/abci++_methods.md

Co-authored-by: Daniel <[email protected]>

* Update spec/abci++/abci++_methods.md

Co-authored-by: Daniel <[email protected]>

* Update spec/abci++/abci++_methods.md

Co-authored-by: Daniel <[email protected]>

* Update spec/abci++/abci++_methods.md

Co-authored-by: Daniel <[email protected]>

* Update spec/abci++/abci++_methods.md

Co-authored-by: Daniel <[email protected]>

* Update spec/abci++/abci++_methods.md

Co-authored-by: Daniel <[email protected]>

* Update spec/abci++/abci++_methods.md

Co-authored-by: Daniel <[email protected]>

* Update spec/abci++/abci++_methods.md

Co-authored-by: Daniel <[email protected]>

* Update spec/abci++/abci++_methods.md

Co-authored-by: Daniel <[email protected]>

* Update spec/abci++/abci++_methods.md

Co-authored-by: Daniel <[email protected]>

* Update spec/abci++/abci++_methods.md

Co-authored-by: Daniel <[email protected]>

* Update spec/abci++/abci++_methods.md

Co-authored-by: Daniel <[email protected]>

* Update spec/abci++/abci++_methods.md

Co-authored-by: Daniel <[email protected]>

* Update spec/abci++/abci++_methods.md

Co-authored-by: Daniel <[email protected]>

* Update spec/abci++/abci++_methods.md

Co-authored-by: Daniel <[email protected]>

* Update spec/abci++/abci++_methods.md

Co-authored-by: Daniel <[email protected]>

* Update spec/abci++/abci++_methods.md

Co-authored-by: Daniel <[email protected]>

* Update spec/abci++/abci++_methods.md

Co-authored-by: Daniel <[email protected]>

* Addressed @cason's comments

* Clarified conditions for `ProcessProposal` call at proposer

Co-authored-by: Callum Waters <[email protected]>
Co-authored-by: M. J. Fromberger <[email protected]>
Co-authored-by: William Banfield <[email protected]>
Co-authored-by: Daniel <[email protected]>
(cherry picked from commit 331860c)

* Fixed merge conflicts

Co-authored-by: Sergio Mena <[email protected]>

* make proto-gen (and again)

* make build

* fix UTs

Co-authored-by: William Banfield <[email protected]>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
@sergio-mena
Remove unused code in internal directory (#9223)
c4eb611
@sergio-mena @tac0turtle
ABCI types.proto. Handle remaining discrepancies (#9224)
cb570f6 
* [cherrypicked] version: add abci version to handshake (#5706)

- add `AbciVersion` RequestInfo

Closes: #2804

* make proto-gen

* Bump ABCI version: Prepare and Process proposal

Co-authored-by: Marko <[email protected]>
@sergio-mena
Merge branch main into feature/abci++ppp
7dc4f93
@sergio-mena
Merge pull request #9236 from tendermint/sergio/merging-main-abci++ppp
670abbc 
Merge `main` into `feature/abci++ppp`
@samricotta
abci: Remove SetOption #5447 #9091 (#9266)
9993514 
* Remove set option for abci
@williambanfield
consensus: backport abci and consensus metrics (#9273)
7bd86ec 
Partial backport of #8480
@sergio-mena @jmalicevic @hvanz
abci-cli: PrepareProposal and ProcessProposal (#9281)
622b930 
* [cherrypicked] abci-cli: added `PrepareProposal` command to cli (#8656)

* Prepare prosal cli

* [cherrypicked + fixes] abci-cli: Add `process_proposal` command to abci-cli (#8901)

* Add `process_proposal` command to abci-cli

* Added process proposal to the 'tutorial' examples

* Added entry in CHANGELOG_PENDING.md

* Allow empty blocks in PrepareProposal, ProcessProposal, and FinalizeBlock

* Fix minimum arguments

* Add tests for empty block

* Updated abci-cli doc

Co-authored-by: Sergio Mena <[email protected]>
Co-authored-by: Jasmina Malicevic <[email protected]>

* Addressed @williambanfield's comment

Co-authored-by: Jasmina Malicevic <[email protected]>
Co-authored-by: Hernán Vanzetto <[email protected]>
@sergio-mena
Restore e2e nightly workflow on feature/abci++ppp branch (#9280)
fb9afcc
@williambanfield
abci++: update PrepareProposal API to accept just a list of bytes (#9283
1a4d939 
)

This changes the ResponsePrepareProposal type, substituting the []TxRecord for just []bytes. This change is made in the .proto file and in all necessary additional places in the code.
@cmwaters
e2e: add abci delays (#9254)
0ca3a89
@cmwaters
proto: deduplicate consensus params (#9287)
2d8df1b
@cmwaters
e2e: add evidence tests (#9292)
b37f062
@sergio-mena
Merge branch 'feature/abci++ppp'
50b5c23
@sergio-mena sergio-mena requested a review from ebuchman as a code owner August 22, 2022 15:23
@sergio-mena sergio-mena requested a review from a team August 22, 2022 15:23
tychoish
tychoish reviewed Aug 22, 2022
View reviewed changes
Copy link
Contributor

@tychoish tychoish left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this all seems good, just some minor quibbles.

I would have expected that the SetOption removal pieces would have been able to happen in an orthogonal PR, but it seems like this is fine too.

.gitignore Show resolved Hide resolved
CHANGELOG_PENDING.md
Comment on lines +21 to +22
- [abci] \#8656 Added cli command for `PrepareProposal`. (@jmalicevic)
- [abci] \#8901 Added cli command for `ProcessProposal`. (@hvanz)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we combine these lines

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am preparing a CHANGELOG PR spanning all PrepareProposal/ProcessProposal changes that will deal with this

abci/example/kvstore/kvstore.go
Comment on lines +190 to +191
func (app *Application) ProcessProposal(
req types.RequestProcessProposal) types.ResponseProcessProposal {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think splitting this makes it harder to read, and we haven't typically enforced line lengths.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As this is a merge commit, I prefer not to tackle these minor issues now, but in a subsequent PR.
Recall this is already reviewed code (on the feature branch), and someone decided this was good to go.

@sergio-mena
Copy link
Contributor Author

sergio-mena commented Aug 22, 2022 •
edited
Loading

I would have expected that the SetOption removal pieces would have been able to happen in an orthogonal PR, but it seems like this is fine too.

Yeah we hesitated whether to base the PR on main or on feature/abci++ppp. In the end we went for the latter, as other changes in ABCI's types.proto were going on, and wanted to avoid conflicts at the moment of merging the branch into main (i.e., now)

@sergio-mena sergio-mena self-assigned this Aug 22, 2022
@sergio-mena sergio-mena merged commit 50b5c23 into main Aug 22, 2022
@sergio-mena sergio-mena deleted the sergio/merge_abci++ppp_into_main branch August 22, 2022 17:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tychoish tychoish tychoish left review comments

@thanethomson thanethomson thanethomson approved these changes

@ebuchman ebuchman Awaiting requested review from ebuchman

Assignees

@sergio-mena sergio-mena

Labels
None yet
Projects
Tendermint
Status: Done/Merged
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Merge the feature/abci++ppp branch into main once all relevant work from #9053 is done
6 participants
@sergio-mena @tychoish @thanethomson @cmwaters @samricotta @williambanfield