Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: state listener observe writes at wrong time (backport #13516) #14139

Merged
merged 9 commits into from
Dec 6, 2022
Merged

fix: state listener observe writes at wrong time (backport #13516) #14139

merged 9 commits into from
Dec 6, 2022

Conversation

mergify[bot]
Copy link
Contributor

@mergify mergify bot commented Dec 2, 2022

This is an automatic backport of pull request #13516 done by Mergify.
Cherry-pick of 1f91ee2 has failed:

On branch mergify/bp/release/v0.45.x/pr-13516
Your branch is up to date with 'origin/release/v0.45.x'.

You are currently cherry-picking commit 1f91ee2ee.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   proto/cosmos/base/store/v1beta1/listening.proto
	modified:   store/cachekv/store.go
	modified:   store/dbadapter/store.go
	modified:   store/dbadapter/store_test.go
	modified:   store/gaskv/store.go
	modified:   store/gaskv/store_test.go
	modified:   store/iavl/store_test.go
	modified:   store/listenkv/store.go
	modified:   store/listenkv/store_test.go
	modified:   store/mem/mem_test.go
	modified:   store/prefix/store.go
	modified:   store/prefix/store_test.go
	modified:   store/rootmulti/store_test.go
	modified:   store/tracekv/store.go
	modified:   store/tracekv/store_test.go
	modified:   store/types/listening.go

Unmerged paths:
  (use "git add/rm <file>..." as appropriate to mark resolution)
	both modified:   CHANGELOG.md
	deleted by us:   api/cosmos/base/store/v1beta1/listening.pulsar.go
	both modified:   baseapp/abci.go
	both modified:   baseapp/streaming.go
	both modified:   docs/architecture/adr-038-state-listening.md
	both modified:   server/config/config.go
	both modified:   server/config/toml.go
	both modified:   simapp/app.go
	deleted by us:   simapp/app_v2.go
	both modified:   store/cachemulti/store.go
	both modified:   store/iavl/store.go
	both modified:   store/mem/store.go
	both modified:   store/rootmulti/store.go
	both modified:   store/streaming/constructor.go
	both modified:   store/streaming/constructor_test.go
	both modified:   store/streaming/file/README.md
	both modified:   store/streaming/file/service.go
	both modified:   store/streaming/file/service_test.go
	both modified:   store/types/listening.pb.go
	both modified:   store/types/store.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/github/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

Mergify commands and options

More conditions and actions can be found in the documentation.

You can also trigger Mergify actions by commenting on this pull request:

  • @Mergifyio refresh will re-evaluate the rules
  • @Mergifyio rebase will rebase this PR on its base branch
  • @Mergifyio update will merge the base branch into this PR
  • @Mergifyio backport <destination> will backport this PR on <destination> branch

Additionally, on Mergify dashboard you can:

  • look at your merge queues
  • generate the Mergify configuration with the config editor.

Finally, you can contact us on https://mergify.com

@yihuang @mergify
fix: state listener observe writes at wrong time (#13516)
223554c 
* fix: state listener observe writes at wrong time

Closes: #13457

Currently state listener is notified when the cache store write, which happens in commit event only, which breaks the current design.
The solution (as discussed in the issue) is to listen state writes on rootmulti store only.

It also changes the file streamer to output single data file for the writes in the whole block, since we can't distinguish writes from different stage of abci events.

It adds new config items for file streamer:
- streamers.file.output-metadata
- streamers.file.stop-node-on-error
- streamers.file.fsync

* synchronous abci call, and format doc

* fix comment

* update file streamer readme and fix typos

* typo

* fix: state listener observe writes at wrong time

Closes: #13457

Currently state listener is notified when the cache store write, which happens in commit event only, which breaks the current design.
The solution (as discussed in the issue) is to listen state writes on rootmulti store only.

It also changes the file streamer to output single data file for the writes in the whole block, since we can't distinguish writes from different stage of abci events.

It adds new config items for file streamer:
- streamers.file.output-metadata
- streamers.file.stop-node-on-error
- streamers.file.fsync

synchronous abci call, and format doc

fix comment

update file streamer readme and fix typos

typo

* improve UX of file streamer, make it immediately usable after enabled

- set default value to write_dir.
- make write_dir based on home directory by default.
- auto-create the directory if not exists.

* get homePage from opts

Co-authored-by: Marko <[email protected]>
(cherry picked from commit 1f91ee2)

# Conflicts:
#	CHANGELOG.md
#	api/cosmos/base/store/v1beta1/listening.pulsar.go
#	baseapp/abci.go
#	baseapp/streaming.go
#	docs/architecture/adr-038-state-listening.md
#	server/config/config.go
#	server/config/toml.go
#	simapp/app.go
#	simapp/app_v2.go
#	store/cachemulti/store.go
#	store/iavl/store.go
#	store/mem/store.go
#	store/rootmulti/store.go
#	store/streaming/constructor.go
#	store/streaming/constructor_test.go
#	store/streaming/file/README.md
#	store/streaming/file/service.go
#	store/streaming/file/service_test.go
#	store/types/listening.pb.go
#	store/types/store.go
@mergify mergify bot requested review from aaronc and alexanderbez as code owners December 2, 2022 14:45
@mergify mergify bot added the conflicts label Dec 2, 2022
@yihuang yihuang added C:Store and removed conflicts labels Dec 3, 2022
@tac0turtle tac0turtle enabled auto-merge (squash) December 3, 2022 21:02
@tac0turtle
Copy link
Member

seems like build Is failing

@tac0turtle tac0turtle merged commit 4a62609 into release/v0.45.x Dec 6, 2022
@tac0turtle tac0turtle deleted the mergify/bp/release/v0.45.x/pr-13516 branch December 6, 2022 07:00
@faddat faddat mentioned this pull request Mar 23, 2023
Closed
19 tasks
@faddat faddat mentioned this pull request Dec 11, 2023
Draft
19 tasks
JeancarloBarrios pushed a commit to agoric-labs/cosmos-sdk that referenced this pull request Sep 28, 2024
@mergify @yihuang
@julienrbrt @JeancarloBarrios
fix: state listener observe writes at wrong time (backport cosmos#13516…
4549180 
…) (cosmos#14139)

* fix: state listener observe writes at wrong time (cosmos#13516)

* fix: state listener observe writes at wrong time

Closes: cosmos#13457

Currently state listener is notified when the cache store write, which happens in commit event only, which breaks the current design.
The solution (as discussed in the issue) is to listen state writes on rootmulti store only.

It also changes the file streamer to output single data file for the writes in the whole block, since we can't distinguish writes from different stage of abci events.

It adds new config items for file streamer:
- streamers.file.output-metadata
- streamers.file.stop-node-on-error
- streamers.file.fsync

* synchronous abci call, and format doc

* fix comment

* update file streamer readme and fix typos

* typo

* fix: state listener observe writes at wrong time

Closes: cosmos#13457

Currently state listener is notified when the cache store write, which happens in commit event only, which breaks the current design.
The solution (as discussed in the issue) is to listen state writes on rootmulti store only.

It also changes the file streamer to output single data file for the writes in the whole block, since we can't distinguish writes from different stage of abci events.

It adds new config items for file streamer:
- streamers.file.output-metadata
- streamers.file.stop-node-on-error
- streamers.file.fsync

synchronous abci call, and format doc

fix comment

update file streamer readme and fix typos

typo

* improve UX of file streamer, make it immediately usable after enabled

- set default value to write_dir.
- make write_dir based on home directory by default.
- auto-create the directory if not exists.

* get homePage from opts

Co-authored-by: Marko <[email protected]>
(cherry picked from commit 1f91ee2)

# Conflicts:
#	CHANGELOG.md
#	api/cosmos/base/store/v1beta1/listening.pulsar.go
#	baseapp/abci.go
#	baseapp/streaming.go
#	docs/architecture/adr-038-state-listening.md
#	server/config/config.go
#	server/config/toml.go
#	simapp/app.go
#	simapp/app_v2.go
#	store/cachemulti/store.go
#	store/iavl/store.go
#	store/mem/store.go
#	store/rootmulti/store.go
#	store/streaming/constructor.go
#	store/streaming/constructor_test.go
#	store/streaming/file/README.md
#	store/streaming/file/service.go
#	store/streaming/file/service_test.go
#	store/types/listening.pb.go
#	store/types/store.go

* `make proto-gen`, update changelog, delete uncessary files

* fix conflicts

* fix GetConfig

Co-authored-by: yihuang <[email protected]>
Co-authored-by: Julien Robert <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tac0turtle tac0turtle tac0turtle approved these changes

@aaronc aaronc Awaiting requested review from aaronc aaronc is a code owner

@alexanderbez alexanderbez Awaiting requested review from alexanderbez

Assignees

@yihuang yihuang

Labels
C:Store Type: ADR
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@tac0turtle @yihuang @julienrbrt