Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FastHeadersSyncFeed explicitly flush before setting metadata #8103

Draft
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

Scooletz
Copy link
Contributor

@Scooletz Scooletz commented Jan 24, 2025

Closes #8029

This PR makes the flush of the underlying databases explicit for HeadersSyncFeed inserting headers. It's a change similar to a fix done in #6227 which explicitly calls the flush after a specific boundary is breached. This PR flushes the headers after each batch. If a performance penalty occurs, it should be done in an amortized way, every N batches set.

In addition BodiesSyncFeed is modified, so that it Flushes the underlying database in the same way HeadersSyncFeed does it. It calls the .Flush with a specific reason.

Changes

  • explicit call out to .Flush of the underlying database

Types of changes

What types of changes does your code introduce?

  • Bugfix (a non-breaking change that fixes an issue)
  • New feature (a non-breaking change that adds functionality)
  • Breaking change (a change that causes existing functionality not to work as expected)
  • Optimization
  • Refactoring
  • Documentation update
  • Build-related changes
  • Other: Description

Testing

Requires testing

  • Yes
  • No

If yes, did you write tests?

  • Yes
  • No

Notes on testing

Optional. Remove if not applicable.

Documentation

Requires documentation update

  • Yes
  • No

If yes, link the PR to the docs update or the issue with the details labeled docs. Remove if not applicable.

Requires explanation in Release Notes

  • Yes
  • No

If yes, fill in the details here. Remove if not applicable.

Remarks

Optional. Remove if not applicable.

@Scooletz Scooletz force-pushed the headers-feed-flush-behavior branch from 9a0ef1f to c7f2a1d Compare January 24, 2025 16:12
@Scooletz Scooletz force-pushed the headers-feed-flush-behavior branch from c7f2a1d to 5df524c Compare January 24, 2025 16:15
@Scooletz Scooletz requested a review from asdacap January 24, 2025 16:18
@Scooletz
Copy link
Contributor Author

@asdacap You authored #6227 and I think you'd be most suited to provide the initial review. If you could, it'd be great.

@asdacap
Copy link
Contributor

asdacap commented Jan 24, 2025

Do you have some performance figure? Also, where is the NoWAL flag in fast header? If that is not use, there is not much use for an explicit flush.

The thing about blocks and receipts is that, once the compaction writes was reduced enough, the portion of the writes for the WAL file was large enough that it is worth optimizing away.

The thing about headers is that, it is not parallelized when inserting because of the total difficulty calculation, so it straight up can't saturate the IO bandwidth. Latency, sure, but not bandwith. Having a way insert batch of headers instead of one by one would help it a lot.

@Scooletz
Copy link
Contributor Author

Do you have some performance figure? Also, where is the NoWAL flag in fast header? If that is not use, there is not much use for an explicit flush.

I'll clear it out.

The thing about blocks and receipts is that, once the compaction writes was reduced enough, the portion of the writes for the WAL file was large enough that it is worth optimizing away.

I see.

The thing about headers is that, it is not parallelized when inserting because of the total difficulty calculation, so it straight up can't saturate the IO bandwidth. Latency, sure, but not bandwith. Having a way insert batch of headers instead of one by one would help it a lot.

This was not intended to be addressed here, but surely I'll take a look.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Chiado node after several early restarts crashed Old Headers with orphaned block exception
2 participants