-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
task(buffers): ensure disk_v1
buffer uses same directory as before
#10430
Comments
Well, running the same configuration (using the disk v1 buffer) on both 0.18.1 and 0.19.0 shows differing data directories, so that's a definite regression. Ugh. :( |
Closed via #10826. |
Manual resolution of simultaneous old/new disk buffersAs part of 0.19.0, we released some changes to the buffer code that was an incremental part of our overall work of improving buffers. This work was intended to be backwards-compatible and transparent to you, the end user. Unfortunately, we (me, really) introduced a regression that unintentionally changed the data directory path for disk buffers. We pushed a subsequent partial fix -- #10826 -- but if you're here, it's because you've hit a particular corner case that we could not handle in the partial fix. The regression itself essentially had the effect of creating a new buffer: since the data directory path changed, it would leave the old data in a directory that ended in What this means for you, as a user, is that you now have orphaned data still present in the "old" buffer -- the data directory suffixed with My data is ephemeral / old enough that I don't care about it anymore:If you do not care about this data, potentially because now it's too old to matter, then you can simply delete the old data directory, the one suffixed with I want to process all the data leftover in the old buffer:If you want to process all of the messages in the "old" buffer, you can follow the steps below. For the sake of example, we use a sink called In TOML: # `my_sink_id` is the ID of the sink.
[sinks.my_sink_id]
... In YAML: # `my_sink_id` is the ID of the sink.
sinks:
my_sink_id:
... Additionally, some of the log messages will refer to a full directory, which will be rooted at Alright, now that we know how to grab the ID of the sink, the actual steps:
Quiescing Vector to allow buffer to drain:The most complicated step here is step 5, or draining the old buffer. This requires ensuring that the buffer is fully drained before moving on, which can be hard to accomplish without also figuring out how to stop accepting events in Vector. One possible option is simply duplicating your configuration and changing the source(s) such that are no longer valid. This could mean listening on a different port for a source that listens for HTTP requests, or on a specific TCP port, etc. In the case of sources that pull their data from an external source, it may mean pointing the source at a non-existent bucket/topic/etc so that it runs but has no chance to actually get any events. In certain circumstances, these options may be too heavyweight, so another option could be to simply remove everything but the sink in question from the duplicated config, and add a dummy source -- the [sources.dummy_source]
type = "socket"
address = "0.0.0.0:60000"
mode = "tcp"
# IMPORTANT: The sink ID _must_ stay the same, as it relates to which data directories it uses on disk.
[sinks.my_sink_id]
inputs = ["dummy_source"]
... This example uses a very high port that should not conflict with much of anything, but there's a chance you may need to pick another port if something is already listening on that one. Internal remediationFirstly, we're very sorry. It sucks that this bug managed to slip through our review process, and made it out into a release. While bugs are inevitable, they're especially impactful when they're related to the very features we ship in Vector to help you avoid data loss in the first place, like buffers. We've already opened #10895 to better guard around this particular issue happening again, and will continue to try and find ways to better codify and test for consistent user experience between releases. |
We didn't properly account for this ahead of time, so we need to make sure the directories used between 0.18.x and master match when the buffer type is
disk
(which is the alias fordisk_v1
), otherwise we'd have introduced a regression.For users who encountered a warning message after upgrading to 0.19.1+:
If you've upgraded to 0.19.1+, you may be here after seeing an error message similar to this:
Scroll down (or click here) to see the comment about how to manually resolve this issue.
The text was updated successfully, but these errors were encountered: