Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle out of disk space failures better #3028

Open
4 tasks
whyrusleeping opened this issue Aug 2, 2016 · 2 comments
Open
4 tasks

Handle out of disk space failures better #3028

whyrusleeping opened this issue Aug 2, 2016 · 2 comments
Labels
kind/bug A bug in existing code (including security flaws) topic/repo Topic repo

Comments

@whyrusleeping
Copy link
Member

The gateways have now gotten into bad states from running out of disk space a few times. Fetching content from gateways that are out of space results in something like this: https://a.uguu.se/UnPPwtbEJHS1.png

We should:

  • Catch and return these errors if we encounter them.
    • Randomly closing the connection on users isnt good behaviour
  • Make sure that garbage collections are run frequently enough
    • should also trigger a garbage collection if an 'out of disk space' error is encountered
  • add tests for these conditions using either sized subvolumes (btrfs, zfs) or fixed size tmpfs mounts.

And while we're at it:

  • Respect set 'repo size limitations'
@whyrusleeping
Copy link
Member Author

Another issue that just happened and is related:

Node ran out of disk space and seemed to crash, docker attempted to restart it and failed because it needed disk space on startup and that broke it.

@whyrusleeping whyrusleeping added kind/bug A bug in existing code (including security flaws) topic/repo Topic repo labels Aug 2, 2016
@whyrusleeping
Copy link
Member Author

Another observation: It looks like if a node runs out of disk space and doesnt crash, it gets into a bad state where even if space is freed up, it thinks its out of space still.

@whyrusleeping whyrusleeping added the help wanted Seeking public contribution on this issue label Aug 19, 2016
@Stebalien Stebalien removed the help wanted Seeking public contribution on this issue label Feb 20, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug A bug in existing code (including security flaws) topic/repo Topic repo
Projects
None yet
Development

No branches or pull requests

2 participants