Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[bugfix] Close reader gracefully when streaming recache of remote media to fileserver api caller #1281

Merged
merged 7 commits into from
Dec 21, 2022

Conversation

tsmethurst
Copy link
Contributor

@tsmethurst tsmethurst commented Dec 18, 2022

Description

If this is a code change, please include a summary of what you've coded, and link to the issue(s) it closes/implements.

If this is a documentation change, please briefly describe what you've changed and why.

This PR implements graceful failure when streaming an image to an API client during a media recache, in cases where the media is no longer present on a remote instance, or some other failure occurs.

Here's the situation before:

  1. a client request a piece of media that was cached previously, but needs to be recached now
  2. the async Data function is passed back to the worker, with a tee reader pipe jammed into it which the API caller will slurp from as the media is recached into storage
  3. the caller waits for data to be blasted into the pipe
  4. the Data function fails
  5. the pipe stays open, the caller keeps waiting for that data, which will never come
  6. request eventually times out (depending on reverse proxy / client settings), but the gin request hangs around inside gts forever blocked

With these changes now:

  1. a client request a piece of media that was cached previously, but needs to be recached now
  2. the async Data function is passed back to the worker, with a tee reader pipe jammed into it which the API caller will slurp from as the media is recached into storage
  3. the caller waits for data to be blasted into the pipe
  4. the Data function fails, and the pipe reader is closed when this happens
  5. the caller in the API function checks if any data is coming through the pipe, or if there's an EOF
  6. in case of error, a 404 is returned to the caller immediately and the request completes + gets cleaned up

Checklist

Please put an x inside each checkbox to indicate that you've read and followed it: [ ] -> [x]

If this is a documentation change, only the first checkbox must be filled (you can delete the others if you want).

  • I/we have read the GoToSocial contribution guidelines.
  • I/we have discussed the proposed changes already, either in an issue on the repository, or in the Matrix chat.
  • I/we have performed a self-review of added code.
  • I/we have written code that is legible and maintainable by others.
  • I/we have commented the added code, particularly in hard-to-understand areas.
  • I/we have made any necessary changes to documentation (N/A)
  • I/we have added tests that cover new code.
  • I/we have run tests and they pass locally with the changes.
  • I/we have run go fmt ./... and golangci-lint run.

@tsmethurst tsmethurst changed the title Everlasting attachment wait [ugfix] Close reader gracefully when streaming recache of remote media to fileserver api caller Dec 18, 2022
@tsmethurst tsmethurst marked this pull request as ready for review December 19, 2022 10:42
@tsmethurst tsmethurst changed the title [ugfix] Close reader gracefully when streaming recache of remote media to fileserver api caller [bugfix] Close reader gracefully when streaming recache of remote media to fileserver api caller Dec 19, 2022
…uffering while piping recache stream

Signed-off-by: kim <[email protected]>
@NyaaaWhatsUpDoc
Copy link
Member

Checked over and approved this earlier. Once you're happy with the extra few changes I added @tsmethurst this is good to go :)

@tsmethurst tsmethurst merged commit 6ebdc30 into main Dec 21, 2022
@tsmethurst tsmethurst deleted the everlasting_attachment_wait branch December 21, 2022 10:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants