Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ensure sync results (from background worker) are garbage collected #2209

Open
githubbob42 opened this issue May 4, 2019 · 1 comment
Open
Labels

Comments

@githubbob42
Copy link
Owner

Mingle Card: 2452
See #2439 for story details.

Task

The sync background worker will store sync results in a database (probably mongodb) until they are returned to the client.

Once the client has finished the sync (or, in the case of disconnecting clients) after a reasonable period of time, we may need to remove the old sync results.

This card may not be required if a capped collection is used for sync results.

A TTL (time-to-live) index may be an appropriate choice for expiring old sync results.

The sync background worker will store sync results in a database until they are returned to the client. Once the client has finished the sync or (in the case of disconnecting clients) after a reasonable period of time, we need to remove the old sync results.

Since we're using gridfs to store sync results (in order to bypass the 16MB storage limit of traditional BSON document storage), we cannot use a TTL or capped collection to manage cleanup of old sync packets.

Instead, this PR defines a task for the heroku scheduler.

Setting up the heroku scheduler

installation

$ heroku addons:add scheduler:standard 

configure scheduler

$ heroku addons:open scheduler 

image

On the dashboard page click the Add Job... link, in the text box write the name of the file (without any extension) that defines the task located in the bin folder (in this case it is: cleanup-sync-packets). From the drop-down list select the frequency and adjust next run time. Commit the changes by clicking the save button.

NOTE: By default, the cleanup-sync-packets task will remove any sync packets that are over an hour old. The window can be changed from one hour to any other duration using theFETCH_RESULT_EXPIRES_AFTER environment variable (which accepts a value in milliseconds).

@githubbob42
Copy link
Owner Author

Story: #2196 User initiates a sync request on the client (Mingle)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant