-
Notifications
You must be signed in to change notification settings - Fork 3
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Move user retirement code to edx-platform and drop it from Tubular #881
Comments
Please don't bring over the mongo structure pruning code–just kill it entirely. It's potentially dangerous to run since we shifted the indexes into MySQL. 2U stopped running it a while back. |
PR is ready for code review: openedx/edx-platform#34063
Script I used
|
@farhan I think we should still remove them to reduce confusion. |
@ormsbee just joining the conversation late, I'm curious about the way 2U is handling the Mongo DB structures, if they stopped running the script, the structures keep growing with no control? Do they have a different mechanism to prune the old structures? I'm asking this because this tool helps us to solve issues with very large databases that contain a huge amount of old structures which make the backups difficult to run. |
Please don't run pruning code. I'm out of office now, will explain more next week, but pruning might not be a safe operation any longer. |
Ticket Update:
cc: @feanil |
Okay, I think pruning is still safe because we're writing the index to both places: https://github.com/openedx/edx-platform/blob/fca51419f8facbcb0e6eaad7750bafee23f92509/xmodule/modulestore/split_mongo/mongo_connection.py#L756-L759 My original concern is this: The pruning code currently only looks at MongoDB. It was written at a really low level for performance reasons, and it predates the We should probably change the comment in that code above to indicate that it's still necessary for cleanup safety. @bradenmacdonald: Does this sound right to you? |
@ormsbee it sounds like that mean it's worth moving it out of here before we deprecate this repo? |
@ormsbee That sounds right. Also I don't know if it's happened at all recently, but we have observed some instances in the past where the two were not in sync. Since they're separate database systems, there's no way to completely guarantee consistency. |
|
@robrap everything but |
@feanil and @ormsbee, so can I nuke this mongo-pruner job within our internal jenkins jobs? I read the initial description without following up on the comments and assumed that this |
@ohnickmoy if we're gonna keep it, we'll probably move it to edx-platform so you won't have to revert the job, we can just move that script. I'll check in with Dave today on it to figure out what the next steps should be. |
@ohnickmoy: I thought edx.org had stopped running this pruning a while back. It's definitely still running today? I feel conflicted about this one because while the risk of losing sync is low, the consequences would be really bad, since it would mean data loss of actively used content. @feanil: I guess the real thing to do here would be to make the pruner safe by having it check for and pull from the MySQL table for active version information, so that it's always has the right picture (or can at least abort if it sees a mismatch). So maybe let's bring the @ohnickmoy: I still think the recurring pruner job should be stopped until that fix happens ^. |
@farhan can you mane one more PR to move over the structures.py file and update the relevant docs on the tubular side? |
Probably goes without saying, but bringing over Thank you for your patience. |
@ormsbee, from what I can see on tools-jenkins, there have been recent runs in the last month. I'll take a look closer if it's running on a schedule or if it's been run manually EDIT: Yep running on a schedule. All three environments |
Update deprecations are in place for most things, we're in the process of moving the pruning code: openedx/edx-platform#34328 Once that's done, we should be good to archive the tubular repo. |
@feanil, im reading the PR. in sum, the structures.py job is not ready to migrate yet until merged? |
@ohnickmoy correct, but that PR has now merged so that job should be good to migrate now. |
PR that added the final deprecation warnings before we archived the repo: openedx-unsupported/tubular#744 |
thanks @feanil. Will get to it |
Context
The tubular repo mostly contains tools that are operationally relevant to the edx/2U org. However there are a few scripts that are relevant to the entire community.
Tasks
Relevant Scripts
Note: There will be other library code that will also need to be moved like the
edx_apy.py
. I didn't list everything that needs to be moved, just the top level scripts that need to be moved. You should move everything we need to make them work.Acceptance Criteria
Other Useful Links
The text was updated successfully, but these errors were encountered: