Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move storage off of local disk #84

Closed
cdsmith opened this issue Nov 4, 2014 · 6 comments
Closed

Move storage off of local disk #84

cdsmith opened this issue Nov 4, 2014 · 6 comments

Comments

@cdsmith
Copy link
Collaborator

cdsmith commented Nov 4, 2014

As is, the CodeWorld server reads and writes files on local disk, in the user/ and projects/ directories. This makes it impossible to scale up to more than one server. Instead, we should use some kind of shared storage API (database, etc.) that can be scaled, and shared by many front end systems. Existing projects will need to be migrated into the new store.

@cdsmith
Copy link
Collaborator Author

cdsmith commented Aug 6, 2015

In a slight improvement, I've now moved the storage to a single data directory, which can be symlinked where ever you like, including (in theory, anyway) a shared network volume.

@cdsmith
Copy link
Collaborator Author

cdsmith commented Sep 25, 2016

Should be made easier by commit 0a82241, which now builds from a temporary directory rather than in-place. I'm waiting on fix to be pushed to hackage from brendanhay/gogol#35, before I can move everything over to GCS.

When I do this, I'll make a pluggable storage layer, so that it's still possible to develop locally with storage on disk.

@cdsmith
Copy link
Collaborator Author

cdsmith commented Nov 21, 2016

I poked at this a little this weekend. The gogol bug is fixed, and I can successfully read and write to/from cloud storage from a test app. Great! I've started to build a compatibility layer now, so that it becomes possible to run on cloud storage, or from local disk. Getting there!

I'm focusing on this, because it seems to be the next step to moving off of Google Compute Engine, and onto Google Container Engine. I expect that will help a lot in scalability, and hopefully in cost, as well.

@cdsmith
Copy link
Collaborator Author

cdsmith commented Mar 2, 2017

See #448

@cdsmith
Copy link
Collaborator Author

cdsmith commented Jul 16, 2018

Rethinking the approach here. We really just want a network drive, and switching to Google Cloud Storage for it is not worth the bother. There are now other people running CodeWorld servers, and I can't trust them all to run it on Google Cloud.

So I should just use a shared network drive. Too bad Google Cloud doesn't make that easy...

@cdsmith
Copy link
Collaborator Author

cdsmith commented Jul 17, 2018

This is done. CodeWorld data is now stored on an NFS volume.

@cdsmith cdsmith closed this as completed Jul 17, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant