Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make user-partitioning strategy configurable #33

Open
dcrankshaw opened this issue Mar 30, 2015 · 1 comment
Open

Make user-partitioning strategy configurable #33

dcrankshaw opened this issue Mar 30, 2015 · 1 comment
Labels

Comments

@dcrankshaw
Copy link
Contributor

The user partitioning strategy in use is needed by several different parts of the system. I would like to standardize this and make sure it only gets defined in one place.

Here is a partial list of places where the partitioning info is necessary:

  • The client needs to know the partitioning strategy to direct the request to the correct velox partition (this means hostname currently, as right now there is one Velox partition per machine).
  • The server's need to know for a given incoming request whether they are responsible for that user. If it is a user that is directed to the right machine and Velox can't find a model for them, we can assume it is a new user and serve the request. If the request was directed to the wrong machine, then Velox should return an error message.
  • If Spark retrains user models as well as feature models, each Velox partition should only read the updated models for the users that belong to that partition.
  • If adding a new Velox machine to the serving cluster, the partitioning strategy is needed to determine how to migrate users to the new cluster member. Similarly, this info is needed for failover if a machine goes down.
@dcrankshaw dcrankshaw added this to the 0.1 Alpha Release milestone Mar 30, 2015
@dcrankshaw dcrankshaw self-assigned this Mar 31, 2015
@tomerk
Copy link
Contributor

tomerk commented Apr 3, 2015

After talking with @dcrankshaw: We shouldn't fail a request that's sent to the wrong partition, we should forward it to the correct partition & log that it needed to do a hop to the correct partition

@dcrankshaw dcrankshaw removed this from the 0.1 Alpha Release milestone May 4, 2015
@dcrankshaw dcrankshaw removed their assignment May 4, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants