Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Create SparkConf/SparkContext/SparkSession for Jobs #49

Closed
KenSuenobu opened this issue May 20, 2018 · 1 comment
Closed

Create SparkConf/SparkContext/SparkSession for Jobs #49

KenSuenobu opened this issue May 20, 2018 · 1 comment
Labels
difficult Feature that may be difficult to implement distribution Distributed architecture work enhancement New feature or request help wanted Extra attention is needed
Milestone
0.2.0

Comments

@KenSuenobu
Copy link
Owner

KenSuenobu commented May 20, 2018 •
edited
Loading

Allow the ability for a SparkConf and/or SparkContext to be created once, and shared amongst Jobs (not Tasks). This will allow for a Job to be submitted to a running server, allow for the job to wait, then kick off multiple Tasks based on the DAG. Apply to SparkSession, as that is where the context is created.

This will require a lot of testing, as this can get very complex very quickly.
@KenSuenobu KenSuenobu added enhancement New feature or request help wanted Extra attention is needed difficult Feature that may be difficult to implement distribution Distributed architecture work labels May 20, 2018
@KenSuenobu KenSuenobu added this to the 0.2.0 milestone May 20, 2018
@KenSuenobu KenSuenobu changed the title Create SparkConf/SparkContext for Jobs Create SparkConf/SparkContext/SparkSession for Jobs May 30, 2018
@KenSuenobu
Copy link
Owner Author

Most of the tests will be in the form of Job tests and SparkTask tests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
difficult Feature that may be difficult to implement distribution Distributed architecture work enhancement New feature or request help wanted Extra attention is needed
Projects
None yet
Milestone
0.2.0
Development

No branches or pull requests
1 participant
@KenSuenobu