Add events Spark Standalone support to the spark agent check #2752
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Added eventing logic for job and stage lifecycles - events work for both YARN and Standalone for both jobs and stages for Spark Standalone
|
@wjsl this is some great stuff. I was actually aware of this limitation and knew sooner or later we'd have to work on something like this. It's quite a PR, so we'll have to review carefully, but this looks great. Thank you so much. |
| ''' | ||
| Figures out what mode we're in and fetches running apps | ||
| ''' | ||
| cluster_mode = instance.get(SPARK_CLUSTER_MODE) |
There was a problem hiding this comment.
We should probably default to SPARK_YARN_MODE - after maybe printing a warning. This is to avoid any backward compatibility breakage for existing customers.
|
Really beautiful PR, great code and some great refactoring @wjsl Just a few minor comments. |
| SPARK_YARN_MODE = 'spark_yarn_mode' | ||
|
|
||
| SPARK_STANDALONE_MASTER = 'spark_standalone_master_uri' | ||
| SPARK_STANDALONE_SERVICE_CHECK = 'spark_standalone_master' |
There was a problem hiding this comment.
This should probably be something along the lines of spark.standalone_master.can_connect
|
Superseded by #2786 - closing. Comments herein still valid. |
zachradtka
pushed a commit
to zdata-inc/dd-agent
that referenced
this pull request
Aug 26, 2016
Specific changes: - Changed `resourcemanager_uri` to `spark_url` - Removed constant SPARK_STANDALONE_MASTER as it was not used - Utilize ''.format() in events - Updated value of SPARK_STANDALONE_SERVICE_CHECK - Moved event titles after if/else logic - Added status checks - General reformatting to standardize spacing and quote types
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Enabled Spark metrics to be gathered when clusters are provisioned using Spark's Standalone cluster mode. Currently the agent checks works when Spark is running with Apache YARN to provision its resources. This PR re-uses the bulk of the collection logic, but retrofits an adapter to locate application servers via Spark Standalone's APIs.
Events have also been added. The agent can now raise events when both jobs and stages have status transitions (ie, moving from WAITING to RUNNING to COMPLETED or FAILED).
Testing Guidelines
Run the mock test using
nosetests. Note that the test fit was only modified to continue force the agent to pretend it is using YARN.