[SPARK-7217][STREAMING] Add configuration to control the default behavior of StreamingContext.stop() implicitly calling SparkContext.stop() #5929
Conversation
|
Merged build triggered. |
|
Merged build started. |
|
Test build #31928 has started for PR 5929 at commit |
|
LGTM |
|
Test build #31928 has finished for PR 5929 at commit
|
|
Merged build finished. Test FAILed. |
|
Test FAILed. |
| * to be processed). | ||
| * | ||
| * @param stopSparkContext if true, stops the associated SparkContext. The underlying SparkContext | ||
| * will be stopped regardless of whether this StreamingContext has been | ||
| * started. | ||
| */ | ||
| def stop(stopSparkContext: Boolean = true): Unit = synchronized { | ||
| def stop(stopSparkContext: Boolean): Unit = synchronized { |
There was a problem hiding this comment.
Hmm, this is actually api incompatible change no? I am wondering if there is a workaround here which does not break compat.
There was a problem hiding this comment.
I thought it wasnt, but it seems to be. I will implement it differently.
On May 5, 2015 6:16 PM, "Hari Shreedharan" [email protected] wrote:
In
streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala
#5929 (comment):* to be processed). * * @param stopSparkContext if true, stops the associated SparkContext. The underlying SparkContext * will be stopped regardless of whether this StreamingContext has been * started. */
- def stop(stopSparkContext: Boolean = true): Unit = synchronized {
- def stop(stopSparkContext: Boolean): Unit = synchronized {
Hmm, this is actually api incompatible change no? I am wondering if there
is a workaround here which does not break compat.—
Reply to this email directly or view it on GitHub
https://github.com/apache/spark/pull/5929/files#r29728863.
|
Merged build triggered. |
|
Merged build started. |
|
Test build #31939 has started for PR 5929 at commit |
|
Test build #31939 has finished for PR 5929 at commit
|
|
Merged build finished. Test PASSed. |
|
Test PASSed. |
|
Seems OK to me. I agree with leaving it "hidden". |
|
@srowen Yep, that's the plan for now. On Tue, May 5, 2015 at 10:37 PM, Sean Owen [email protected] wrote:
|
|
I am merging this. Thanks @srowen @harishreedharan for taking a look. |
…avior of StreamingContext.stop() implicitly calling SparkContext.stop() In environments like notebooks, the SparkContext is managed by the underlying infrastructure and it is expected that the SparkContext will not be stopped. However, StreamingContext.stop() calls SparkContext.stop() as a non-intuitive side-effect. This PR adds a configuration in SparkConf that sets the default StreamingContext stop behavior. It should be such that the existing behavior does not change for existing users. Author: Tathagata Das <[email protected]> Closes #5929 from tdas/SPARK-7217 and squashes the following commits: 869a763 [Tathagata Das] Changed implementation. 685fe00 [Tathagata Das] Added configuration (cherry picked from commit 01187f5) Signed-off-by: Tathagata Das <[email protected]>
…avior of StreamingContext.stop() implicitly calling SparkContext.stop() In environments like notebooks, the SparkContext is managed by the underlying infrastructure and it is expected that the SparkContext will not be stopped. However, StreamingContext.stop() calls SparkContext.stop() as a non-intuitive side-effect. This PR adds a configuration in SparkConf that sets the default StreamingContext stop behavior. It should be such that the existing behavior does not change for existing users. Author: Tathagata Das <[email protected]> Closes apache#5929 from tdas/SPARK-7217 and squashes the following commits: 869a763 [Tathagata Das] Changed implementation. 685fe00 [Tathagata Das] Added configuration
…avior of StreamingContext.stop() implicitly calling SparkContext.stop() In environments like notebooks, the SparkContext is managed by the underlying infrastructure and it is expected that the SparkContext will not be stopped. However, StreamingContext.stop() calls SparkContext.stop() as a non-intuitive side-effect. This PR adds a configuration in SparkConf that sets the default StreamingContext stop behavior. It should be such that the existing behavior does not change for existing users. Author: Tathagata Das <[email protected]> Closes apache#5929 from tdas/SPARK-7217 and squashes the following commits: 869a763 [Tathagata Das] Changed implementation. 685fe00 [Tathagata Das] Added configuration
…avior of StreamingContext.stop() implicitly calling SparkContext.stop() In environments like notebooks, the SparkContext is managed by the underlying infrastructure and it is expected that the SparkContext will not be stopped. However, StreamingContext.stop() calls SparkContext.stop() as a non-intuitive side-effect. This PR adds a configuration in SparkConf that sets the default StreamingContext stop behavior. It should be such that the existing behavior does not change for existing users. Author: Tathagata Das <[email protected]> Closes apache#5929 from tdas/SPARK-7217 and squashes the following commits: 869a763 [Tathagata Das] Changed implementation. 685fe00 [Tathagata Das] Added configuration
In environments like notebooks, the SparkContext is managed by the underlying infrastructure and it is expected that the SparkContext will not be stopped. However, StreamingContext.stop() calls SparkContext.stop() as a non-intuitive side-effect. This PR adds a configuration in SparkConf that sets the default StreamingContext stop behavior. It should be such that the existing behavior does not change for existing users.