Skip to content
This repository has been archived by the owner on Jan 20, 2022. It is now read-only.

Refactor SummerBuilder to make it possible to use Counters on SummerBuilder creation #738

Merged
merged 4 commits into from
Jun 30, 2017

Conversation

ttim
Copy link
Collaborator

@ttim ttim commented Jun 27, 2017

Currently the way to customize Summer instances is not perfect - we have an old 'legacy' way based on a lot of different options and 'new' way with setting SummerBuilder directly.
At the same time with new way it's impossible to fully utilize Counters because jobId is needed to create them. Also new way is quite verbose.

This PR adds a way to use Counters in SummerBuilder and couple of SummerBuilders: Null, sync and async.

Verified

This commit was signed with the committer’s verified signature.
ssheikin Sasha Sheikin
…mmer` creation
@ttim
Copy link
Collaborator Author

ttim commented Jun 27, 2017

There is an issue for that: #711

@ttim ttim requested review from pankajroark and johnynek June 27, 2017 06:49
@codecov-io
Copy link

codecov-io commented Jun 27, 2017

Codecov Report

Merging #738 into develop will decrease coverage by 0.13%.
The diff coverage is 39.28%.

Impacted file tree graph

@@             Coverage Diff             @@
##           develop     #738      +/-   ##
===========================================
- Coverage    71.99%   71.86%   -0.14%     
===========================================
  Files          152      153       +1     
  Lines         3721     3735      +14     
  Branches       208      202       -6     
===========================================
+ Hits          2679     2684       +5     
- Misses        1042     1051       +9
Impacted Files Coverage Δ
...om/twitter/summingbird/online/option/AllOpts.scala 62.5% <0%> (-20.84%) ⬇️
...om/twitter/summingbird/online/option/Summers.scala 32.35% <32.35%> (ø)
...la/com/twitter/summingbird/storm/BuildSummer.scala 50% <55%> (+6.52%) ⬆️
.../twitter/summingbird/memory/ConcurrentMemory.scala 97.14% <0%> (+1.42%) ⬆️
.../main/scala/com/twitter/summingbird/Producer.scala 77.27% <0%> (+1.51%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6f8dd8f...22532c0. Read the comment docs.

@ttim
Copy link
Collaborator Author

ttim commented Jun 27, 2017

@johnynek I know here is quite a lot of code without test coverage, if you think it worth it to add tests on that - I can do this. But personally I think it's ok to leave this kind of code without tests.

case class SummerConstructor(get: SummerBuilder)
case class SummerConstructor(get: SummerConstructorSpec)

trait SummerConstructorSpec {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is it called SummerConstructorSpec, there's no SummerConstructor in definition.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After offline discussion renamed it to SummerWithCountersBuilder.

case class SummerConstructor(get: SummerConstructorSpec)

trait SummerConstructorSpec {
def builder(counter: Name => Counter with Incrementor): SummerBuilder
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tbh, I think we should just change the definition of SummerBuilder to take the counterBuilder as argument and default to a no-op counterBuilder. SummerBuilder is used very rarely and I think it's ok to break it. There are only a couple places that we'll need to fix at Twitter. Seems better than adding a level of indirection here.
@johnynek wdyt?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did this change in backward compatible manner - it's still possible to create SummerWithCountersBuilder out of plain SummerBuilder.

val valueCombinerCrushSize = option(DEFAULT_VALUE_COMBINER_CACHE_SIZE)
val doCompact = option(CompactValues.default)
Summers.async(
cacheSize, flushFrequency, softMemoryFlush, asyncPoolSize, doCompact, valueCombinerCrushSize
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

too long, move each param to separate line.

Copy link
Collaborator

@johnynek johnynek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

some minor comments, but it looks like a nice refactor.

case class SummerConstructor(get: SummerWithCountersBuilder)

trait SummerWithCountersBuilder {
def create(counter: Name => Counter with Incrementor): SummerBuilder
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not crazy about with what about Name => (Counter, Incrementor) do they have to be bound together?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I prefer to keep it this way (or maybe change to just Incrementor). It should be the same object, not both of them.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed to just Incrementor.

case class SummerConstructor(get: SummerBuilder)
case class SummerConstructor(get: SummerWithCountersBuilder)

trait SummerWithCountersBuilder {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you probably want this to extend Serializable or we may have some corner case issues.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually I prefer to keep it non serializable, this code (creating actual SummerBuilder with Counters) should happen only on submitter node, and SummerBuilder is an object which should be serializable (and it is).

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Commented in code

def apply(builder: StormTopologyBuilder, node: StormNode): SummerBuilder = {
val summerBuilder = builder.get[SummerConstructor](node)
.map { case (_, constructor) => constructor.get }.getOrElse {
logger.info(s"[${builder.getNodeName(node)}] use legacy way of getting summer builder")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we indent here?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure

@ttim
Copy link
Collaborator Author

ttim commented Jun 30, 2017

@johnynek @pankajroark can you look again did I point all your comments? Thanks!

@johnynek
Copy link
Collaborator

👍

@ttim ttim merged commit c272b2a into twitter:develop Jun 30, 2017
@ttim ttim deleted the summer_build_params branch July 1, 2017 00:27
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants