Track state in summingbird-online as an Iterator rather than a Seq. #703

pankajroark · 2016-12-24T08:51:52Z

Fix for #689 . This should avoid n^2 compute complexity when summing single element lists of Storm tuples.

…his should avoid n^2 compute comlexity when summing single element lists of Storm tuples.

johnynek · 2016-12-24T18:48:21Z

could we use Stream instead? the mutability of an Iterator makes it pretty costly to verify that it is not buggy with a code review. I really hate to have a mutable object on an API unless the performance is really necessitating it?

pankajroark · 2016-12-25T01:35:44Z

Good idea. Changed to use stream instead of iterator.

pankajroark · 2016-12-25T04:15:19Z

tbh Iterator is 4x faster than stream in this microbenchmark but both are way faster than List, so stream seems fine:
val s = (0 to 10000).toList

@benchmark
def listConcat(): List[Int] = {
s ++ List(0)
}

@benchmark
def streamConcat(): Stream[Int] = {
s.toStream ++ Stream(0)
}

@benchmark
def iterConcat(): Iterator[Int] = {
s.toIterator ++ Iterator.single(0)
}

Results:
[info] ToBenchmark.iterConcat thrpt 4 85182247.256 ± 5840222.962 ops/s
[info] ToBenchmark.listConcat thrpt 4 9937.918 ± 26498.037 ops/s
[info] ToBenchmark.streamConcat thrpt 4 22019587.193 ± 7199199.008 ops/s

johnynek · 2016-12-25T04:19:46Z

I would bet Batched will be as fast or faster than Iterator.

…

On Sat, Dec 24, 2016 at 18:15 Pankaj Gupta ***@***.***> wrote: tbh Iterator is 4x faster than stream in this microbenchmark but both are way faster than List, so stream seems fine: val s = (0 to 10000).toList @benchmark <https://github.com/Benchmark> def listConcat(): List[Int] = { s ++ List(0) } @benchmark <https://github.com/Benchmark> def streamConcat(): Stream[Int] = { s.toStream ++ Stream(0) } @benchmark <https://github.com/Benchmark> def iterConcat(): Iterator[Int] = { s.toIterator ++ Iterator.single(0) } Results: [info] ToBenchmark.iterConcat thrpt 4 85182247.256 ± 5840222.962 ops/s [info] ToBenchmark.listConcat thrpt 4 9937.918 ± 26498.037 ops/s [info] ToBenchmark.streamConcat thrpt 4 22019587.193 ± 7199199.008 ops/s — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#703 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEJdpCK_zBnqzN-dyCdF8YtGW_3fhfxks5rLe3YgaJpZM4LVMH_> .

pankajroark · 2016-12-25T04:37:42Z

I tried Batched but the code started getting complicated because of https://github.com/twitter/summingbird/blob/develop/summingbird-online/src/main/scala/com/twitter/summingbird/online/executor/AsyncBase.scala#L46

We need to be able to pass an empty Batched here. Batched itself doesn't have a zero value it relies on the contained type. So we either have to use Option[Batched] which wouldn't be good for performance, always having to wrap/unwrap. Or we have to have the monoid of S available in this abstract class, which would require further code changes.

pankajroark · 2016-12-25T04:47:21Z

Just for comparison iterator and Batched are indeed comparable in perf. I'd really like to use Batched if we could find a simple way or iterator otherwise. Combining input state is in the hot path.:
val s = (0 to 10000).toList
val bs = Batched.items(s).get
val is = s.toIterator
val ss = s.toStream

@benchmark
def listConcat(): List[Int] = {
s ++ List(0)
}

@benchmark
def streamConcat(): Stream[Int] = {
ss ++ Stream(0)
}

@benchmark
def iterConcat(): Iterator[Int] = {
is ++ Iterator.single(0)
}

@benchmark
def batchedConcat(): Batched[Int] = {
bs.combine(Batched(0))
}

[info] ToBenchmark.batchedConcat thrpt 4 126300963.122 ± 46804561.489 ops/s
[info] ToBenchmark.iterConcat thrpt 4 121149476.337 ± 46339829.598 ops/s
[info] ToBenchmark.listConcat thrpt 4 9380.640 ± 23140.126 ops/s
[info] ToBenchmark.streamConcat thrpt 4 24118157.945 ± 7172767.691 ops/s

pankajroark · 2016-12-31T02:56:21Z

Any suggestions on the next steps here. I'm ok with Stream, it's still much better than List in this case.

johnynek · 2017-01-02T18:04:03Z

I have not investigated why the tests are red.

If we can make them green with something faster. I'd be happy to do that.

I don't think using Batched would be too much work. Or we could copy this code or add this dependency:

https://github.com/non/chain/blob/master/src/main/scala/chain/Chain.scala

It is a single file library that is the more general version of Batched (it has empty), it was written by @non who also wrote Batched.

Do we really need an empty Batched? I would imagine that a Monoid[Option[Batched[T]]] would be almost as fast (still faster than Stream).

It is up to you. I think killing the O(N^2) is most important. Losing a constant factor of 4 is probably not a huge deal if you don't want to work on this other stuff.

Not using a mutable data structure is pretty important to me since this code has been worked on by many people now, and it is much easier to make a mistake with mutable APIs.

pankajroark · 2017-01-02T19:09:55Z

Thanks let me try chained. I agree, constant factor of 4 is insignificant compared to n^2. Tests are failing the Mima check.

…

On Mon, Jan 2, 2017 at 10:04 AM P. Oscar Boykin ***@***.***> wrote: I have not investigated why the tests are red. If we can make them green with something faster. I'd be happy to do that. I don't think using Batched would be too much work. Or we could copy this code or add this dependency: https://github.com/non/chain/blob/master/src/main/scala/chain/Chain.scala It is a single file library that is the more general version of Batched (it has empty), it was written by @non <https://github.com/non> who also wrote Batched. Do we really need an empty Batched? I would imagine that a Monoid[Option[Batched[T]]] would be almost as fast (still faster than Stream). It is up to you. I think killing the O(N^2) is most important. Losing a constant factor of 4 is probably not a huge deal if you don't want to work on this other stuff. Not using a mutable data structure is pretty important to me since this code has been worked on by many people now, and it is much easier to make a mistake with mutable APIs. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#703 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAojhv2OALNrtkVtF2XM1awk5KsqCOuAks5rOTwWgaJpZM4LVMH_> .

johnynek · 2017-01-02T19:36:59Z

Can you add the exclusions to so we can keep the tests green? It prints what you need to add to the build to exclude those methods from erroring. On Mon, Jan 2, 2017 at 9:09 AM Pankaj Gupta <[email protected]> wrote:

…

Thanks let me try chained. I agree, constant factor of 4 is insignificant compared to n^2. Tests are failing the Mima check. On Mon, Jan 2, 2017 at 10:04 AM P. Oscar Boykin ***@***.***> wrote: > I have not investigated why the tests are red. > > If we can make them green with something faster. I'd be happy to do that. > > I don't think using Batched would be too much work. Or we could copy this > code or add this dependency: > > https://github.com/non/chain/blob/master/src/main/scala/chain/Chain.scala > > It is a single file library that is the more general version of Batched > (it has empty), it was written by @non <https://github.com/non> who also > wrote Batched. > > Do we really need an empty Batched? I would imagine that a > Monoid[Option[Batched[T]]] would be almost as fast (still faster than > Stream). > > It is up to you. I think killing the O(N^2) is most important. Losing a > constant factor of 4 is probably not a huge deal if you don't want to work > on this other stuff. > > Not using a mutable data structure is pretty important to me since this > code has been worked on by many people now, and it is much easier to make a > mistake with mutable APIs. > > — > You are receiving this because you authored the thread. > > > Reply to this email directly, view it on GitHub > <#703 (comment) >, > or mute the thread > < https://github.com/notifications/unsubscribe-auth/AAojhv2OALNrtkVtF2XM1awk5KsqCOuAks5rOTwWgaJpZM4LVMH_ > > . > — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#703 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEJdsU6cNEO-Wai0iYGYHNXOgmudGDDks5rOUuEgaJpZM4LVMH_> .

pankajroark · 2017-01-02T19:47:01Z

Sure I'll take a look later today. On Mon, Jan 2, 2017 at 11:37 AM P. Oscar Boykin <[email protected]> wrote:

…

Can you add the exclusions to so we can keep the tests green? It prints what you need to add to the build to exclude those methods from erroring. On Mon, Jan 2, 2017 at 9:09 AM Pankaj Gupta ***@***.***> wrote: > Thanks let me try chained. I agree, constant factor of 4 is insignificant > compared to n^2. Tests are failing the Mima check. > > On Mon, Jan 2, 2017 at 10:04 AM P. Oscar Boykin < ***@***.***> > wrote: > > > I have not investigated why the tests are red. > > > > If we can make them green with something faster. I'd be happy to do that. > > > > I don't think using Batched would be too much work. Or we could copy this > > code or add this dependency: > > > > > https://github.com/non/chain/blob/master/src/main/scala/chain/Chain.scala > > > > It is a single file library that is the more general version of Batched > > (it has empty), it was written by @non <https://github.com/non> who also > > wrote Batched. > > > > Do we really need an empty Batched? I would imagine that a > > Monoid[Option[Batched[T]]] would be almost as fast (still faster than > > Stream). > > > > It is up to you. I think killing the O(N^2) is most important. Losing a > > constant factor of 4 is probably not a huge deal if you don't want to > work > > on this other stuff. > > > > Not using a mutable data structure is pretty important to me since this > > code has been worked on by many people now, and it is much easier to > make a > > mistake with mutable APIs. > > > > — > > You are receiving this because you authored the thread. > > > > > > Reply to this email directly, view it on GitHub > > < #703 (comment) > >, > > or mute the thread > > < > https://github.com/notifications/unsubscribe-auth/AAojhv2OALNrtkVtF2XM1awk5KsqCOuAks5rOTwWgaJpZM4LVMH_ > > > > . > > > > — > You are receiving this because you commented. > > > Reply to this email directly, view it on GitHub > <#703 (comment) >, > or mute the thread > < https://github.com/notifications/unsubscribe-auth/AAEJdsU6cNEO-Wai0iYGYHNXOgmudGDDks5rOUuEgaJpZM4LVMH_ > > . > — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#703 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAojht74AChMPen2r3kCCNADToPp9s9Sks5rOVHggaJpZM4LVMH_> .

pankajroark · 2017-01-03T03:30:09Z

Chain seems good:
[info] ToBenchmark.batchedConcat thrpt 4 133738359.414 ± 3281958.077 ops/s
[info] ToBenchmark.chainConcat thrpt 4 90914620.450 ± 19299667.614 ops/s
[info] ToBenchmark.iterConcat thrpt 4 122448980.991 ± 38379171.178 ops/s
[info] ToBenchmark.streamConcat thrpt 4 26017386.004 ± 6404231.437 ops/s

I've updated the review with now using chain.

pankajroark · 2017-01-04T01:00:08Z

Tests pass now, just waiting for shipit to merge.

johnynek · 2017-01-04T01:19:56Z

summingbird-storm/src/main/scala/com/twitter/summingbird/storm/BaseBolt.scala

@@ -149,12 +150,13 @@ case class BaseBolt[I, O](jobID: JobId,
    }
  }

-  private def finish(inputs: Stream[InputState[Tuple]], results: TraversableOnce[O]) {
+  private def finish(inputs: Chain[InputState[Tuple]], results: TraversableOnce[O]) {
+    val tuples = inputs.iterator.map(_.state).toList


shouldn't we move this line to 158 (only materialize the List if we have dependants and we anchor?

My reasoning was that we need to iterate through the chain anyway to get the size for the log statement at the end of function but I agree, materialization of list should be avoided as well in those cases. Proposed a fix.

johnynek · 2017-01-04T01:43:57Z

summingbird-storm/src/main/scala/com/twitter/summingbird/storm/BaseBolt.scala

    var emitCount = 0
    if (hasDependants) {
      if (anchorTuples.anchor) {
        results.foreach { result =>
+          val tuples = inputs.iterator.map(_.state).toList
+          numTuples = Some(tuples.size)


list .size is O(N) not O(1). So if we are going to do this, why use the var and not just use use inputs.iterator.size.

Good point. We should be able to calculate the size and list in one pass though. Let me try.

johnynek · 2017-01-04T01:44:47Z

summingbird-storm/src/main/scala/com/twitter/summingbird/storm/BaseBolt.scala

    var emitCount = 0
    if (hasDependants) {
      if (anchorTuples.anchor) {
        results.foreach { result =>
+          val tuples = inputs.iterator.map(_.state).toList


don't we want this above the foreach? We don't want to recompute tuples for each result. We just want it once and then reuse for all results, no?

Yeah, good idea. I believe this was how it was in original code too so I didn't notice. Let me try to fix this.

codecov-io · 2017-01-04T03:51:57Z

Current coverage is 70.96% (diff: 80.95%)

No coverage report found for develop at 9a80b22.

Powered by Codecov. Last update 9a80b22...a90c9da

johnynek · 2017-01-04T04:49:10Z

👍

ttim · 2017-01-04T17:32:25Z

What is a reason to expose Chain on a level of OperationContainer ? I'm not strictly against that but Traversable or even TraversableOnce seems more appropriate for me.

To be more precise - AsyncSummer instances should be over Chain to avoid N^2 complexity while everything else should be over Traversable/TraversableOnce. What do you think?

johnynek · 2017-01-04T17:42:15Z

I think making a follow up PR that minimized the scope of visibility of Chain (maybe even walking back some of the mima exclusions) would be fine. But if we have to copy to do it, I would not.

For instance, you can go: Iterable[T] => Chain[T] but not Iterator[T] => Chain[T] without a copy.

So, maybe we could use Iterable[T] in some cases, and internally use a Chain[T] to do fast concat.

That said, these are fairly "private" classes in that 99% of summingbird users would never use them. In fact, likely only storm/heron platform would use them.

ttim · 2017-01-04T17:55:03Z

I agree.

Regarding to copying - we don't need Iterator[T] => Chain[T] transformation because OperationContainer#execute accepts single state element. But we need Chain[T] => Iterable[T] transformation which is the same (in terms of copying) as Chain[T] => Iterator[T].

pankajroark · 2017-01-04T18:41:24Z

I agree exposing chain at OperationContainer is not ideal and also that OperationContainer is used only in storm/heron platform right now, so I feel it's better to keep it simple, as is, for now. Let me create an issue to capture this though.

Track state in summingbird-online as an Iterator rather than a Seq. T…

50eb9b4

…his should avoid n^2 compute comlexity when summing single element lists of Storm tuples.

Use Stream instead of iterator for tracking InputState

84842a4

Rename semigroup.

2188e3e

Use chain instead of stream

8ca59c6

Add mima exclusions.

17e5e63

johnynek suggested changes Jan 4, 2017

View reviewed changes

Avoid materializing list to get tuple count in some cases.

e40133b

johnynek reviewed Jan 4, 2017

View reviewed changes

pankajroark added 2 commits January 3, 2017 18:09

Oscar's comments.

8e42a62

Simplify

a90c9da

pankajroark merged commit 77b65d5 into develop Jan 4, 2017

johnynek deleted the pg/inputstate_seq_opt branch January 4, 2017 17:42

pankajroark mentioned this pull request Jan 4, 2017

Operation container should use a more general container than Chain #705

Open

ameet20 mentioned this pull request Jan 24, 2019

Use more container than chain #773

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Track state in summingbird-online as an Iterator rather than a Seq. #703

Track state in summingbird-online as an Iterator rather than a Seq. #703

pankajroark commented Dec 24, 2016

johnynek commented Dec 24, 2016

pankajroark commented Dec 25, 2016

pankajroark commented Dec 25, 2016

johnynek commented Dec 25, 2016 via email

pankajroark commented Dec 25, 2016

pankajroark commented Dec 25, 2016

pankajroark commented Dec 31, 2016

johnynek commented Jan 2, 2017

pankajroark commented Jan 2, 2017 via email

johnynek commented Jan 2, 2017 via email

pankajroark commented Jan 2, 2017 via email

pankajroark commented Jan 3, 2017

pankajroark commented Jan 4, 2017

johnynek Jan 4, 2017

pankajroark Jan 4, 2017

johnynek Jan 4, 2017

pankajroark Jan 4, 2017

johnynek Jan 4, 2017

pankajroark Jan 4, 2017

codecov-io commented Jan 4, 2017 •

edited

Loading

johnynek commented Jan 4, 2017

ttim commented Jan 4, 2017

johnynek commented Jan 4, 2017

ttim commented Jan 4, 2017

pankajroark commented Jan 4, 2017

Track state in summingbird-online as an Iterator rather than a Seq. #703

Track state in summingbird-online as an Iterator rather than a Seq. #703

Conversation

pankajroark commented Dec 24, 2016

johnynek commented Dec 24, 2016

pankajroark commented Dec 25, 2016

pankajroark commented Dec 25, 2016

johnynek commented Dec 25, 2016 via email

pankajroark commented Dec 25, 2016

pankajroark commented Dec 25, 2016

pankajroark commented Dec 31, 2016

johnynek commented Jan 2, 2017

pankajroark commented Jan 2, 2017 via email

johnynek commented Jan 2, 2017 via email

pankajroark commented Jan 2, 2017 via email

pankajroark commented Jan 3, 2017

pankajroark commented Jan 4, 2017

johnynek Jan 4, 2017

Choose a reason for hiding this comment

pankajroark Jan 4, 2017

Choose a reason for hiding this comment

johnynek Jan 4, 2017

Choose a reason for hiding this comment

pankajroark Jan 4, 2017

Choose a reason for hiding this comment

johnynek Jan 4, 2017

Choose a reason for hiding this comment

pankajroark Jan 4, 2017

Choose a reason for hiding this comment

codecov-io commented Jan 4, 2017 • edited Loading

Current coverage is 70.96% (diff: 80.95%)

johnynek commented Jan 4, 2017

ttim commented Jan 4, 2017

johnynek commented Jan 4, 2017

ttim commented Jan 4, 2017

pankajroark commented Jan 4, 2017

codecov-io commented Jan 4, 2017 •

edited

Loading