Skip to content
This repository has been archived by the owner on Nov 7, 2022. It is now read-only.

Add batching to queued sender implementations #361

Merged
merged 22 commits into from
Feb 12, 2019
Merged

Add batching to queued sender implementations #361

merged 22 commits into from
Feb 12, 2019

Conversation

sjkaris
Copy link

@sjkaris sjkaris commented Feb 5, 2019

Adds batching (as a feature of queuing) to the current queued sender implementations.

Copy link
Contributor

@songy23 songy23 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: maybe split to multiple PRs for updates to each exporter. This PR is a bit too large.

@sjkaris sjkaris changed the title Add batching to all exporter implementations Add batching to queued sender implementations Feb 5, 2019
@sjkaris
Copy link
Author

sjkaris commented Feb 5, 2019

@songy23 Makes sense, split out the exporter changes (this will just add to the current queued senders), and will make a followup PR containing the changes to allow the other exporters to be batched/queued. Ultimately with the discussion on friday, I can see if we just want to forgo any of these smaller config changes, and just refactor the entire configuration scheme in one go

@sjkaris sjkaris closed this Feb 6, 2019
@sjkaris sjkaris reopened this Feb 6, 2019
@sjkaris sjkaris closed this Feb 11, 2019
@sjkaris sjkaris reopened this Feb 11, 2019
go.mod Outdated Show resolved Hide resolved
Copy link

@pjanotti pjanotti left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @sjkaris, LGTM. Just a few Qs.

Copy link
Member

@odeke-em odeke-em left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Am still reviewing this PR but providing feedback as we go on.

cmd/occollector/app/builder/processor_builder.go Outdated Show resolved Hide resolved
cmd/occollector/app/collector/processors.go Outdated Show resolved Hide resolved
cmd/occollector/app/collector/processors.go Outdated Show resolved Hide resolved
internal/collector/processor/nodebatcher/metrics.go Outdated Show resolved Hide resolved
internal/collector/processor/nodebatcher/metrics.go Outdated Show resolved Hide resolved
internal/collector/processor/nodebatcher/node_batcher.go Outdated Show resolved Hide resolved
internal/collector/processor/nodebatcher/node_batcher.go Outdated Show resolved Hide resolved
// atomic.LoadPointer only takes unsafe.Pointer interfaces. We do not use unsafe
// to skirt around the golang type system.
b = (*batch)(atomic.LoadPointer((*unsafe.Pointer)(unsafe.Pointer(&nb.currBatch))))
cut, closed = b.add(spans)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So am assuming cut here means performedBatching? If so, please use a clearer term like performedBatching.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

used cutBatch instead (mirroring the function name) lmk what you think

@sjkaris sjkaris closed this Feb 12, 2019
@sjkaris sjkaris reopened this Feb 12, 2019
@sjkaris sjkaris closed this Feb 12, 2019
@sjkaris sjkaris reopened this Feb 12, 2019

closed uint32
growMu sync.Mutex
pending int32
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using atomics might fail due to the arrangement of these fields due to potential alignment issue

Field 64 bit size(bytes) 32 bit size(bytes)
items 16 8
currCap 4 4
nextEmptyItem 4 4
sendItemsSize 4 4
close 4 4
growMu 8 8
pending 4 4

Please examine and adjust accordingly, you might want to ensure that all the mutex is first, then integers and then items.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

or rather, integers first, then items and then lastly the mutex but just make sure that anything that gets loaded or stored is either on a 4 or 8 alignment boundary.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Discussed offline -- using 32 bit ints is safe for both 32 and 64 bit systems since golang keeps stuff word-aligned. Moved lastSent to top for alignment since it is 64 bits and therefore needs to be 64-bit aligned for 32 bit systems.

Copy link
Member

@odeke-em odeke-em left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM generally and thank you @sjkaris!

Please take a look at my suggestions to ensure atomics don't crash due to alignment.

@sjkaris sjkaris closed this Feb 12, 2019
@sjkaris sjkaris reopened this Feb 12, 2019
@sjkaris sjkaris closed this Feb 12, 2019
@sjkaris sjkaris reopened this Feb 12, 2019
@sjkaris
Copy link
Author

sjkaris commented Feb 12, 2019

travis sometimes hangs on testing exporters, created #375

@sjkaris sjkaris closed this Feb 12, 2019
@sjkaris sjkaris reopened this Feb 12, 2019
@sjkaris sjkaris merged commit 85ab293 into census-instrumentation:master Feb 12, 2019
@sjkaris sjkaris deleted the add-generic-queueing branch February 12, 2019 21:55
@flands flands added this to the 0.1.2 milestone Feb 13, 2019
TickTime *time.Duration `mapstructure:"tick-time,omitempty"`
// RemoveAfterTicks is the number of ticks that must pass without a span arriving
// from a node after which the batcher for that node will be deleted. This is an
// advanved configuration option.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

advanced

fivesheep pushed a commit to fivesheep/opencensus-service that referenced this pull request Jun 12, 2019
…#361)

Adds batching (as a feature of queuing) to the current queued sender implementations.

Testing Done: unit tests
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants