-
Notifications
You must be signed in to change notification settings - Fork 17
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cross-actor, shm, (tick) ring buffer #107
Labels
data-layer
real-time and historical data processing and storage
feature-request
New feature or request
integration
external stack and/or lib augmentations
Comments
goodboy
added
feature-request
New feature or request
help wanted
integration
external stack and/or lib augmentations
data-layer
real-time and historical data processing and storage
labels
Aug 20, 2020
goodboy
added a commit
that referenced
this issue
Sep 16, 2020
This adds a shared memory "incrementing array" sub-sys interface for single writer, multi-reader style data passing. The main motivation is to avoid multiple copies of the same `numpy` array across actors (plus now we can start being fancy like ray). There still seems to be some odd issues with the "resource tracker" complaining at teardown (likely partially to do with SIGINT stuff) so some further digging in the stdlib code is likely coming. Pertains to #107 and #98
goodboy
added a commit
that referenced
this issue
Sep 26, 2020
This adds a shared memory "incrementing array" sub-sys interface for single writer, multi-reader style data passing. The main motivation is to avoid multiple copies of the same `numpy` array across actors (plus now we can start being fancy like ray). There still seems to be some odd issues with the "resource tracker" complaining at teardown (likely partially to do with SIGINT stuff) so some further digging in the stdlib code is likely coming. Pertains to #107 and #98
goodboy
added a commit
that referenced
this issue
Sep 29, 2020
This adds a shared memory "incrementing array" sub-sys interface for single writer, multi-reader style data passing. The main motivation is to avoid multiple copies of the same `numpy` array across actors (plus now we can start being fancy like ray). There still seems to be some odd issues with the "resource tracker" complaining at teardown (likely partially to do with SIGINT stuff) so some further digging in the stdlib code is likely coming. Pertains to #107 and #98
goodboy
added a commit
that referenced
this issue
Oct 2, 2020
This adds a shared memory "incrementing array" sub-sys interface for single writer, multi-reader style data passing. The main motivation is to avoid multiple copies of the same `numpy` array across actors (plus now we can start being fancy like ray). There still seems to be some odd issues with the "resource tracker" complaining at teardown (likely partially to do with SIGINT stuff) so some further digging in the stdlib code is likely coming. Pertains to #107 and #98
data_pipe is another one to check out (though not sure it'll have Looks to imply it has some disruptor style examples. |
goodboy
added a commit
that referenced
this issue
Jun 15, 2021
Adding binance's "hft" ws feeds has resulted in a lot of context switching in our Qt charts, so much so it's chewin CPU and definitely worth it to throttle to the detected display rate as per discussion in issue #192. This is a first very very naive attempt at throttling L1 tick feeds on the `brokerd` end (producer side) using a constant and uniform delivery rate by way of a `trio` task + mem chan. The new func is `data._sampling.uniform_rate_send()`. Basically if a client request a feed and provides a throttle rate we just spawn a task and queue up ticks until approximately the next display rate's worth period of time has passed before forwarding. It's definitely nothing fancy but does provide fodder and a start point for an up and coming queueing eng to start digging into both #107 and #109 ;)
Merged
goodboy
changed the title
Numpy compatible shared mem ring buffer
Cross-actor, shared mem, (tick) ring buffer
Mar 6, 2023
goodboy
changed the title
Cross-actor, shared mem, (tick) ring buffer
Cross-actor, shm, (tick) ring buffer
Aug 4, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
data-layer
real-time and historical data processing and storage
feature-request
New feature or request
integration
external stack and/or lib augmentations
As part of the fsp subsystem design we're likely going to want to implement some shared memory systems for low latency multi-time frame calcs. I've started a couple design ideas about this in #98 but I do think for sure part of this system will require a fast ring buffer for
numpy
data:Here's a starter list of projects to try after some very brief searching
committed to long ago
numpy
pyringbuf
a c-ext implementing a character buffercommitted recently
pyring
pure python impl and has a multi-proc c-types exampleringbuf
looks to be built for real-time audio but is single producer/consumer (which is actually fine for our use case afaik) but is built as a cython wrapper aroundboost
(ala C++) libs;numpy
example looks goodringbuf
in rust targeting some missing features withctypes
wrapping/integration for pythonfuther resources
scipy
mailing list discussion from 09'pyqtgraph
and anumpy
usingnp.roll()
.It may just end up we take a lang from #106 and implement one or try out some designs from above on the new
SharedMemory
type which has an example using anumpy
array. An example wrapper for an older version of this from the scipy cookbooks is here.disruptor style
The "lurkers" said LMAX already did this best with their
disruptor
project. I think we can make a very cool variant of this with actors,numpy
andnumba
:disruptor
link 🏄numpy
support at all).As always, lurkers please chime.
The text was updated successfully, but these errors were encountered: