Refactor HLG materialization and processing into a shared utility #6
Conversation
|
@mrocklin - Let me know if you still wanted help here, or if you want to go in a different direction |
|
Hey @rjzamora ! I still think that Nuking HLG serialization and embracing Pickle is probably the right move. I personally probably won't push on this (my dev time is low these days) but I think that it's something that @fjetter might be interested in. Can I interest you two in chatting directly sometime? |
|
Florian and I were talking about this just yesterday in fact. I think that it might make sense to prioritize this somewhat soonish. |
Sounds good - Several of us at NVIDIA will be happy to collaborate on this effort :) |
|
I'm probably going to defer engagement here to Florian. As a heads-up he
is off due to public holidays in his region until Wednesday.
…On Fri, Oct 28, 2022 at 1:31 PM Richard (Rick) Zamora < ***@***.***> wrote:
Florian and I were talking about this just yesterday in fact. I think that
it might make sense to prioritize this somewhat soonish.
Sounds good - Several of us at NVIDIA will be happy to collaborate on this
effort :)
—
Reply to this email directly, view it on GitHub
<#6 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AACKZTAJEUMGGPSC7KAIGQLWFQLWLANCNFSM57X2XJNA>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Uses
"update-graph"instead of"update-graph-hlg"when the"distributed.scheduler.pickle"config isFalse. This just runs the same graph materialization and processing code on client instead of on the scheduler.Note that we may want to use a config option that is distinct from scheduler/pickle, since the real decision is whether to materialize the graph on the client or scheduler.
@mrocklin - I know you didn't plan to support the "non-pickle" code path. However, this seems like a relatively simple escape hatch to support for now? Let me know what you think.