high level roadmap for Arrow / Datafusion

[alamb]

@matthewmturner and I we think it would be valuable to try making a high level roadmap for Arrow / Datafusion. Specifically, and how this differs from the existing roadmap, we are looking for a rough idea of what people plan on working on with a best efforts guess when they hope to finish. Of course nothing is binding, the goal is to make visible peoples plans so that the developers as a whole can better organize / plan efforts and users have a rough idea of feature roadmap.

There is a roadmap (https://github.com/apache/arrow-datafusion/blob/master/docs/source/specification/roadmap.md) but im interested in seeing how that translates to a more tangible plan. im cognizant of not wanting to have something that is too much of a burden to maintain, but some public strategy / structure / plan could be helpful.

Assuming we get enough feedback, we can merge everything in a unified plan.

started a doc here: https://docs.google.com/document/d/1t64vZwZnXm9MyFj2qz3xcAkSxK3Wu12giS3KrS4nDE0/edit

[alamb]

Previous attempt at the same type of exercise from a year ago:

https://docs.google.com/document/d/1qspsOM_dknOxJKdGvKbC1aoVoO0M3i6x1CIo58mmN2Y

[xudong963]

Make sense to me.
BTW, I don't have edit permission in https://docs.google.com/document/d/1t64vZwZnXm9MyFj2qz3xcAkSxK3Wu12giS3KrS4nDE0/edit

[matthewmturner]

Make sense to me.

BTW, I don't have edit permission in https://docs.google.com/document/d/1t64vZwZnXm9MyFj2qz3xcAkSxK3Wu12giS3KrS4nDE0/edit

@xudong963 can you try now?

[xudong963]

Make sense to me.
BTW, I don't have edit permission in https://docs.google.com/document/d/1t64vZwZnXm9MyFj2qz3xcAkSxK3Wu12giS3KrS4nDE0/edit

@xudong963 can you try now?

all right

[alamb]

I plan to add my content tomorrow, FWIW -- thank you for organizing this @matthewmturner

[liukun4515]

I plan to add my content tomorrow, FWIW -- thank you for organizing this @matthewmturner

I give some suggestions on your plan. @alamb

[xudong963]

I want a detailed evaluation of my time and what I want to do, which is expected to be completed this week. BTW, I'm at Oncall this week, so there's not a lot of time to get involved with Datafusion.

[matthewmturner]

@Dandandan @pjmore do you think the work on tokomak optimizer could be added to roadmap?

[liukun4515]

@Dandandan @pjmore do you think the work on tokomak optimizer could be added to roadmap?

is there a related issue or pull request？

[houqp]

@liukun4515 see #440

[liukun4515]

@liukun4515 see #440

thanks

[matthewmturner]

@hntd187 @xudong963 let me know if you think worth adding streaming to the roadmap (https://docs.google.com/document/d/1t64vZwZnXm9MyFj2qz3xcAkSxK3Wu12giS3KrS4nDE0/edit).

If so, and you both think its reasonable, I could add to Q1 something like "Begin prototyping a stream provider"

[hntd187]

@matthewmturner I'm fine with it, I know design discussions can take time through this venue so if a prototype is working and we can at least have a general idea of the direction/design we want then I’d consider that a very successful Q1.

[pjmore]

@matthewmturner I've added a little blurb to the roadmap about the optimizer. I'm not entirely sure if the optimizer as it exists is useful so I made it clear that it is in an experimental context.

[matthewmturner]

@yahoNanJing no pressure, but ive seen you on multiple ballista related topics and was wondering if you perhaps had any ballista related goals that you would like to add to our 2022 Q1 roadmap. doc is here in case you wanted to see what was already there https://docs.google.com/document/d/1t64vZwZnXm9MyFj2qz3xcAkSxK3Wu12giS3KrS4nDE0/edit.

[matthewmturner]

As an update on this, I'm currently away but I plan on submitting PR to include roadmap in README later this week.

[matthewmturner]

Working on PR for this now.