Add a Table Data Type!

[oparisblue]

The Table format stores CSV-style tables.

We want to support the following operations:

• CSV Upload => make table
• Filter / Query / Aggregate functions:
  • Drop Column
  • Filter Rows
  • Add Column
  • Sum / Average / etc
  • Group
  • Limit
  • etc

But, this seems overly limiting to the user, and seems to pose some intricate UI problems. For example, what if you want to sum just all of the odd rows? Perhaps, you would filter out the odd rows, and then run the stock sum function (of note: what column does this sum???). But this just pushes the problem back a level - how do we choose what gets filter?

Therefore, there will just be one key node: "For Each Row".

What's special about this node is that it contains a "subgraph", e.g. a graph which will be executed for each node. This subgraph will have unremovable "Input" and "Output" nodes in it, which items can an be piped in and out of each time. The input node has a dynamic number of outputs (one for each column), as well as a counter (e.g. what number row is this) and a total (e.g. how many rows are there), and the accumulator (initially the base case, but contains all of the previous modifications too). The output node accepts just one input: something of the same type as the base case / accumulator.

If you're used to .reduce() in JavaScript, or foldl / foldr in other languages, then this should be familiar, and clearly it provides the flexibility that's needed - e.g. for our previous example, we could do something like "0" as the base case, and then e.g. (accumulator + (column value * (row counter % 2))) in the subgraph.

However, as this could be confusing for some of the more simple operations, I think we should offer both: "For Each Row" for complex cases, and then basic nodes like "Sum", "Filter", etc, with dropdown menus / custom UIs to set up the basic tasks.

---

To-do:

• CSV Upload => make table
• For-each node and subgraph
• Abstract "Aggregate" node with four nodes extending it: Min, Max, Average, and Sum (or more if we think they're useful)
• Drop Column node (by number)

potentially more --- feel free to edit!!

---

Eventual:

• Excel Upload => make table