Use Tables.jl interface and ag-grid #6

pfitzseb · 2018-10-19T15:14:12Z

This works fine for small(ish) tables already (< 100k rows). For everything bigger than that we should use the Infinite View Model instead, but that requires more communication between Julia and the browser. We'd also need to implement sorting and filtering on the Julia side in that case (or just disable it).

davidanthoff · 2018-10-20T02:11:57Z

Could we make sure that TableTraits.jl sources keep working? The old version supports any TableTraits.jl source, so it would be great to not remove that functionality.

pfitzseb · 2018-10-21T09:38:38Z

Yeah, definitely. My main goals for this PR are to

support all table interfaces
be able to display arbitrarily large tables easily

Editing and Julia-side filtering/sorting can come after that.

piever · 2018-10-21T11:41:05Z

For inspiration for filtering you can look at TableWidgets (see for example here for the demo at juliacon https://youtu.be/wFzkNSoKo0U?t=640), in particular the 3 selectors (discrete, categorical and arbitrary).

I think it'd be nice to do this using the Widgets interface (see here), which means that showtable would return a Widget object like:

Widget{:table}([:input => t], output = o, scope = w, layout = scope)

where the :input is the table the user is passing, the scope is the WebIO scope you are creating and output is the result of filtering and sorting. In this way the user can "save" the result of manipulations on the table. They could even feed the widget as observable input to another widget to create a pipeline.

pfitzseb · 2018-10-22T11:53:41Z

pfitzseb · 2018-10-22T15:46:50Z

Works remarkably well with "big data":

davidanthoff

Nice! I've added some comments, there are a whole bunch of queries this won't handle as is.

Also, I've tried this from the REPL, but nothing happened, same for Jupyter Lab. Things did show up in Jupyter notebook.

davidanthoff · 2018-10-22T16:42:34Z

src/TableView.jl

+function showtable(x; dark = false)
+    if Tables.istable(typeof(x))
+        return _showtable(x, dark)
+    elseif TableTraits.isiterabletable(x)


This also needs to handle the case where TableTraits.isiterabletable returns missing, like the DataFrame constructor here.

Those cases are a bit tricky to generate right now [*] but they will become much more common with hopefully the next release of Query.jl. These are essentially cases where Query.jl returns a lazy iterator but wasn't able to use type inference to determine the eltype of the iterator, and so a source needs to be able to handle that case. The general strategy in those cases is to look at the type of the first iterated element to figure out the schema. For iterable tables, there is a rule that they must iterate only elements of the same type, so looking at the first element should be enough to handle all iterable table sources.

[*] The shortest example right now is TableView.showtable(rand()>0.00000000001 ? i : true for i in df). I think there is a plan to handle to subsume generators under Tables.istable (that I disagree with), at which point this particular example would be moot. But once I've made the change in Query.jl that I mentioned, this will show up independently of this example.

Ah, I saw that case in the DataFrame constructor but couldn't figure out what it meant.

Isn't isiterabletable a bit of a misnomer it can return missing for something that most definitely is an iterable table, albeit with undetermined eltype?

Well, in general there are cases where isiterabletable just can't figure out whether something is a table or not. A generator that returns a stream of Ints is not a table, but it doesn't have an element type, so you can't tell until you start to iterate. So, of course a source either is or is not a table, but we don't know that when we call isiterabletable, so missing seemed appropriate?

What is missing is the information whether something is a table or not.

Fair enough, I guess -- I definitely haven't put as much thought into it as you :)

davidanthoff · 2018-10-22T16:45:55Z

src/TableView.jl

-function showna(xs::Columns)
-    rows(map(showna, columns(xs)))
+function _showtable(table, dark)
+    length(Tables.rows(table)) > 10_000 ? _showtable_async(table, dark) :


This should check IteratorSize before calling length. Any Query.jl query that includes a @filter will have IteratorSize(q)==SizeUnknown(), and so a sink needs to be able to handle sources with unknown length.

davidanthoff · 2018-10-22T16:46:38Z

src/TableView.jl

-    headers = colnames(subt)
-    cols = [merge(Dict(:data=>n), get(colopts, n, Dict())) for n in headers]
+function _showtable_sync(table, dark)
+    schema = Tables.schema(table)


There needs to be some code path that handles situation where no schema is available. I think this function returns nothing in that case, but I'm not 100% sure.

Yeah, it does. I guess we should just not print a header in that case? Or is there any other mechanism to handle schemaless tables?

Well, they are not really schema less, it is just that you don't know the schema until you start iterating. So the strategy would be to iterate the first element, look at the type of that and that should tell you what the column names and types are.

davidanthoff · 2018-10-22T16:47:02Z

src/TableView.jl

-         ks = collect(ks)
-         vs = collect(vs)
+function _showtable_async(table, dark)
+    schema = Tables.schema(table)


Same comment here, needs to handle schemaless sources.

davidanthoff · 2018-10-22T16:55:40Z

Just one other general pointer: I've started adding support for the application/vnd.dataresource+json MIME type to Queryverse. nteract has a native table viewer built in for that MIME type that is pretty nice. I think the Jupyter Lab folks are also considering adding native support, although it is unclear to me how active that effort is.

Currently the only package that exports this support on my end is VegaDatasets.jl. If you open a notebook in nteract and then run:

using VegaDatasets

dataset("cars")

you can see how that looks.

pfitzseb · 2018-10-22T20:06:18Z

Thanks for the comments, @davidanthoff! I think I've addressed most of those.

Also, I've tried this from the REPL, but nothing happened, same for Jupyter Lab. Things did show up in Jupyter notebook.

That's a WebIO problem, I think, but I hope that'll be solved soon.

shashi · 2018-10-23T21:24:26Z

Awesome stuff. cc @andreasnoack @joshday could you check your tables?

piever · 2018-10-24T10:58:06Z

Really nice work!

I like the performance a lot for normal datasets, but I'm having issues to get it to work efficiently for a Observable of tables. Meaning something like:

using RDatasets, Blink, TableView, Observables
school = RDatasets.dataset("mlmRev","Hsb82")
w = Window()
body!(w, map(TableView.showtable, Observable(school)))

Gives me an unresponsive window for quite some time (whereas using it directly on school is pretty fast). I wonder where the performance issue comes from and whether it's easy enough to add support for Observables in TableView.showtable directly (so that TableView.showtable(Observable(school)) would work already).

EDIT: actually checking out Simon's branch JuliaGizmos/WebIO.jl#208 in WebIO seems to mitigate the performance issue. Still, if adding support for Observables is straightforward, it'd definitely be nice to have.

shashi · 2018-11-05T16:37:39Z

src/TableView.jl


-end # module
+    String(take!(io))
+end


I love this!

The unfortunate thing is that I can't figure out how to transfer the data with WebIO without an additional JSON.parse on the javascript side. Any ideas on how to circumvent that?

I think there is a type in JSON.jl called JSONString which was introduced for this purpose

shashi · 2018-11-05T16:38:37Z

deps/build.log

+  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
+                                 Dload  Upload   Total   Spent    Left  Speed
+  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0100    84  100    84    0     0    497      0 --:--:-- --:--:-- --:--:--   497
+100 91448  100 91448    0     0   493k      0 --:--:-- --:--:-- --:--:--  493k


Minor nit pick but maybe this should be omitted and in .gitignore?

shashi · 2018-11-05T16:43:43Z

src/TableView.jl

+    empty!(ag_grid_imports)
+    for f in ["ag-grid.js", "ag-grid.css", "ag-grid-light.css", "ag-grid-dark.css"]
+        path = normpath(joinpath(@__DIR__, "..", "deps", "ag-grid", f))
+        AssetRegistry.register(path)


If you omit this line, I think we can drop the explicit dependency on AssetRegistry. WebIO will automatically make this translation if it sees a file path. (I don't know if that's a 100% good idea, but it seemed like it before; let me know if you have any opinions about this!).

Also, if you just register joinpath(@__DIR__, "..", "deps", "ag-grid") then you can serve any file under that path (this is useful if your files import other files under this directory) this does require AssetRegistry dependency though.

However, I'm fine with merging this as it is.

Alright, I've removed the dependency on AssetRegistry now. It's probably safer to have it, but dunno.

pfitzseb · 2018-11-05T20:13:09Z

src/TableView.jl

+        names = schema.names
+        types = schema.types
+    end
+    w = Scope(imports = ag_grid_imports)


@shashi Is there a way to directly style the div introduced by the Scope? This results in something like

<div class="webio-mountpoint interactbulma" data-webio-mountpoint="13462147745702835718"> <div class="webio-scope" data-webio-scope-id="scope-4acc84b9-6adc-4a65-8534-e785737a7c2d"> <div id="grid" class="ag-theme-balham" style="height: 100%; min-height: 200px; width: 100%;">

right now, but I'd also like to give the Scope (and the mountpoint, I guess) a style to allow piping through the parent's width and height.

We don't have that right now, but the way to do it would be allow s(style=Dict(:foo => bar)) to work where s is a Scope (just like it works for Node)...

shashi · 2018-11-06T21:21:32Z

Nice! Can we merge this? (you can do it when you're ready)

pfitzseb · 2018-11-07T10:26:03Z

I can't figure out how to properly style this such that it looks good in various frontends (i.e. without having to actually specify a width or height that's not relative to the parent container).

and ag-grid

also show coltype as tooltip

with more than 10k rows

pfitzseb · 2019-02-19T16:49:45Z

davidanthoff reviewed Oct 22, 2018

View reviewed changes

shashi reviewed Nov 5, 2018

View reviewed changes

pfitzseb commented Nov 5, 2018

View reviewed changes

tkf mentioned this pull request Feb 3, 2019

Support html and markdown MIMEs queryverse/ElectronDisplay.jl#8

Merged

pfitzseb added 11 commits February 19, 2019 17:13

use Tables.jl interface

06ef085

and ag-grid

support TableTraits interface

779a0ac

also show coltype as tooltip

infinite scrolling for big tables

14e6c37

with more than 10k rows

clean up code a bit

7e807a2

better support for queryverse

f7f1dab

better header for schemaless tables

6e6706a

use assetregistry to serve files

7234810

remove assetregistry dep

e396d1c

no double json conversion

b97defd

misc updates

c40a49b

cleanup

be1a96d

pfitzseb force-pushed the sp/tables branch from 0afc8b7 to be1a96d Compare February 19, 2019 16:15

export showtable, basic tests

d0f5e96

pfitzseb merged commit 3179e08 into master Feb 19, 2019

pfitzseb deleted the sp/tables branch February 19, 2019 16:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use Tables.jl interface and ag-grid #6

Use Tables.jl interface and ag-grid #6

pfitzseb commented Oct 19, 2018 •

edited

Loading

davidanthoff commented Oct 20, 2018

pfitzseb commented Oct 21, 2018

piever commented Oct 21, 2018

pfitzseb commented Oct 22, 2018

pfitzseb commented Oct 22, 2018

davidanthoff left a comment

davidanthoff Oct 22, 2018

pfitzseb Oct 22, 2018

davidanthoff Oct 22, 2018

pfitzseb Oct 22, 2018

davidanthoff Oct 22, 2018

davidanthoff Oct 22, 2018

pfitzseb Oct 22, 2018

davidanthoff Oct 22, 2018

davidanthoff Oct 22, 2018

davidanthoff commented Oct 22, 2018

pfitzseb commented Oct 22, 2018

shashi commented Oct 23, 2018

piever commented Oct 24, 2018 •

edited

Loading

shashi Nov 5, 2018

pfitzseb Nov 5, 2018

shashi Nov 6, 2018

shashi Nov 5, 2018

pfitzseb Nov 5, 2018

shashi Nov 5, 2018

pfitzseb Nov 5, 2018

pfitzseb Nov 5, 2018

shashi Nov 5, 2018

shashi commented Nov 6, 2018

pfitzseb commented Nov 7, 2018

pfitzseb commented Feb 19, 2019

Use Tables.jl interface and ag-grid #6

Use Tables.jl interface and ag-grid #6

Conversation

pfitzseb commented Oct 19, 2018 • edited Loading

davidanthoff commented Oct 20, 2018

pfitzseb commented Oct 21, 2018

piever commented Oct 21, 2018

pfitzseb commented Oct 22, 2018

pfitzseb commented Oct 22, 2018

davidanthoff left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davidanthoff commented Oct 22, 2018

pfitzseb commented Oct 22, 2018

shashi commented Oct 23, 2018

piever commented Oct 24, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shashi commented Nov 6, 2018

pfitzseb commented Nov 7, 2018

pfitzseb commented Feb 19, 2019

pfitzseb commented Oct 19, 2018 •

edited

Loading

piever commented Oct 24, 2018 •

edited

Loading