Add `CREATE VIEW` #2279

matthewmturner · 2022-04-20T05:16:19Z

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

matthewmturner · 2022-04-21T17:20:00Z

My idea here is to create a new ViewTable struct (similar concept to how MemTable is used but obviously not in memory) which of course would have TableType::View and we just use that with the existing register_table functionality within ExecutionContext / SchemaProvider. @alamb does that sound good from your end?

alamb · 2022-04-21T21:21:57Z

Thanks @matthewmturner -- I'll try and give it a look tomorrow

matthewmturner · 2022-04-21T21:25:38Z

@alamb I actually hadn't even made it to the point of implementing that yet - just wanted to see if conceptually you thought that was the right approach. Wasn't expecting you to review anything yet. If you aren't sure and it would require you looking into it I can just give it a shot and get back to you. I don't want you to have to do unnecessary work.

alamb · 2022-04-22T19:46:43Z

Sorry for the late review @matthewmturner

At a high level, I would expect a VIEW to be represented by a query -- maybe as a SQL string / parsed Query or perhaps a LogicalPlan

So let's say we have a query like this

select sum(y) from T where a > 5 group by a;

If T is a table, I would expect the plan to look like

GroupBy (gby a, sum(y))
  Filter(a > 5)
    TableScan T

If T is a view,

create view T as select  a, y from bar where y > 10000

I would expect the plan to look like:

GroupBy (gby a, sum(y))
  Filter(a > 5)
    Project (a, y) <-- the LogicalPlan for the View is pasted in here
      Filter (y > 10000)
       TableScan bar

The question then becomes how do you want to get the LogicalPlan for the view when it is referenced. Storing a SQL string might be the simplest, but I am not sure how that would work with the DataFrame API

But then again, I am not sure a view makes sense in the context of a dataframe -- the user would just clone the DataFrame (which is a wrapper around a LogicalPlan 🤔 )

matthewmturner · 2022-05-04T14:56:09Z

I apologize I have been quite busy lately and havent been able to continue my efforts on this. Im really hoping to get this in for 8.0 release, I will try to work on this over the weekend.

alamb · 2022-05-05T00:19:57Z

I apologize I have been quite busy lately ...

No worries at all - I totally understand!

matthewmturner · 2022-05-10T04:12:12Z

@alamb would you mind checking this out and see if its in the right direction? I still have to resolve conflicts and update ballista but wanted to make sure this was at least getting close before proceeding.

matthewmturner · 2022-05-10T17:38:33Z

@andygrove FYI I'm hoping to get this in for 8.0 release. ill be continuing my work on it tonight,

alamb

I think this is looking very much on the right track @matthewmturner 👍

alamb · 2022-05-10T20:27:11Z

datafusion/core/src/datasource/view.rs

+/// An implementation of `TableProvider` that uses the object store
+/// or file system listing capability to get the list of files.


The implementation uses another LogicalPlan

alamb · 2022-05-10T20:27:35Z

datafusion/core/src/datasource/view.rs

+        filters: &[Expr],
+        limit: Option<usize>,
+    ) -> Result<Arc<dyn ExecutionPlan>> {
+        self.context.create_physical_plan(&self.logical_plan).await


alamb · 2022-05-10T20:30:52Z

datafusion/core/src/datasource/view.rs

+
+        assert_batches_eq!(expected, &results);
+
+        let results = session_ctx


I think more interesting tests might be to create a view and then run queries against the view (also adding things like filters to on the view

For example

select coumn1, count(colum2) from xyz where column3 > 2

andygrove · 2022-05-10T22:05:29Z

I just took a quick look through this and LGTM ❤️

Dandandan · 2022-05-10T23:26:23Z

datafusion/core/src/datasource/view.rs

+    /// To create ExecutionPlan
+    context: SessionContext,
+    /// LogicalPlan of the view
+    logical_plan: LogicalPlan,


Original SQL might be useful too to store, to preserve formatting when doing things like describe view.

Indeed that would be nice, do you think we could do it as a follow on PR?

hi, I'm new to datafusion and maybe I can help with the follow work ?

👋 welcome @Veeupup . That would be great. I will try and write up a ticket later today that describes the work in some more detail in case that is helpful

Thank you!

#2529

@Dandandan not sure if you had specific ideas on what command you wanted to show view definitions.

matthewmturner · 2022-05-11T12:42:45Z

Ah - i believe the tpch tests use a published version of ballista. so i dont think we can run Q15 until our next release. let me know if im misunderstanding, i didnt have time to dig too far into it.

andygrove · 2022-05-11T13:06:03Z

Ah - i believe the tpch tests use a published version of ballista. so i dont think we can run Q15 until our next release. let me know if im misunderstanding, i didnt have time to dig too far into it.

q15 was definitely a stretch goal for this PR 😄 I will take a look

andygrove · 2022-05-11T13:07:45Z

RAT is failing due to an empty file @ datafusion/core/src/physical_plan/view.rs

matthewmturner · 2022-05-11T13:43:22Z

@andygrove thanks - i removed.

matthewmturner · 2022-05-11T14:34:49Z

@andygrove @alamb hopefully this is good now.

was i able to complete it in time for the 8.0 release?

andygrove

LGTM. Thanks @matthewmturner

matthewmturner · 2022-05-11T14:47:28Z

@Igosuki FYI

github-actions bot added the datafusion Changes in the datafusion crate label Apr 20, 2022

xudong963 added the enhancement New feature or request label Apr 23, 2022

alamb reviewed May 10, 2022

View reviewed changes

matthewmturner force-pushed the add_create_view branch from 48c4349 to 0ba010f Compare May 10, 2022 22:24

github-actions bot added the ballista label May 10, 2022

Dandandan reviewed May 10, 2022

View reviewed changes

matthewmturner marked this pull request as ready for review May 10, 2022 23:42

matthewmturner mentioned this pull request May 10, 2022

MINOR: Enable multi-statement benchmark queries #2507

Merged

matthewmturner added 7 commits May 10, 2022 22:56

Initial commit

cd4c188

First passing test

a2c2aee

Add OR REPLACE and more tests

4e2b54a

Update doc comment

aa313d2

More tests

eeeb487

Add CreateView to Ballista

3eedfe5

Include Q15 for TPCH

53238a1

matthewmturner force-pushed the add_create_view branch from 335be74 to 53238a1 Compare May 11, 2022 02:56

Ignore q15

30c2abf

Delete view physical plan

c45353e

andygrove approved these changes May 11, 2022

View reviewed changes

andygrove merged commit 19d937a into apache:master May 11, 2022

alamb mentioned this pull request May 13, 2022

Implement some way to show the sql used to create a view #2529

Closed

alamb mentioned this pull request Aug 9, 2022

TPC-H Query 15 #166

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `CREATE VIEW` #2279

Add `CREATE VIEW` #2279

matthewmturner commented Apr 20, 2022

matthewmturner commented Apr 21, 2022

alamb commented Apr 21, 2022

matthewmturner commented Apr 21, 2022

alamb commented Apr 22, 2022

matthewmturner commented May 4, 2022

alamb commented May 5, 2022

matthewmturner commented May 10, 2022

matthewmturner commented May 10, 2022 •

edited

Loading

alamb left a comment

alamb May 10, 2022

alamb May 10, 2022

alamb May 10, 2022

andygrove commented May 10, 2022

Dandandan May 10, 2022

matthewmturner May 11, 2022

alamb May 11, 2022

Veeupup May 12, 2022

alamb May 12, 2022

alamb May 13, 2022

matthewmturner commented May 11, 2022

andygrove commented May 11, 2022

andygrove commented May 11, 2022

matthewmturner commented May 11, 2022

matthewmturner commented May 11, 2022

andygrove left a comment

matthewmturner commented May 11, 2022

		/// An implementation of `TableProvider` that uses the object store
		/// or file system listing capability to get the list of files.


		assert_batches_eq!(expected, &results);

		let results = session_ctx

Add CREATE VIEW #2279

Add CREATE VIEW #2279

Conversation

matthewmturner commented Apr 20, 2022

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

matthewmturner commented Apr 21, 2022

alamb commented Apr 21, 2022

matthewmturner commented Apr 21, 2022

alamb commented Apr 22, 2022

matthewmturner commented May 4, 2022

alamb commented May 5, 2022

matthewmturner commented May 10, 2022

matthewmturner commented May 10, 2022 • edited Loading

alamb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andygrove commented May 10, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

matthewmturner commented May 11, 2022

andygrove commented May 11, 2022

andygrove commented May 11, 2022

matthewmturner commented May 11, 2022

matthewmturner commented May 11, 2022

andygrove left a comment

Choose a reason for hiding this comment

matthewmturner commented May 11, 2022

Add `CREATE VIEW` #2279

Add `CREATE VIEW` #2279

matthewmturner commented May 10, 2022 •

edited

Loading