Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support unparsing UNNEST plan to UNNEST table factor SQL #13660

Merged
merged 7 commits into from
Dec 11, 2024

Conversation

goldmedal
Copy link
Contributor

Which issue does this PR close?

Closes #13601

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

@github-actions github-actions bot added the sql SQL Planner label Dec 5, 2024
Comment on lines +565 to +566
sql: "SELECT UNNEST([1,2,3])",
expected: r#"SELECT * FROM UNNEST([1, 2, 3])"#,
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because the logical plan of SELECT UNNEST([1,2,3]) and SELECT * FROM UNNEST([1, 2, 3]) is the same, if the with_unnest_as_table_factor is enabled, we can't avoid to unparse it to the table factor SQL. However, I guess it's fine because they are the same in most scenarios. 🤔

@goldmedal goldmedal marked this pull request as ready for review December 5, 2024 11:34
@goldmedal goldmedal requested review from alamb and findepi December 10, 2024 04:20
/// Allow to unparse the unnest plan as [ast::TableFactor::UNNEST].
///
/// Some dialects like BigQuery require UNNEST to be used in the FROM clause but
/// the LogicalPlan planner always put UNNEST in the SELECT clause. This flag allows
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

put -> puts

well, UNNEST is a lateral relational operator. It doesn't belong to the select clause.
It's probably yet another witness of the problem that the LP is half way between AST and a good IR (which we don't have yet)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

well, UNNEST is a lateral relational operator. It doesn't belong to the select clause. It's probably yet another witness of the problem that the LP is half way between AST and a good IR (which we don't have yet)

Interesting.
The logical planner plans FROM unnest([1,2,3]) to

Projection: unnest_placeholder(make_array(Int64(1),Int64(2),Int64(3)),depth=1) AS UNNEST(make_array(Int64(1),Int64(2),Int64(3)))
    Unnest: lists[unnest_placeholder(make_array(Int64(1),Int64(2),Int64(3)))|depth=1] structs[]
      Projection: make_array(Int64(1), Int64(2), Int64(3)) AS unnest_placeholder(make_array(Int64(1),Int64(2),Int64(3)))
        EmptyRelation

Do you think it's a good plan? What would a good IR look like? 🤔

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not unreasonable.
Looking at this the "unnest_placeholder" looks more like "unnest input".

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

actually it's both: unnest input and unnest output. confusing.

datafusion/sql/src/unparser/plan.rs Outdated Show resolved Hide resolved
datafusion/sql/src/unparser/plan.rs Outdated Show resolved Hide resolved
datafusion/sql/tests/cases/plan_to_sql.rs Show resolved Hide resolved
@github-actions github-actions bot added the sqllogictest SQL Logic Tests (.slt) label Dec 10, 2024
@goldmedal goldmedal merged commit 08119e6 into apache:main Dec 11, 2024
25 checks passed
@goldmedal goldmedal deleted the feature/13601-bigquery-unnest branch December 11, 2024 15:27
@goldmedal
Copy link
Contributor Author

Thanks @findepi for reviewing 👍

@alamb
Copy link
Contributor

alamb commented Dec 11, 2024

🚀

zhuliquan pushed a commit to zhuliquan/datafusion that referenced this pull request Dec 11, 2024
…13660)

* add `unnest_as_table_factor` and `UnnestRelationBuilder`

* unparse unnest as table factor

* fix typo

* add tests for the default configs

* add a static const for unnest_placeholder

* fix tests

* fix tests
zhuliquan pushed a commit to zhuliquan/datafusion that referenced this pull request Dec 15, 2024
…13660)

* add `unnest_as_table_factor` and `UnnestRelationBuilder`

* unparse unnest as table factor

* fix typo

* add tests for the default configs

* add a static const for unnest_placeholder

* fix tests

* fix tests
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
sql SQL Planner sqllogictest SQL Logic Tests (.slt)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Allow to unparse UNNEST plan back to a table function SQL text
3 participants