Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support scala 2.13 (to enable full spark 3.2.0 support) #143

Closed
wants to merge 3 commits into from

Conversation

skestle
Copy link
Contributor

@skestle skestle commented May 13, 2021

Issues with:

  • StructType string changes
  • Fragile schema tests because scala's .groupBy function is not deterministic in ordering, and deep schemas don't resolve
    The existing tests on 2.12 should be able to be falsified just be adding a few more foo.* columns in the nested DataFrame

@skestle
Copy link
Contributor Author

skestle commented May 13, 2021

In terms of a schema-equality breaking test - are the following 2 equal?:

  • root
    • foo
      • baz
      • bar
  • root
    • foo
      • bar
      • baz

I believe they are (given that we make a lookup map for schema equality). I can add the test tomorrow if you like, but I'm not sure I want to do the code fix ;). Perhaps deep is what we actually want there ;D?

@skestle
Copy link
Contributor Author

skestle commented May 14, 2021

See #144

Enables nested structures to be compared regardless of order.
@skestle skestle force-pushed the spark-3.2-draft branch from 11fcd2a to c31c7b2 Compare May 14, 2021 09:24
@MrPowers
Copy link
Collaborator

Glad we're experimenting with Scala 2.13 😄

I'm glad we're doing this work to see what bugs need to be fixed before Scala 2.13 gets released. Better to get out ahead of them.

@skestle skestle force-pushed the spark-3.2-draft branch from c31c7b2 to efa2803 Compare May 25, 2021 23:44
@skestle skestle changed the title Updated to scala 2.13 and spark 3.2.0-SNAPSHOT Support scala 2.13 (to enable full spark 3.2.0 support) May 25, 2021
@MrPowers
Copy link
Collaborator

@skestle - do we still need this PR?

@skestle
Copy link
Contributor Author

skestle commented Jun 12, 2021

Nah. It'd be rather complicated supporting 3.0/2.12 and 3.2/2.13...

@skestle skestle closed this Jun 12, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants