Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for serializing Spark's DecimalType #947

Merged
merged 6 commits into from
Dec 2, 2024

Conversation

engechas
Copy link
Contributor

@engechas engechas commented Nov 26, 2024

Description

Adds support for serializing Spark's DecimalType in FlintDataType

Related Issues

List any issues this PR will resolve, e.g. Resolves [...].

Check List

  • Updated documentation (docs/ppl-lang/README.md)
  • Implemented unit tests
  • Implemented tests for combination with other commands
  • New added source code should include a copyright header
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Chase Engelbrecht <[email protected]>
Signed-off-by: Chase Engelbrecht <[email protected]>
@dai-chen dai-chen added enhancement New feature or request 0.7 labels Nov 26, 2024
@engechas
Copy link
Contributor Author

engechas commented Dec 1, 2024

Could we backport this change to 0.6?

Copy link
Collaborator

@dai-chen dai-chen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the changes!

@noCharger noCharger merged commit 0943f1f into opensearch-project:main Dec 2, 2024
5 checks passed
@opensearch-trigger-bot
Copy link

The backport to 0.6 failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/opensearch-spark/backport-0.6 0.6
# Navigate to the new working tree
pushd ../.worktrees/opensearch-spark/backport-0.6
# Create a new branch
git switch --create backport/backport-947-to-0.6
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 0943f1fbe893349688aaa84c1d1fcab69156d264
# Push it to GitHub
git push --set-upstream origin backport/backport-947-to-0.6
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/opensearch-spark/backport-0.6

Then, create a pull request where the base branch is 0.6 and the compare/head branch is backport/backport-947-to-0.6.

@noCharger
Copy link
Collaborator

@engechas could you backport to 0.6 manually?

engechas added a commit to engechas/opensearch-spark that referenced this pull request Dec 2, 2024
* Add support for serializing DecimalType in FlintDataType

Signed-off-by: Chase Engelbrecht <[email protected]>

* Fix checkstyle

Signed-off-by: Chase Engelbrecht <[email protected]>

* Add documentation on the new serialization behavior

Signed-off-by: Chase Engelbrecht <[email protected]>

* Fix integ test

Signed-off-by: Chase Engelbrecht <[email protected]>

* Actually fix integ tests

Signed-off-by: Chase Engelbrecht <[email protected]>

* Move the decimal and map IT to the base suite instead of the iceberg suite

Signed-off-by: Chase Engelbrecht <[email protected]>

---------

Signed-off-by: Chase Engelbrecht <[email protected]>
(cherry picked from commit 0943f1f)
@engechas
Copy link
Contributor Author

engechas commented Dec 2, 2024

Created #962 to manually backport to 0.6

dai-chen pushed a commit that referenced this pull request Dec 3, 2024
* Add support for serializing DecimalType in FlintDataType

Signed-off-by: Chase Engelbrecht <[email protected]>

* Fix checkstyle

Signed-off-by: Chase Engelbrecht <[email protected]>

* Add documentation on the new serialization behavior

Signed-off-by: Chase Engelbrecht <[email protected]>

* Fix integ test

Signed-off-by: Chase Engelbrecht <[email protected]>

* Actually fix integ tests

Signed-off-by: Chase Engelbrecht <[email protected]>

* Move the decimal and map IT to the base suite instead of the iceberg suite

Signed-off-by: Chase Engelbrecht <[email protected]>

---------

Signed-off-by: Chase Engelbrecht <[email protected]>
(cherry picked from commit 0943f1f)
saranrajnk pushed a commit to saranrajnk/opensearch-spark that referenced this pull request Dec 3, 2024
…) (opensearch-project#962)

* Add support for serializing DecimalType in FlintDataType

Signed-off-by: Chase Engelbrecht <[email protected]>

* Fix checkstyle

Signed-off-by: Chase Engelbrecht <[email protected]>

* Add documentation on the new serialization behavior

Signed-off-by: Chase Engelbrecht <[email protected]>

* Fix integ test

Signed-off-by: Chase Engelbrecht <[email protected]>

* Actually fix integ tests

Signed-off-by: Chase Engelbrecht <[email protected]>

* Move the decimal and map IT to the base suite instead of the iceberg suite

Signed-off-by: Chase Engelbrecht <[email protected]>

---------

Signed-off-by: Chase Engelbrecht <[email protected]>
(cherry picked from commit 0943f1f)
kenrickyap pushed a commit to Bit-Quill/opensearch-spark that referenced this pull request Dec 11, 2024
* Add support for serializing DecimalType in FlintDataType

Signed-off-by: Chase Engelbrecht <[email protected]>

* Fix checkstyle

Signed-off-by: Chase Engelbrecht <[email protected]>

* Add documentation on the new serialization behavior

Signed-off-by: Chase Engelbrecht <[email protected]>

* Fix integ test

Signed-off-by: Chase Engelbrecht <[email protected]>

* Actually fix integ tests

Signed-off-by: Chase Engelbrecht <[email protected]>

* Move the decimal and map IT to the base suite instead of the iceberg suite

Signed-off-by: Chase Engelbrecht <[email protected]>

---------

Signed-off-by: Chase Engelbrecht <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants