Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: reset the df schema after read to fix the "not null" issue when loading data #1341

Merged
merged 1 commit into from
Mar 1, 2022

Conversation

vagetablechicken
Copy link
Collaborator

No description provided.

If read non-streaming files, spark will set all schema fields to nullable. We should reset it.
Copy link
Collaborator

@tobegit3hub tobegit3hub left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

Linux Test Report

     102 files       228 suites   43m 0s ⏱️
  8 593 tests   8 590 ✔️ 3 💤 0
12 715 runs  12 712 ✔️ 3 💤 0

Results for commit 9f88fb7.

@codecov
Copy link

codecov bot commented Feb 28, 2022

Codecov Report

Merging #1341 (9f88fb7) into branch-0.4 (d4a07c5) will decrease coverage by 0.05%.
The diff coverage is 77.61%.

Impacted file tree graph

@@               Coverage Diff                @@
##             branch-0.4    #1341      +/-   ##
================================================
- Coverage         65.50%   65.45%   -0.06%     
  Complexity          222      222              
================================================
  Files               570      570              
  Lines            106347   106386      +39     
  Branches            841      854      +13     
================================================
- Hits              69664    69636      -28     
- Misses            36537    36604      +67     
  Partials            146      146              
Impacted Files Coverage Δ
...openmldb/taskmanager/config/TaskManagerConfig.java 0.00% <0.00%> (ø)
...m/_4paradigm/openmldb/taskmanager/dao/JobInfo.java 0.00% <0.00%> (ø)
...nmldb/taskmanager/server/impl/TaskManagerImpl.java 0.00% <0.00%> (ø)
...paradigm/openmldb/taskmanager/JobInfoManager.scala 0.00% <0.00%> (ø)
src/cmd/sql_cmd.h 21.41% <ø> (-0.10%) ⬇️
..._4paradigm/openmldb/batch/utils/HybridseUtil.scala 60.90% <38.46%> (-2.44%) ⬇️
...4paradigm/openmldb/batch/nodes/WindowAggPlan.scala 73.89% <66.66%> (-0.24%) ⬇️
...4paradigm/openmldb/batch/api/OpenmldbSession.scala 57.57% <94.11%> (+4.70%) ⬆️
..._4paradigm/openmldb/batch/nodes/LoadDataPlan.scala 60.37% <100.00%> (+4.58%) ⬆️
..._4paradigm/openmldb/batch/utils/SparkRowUtil.scala 66.66% <100.00%> (+14.03%) ⬆️
... and 14 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1977e88...9f88fb7. Read the comment docs.

@dl239 dl239 merged commit 4dd3479 into 4paradigm:branch-0.4 Mar 1, 2022
@lumianph lumianph changed the title fix: reset the df schema after read fix: reset the df schema after read to fix the "not null" issue when loading data Mar 14, 2022
@lumianph lumianph mentioned this pull request Mar 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants