Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SUPPORT] command line HiveSyncTool: parsing with JCommander broken #6937

Closed
matthiasdg opened this issue Oct 13, 2022 · 4 comments · Fixed by #7068
Closed

[SUPPORT] command line HiveSyncTool: parsing with JCommander broken #6937

matthiasdg opened this issue Oct 13, 2022 · 4 comments · Fixed by #7068
Assignees
Labels
meta-sync priority:major degraded perf; unable to move forward; potential bugs

Comments

@matthiasdg
Copy link

Describe the problem you faced

Upgraded to hudi 0.12 here (from 0.10) and it seems some things changed to the HiveSyncTool, more specifically to HiveSyncConfig.java
Where before there were defaults for some Boolean parameters, this is now no longer the case. As an example:

    @Parameter(names = {"--spark-datasource"}, description = "Whether sync this table as spark data source table.")
    public Boolean syncAsSparkDataSourceTable;

used to be

  @Parameter(names = {"--spark-datasource"}, description = "Whether sync this table as spark data source table.")
  public Boolean syncAsSparkDataSourceTable = true;

When I try to run the synctool commandline, I now get NPE's as soon as I include one of these Boolean parameters.
I found cbeust/jcommander#447
This does not appear to be fixed in later JCommander releases at first sight (did a quick test overriding the dependencies).
Haven't tested yet whether switching to boolean as suggested in that issue fixes it. Reverting to default values as before also has the issue cbeust/jcommander#378 (so it then just flips the set default which is weird). Specifying an arity 1 so the value has to be set is maybe another option (https://jcommander.org/).

To Reproduce

Steps to reproduce the behavior:

  1. Run HiveSyncTool main and specify one of the boolean parameters; e.g. --spark-datasource, --ignore-exceptions...
  2. a NullPointerException will be thrown

Expected behavior
Should run fine

Environment Description

  • Hudi version : 0.12

Stacktrace

java.lang.NullPointerException:
[info]   at org.apache.hudi.com.beust.jcommander.JCommander.parseValues(JCommander.java:718)
[info]   at org.apache.hudi.com.beust.jcommander.JCommander.parse(JCommander.java:340)
[info]   at org.apache.hudi.com.beust.jcommander.JCommander.parse(JCommander.java:319)
[info]   at org.apache.hudi.hive.HiveSyncTool.main(HiveSyncTool.java:353)
@xushiyan xushiyan self-assigned this Oct 13, 2022
@xushiyan xushiyan added meta-sync priority:major degraded perf; unable to move forward; potential bugs labels Oct 13, 2022
@xushiyan xushiyan moved this to Awaiting Triage in Hudi Issue Support Oct 13, 2022
@yihua
Copy link
Contributor

yihua commented Oct 14, 2022

@matthiasdg thanks for reporting the issue. We'll try to reproduce it.

@xicm
Copy link
Contributor

xicm commented Oct 26, 2022

Hi, I can fix this bug.

@xicm
Copy link
Contributor

xicm commented Oct 26, 2022

#7068
@yihua @xushiyan Have a review, if you are available.

@xushiyan xushiyan moved this from Awaiting Triage to Triaged in Hudi Issue Support Oct 27, 2022
@xushiyan xushiyan linked a pull request Oct 27, 2022 that will close this issue
4 tasks
@xushiyan
Copy link
Member

#7068 @yihua @xushiyan Have a review, if you are available.

Thanks for taking this up @xicm

Repository owner moved this from 🏁 Triaged to ✅ Done in Hudi Issue Support Oct 31, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
meta-sync priority:major degraded perf; unable to move forward; potential bugs
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

4 participants