Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[HUDI-1176]Upgrade tp log4j2 #1946

Closed
wants to merge 3 commits into from
Closed

[HUDI-1176]Upgrade tp log4j2 #1946

wants to merge 3 commits into from

Conversation

hddong
Copy link
Contributor

@hddong hddong commented Aug 11, 2020

Tips

What is the purpose of the pull request

Now, in some modules(like cli, utilities) use log4j2, and it cannot correct load config file with error log: ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.

Brief change log

(for example:)

  • Support log4j2 config

Verify this pull request

This pull request is a trivial rework / code cleanup without any test coverage.

Committer checklist

  • Has a corresponding JIRA in PR title & commit

  • Commit message is descriptive of the change

  • CI is green

  • Necessary doc changes done or have another open PR

  • For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

@hddong hddong force-pushed the fix-log4j2 branch 2 times, most recently from 52638b7 to 5b62ec4 Compare August 12, 2020 08:03
@yanghua yanghua self-requested a review August 25, 2020 11:15
@yanghua
Copy link
Contributor

yanghua commented Aug 25, 2020

@wangxianghu Can you help to verify and review this PR?

@wangxianghu
Copy link
Contributor

@wangxianghu Can you help to verify and review this PR?

sure, will do

appender.console.name = consoleLogger
# CONSOLE uses PatternLayout.
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = %-4r [%t] %-5p %c %x - %m%n
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @hddong , thanks for your contribution.
Can you explain why you configured two formats? If there is no special reason, keep them in the same format might be better.
The rest LGTM cc@yanghua

@vinothchandar
Copy link
Member

Should we just standardize on log4j instead? (wondering what happened to our slf4j efforts. )

@yanghua
Copy link
Contributor

yanghua commented Sep 1, 2020

Should we just standardize on log4j instead? (wondering what happened to our slf4j efforts. )

Yes, it would be better to support slf4j as a facade of multiple log frameworks. @hddong Do you have time to support slf4j for hudi? If not, @wangxianghu can you take over this work?

@hddong
Copy link
Contributor Author

hddong commented Sep 1, 2020

@yanghua yes, I will address this these two days.

@yanghua
Copy link
Contributor

yanghua commented Sep 1, 2020

@yanghua yes, I will address this these two days.

Sounds good. Any questions please contact @leesf or let me know. thanks.

@hddong
Copy link
Contributor Author

hddong commented Sep 3, 2020

@yanghua @vinothchandar @wangxianghu : make it only use log4j now, please have a review when free.

@yanghua
Copy link
Contributor

yanghua commented Sep 3, 2020

@yanghua @vinothchandar @wangxianghu : make it only use log4j now, please have a review when free.

@wangxianghu please help to verify, when you have time.

@wangxianghu
Copy link
Contributor

@yanghua @vinothchandar @wangxianghu : make it only use log4j now, please have a review when free.

@wangxianghu please help to verify, when you have time.

Ack, will do

@vinothchandar
Copy link
Member

I am bit confused about this PR.
Checked out log4j 2, seems like it has some nice advantages. Should we just move to log4j 2?
https://logging.apache.org/log4j/2.x/
https://logging.apache.org/log4j/2.x/manual/messages.html
https://logging.apache.org/log4j/2.x/manual/api.html#LambdaSupport

@hddong
Copy link
Contributor Author

hddong commented Oct 12, 2020

@vinothchandar : yes, +1 for move to log4j2. I will do it if necessary.

@wangxianghu
Copy link
Contributor

@vinothchandar : yes, +1 for move to log4j2. I will do it if necessary.

+1 on moving to log4j2.

@vinothchandar
Copy link
Member

We have discussed log4j, slf4j and log4j2 now. Should we just upgrade to log4j2 across the board and be done with it? :) Is that what this PR does

@hddong
Copy link
Contributor Author

hddong commented Feb 7, 2021

@vinothchandar: this PR just upgrade some modules, I will update this PR for all modules.

@codecov-io
Copy link

codecov-io commented Feb 8, 2021

Codecov Report

Merging #1946 (77d3697) into master (e692c70) will decrease coverage by 0.06%.
The diff coverage is 100.00%.

Impacted file tree graph

@@             Coverage Diff              @@
##             master    #1946      +/-   ##
============================================
- Coverage     52.26%   52.19%   -0.07%     
+ Complexity     3679     3678       -1     
============================================
  Files           484      484              
  Lines         23079    23079              
  Branches       2457     2457              
============================================
- Hits          12063    12047      -16     
- Misses         9952     9969      +17     
+ Partials       1064     1063       -1     
Flag Coverage Δ Complexity Δ
hudicli 36.94% <100.00%> (ø) 0.00 <1.00> (ø)
hudiclient ∅ <ø> (∅) 0.00 <ø> (ø)
hudicommon 50.79% <100.00%> (+0.01%) 0.00 <1.00> (ø)
hudiflink 56.47% <ø> (-0.12%) 0.00 <ø> (ø)
hudihadoopmr 33.44% <ø> (ø) 0.00 <ø> (ø)
hudisparkdatasource 70.54% <ø> (-0.79%) 0.00 <ø> (ø)
hudisync 45.47% <ø> (ø) 0.00 <ø> (ø)
huditimelineservice 64.36% <ø> (ø) 0.00 <ø> (ø)
hudiutilities 69.74% <ø> (ø) 0.00 <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ Complexity Δ
...rg/apache/hudi/cli/commands/CompactionCommand.java 0.80% <ø> (ø) 2.00 <0.00> (ø)
...va/org/apache/hudi/cli/commands/ExportCommand.java 1.09% <ø> (ø) 1.00 <0.00> (ø)
...g/apache/hudi/cli/utils/SparkTempViewProvider.java 59.67% <ø> (ø) 12.00 <0.00> (ø)
...di/common/bootstrap/index/HFileBootstrapIndex.java 81.48% <ø> (ø) 18.00 <0.00> (ø)
...hudi/common/config/DFSPropertiesConfiguration.java 78.04% <ø> (ø) 14.00 <0.00> (ø)
...c/main/java/org/apache/hudi/common/fs/FSUtils.java 47.34% <ø> (ø) 57.00 <0.00> (ø)
...pache/hudi/common/fs/FailSafeConsistencyGuard.java 0.00% <ø> (ø) 0.00 <0.00> (ø)
...che/hudi/common/fs/OptimisticConsistencyGuard.java 0.00% <ø> (ø) 0.00 <0.00> (ø)
...apache/hudi/common/model/HoodieCommitMetadata.java 44.87% <ø> (ø) 38.00 <0.00> (ø)
...che/hudi/common/model/HoodiePartitionMetadata.java 66.66% <ø> (ø) 7.00 <0.00> (ø)
... and 101 more

@hddong hddong marked this pull request as draft February 20, 2021 02:27
@hddong hddong force-pushed the fix-log4j2 branch 2 times, most recently from c625d8b to 2fb88d1 Compare February 22, 2021 02:36
@hddong hddong marked this pull request as ready for review February 22, 2021 02:36
@hddong hddong changed the title [HUDI-1176]Support log4j2 config [HUDI-1176]Upgrade tp log4j2 Feb 24, 2021
@hddong
Copy link
Contributor Author

hddong commented Feb 24, 2021

@yanghua @vinothchandar this PR is ready for review.

@hddong
Copy link
Contributor Author

hddong commented Mar 23, 2021

@yanghua @vinothchandar: pls have a review when free.

@wangxianghu wangxianghu self-assigned this Mar 24, 2021
@hddong hddong closed this Mar 25, 2021
@hddong hddong reopened this Mar 25, 2021
@wangxianghu
Copy link
Contributor

Hi @hddong,I found the newest version of Apache Log4j is 2.14.1 for now, how about using 2.13.3 instead of 2.11.0 ?
Besides, I ran the unit test locally and found some No appenders could be found for logger (org.apache.hadoop.util.Shell) warnings, It seems log4j2.properties is missed

@hddong
Copy link
Contributor Author

hddong commented Apr 8, 2021

@wangxianghu: had upgrade to 2.13.3 and fix the warning.

@wangxianghu
Copy link
Contributor

wangxianghu commented Apr 10, 2021

@wangxianghu: had upgrade to 2.13.3 and fix the warning.

Hi @hddong thanks for your patient.
we still need log4j2.properties or log4j2.xml configs right?

@hddong
Copy link
Contributor Author

hddong commented Apr 14, 2021

we still need log4j2.properties or log4j2.xml configs right?

Yes, dependent on log4j2 config files now.

@vinothchandar
Copy link
Member

cc @xushiyan might be good to have this done at some point.

@xushiyan
Copy link
Member

@hddong sorry that this PR was delayed and resulted in this many conflicts. would you take a look at the conflicts see if resolvable? or you mind opening a new one?

@hddong
Copy link
Contributor Author

hddong commented Oct 25, 2021

@xushiyan : I will resolve these conflicts.

@hudi-bot
Copy link

hudi-bot commented Nov 5, 2021

CI report:

Bot commands @hudi-bot supports the following commands:
  • @hudi-bot run azure re-run the last Azure build

@hddong hddong marked this pull request as draft December 24, 2021 06:19
@umehrot2
Copy link
Contributor

@hddong are you still going to work on this ?

If you don't have the bandwidth, I would be happy to drive this to completion and upgrade this further to Log4j 2.17.1 which has fixed several of the major CVEs that have been encountered with Log4j2 recently. Please let me know.

@vinothchandar
Copy link
Member

Closing in favor of #5366

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants