Fluentd for Audit Logging

[qiyueyao]

This is to complete this issue request Add documentation support for Fluentd integration with Audit Logging

• Added a cookbook for using Fluentd with Antrea
• Changed go get to go install in file hack/update-toc.sh and hack/update-spelling.sh
• Changed set jvm guide link to 7.8 version to avoid 404 status

After investigating with top, the CPU consumption percentage does not have significant change after deploying Fluentd.

[codecov-commenter]

Codecov Report

Merging #2853 (d5152b0) into main (0feed23) will increase coverage by 19.45%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##             main    #2853       +/-   ##
===========================================
+ Coverage   40.23%   59.69%   +19.45%     
===========================================
  Files         166      289      +123     
  Lines       20693    24540     +3847     
===========================================
+ Hits         8326    14648     +6322     
+ Misses      11558     8322     -3236     
- Partials      809     1570      +761     

Flag	Coverage Δ
kind-e2e-tests	46.25% <ø> (?)
unit-tests	40.20% <ø> (-0.04%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
pkg/apiserver/storage/ram/store.go	78.35% <0.00%> (ø)
pkg/apiserver/handlers/webhook/convert_crd.go	2.56% <0.00%> (ø)
pkg/apiserver/handlers/webhook/mutation_labels.go	24.71% <0.00%> (ø)
pkg/ipfix/ipfix_collector.go	81.81% <0.00%> (ø)
...agent/apiserver/handlers/appliedtogroup/handler.go	0.00% <0.00%> (ø)
pkg/legacyapis/controlplane/register.go	88.23% <0.00%> (ø)
...lientset/versioned/typed/security/v1alpha1/tier.go	27.39% <0.00%> (ø)
pkg/legacyapis/stats/v1alpha1/register.go	90.90% <0.00%> (ø)
pkg/signals/signals.go	100.00% <0.00%> (ø)
...versions/security/v1alpha1/clusternetworkpolicy.go	64.28% <0.00%> (ø)
... and 227 more

[antoninbas]

I think https://github.com/antrea-io/antrea/tree/main/docs/cookbooks/ may be a better location for this stuff (with a reference in docs/antrea-network-policy.md). @abhiraut?

[abhiraut]

I think https://github.com/antrea-io/antrea/tree/main/docs/cookbooks/ may be a better location for this stuff (with a reference in docs/antrea-network-policy.md). @abhiraut?

that's a good idea.. probably cookbook/fluentd-integration or something like that?

[qiyueyao]

I think https://github.com/antrea-io/antrea/tree/main/docs/cookbooks/ may be a better location for this stuff (with a reference in docs/antrea-network-policy.md). @abhiraut?

that's a good idea.. probably cookbook/fluentd-integration or something like that?

Sounds good, shall I move the YAML files from build/YAMLs/audit-logging to this folder as well?

[antoninbas]

maybe cookbooks/netpol-logs-fluentd?

yes, please move the YAML manifests there as well

[abhiraut]

I think https://github.com/antrea-io/antrea/tree/main/docs/cookbooks/ may be a better location for this stuff (with a reference in docs/antrea-network-policy.md). @abhiraut?

that's a good idea.. probably cookbook/fluentd-integration or something like that?

Sounds good, shall I move the YAML files from build/YAMLs/audit-logging to this folder as well?

you can take a look at the cookbooks/multus section to see how it is organized

[abhiraut]

maybe cookbooks/netpol-logs-fluentd?

yes, please move the YAML manifests there as well

do we think we could extend fluentd integration beyond netpol in future?

[qiyueyao]

maybe cookbooks/netpol-logs-fluentd?
yes, please move the YAML manifests there as well

do we think we could extend fluentd integration beyond netpol in future?

Yes, it could tail any file as long as we specify the correct parse in kubernetes.conf.

[antoninbas]

Could you use more consistent line wrapping (~80 chars) in the README?

[qiyueyao]

@zyiou This link currently shows 404 that causes markdown test to fail, shall I change it to current heap size setting or version 7.8 heap size setting? Thanks!

[zyiou]

@zyiou This link currently shows 404 that causes markdown test to fail, shall I change it to current heap size setting or version 7.8 heap size setting? Thanks!

Please change it to latter one https://www.elastic.co/guide/en/elasticsearch/reference/7.8/heap-size.html. Thanks for updating this link.

[antoninbas]

@zyiou @qiyueyao Quan already changed the link in #2919 to avoid CI failures. Feel free to update the link again in this PR if you think Quan didn't pick the most appropriate one.

[zyiou]

@zyiou @qiyueyao Quan already changed the link in #2919 to avoid CI failures. Feel free to update the link again in this PR if you think Quan didn't pick the most appropriate one.

I think we should keep 7.8.0 version of the link since it is consistent with the images we are using. We can keep this change in the PR. Thanks!

[tnqn]

@qiyueyao @antoninbas @abhiraut do we plan to include it in v1.4?

[antoninbas]

I found some small issues

@tnqn this can be postponed to 1.5

[antoninbas]

There is still a lot of DeamonSet typos (instead of DaemonSet)

[antoninbas]

/skip-all

[antoninbas]

@qiyueyao the DCO check is failing. It may be a good opportunity to squash the commits since you need to sign them anyway.

[qiyueyao]

@qiyueyao the DCO check is failing. It may be a good opportunity to squash the commits since you need to sign them anyway.

Thanks for the suggestion! Cleaned up the commits and checked sign offs.

[antoninbas]

/skip-all

[antoninbas]

@qiyueyao next time please squash as one commit only and edit the resulting commit message as needed