add functionality to scrape pods per kubernetes node

[mg03]

Required for all PRs:

• [ y] Signed CLA.
• [ y] Associated README.md updated.
• [ y] Has appropriate unit tests.

Fixes #6950

[mg03]

@danielnelson : please review my PR, thank you

[mg03]

@jackzampolin : Hi, Im tagging you since I see you are the primary moderator on community.influxdata.com and I thought you might have some suggestion/s . Would you know who could possibly help with PR review ?

[mg03]

@danielnelson : hi would it be possible for you to take a look at this PR and let me know your thoughts on the same.

[danielnelson]

Code looks good but I do feel that this method is very specific to your case and that we should try to generalize it more.

I'm not a Kubernetes expert, but perhaps we could provide access to the labelSelector option when starting the watch. With this we could watch any custom subset of pods and as a bonus it should perform better since it doesn't need to skip over unmatched pods. However, this would require having the right labels set, maybe this technique can work: https://stackoverflow.com/questions/40131682/how-to-set-label-to-kubernetes-node-at-creation-time

[mg03]

@danielnelson : thank you for the feedback.

1. Agree the code is specific
2. Labels on nodes do not help filter pods during a watch. Only pod attributes can be used to filter during a watch.
3. Assuming we go with filtering on pod label selector, still we cannot hone in on filtering by pod running on a specific node, the reason is that the nodename attribute is not part of pod labels. Labels are under metadata parent field. the nodename is under the spec parent field.
4. Due to above may I suggest if we can have both label selectors and field selectors, so that users can filter on both or any or none of the filters during watch cycle.
   Let me know if you are ok with my suggestion in point 4) above and I can work towards it ....

[danielnelson]

That sounds great, thanks.

[mg03]

@danielnelson : As per our discussion above, Ive updated the code to support Kubernetes label and field selectors. Please review and let me know your thoughts.

[mg03]

@danielnelson : would it be possible for you to take a look at the PR, which is updated as per our conversation about using Kubernetes label and field selectors?

[mg03]

@danielnelson : The functionality implemented here is important for adoption within our org, as such please provide guidance on the path forward on this PR.

[mg03]

@danielnelson : I have made the changes as requested, please review.

[mg03]

@danielnelson : updated the PR with changes as requested, please review.

[bamvenkatesh]

@mg03 - Hi, I was just curious why choose DaemonSet instead of a deployment for Telegraf.

Like @russorat asked, Is there a specific reason you don't want to run a single telegraf to handle the scraping?