Track CVEs for kubernetes dependencies...

[brendandburns]

Kubernetes has a very large number of golang library dependencies. While there is some work to track and ensure license compatability, there is little to know work done to track vulnerabilities in these library dependencies.

Indeed, I don't know of a database (something like https://ossindex.sonatype.org/) for go libraries that we could use. (perhaps the CNCF can help here...)

But the lack of tools and databases isn't an excuse.

We need to do a better job here of tracking, reporting and updating our dependencies to fix known relevant security issues.

And ultimately, we also need to do a periodic audit to make sure that we aren't importing vulnerabilities into the codebase.

@philips @spiffxp @kubernetes/steering-committee

[jayunit100]

Agree , one highly visible critical CVE unchecked could really hurt a data center and even if not, if public it would hurt the growth and adoption of kubernetes in enterprises.

Should this
Be part of the build process ? We may be able to Integrate with blackduck copilot .

[dims]

@jayunit100 AFAICT copilot does not support golang (looking at https://copilot.blackducksoftware.com/)

[cblecker]

cc: @kubernetes/product-security-team

I know GitHub does this for other languages.. I wonder if they will start doing it for Go 🤔.

[brendandburns]

@jessfraz Do you know anything about tools that Github might be working on?

[jessfraz]

yes there is work on better tooling around this, so if you can hang tight for a little bit (maybe a few months but don't quote me on that) it will get better

[jessfraz]

I had bounced some ideas off @cblecker and the PST about our plans here with github integrations but happy to discuss with anyone else as well :) don't want to necessarily make promises in public in writing just in case things change a little but we would love feedback on the ideas going forward

[brendandburns]

@jessfraz any chance you can add me to the email?

[jessfraz]

Added!

[gjohnson]

In the interim, would something like dependabot work?

[tallclair]

• A lot of our dependencies are not mature enough to get CVEs for security issues. That doesn't mean we shouldn't care about the vulnerability.
• We also want to have insight into non-security bug fixes. See this slide: https://docs.google.com/presentation/d/1i3Qnt2yQOfqVLXXkyiaQ_6HOj5ZqkeWQ2wRmT1U2aBs/edit#slide=id.g499934ca8f_0_72
• We should also work on trimming dependencies to reduce the number we need to worry about

[bgrant0607]

On trimming dependencies: that's one of many reasons why I want provider-specific code (e.g., cloud providers, volume sources) out of tree.

[philips]

@brendanburns Did you ever see any response on the CNCF Service Desk request for Go dependency CVE tracking?

@tallclair Is there even a way for Go dependencies to express metadata about security right now?

[tallclair]

Not that I know of... Maybe it's something the go team could help us solve?

[philips]

@tallclair Yea, I think we are in a bit of a chicken and egg situation. If there is no way to express security issues then it is impossible to build tooling for it.

Anyone know who on the Go team to talk to? The only person I know is @bradfitz 👋

[tallclair]

Yea, I think we are in a bit of a chicken and egg situation.

I think there are some hacky things we could do, like creating a dashboard for commits of the dependencies. For other dependencies in github, we could also track issues & releases.

[Sajmani]

I'll loop in a few Go team members here; CVE tracking is definitely something we're thinking about. It would be useful to us to know what form this needs to take to integrate well with the k8s workflow.

CC @rsc @andybons @FiloSottile @julieqiu @spf13

[dims]

@Sajmani not sure if you have seen this already, just in case - https://github.com/kubernetes/sig-release/blob/master/security-release-process-documentation/security-release-process.md#security-release-process

[caniszczyk]

nothing was every filed to the CNCF servicedesk but note that FOSSA which Kubernetes uses for licensing scanning already supports security scanning

CNCF also is happy to fund other tools like Snyk that do this type of work

[dims]

@caniszczyk Snyk does seem interesting, let's see what others say. one issue is that they support dep/govendor and we are currently using godep

• https://snyk.io/docs/snyk-for-golang/
• https://snyk.io/vuln?type=golang

[caniszczyk]

@dims ya, I mean since we're already doing the FOSSA scanning for licensing checks it shouldn't be too hard to do the CVE scan too, they are more than happy to help with this, but we leave it up to the project to choose what tools work best

[philips]

@caniszczyk This was filed with the service desk. Maybe something is wrong with the CNCF service desk? Spam processing? @brendanburns emailed the service desk ([email protected]) with the title "Request for CVE/Vulnerability tracking for golang packages." on November 28th 2018. I was cc'd on the request and sent a followup email yesterday.

…

On Wed, Feb 27, 2019 at 6:54 AM Chris Aniszczyk ***@***.***> wrote: nothing was every filed to the CNCF servicedesk but note that FOSSA which Kubernetes uses for licensing scanning already supports security scanning CNCF also is happy to fund other tools like Snyk that do this type of work — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2992 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AACDCN7osFvCZHOxts8UwD_FjXHunrPaks5vRpwKgaJpZM4Y5Bmu> .

[caniszczyk]

forward me the email and I'll look into it On Wed, Feb 27, 2019 at 11:33 AM Brandon Philips <[email protected]> wrote:

…

@caniszczyk This was filed with the service desk. Maybe something is wrong with the CNCF service desk? Spam processing? @brendanburns emailed the service desk ***@***.***) with the title "Request for CVE/Vulnerability tracking for golang packages." on November 28th 2018. I was cc'd on the request and sent a followup email yesterday. On Wed, Feb 27, 2019 at 6:54 AM Chris Aniszczyk ***@***.***> wrote: > nothing was every filed to the CNCF servicedesk but note that FOSSA which > Kubernetes uses for licensing scanning already supports security scanning > > CNCF also is happy to fund other tools like Snyk that do this type of work > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > < #2992 (comment) >, > or mute the thread > < https://github.com/notifications/unsubscribe-auth/AACDCN7osFvCZHOxts8UwD_FjXHunrPaks5vRpwKgaJpZM4Y5Bmu > > . > — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2992 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAD5IYHUCmbjrJ7fNUWu9hb82VuTYwDrks5vRsFfgaJpZM4Y5Bmu> .

-- Cheers, Chris Aniszczyk http://aniszczyk.org +1 512 961 6719

[philips]

@dims @caniszczyk The problem is that the data set FOSSA and synk are working from is super duper small. There are many libraries Kubernetes depends on that we have fixed issues in that don't show up in the snyk data set.

To fix this we have to work with the Go community to enable publishing of this metadata. This is not a vendor selection issue at this point. It is a missing metadata issue.

[philips]

@caniszczyk Sent. For context here was the request headers.

From: Brendan Burns
Date: Wed, Nov 28, 2018, 10:35 PM
To: [email protected]
Subject: Request for CVE/Vulnerability tracking for golang packages.

[philips]

For those following along the CNCF Service desk silently ignores emails from addresses not pre-registered with it. https://twitter.com/BrandonPhilips/status/1102992489072349184

[tom-snyk]

@dims happy to get some of our team involved to learn your requirements! @philips appreciate this is a community publishing issue to; however, we do have some unique harvest capabilities that might be able to help source more. Let us know if you want to chat with our engineers.

[nikhita]

/assign @navidshaikh

[tallclair]

Hopefully we can leverage https://go.googlesource.com/proposal/+/master/design/draft-vulndb.md for this in the future

[BenTheElder]

@tallclair +1 -- I've been in contact with the google team working on this, they've actually been using kubernetes/kubernetes as a test case given our large dependency tree 👍

[PushkarJ]

I am working with @navidshaikh on this from sig-security In terms of triage options of CVEs post identification regardless of tools, would love to hear some feedback on this proposal: https://docs.google.com/document/d/1V-IUJVYq4SGizNpZXA9GPBPeCJ-oH9p5V747o51NYvo/edit

Thanks @tallclair for your earlier feedback on slack for this proposal.

/sig security

[BenTheElder]

vulndb is now somewhat exposed in https://deps.dev/
however, it's not really setup to monitor a repo @ HEAD AFAICT.
There's been some progress recently on standing up snyk I think (some recent discussion in #sig-testing, #code-organization, #wg-k8s-infra).

[Sajmani]

A prototype CLI for reading the Go vulndb is here:
https://pkg.go.dev/golang.org/x/exp/vulndb/govulncheck
This is not yet ready for production use, but we welcome early adopter feedback.

[dims]

@Sajmani thanks! will poke at it when we get a chance

[PushkarJ]

A prototype CLI for reading the Go vulndb is here:
https://pkg.go.dev/golang.org/x/exp/vulndb/govulncheck
This is not yet ready for production use, but we welcome early adopter feedback.

@vinayakankugoyal is this the same tool you are doing a demo about in next SIG Security Tooling meeting (08/17)?

[vinayakankugoyal]

Yeah its https://pkg.go.dev/golang.org/x/exp/vulndb/govulncheck

[vinayakankugoyal]

cc @zpavlinovic (who will be doing the demo)

[justaugustus]

@vinayakankugoyal is this the same tool you are doing a demo about in next SIG Security Tooling meeting (08/17)?

@kubernetes/release-engineering -- FYI, in case anyone is able to attend this meeting.

Adding current RelEng Tech Leads as assignees:
/unassign @justaugustus @hasheddan
/assign @cpanato @puerco

[tabbysable]

/sig security

[dims]

We have snyk:
https://testgrid.k8s.io/sig-security-snyk-scan#ci-kubernetes-snyk-master&width=20

We have presubmit blocking ci job that runs govulncheck using this script:
https://testgrid.k8s.io/presubmits-kubernetes-blocking#pull-kubernetes-verify&width=20