guidelines for prioritizing projects for security assessements

[ultrasaurus]

We need guidelines on how to prioritize projects for security assessments when there are multiple projects interested in engaging with us on an assessment.

see draft guidelines -- these need to be moved into a PR, linking from here for visibility

Related:

• Assessment Listing Assessments Listing #206
• Annual review process: document annual review process to update security assessment #152

[ultrasaurus]

see draft guidelines

[ultrasaurus]

related issue on annual review -- maybe we should get this reviewed/merged first: #152

[rficcaglia]

my $.02 is that the CNCF needs to endorse the security assessment in some concrete way.

either officially require it for some part of the process, or at the very least give it some "fast track" consideration (a thumb on the scale you might say).

from discussions I've had with project teams, they were/are hesitant to dedicate time and resources away from other activities (including other CNCF presentations) to do the security assessment if it's not required.

similarly, discussions with potential volunteers suggest that volunteers are very interested in high profile CNCF projects, but not necessarily any random project that comes asking.

on one of the Wed calls, Dan and I debated this at some length and he made a valid point --- security audits are very expensive (in terms of time, or as a proxy for time, money) --- and as such having volunteers donate their time (and forego money) ought to generate some tangible reward. If you are working on a high impact, high visibility CNCF project with the explicit recognition and acceptance of the CNCF TOC, this could be very rewarding in terms of community recognition, or professional recognition, or generating consulting gigs, etc etc.

in short there has to be a carrot, a stick, or both from the CNCF for both the projects and volunteers :)

[ultrasaurus]

The security assessments are absolutely endorsed and supported by the TOC. They are very excited about our work in this area. We decided (and TOC thought it was a good idea) that we would do 5 assessments (#167) and then make some process improvements before considering whether/how to encourage/evangelize/require projects to participate.

We're about to finish assessment No. 2, and it took way more calendar time and a little bit more effort than we would like. I hope/expect that by the time we get through five of them, that we'll be able to execute more consistently and then can effectively set expectations (for project & TOC) about both the effort and the outcome. Until then, I think it is appropriate to engage with projects that are excited to engage with us (because they see value in the process already or are willing to devote time to help us figure it out) and volunteers who find it intrinsically interesting without any extrinsic reward.

I know it can be a little frustrating because it seems to move forward in fits and starts right now, but I hope we'll be able to smooth out the frustrating parts, while keeping the interesting / challenging parts of the process. Personally, I have found the experience of being a security reviewer to be very rewarding -- I've learned about specific projects and also learned from the experience of other reviewers how they think about threats and risks in the cloud ecosystem.

[rficcaglia]

absolutely endorsed and supported by the TOC

that is good to know! I don't think that has been communicated to the group before. but to drill down, how exactly has it been endorsed? ie if it is not required for sandbox, incubation or other milestones, is it "recommended"? if so, is there actually a document in the TOC repo saying to the effect "we recommend the assessment for CNCF projects"?

appropriate to engage with projects that are excited to engage with us

are there any projects who have come forward yet, ie are ready to schedule a concrete start date? Falco and NSM have not committed to a concrete start date.

learned from the experience of other reviewers how they think about threats and risks

On that note I have reached out to Radically Open Security to ask Dr. Rieback to present to the group on Dec 4 specifically to hear her experiences launching a transparent and non-profit security audit process, and specifically her experiences with Mozilla and other high visibility projects. Perhaps there are best practices that we can learn from her experiences.

[ultrasaurus]

assigning this issue to @pragashj who is currently reviewing the doc -- additional improvements, if needed, can be made will another PR

cc @jbeda @lizrice for visibility, this was covered by @dshaw in today's TOC meeting

[stale]

This issue has been automatically marked as inactive because it has not had recent activity.

[anvega]

#296 was long ago merged.