From 1b006145085c142704f202db727b47991267b057 Mon Sep 17 00:00:00 2001 From: Laurie Tyzenhaus <33037086+laurie-tyz@users.noreply.github.com> Date: Tue, 15 Dec 2020 14:37:45 -0500 Subject: [PATCH 1/5] Update 040_treesForVulMgmt.md --- doc/version_1/040_treesForVulMgmt.md | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/doc/version_1/040_treesForVulMgmt.md b/doc/version_1/040_treesForVulMgmt.md index ce7e527c..4ff340a0 100644 --- a/doc/version_1/040_treesForVulMgmt.md +++ b/doc/version_1/040_treesForVulMgmt.md @@ -6,7 +6,7 @@ Viable decision guidance for vulnerability management should, at a minimum, cons ## Enumerating Stakeholders -Stakeholders in vulnerability management can be identified within multiple independent axes. For example, they can be identified by their responsibility: whether the organization *supplies*, *deploys*, or *coordinates* patches. Organizations may also be distinguished by type of industry sector. While it might be useful to enumerate all the sectors of the economy, we propose to draft decision points that include those from multiple important sectors. For example, we have safety-related questions in the decision path for all suppliers and deployers, so whether or not the stakeholder is in a safety-critical sector, the decision will be addressed. +Stakeholders in vulnerability management can be identified within multiple independent axes. For example, they can be identified by their responsibility: whether the organization *supplies*, *deploys*, or *coordinates* remediation actions. Organizations may also be distinguished by type of industry sector. While it might be useful to enumerate all the sectors of the economy, we propose to draft decision points that include those from multiple important sectors. For example, we have safety-related questions in the decision path for all suppliers and deployers, so whether or not the stakeholder is in a safety-critical sector, the decision will be addressed. The choice not to segregate the decisions by sector is reinforced by the fact that any given software system might be used by different sectors. It is less likely that one organization has multiple responsibilities within the vulnerability management process. Even if there is overlap within an organization, the two responsibilities are often located in distinct business units with distinct decision-making processes. We can treat the responsibilities as non-overlapping, and, therefore, we can build two decision trees—one for each of the “patch supplier” and “patch deployer” responsibilities in the vulnerability management process. We leave “coordinating patches” as future work. Each of these trees will have different decision points that they take to arrive at a decision about a given unit of work. @@ -30,7 +30,7 @@ To further clairify terms, "Remediaton occurs when the vulnerability is eliminat ### Supplying Patches -At a basic level, the decision at a software development organization is whether to issue a work order and what resources to expend to fix a vulnerability in the organization’s software. Prioritization is required because, at least in the current history of software engineering, the effort to patch all known vulnerabilities will exceed available resources. The organization considers several other factors to build the patch; refactoring a large portion of the code base may be necessary for some patches, while others require relatively small changes. We focus only on the priority of building the patch, and we consider four categories of priority, as outlined in Table 2. +At a basic level, the decision at a software development organization is whether to issue a work order and what resources to expend to remediate a vulnerability in the organization’s software. Prioritization is required because, at least in the current history of software engineering, the effort to patch all known vulnerabilities will exceed available resources. The organization considers several other factors to build the patch; refactoring a large portion of the code base may be necessary for some patches, while others require relatively small changes. We focus only on the priority of building the patch, and we consider four categories of priority, as outlined in Table 2. Table 2: Proposed Meaning for Supplier Priority Outcomes @@ -38,13 +38,13 @@ Table 2: Proposed Meaning for Supplier Priority Outcomes | ------------------ | ----------- | | Defer | Do not work on the patch at present. | | Scheduled | Develop a fix within regularly scheduled maintenance using supplier resources as normal. | -| Out-of-Cycle | Develop a fix out-of-cycle, taking resources away from other projects and releasing the fix as a security patch when it is ready. | +| Out-of-Cycle | Develop remediation out-of-cycle, taking resources away from other projects and releasing the fix as a security patch when it is ready. | | Immediate | Develop and release a fix as quickly as possible, drawing on all available resources, potentially including drawing on or coordinating resources from other parts of the organization. | ### Deploying Patches - Whether or not to deploy an available patch is a binary decision. However, additional defensive mitigations may be necessary sooner than patches are available and may be advisable after patches are applied. We recognize that vulnerability management actions are different when a patch is available and when it is not available. + Whether or not to deploy available remediation is a binary decision. However, additional defensive mitigations may be necessary sooner than patches are available and may be advisable after patches are applied. We recognize that vulnerability management actions are different when a patch is available and when it is not available. -When a patch is available, usually the action is to apply it. When a patch is not yet available, the action space is more diverse, but it should involve mitigating the vulnerability (e.g., shutting down services or applying additional security controls) or accepting the risk of not mitigating the vulnerability. Table 3 displays the action priorities for the deployer, which are similar to the supplier case. +When remediation is available, usually the action is to apply it. When remediation is not yet available, the action space is more diverse, but it should involve mitigating the vulnerability (e.g., shutting down services or applying additional security controls) or accepting the risk of not mitigating the vulnerability. Table 3 displays the action priorities for the deployer, which are similar to the supplier case. @@ -56,8 +56,8 @@ Table 3: Proposed Meaning for Deployer Priority Outcomes | ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | | Defer | Do not act at present. | | Scheduled | Act during regularly scheduled maintenance time. | -| Out-of-cycle | Act more quickly than usual to apply the fix out-of-cycle, during the next available opportunity, working overtime if necessary. | -| Immediate | Act immediately; focus all resources on applying the fix as quickly as possible, including, if necessary, pausing regular organization operations. | +| Out-of-cycle | Act more quickly than usual to apply the remediation out-of-cycle, during the next available opportunity, working overtime if necessary. | +| Immediate | Act immediately; focus all resources on applying the patch as quickly as possible, including, if necessary, pausing regular organization operations. | ### Coordinating Patches In coordinated vulnerability disclosure (CVD), the available decision is whether or not to coordinate a vulnerability report. Vulnerability Response Decision Assistance (VRDA) provides a starting point for a decision tree for this situation.23 VRDA is likely adequate for national-level CSIRTs that do general CVD, but other CSIRT types may have different needs. Future work may elicit those types and make a few different decision options. Specialized coordination organizations exist (e.g., ICS-CERT, which conducts CVD for safety-critical systems). We have not developed a coordination tree in this work, but future work could use our principles and design techniques to refine and evaluate VRDA or some other decision tree for coordinated vulnerability disclosure. The CERT guide to CVD provides something similar for those deciding how to report and disclose vulnerabilities they have discovered [@householder2020cvd, section 6.10]. @@ -65,7 +65,7 @@ In coordinated vulnerability disclosure (CVD), the available decision is whether Within each setting, the decisions are a kind of equivalence class for priority. That is, if an organization must deploy patches for three vulnerabilities, and if these vulnerabilities are all assigned the *scheduled* priority, then the organization can decide which to deploy first. The priority is equivalent. This approach may feel uncomfortable since CVSS gives the appearance of a finer grained priority. CVSS appears to say, “Not just 4.0 to 6.9 is ‘medium’ severity, but 4.6 is more severe than 4.5.” However, as discussed previously (see page 4), CVSS is designed to be accurate only within +/- 0.5, and, in practice, is scored with errors of around +/- 1.5 to 2.5 [@allodi2018effect, see Figure 1]. An error of this magnitude is enough to make all of the “normal” range from 4.0 to 6.9 equivalent, because 5.5 +/- 1.5 is the range 4.0 to 7.0. Our proposal is an improvement over this approach. CVSS errors often cross decision boundaries; in other words, the error range often includes the transition between “high” and “critical” or “medium.” Since our approach keeps the decisions qualitatively defined, this fuzziness does not affect the results. -Returning to the example of an organization with three vulnerabilities to patch that were assigned *scheduled* priority, in SSVC, they can be patched in any order. This is an improvement over CVSS, since based on the scoring errors, CVSS was essentially just giving random fine-grained priorities within qualitative categories anyway. With our system, organizations can be more deliberate about conveniently organizing work that is of equivalent priority. +Returning to the example of an organization with three vulnerabilities to remediate that were assigned *scheduled* priority, in SSVC, they can be patched in any order. This is an improvement over CVSS, since based on the scoring errors, CVSS was essentially just giving random fine-grained priorities within qualitative categories anyway. With our system, organizations can be more deliberate about conveniently organizing work that is of equivalent priority. ### Risk Tolerance and Response Priority @@ -73,14 +73,14 @@ SSVC enables stakeholders to balance and manage their risks themselves. We follow [@ISO73] and define risk as "effect of uncertainty on objectives;" see [@ISO73] for notes on the terms in the definition. For any vulnerability management practice to succeed it must balance at least two risks: -1. Change risk: the potential costs of deploying fixes, which include testing and deployment in addition to any problems that could arise from making changes to production systems. +1. Change risk: the potential costs of deploying remediation, which include testing and deployment in addition to any problems that could arise from making changes to production systems. 2. Vulnerability risk: the potential costs of incidents resulting from exploitation of vulnerable systems To place these risks in context, we follow the SEI's Taxonomy of Operational Cyber Security Risks [@cebula2010taxonomy]. Change risk can be characterized as a combination of Class 2 and/or Class 3 risks. Class 2: Systems and Technology Failures includes hardware, software, and systems risks. Class 3: Failed Internal Processes can arise from process design, process execution, process controls, or supporting processes. Meanwhile, vulnerability risk falls into Subclass 1.2: Actions of People: Deliberate. In developing the decision trees in this document, we had in mind stakeholders with a moderate tolerance for risk. The resulting trees reflect that assumption. Organizations may of course be more or less conservative in their own vulnerability management practices, and we cannot presume to determine how an organization should balance their risk. -We therefore remind our readers that the labels on the trees (defer, immediate, etc.) can and should be customized to suit the needs of individual stakeholders wherever necessary and appropriate. For example, an organization with a high aversion to change risk might choose to accept more vulnerability risk by lowering the overall response labels for many branches in the trees, resulting in fewer vulnerabilities attaining the most urgent response. On the other hand, an organization with a high aversion to vulnerability risk could elevate the priority of many branches to ensure fixes are deployed quickly. +We therefore remind our readers that the labels on the trees (defer, immediate, etc.) can and should be customized to suit the needs of individual stakeholders wherever necessary and appropriate. For example, an organization with a high aversion to change risk might choose to accept more vulnerability risk by lowering the overall response labels for many branches in the trees, resulting in fewer vulnerabilities attaining the most urgent response. On the other hand, an organization with a high aversion to vulnerability risk could elevate the priority of many branches to ensure patches are deployed quickly. ## Scope @@ -94,11 +94,11 @@ We put forward recommendations for each of these. However, users of the decision process may want to define different scopes. Users may define a different scope as long as the scope is consistent across decisions, and are credible, explicit, and accessible to all relevant decision makers. -For example, suppliers often decline to support products beyond a declared end-of-life (EOL) date. In those cases, a supplier could reasonably consider vulnerabilities in those products to be out of scope. However, a deployer may still have active instances of EOL products in their infrastructure. It remains appropriate for a deployer to use SSVC to prioritize their response to such situations, since even if there is no fix forthcoming from the supplier it may be possible for the deployer to mitigate or remediate the vulnerability in other ways, up to and including decommissioning the affected system(s). +For example, suppliers often decline to support products beyond a declared end-of-life (EOL) date. In those cases, a supplier could reasonably consider vulnerabilities in those products to be out of scope. However, a deployer may still have active instances of EOL products in their infrastructure. It remains appropriate for a deployer to use SSVC to prioritize their response to such situations, since even if there is no remediation forthcoming from the supplier it may be possible for the deployer to mitigate or remediate the vulnerability in other ways, up to and including decommissioning the affected system(s). ### Boundaries of the Affected System -One distinction is whether the system of interest is software per se or a cyber-physical system. One problem is that in every practical case, both are involved. Software is what has vulnerabilities and is what vulnerability management is focused on patching. Multiple pieces of software run on any given computer system. To consider software vulnerabilities in a useful scope, we follow prior work and propose that a vulnerability affects “the set of software instructions that executes in an environment with a coherent function and set of permissions” [@spring2015global]. This definition is useful because it lets us keep to common usage and intuition and call the Linux kernel—at least a specific version of it—“one piece” of software. But decision points about safety and mission impact are not about the software per se; they are about the cyber-physical system, of which the software is a part. The term "physical" in "cyber-physical system" should be interpreted broadly; selling stocks or manipulating press outlet content are both best understood as affecting human social institutions. These social institutions do not have much of a corporeal instantiation, but they are physical in the sense that they are not merely software, and so are parts of cyber-physical systems. +One distinction is whether the system of interest is software per se or a cyber-physical system. One problem is that in every practical case, both are involved. Software is what has vulnerabilities and is what vulnerability management is focused on remediation. Multiple pieces of software run on any given computer system. To consider software vulnerabilities in a useful scope, we follow prior work and propose that a vulnerability affects “the set of software instructions that executes in an environment with a coherent function and set of permissions” [@spring2015global]. This definition is useful because it lets us keep to common usage and intuition and call the Linux kernel—at least a specific version of it—“one piece” of software. But decision points about safety and mission impact are not about the software per se; they are about the cyber-physical system, of which the software is a part. The term "physical" in "cyber-physical system" should be interpreted broadly; selling stocks or manipulating press outlet content are both best understood as affecting human social institutions. These social institutions do not have much of a corporeal instantiation, but they are physical in the sense that they are not merely software, and so are parts of cyber-physical systems. The hard part of delineating the boundaries of the affected system is specifying what it means for one system to be a part of another. Just because a computer is bolted to a wall does not mean the computer is part of the wall’s purpose, which is separating physical space. At the same time, an off-premises DNS server may be part of the site security assurance system if the on-premises security cameras rely on the DNS server to connect to the displays at the guard’s desk. We define computer software as part of a cyber-physical system if the two systems are mutually manipulable; that is, changes in the part (the software) will (at least, often) make detectable and relevant changes to the whole (the cyber-physical system), and changes in the whole will (often) make relevant and detectable changes in the part [@spring2018generalization]. @@ -110,7 +110,7 @@ When reasoning about a vulnerability, we assign the vulnerability to the nearest - **More abstract** means it’s at least one level of abstraction above the specific location of the vulnerability. For example, if the vulnerability is localized to a single line of code in a function, then the function, the module, the library, the application, the product, and the system it belongs to are all "more abstract." - - **Discrete** means there is an identifiable thing that can be fixed (e.g., the unit of patching). + - **Discrete** means there is an identifiable thing that can be remediated (e.g., the unit of patching). Products, libraries, and applications tend to be appropriate objects of focus when seeking the right level to analyze the impact of a vulnerability. Hence, when reasoning about the technical impact of a vulnerability localized to a function in a module in an open source library, the nearest relevant discrete abstraction is the library because the patches are made available at the library level. Similarly, analysis of a vulnerability in closed source database software that supports an enterprise resource management (ERM) system would consider the technical impact to the database, not to the ERM system. From d2ce50774403bdc031c426ded72a5d87b1f55740 Mon Sep 17 00:00:00 2001 From: Laurie Tyzenhaus <33037086+laurie-tyz@users.noreply.github.com> Date: Thu, 17 Dec 2020 11:26:47 -0500 Subject: [PATCH 2/5] Update 040_treesForVulMgmt.md Ok. Trying this one again. Changes are as follows: Some "patch" was changed to "remediate" Some "fix" was changed to "remediate" I rolled back the changes from "fix" to "patch". I will open a new branch so @cgyarbrough and I can work through the fix/patch discussion. --- doc/version_1/040_treesForVulMgmt.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/version_1/040_treesForVulMgmt.md b/doc/version_1/040_treesForVulMgmt.md index 4ff340a0..05c6782f 100644 --- a/doc/version_1/040_treesForVulMgmt.md +++ b/doc/version_1/040_treesForVulMgmt.md @@ -57,7 +57,7 @@ Table 3: Proposed Meaning for Deployer Priority Outcomes | Defer | Do not act at present. | | Scheduled | Act during regularly scheduled maintenance time. | | Out-of-cycle | Act more quickly than usual to apply the remediation out-of-cycle, during the next available opportunity, working overtime if necessary. | -| Immediate | Act immediately; focus all resources on applying the patch as quickly as possible, including, if necessary, pausing regular organization operations. | +| Immediate | Act immediately; focus all resources on applying the fix as quickly as possible, including, if necessary, pausing regular organization operations. | ### Coordinating Patches In coordinated vulnerability disclosure (CVD), the available decision is whether or not to coordinate a vulnerability report. Vulnerability Response Decision Assistance (VRDA) provides a starting point for a decision tree for this situation.23 VRDA is likely adequate for national-level CSIRTs that do general CVD, but other CSIRT types may have different needs. Future work may elicit those types and make a few different decision options. Specialized coordination organizations exist (e.g., ICS-CERT, which conducts CVD for safety-critical systems). We have not developed a coordination tree in this work, but future work could use our principles and design techniques to refine and evaluate VRDA or some other decision tree for coordinated vulnerability disclosure. The CERT guide to CVD provides something similar for those deciding how to report and disclose vulnerabilities they have discovered [@householder2020cvd, section 6.10]. @@ -80,7 +80,7 @@ To place these risks in context, we follow the SEI's Taxonomy of Operational Cyb In developing the decision trees in this document, we had in mind stakeholders with a moderate tolerance for risk. The resulting trees reflect that assumption. Organizations may of course be more or less conservative in their own vulnerability management practices, and we cannot presume to determine how an organization should balance their risk. -We therefore remind our readers that the labels on the trees (defer, immediate, etc.) can and should be customized to suit the needs of individual stakeholders wherever necessary and appropriate. For example, an organization with a high aversion to change risk might choose to accept more vulnerability risk by lowering the overall response labels for many branches in the trees, resulting in fewer vulnerabilities attaining the most urgent response. On the other hand, an organization with a high aversion to vulnerability risk could elevate the priority of many branches to ensure patches are deployed quickly. +We therefore remind our readers that the labels on the trees (defer, immediate, etc.) can and should be customized to suit the needs of individual stakeholders wherever necessary and appropriate. For example, an organization with a high aversion to change risk might choose to accept more vulnerability risk by lowering the overall response labels for many branches in the trees, resulting in fewer vulnerabilities attaining the most urgent response. On the other hand, an organization with a high aversion to vulnerability risk could elevate the priority of many branches to ensure fixes are deployed quickly. ## Scope From f3a7771b9763e8f5268969561ec910a3912f9390 Mon Sep 17 00:00:00 2001 From: Laurie Tyzenhaus <33037086+laurie-tyz@users.noreply.github.com> Date: Thu, 17 Dec 2020 13:51:47 -0500 Subject: [PATCH 3/5] Update 047_treesForVulMgmt_4.md --- doc/version_1/047_treesForVulMgmt_4.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/doc/version_1/047_treesForVulMgmt_4.md b/doc/version_1/047_treesForVulMgmt_4.md index 46fb87c3..380a6526 100644 --- a/doc/version_1/047_treesForVulMgmt_4.md +++ b/doc/version_1/047_treesForVulMgmt_4.md @@ -42,7 +42,7 @@ Deployers (Continued from Figure 2 and Figure 3) Stakeholders are encouraged to customize the SSVC decision process to their needs. Indeed, the first part of SSVC stands for "stakeholder-specific." However, certain parts of SSVC are more amenable to customization than others. -In this section, we'll cover what a stakeholder should leave fixed, what we imagine customization looks like, and some advice on building a usable and manageable decision tree based on our experience so far. +In this section, we'll cover what a stakeholder should leave static, what we imagine customization looks like, and some advice on building a usable and manageable decision tree based on our experience so far. We suggest that the decision points, their definitions, and the decision values should not be customized. Different vulnerability management teams inevitably think of topics such as [*Utility*](#utility) to the adversary in slightly different ways. @@ -54,7 +54,7 @@ Which decision points are involved in a vulnerability management team's decision We have provided some examples for different stakeholder communities here. What decision points a team considers reflects what it cares about and the risks prioritizes. Different teams may legitimately prioritize different objectives. -It should be easier for teams to discuss and communicate such differences if the definitions of the decision points remain fixed. +It should be easier for teams to discuss and communicate such differences if the definitions of the decision points remain static. The other aspect of risk management that SSVC allows a team to customize is its risk appetite or risk tolerance. A team's risk appetite is reflected directly by the priority labels for each combination of decision values. @@ -89,9 +89,11 @@ For example, whether exploitation modules are available in ExploitDB, Metasploit Some of the decision points require some substantial upfront analysis effort to gather risk assessment or organizational data. However, once gathered, this information can be efficiently reused across many vulnerabilities and only refreshed occasionally. An obvious example of this is the mission impact decision point. To answer this, a deployer must analyze their essential functions, how they interrelate, and how they are supported. Exposure is similar; answering that decision point requires an asset inventory, adequate understanding of the network topology, and a view of the enforced security controls. Independently operated scans, such as Shodan or Shadowserver, may play a role in evaluating exposure, but the entire exposure question cannot be reduced to a binary question of whether an organization’s assets appear in such databases. Once the deployer has the situational awareness to understand MEFs or exposure, selecting the answer for each individual vulnerability is usually straightforward. -Stakeholders who use the prioritization method should consider releasing the priority with which they handled the vulnerability. This disclosure has various benefits. For example, if the supplier publishes a priority ranking, then deployers could consider that in their decision-making process. One reasonable way to include it is to break ties for the deployer. If a deployer has three “scheduled” vulnerabilities to patch, they may address them in any order. If two vulnerabilities were produced by the supplier as “scheduled” patches, and one was “out-of-cycle,” then the deployer may want to use that information to favor the latter. +Stakeholders who use the prioritization method should consider releasing the priority with which they handled the vulnerability. This disclosure has various benefits. For example, if the supplier publishes a priority ranking, then deployers could consider that in their decision-making process. One reasonable way to include it is to break ties for the deployer. If a deployer has three “scheduled” vulnerabilities to remediate, they may address them in any order. If two vulnerabilities were produced by the supplier as “scheduled” patches, and one was “out-of-cycle,” then the deployer may want to use that information to favor the latter. -In the case where no information is available or the organization has not yet matured its initial situational analysis, we can suggest something like defaults for some decision points. If the deployer does not know their exposure, that means they do not know where the devices are or how they are controlled, so they should assume *Exposure* is **open**. If the decision maker knows nothing about the environment in which the device is used, we suggest assuming a **major** *Safety Impact*. This position is conservative, but software is thoroughly embedded in daily life now, so we suggest that the decision maker provide evidence that no one’s well-being will suffer. The reach of software exploits is no longer limited to a research network. Similarly, with *Mission Impact*, the deployer should assume that the software is in use at the organization for a reason, and that it supports essential functions unless they have evidence otherwise. With a total lack of information, assume **MEF support crippled** as a default. *Exploitation* needs no special default; if adequate searches are made for exploit code and none is found, the answer is **none**. The decision set {**none**, **open**, **MEF crippled**, **major**} results in a scheduled patch application. +In the case where no information is available or the organization has not yet matured its initial situational analysis, we can suggest something like defaults for some decision points. If the deployer does not know their exposure, that means they do not know where the devices are or how they are controlled, so they should assume *Exposure* is **open**. If the decision maker knows nothing about the environment in which the device is used, we suggest assuming a **major** *Safety Impact*. This position is conservative, but software is thoroughly embedded in daily life now, so we suggest that the decision maker provide evidence that no one’s well-being will suffer. The reach of software exploits is no longer limited to a research network. Similarly, with *Mission Impact*, the deployer should assume that the software is in use at the organization for a reason, and that it supports essential functions unless they have evidence otherwise. With a total lack of information, assume ** + +support crippled** as a default. *Exploitation* needs no special default; if adequate searches are made for exploit code and none is found, the answer is **none**. The decision set {**none**, **open**, **MEF crippled**, **major**} results in a scheduled patch application. ## Guidance on Communicating Results From 818c04eb9c86955fae63d69647c6919f12791494 Mon Sep 17 00:00:00 2001 From: Laurie Tyzenhaus <33037086+laurie-tyz@users.noreply.github.com> Date: Thu, 17 Dec 2020 14:15:07 -0500 Subject: [PATCH 4/5] Update 050_evaluationDraftTrees.md Changed "fixed" to "static". Changed "fix" to "correct". Changed "a patch" to "remediation" --- doc/version_1/050_evaluationDraftTrees.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/doc/version_1/050_evaluationDraftTrees.md b/doc/version_1/050_evaluationDraftTrees.md index bc028f96..41c5947c 100644 --- a/doc/version_1/050_evaluationDraftTrees.md +++ b/doc/version_1/050_evaluationDraftTrees.md @@ -8,7 +8,7 @@ We conducted a pilot test on the adequacy of the hypothesized decision trees. Th The pilot study tested inter-rater agreement of decisions reached. Each author played the role of an analyst in both stakeholder groups (supplying and deploying) for nine vulnerabilities. We selected these nine vulnerabilities based on expert analysis, with the goal that the nine cases cover a useful series of vulnerabilities of interest. Specifically, we selected three vulnerabilities to represent safety-critical cases, three to represent regulated-systems cases, and three to represent general computing cases. -During the pilot, we did not form guidance on how to assess the values of the decision points. However, we did standardize the set of evidence that was taken to be true for the point in time representing the evaluation. Given this fixed information sheet, we did not synchronize an evaluation process for how to decide whether *Exploitation*, for example, should take the value of **none**, **PoC**, or **active**. We agreed on the descriptions of the decision points and the descriptions of their values in (a prior version of) Section 4.4. The goal of the pilot was to test the adequacy of those descriptions by evaluating whether the analysts agreed. We improved the decision point descriptions based on the results of the pilot; our changes are documented in Section 5.3. +During the pilot, we did not form guidance on how to assess the values of the decision points. However, we did standardize the set of evidence that was taken to be true for the point in time representing the evaluation. Given this static information sheet, we did not synchronize an evaluation process for how to decide whether *Exploitation*, for example, should take the value of **none**, **PoC**, or **active**. We agreed on the descriptions of the decision points and the descriptions of their values in (a prior version of) Section 4.4. The goal of the pilot was to test the adequacy of those descriptions by evaluating whether the analysts agreed. We improved the decision point descriptions based on the results of the pilot; our changes are documented in Section 5.3. In the design of the pilot, we held constant the information available about the vulnerability. This strategy restricted the analyst to decisions based on the framework given that information. That is, it controlled for differences in information search procedure among the analysts. The information search procedure should be controlled because this pilot was about the framework content, not how people answer questions based on that content. After the framework is more stable, a separate study should be devised that shows how analysts should answer the questions in the framework. The basis for the assessment that information search will be an important aspect in using the evaluation framework is based on our experience while developing the framework. During informal testing, often disagreements about a result involved a disagreement about what the scenario actually was; different information search methods and prior experiences led to different understandings of the scenario. This pilot methodology holds the information and scenario constant to test agreement about the descriptions themselves. This strategy makes constructing a prioritization system more tractable by separating problems in how people search for information from problems in how people make decisions. This paper focuses only on the structure of decision making. Advice about how to search for information about a vulnerability is a separate project that will be part of future work. In some domains, namely exploit availability, we have started that work in parallel. @@ -61,7 +61,7 @@ exploitation=active We evaluated inter-rater agreement in two stages. In the first stage, each analyst independently documented their decisions. This stage produced 18 sets of decisions (nine vulnerabilities across each of two stakeholder groups) per analyst. In the second stage, we met to discuss decision points where at least one analyst differed from the others. If any analyst changed their decision, they appended the information and evidence they gained during this meeting in the “supporting evidence” value in their documentation. No changes to decisions were forced, and prior decisions were not erased, just amended. After the second stage, we calculated some statistical measures of inter-rater agreement to help guide the analysis of problem areas. -To assess agreement, we calculate Fleiss’ kappa, both for the value in the leaf node reached for each case and every decision in a case [@fleiss1973equivalence]. Evaluating individual decisions is complicated slightly because the different paths through the tree mean that a different number of analysts may have evaluated certain items, and Fleiss’ kappa requires a fixed number of raters. “Leaf node reached” is described to pick out the specific path through the tree the analyst selected and to treat that as a label holistically. Measuring agreement based on the path has the drawback that ostensibly similar paths, which agree on 3 of 4 decisions for example, are treated as no more similar than paths that agree on 0 of 4 decisions. So the two measures of agreement (per decision and per path) are complementary, and we report both. +To assess agreement, we calculate Fleiss’ kappa, both for the value in the leaf node reached for each case and every decision in a case [@fleiss1973equivalence]. Evaluating individual decisions is complicated slightly because the different paths through the tree mean that a different number of analysts may have evaluated certain items, and Fleiss’ kappa requires a static number of raters. “Leaf node reached” is described to pick out the specific path through the tree the analyst selected and to treat that as a label holistically. Measuring agreement based on the path has the drawback that ostensibly similar paths, which agree on 3 of 4 decisions for example, are treated as no more similar than paths that agree on 0 of 4 decisions. So the two measures of agreement (per decision and per path) are complementary, and we report both. ### Pilot participant details @@ -192,7 +192,7 @@ The following changes are reflected in Section 4, Decision Trees for Vulnerabili - We clarified that “MEF failure” refers to any **one** essential function failing, not failure of all of them. We changed most severe mission impact to “mission failure” to better reflect the relationship between MEFs and the organization’s mission. - - We removed the “supplier portfolio value” question since it had poor agreement, and there is no clear way to fix it. We replaced this question with *Utility*, which better captures the relevant kinds of value (namely, to the adversary) of the affected component while remaining amenable to pragmatic analysis. + - We removed the “supplier portfolio value” question since it had poor agreement, and there is no clear way to correct it. We replaced this question with *Utility*, which better captures the relevant kinds of value (namely, to the adversary) of the affected component while remaining amenable to pragmatic analysis. - We clarified that “proof of concept” (see *Exploitation*) includes cases in which existing tooling counts as a PoC. The examples listed are suggestive, not exhaustive. @@ -204,11 +204,11 @@ The following changes are reflected in Section 4, Decision Trees for Vulnerabili In this section, we present ideas we tried but rejected for various reasons. We are not presenting this section as the final word on excluding these ideas, but we hope the reasons for excluding them are instructive, will help prevent others from re-inventing the proverbial wheel, and can guide thinking on future work. -Initially, we brainstormed approximately 12 potential decision points, most of which we removed early in our development process through informal testing. These decision points included adversary’s strategic benefit of exploiting the vulnerability, state of legal or regulatory obligations, cost of developing a patch, patch distribution readiness, financial losses to customers due to potential exploitation, and business losses to the deployer. +Initially, we brainstormed approximately 12 potential decision points, most of which we removed early in our development process through informal testing. These decision points included adversary’s strategic benefit of exploiting the vulnerability, state of legal or regulatory obligations, cost of developing remediation, patch distribution readiness, financial losses to customers due to potential exploitation, and business losses to the deployer. Some of these points left marks on other decision points. The decision point “financial losses of customers” led to an amendment of the definition of *Safety* to include “well-being,” such that, for example, bankruptcies of third parties are now a major safety impact. The “business losses to the deployer” decision point is covered as a mission impact insofar as profit is a mission of publicly traded corporations. -Three of the above decision points left no trace on the current system. “State of legal or regulatory obligations,” “cost of developing a patch,” and “patch distribution readiness” were dropped as either being too vaguely defined, too high level, or otherwise not within the scope of decisions by the defined stakeholders. The remaining decision point, “adversary’s strategic benefit of exploiting the vulnerability,” transmuted to a different sense of system value. In an attempt to be more concrete and not speculate about adversary motives, we considered a different sense of value: supplier portfolio value. +Three of the above decision points left no trace on the current system. “State of legal or regulatory obligations,” “cost of developing remediation,” and “patch distribution readiness” were dropped as either being too vaguely defined, too high level, or otherwise not within the scope of decisions by the defined stakeholders. The remaining decision point, “adversary’s strategic benefit of exploiting the vulnerability,” transmuted to a different sense of system value. In an attempt to be more concrete and not speculate about adversary motives, we considered a different sense of value: supplier portfolio value. The only decision point that we have removed following the pilot is developer portfolio value. This notion of value was essentially an over-correction to the flaws identified in the “adversary’s strategic benefit of exploiting the vulnerability” decision point. “Supplier portfolio value” was defined as “the value of the affected component as a part of the developer’s product portfolio. Value is some combination of importance of a given piece of software, number of deployed instances of the software, and how many people rely on each. The developer may also include lifecycle stage (early development, stable release, decommissioning, etc.) as an aspect of value.” It had two possible values: low and high. As Table 13 demonstrates, there was relatively little agreement among the six analysts about how to evaluate this decision point. We replaced this sense of portfolio value with *Utility*, which combines *Value Density* and *Automatability*. From 66b068ffdeea1090d325a0c24aa99ccf5a9efd0f Mon Sep 17 00:00:00 2001 From: Laurie Tyzenhaus <33037086+laurie-tyz@users.noreply.github.com> Date: Thu, 17 Dec 2020 14:20:36 -0500 Subject: [PATCH 5/5] Update 040_treesForVulMgmt.md updated to include "mitigation or" in both areas Jono commented on. --- doc/version_1/040_treesForVulMgmt.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/version_1/040_treesForVulMgmt.md b/doc/version_1/040_treesForVulMgmt.md index 05c6782f..a7d6d5f6 100644 --- a/doc/version_1/040_treesForVulMgmt.md +++ b/doc/version_1/040_treesForVulMgmt.md @@ -38,7 +38,7 @@ Table 2: Proposed Meaning for Supplier Priority Outcomes | ------------------ | ----------- | | Defer | Do not work on the patch at present. | | Scheduled | Develop a fix within regularly scheduled maintenance using supplier resources as normal. | -| Out-of-Cycle | Develop remediation out-of-cycle, taking resources away from other projects and releasing the fix as a security patch when it is ready. | +| Out-of-Cycle | Develop mitigation or remediation out-of-cycle, taking resources away from other projects and releasing the fix as a security patch when it is ready. | | Immediate | Develop and release a fix as quickly as possible, drawing on all available resources, potentially including drawing on or coordinating resources from other parts of the organization. | ### Deploying Patches @@ -56,7 +56,7 @@ Table 3: Proposed Meaning for Deployer Priority Outcomes | ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | | Defer | Do not act at present. | | Scheduled | Act during regularly scheduled maintenance time. | -| Out-of-cycle | Act more quickly than usual to apply the remediation out-of-cycle, during the next available opportunity, working overtime if necessary. | +| Out-of-cycle | Act more quickly than usual to apply the mitigation or remediation out-of-cycle, during the next available opportunity, working overtime if necessary. | | Immediate | Act immediately; focus all resources on applying the fix as quickly as possible, including, if necessary, pausing regular organization operations. | ### Coordinating Patches