Provide mechanism for inlining externalized properties data into the root log

[michaelcfanning]

This required in order to accommodate sarifLog.taxonomies metadata, which will be a log level construct, as the data itself is entirely external to any tool execution (e.g., the CWE definitions).

The property names would be a JSON path to the construct which has been externalized.

@lgolding per discussion

[michaelcfanning]

Disposition: add a taxonomies property to the run object externalPropertyFiles object. Taxonomies is externalizable. Any externalized property reference can utilize the SARIF protocol to refer to data within the log file.

[kupsch]

Below is an example SARIF file, for my proposal to inline external data that can be shared between runs. Add a new top level property whose value is an array of externalPropertyFile objects. The sarif URI protocol is used to reference the location in the array. Each run is unchanged except for the fileLocation of the external property, and the contents of the external file is unchanged. There is no coupling of runs using this scheme, and file location in an external property file is always an externalPropertyFile object.

A further step would be to not require the externalPropertyFiles fileLocation in a run to change its fileLocation, but to inform consumers that if an inlineExternalFiles property exists with a matching instanceGuid then this is the externalPropertyFile object; this makes inline and externalizing simple.

{
  ...
  
  "inlineExternalFiles":  [
    {
      "$schema":
        "http:///json.schemastore.org/sarif-external-property-file-2.0.0",

      "version": "2.0.0",

      "instanceGuid": "22222222-2222-2222-2222-222222222222",

      "run.files": {
        "apple.png": {
          "mimeType": "image/png"
        },
        "banana.png": {
          "mimeType": "image/png"
        }
      }
    }
  ],

  "runs": [
    { 
      "originalUriBaseIds": {
        "INLINE": {
          "uri": "sarif:///inlineExternalPropertyFiles/"
        }
      },

      "externalPropertyFiles": {
        "files": [
          { 
            "fileLocation": {
              "baseUriId": "INLINE",
              "uri": "0",
              "fileIndex": 1
            },
            "instanceGuid": "22222222-2222-2222-2222-222222222222",
            "itemCount": 2
          }
      ]
    }
  ]
}

[ghost]

Michael: did you capture any other changes than these? Per discussion: 1. Remove the “run.” prefix from the property names in an externalProperties object (section 4.3.6 of the spec). 2. Use JPointer in the SARIF protocol (it’s designed to work with URIs). 3. Accept Jim’s proposal with these renames: { ... "inlineExternalProperties": [ { "$schema": "http:///json.schemastore.org/sarif-external-property-file-2.0.0 <http://json.schemastore.org/sarif-external-property-file-2.0.0> ", "version": "2.0.0", "instanceGuid": "22222222-2222-2222-2222-222222222222", "run.files": { "apple.png": { "mimeType": "image/png" }, "banana.png": { "mimeType": "image/png" } } } ], "runs": [ { "originalUriBaseIds": { "INLINE": { "uri": "sarif:///inlineExternalProperties/" } }, "externalPropertyLocators": { "files": [ { "fileLocation": { "baseUriId": "INLINE", "uri": "0", "fileIndex": 1 }, "instanceGuid": "22222222-2222-2222-2222-222222222222", "itemCount": 2 } ] } ] } From: kupsch <[email protected]> Sent: Friday, February 22, 2019 7:23 AM To: oasis-tcs/sarif-spec <[email protected]> Cc: Larry Golding <[email protected]>; Mention <[email protected]> Subject: Re: [oasis-tcs/sarif-spec] Consider moving externalFiles to log object (#321) Below is an example SARIF file, for my proposal to inline external data that can be shared between runs. Add a new top level property whose value is an array of externalPropertyFile objects. The sarif URI protocol is used to reference the location in the array. Each run is unchanged except for the fileLocation of the external property, and the contents of the external file is unchanged. There is no coupling of runs using this scheme, and file location in an external property file is always an externalPropertyFile object. A further step would be to not require the externalPropertyFiles fileLocation in a run to change its fileLocation, but to inform consumers that if an inlineExternalFiles property exists with a matching instanceGuid then this is the externalPropertyFile object; this makes inline and externalizing simple. { ... "inlineExternalFiles": [ { "$schema": "http:///json.schemastore.org/sarif-external-property-file-2.0.0 <http://json.schemastore.org/sarif-external-property-file-2.0.0> ", "version": "2.0.0", "instanceGuid": "22222222-2222-2222-2222-222222222222", "run.files": { "apple.png": { "mimeType": "image/png" }, "banana.png": { "mimeType": "image/png" } } } ], "runs": [ { "originalUriBaseIds": { "INLINE": { "uri": "sarif:///inlineExternalPropertyFiles/" } }, "externalPropertyFiles": { "files": [ { "fileLocation": { "baseUriId": "INLINE", "uri": "0", "fileIndex": 1 }, "instanceGuid": "22222222-2222-2222-2222-222222222222", "itemCount": 2 } ] } ] } — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#321 (comment)> , or mute the thread <https://github.com/notifications/unsubscribe-auth/ALN73XWfCMsGH8HHfID6Pdbkspeh85bdks5vQAtBgaJpZM4aZGsq> . <https://github.com/notifications/beacon/ALN73Xn-gt49OIJGMc2avcCOhchHlwtNks5vQAtBgaJpZM4aZGsq.gif>

[michaelcfanning]

EBALLOT PROPOSAL

Provide a mechanism for inlining externalized property data at the log level. This facility allows for shared data between runs that is itself persisted to the log (e.g., common taxonomy metadata will be useful to store in this way).

API IMPACT

SDK PR: microsoft/sarif-sdk#1319
Add sarifLog.inlineExternalPropertyFiles, an array of externalPropertyFile objects (already defined in the schema)

NOTES

The run-level property run.externalPropertyFiles can specify the location of an inline external property file in two ways.

1. A uri property under run.externalPropertyFiles (for example, externalPropertyFiles/conversion/fileLocation/uri) can specify the sarif scheme and a JSON pointer (e.g., sarif://inlineExternalPropertyFiles/6)
2. An instanceGuid property under run.externalPropertyFiles (for example, externalPropertyFiles/conversion/instanceGuid) can match the instanceGuid property of one of the externalPropertyFile objects in sarifLog.inlineExternalPropertyFiles.

POST-BALLOT NOTES:

• Erratum: The name of the new property should be sarifLog.inlineExternalProperties, not inlineExternalPropertyFiles, and it is an array of externalProperties objects (as defined in §4.3 of the spec), not externalPropertyFile objects.

  If you look at all the examples above, you'll see that they really are arrays of externalProperties objects. @kupsch's first example proposed to name the new property inlineExternalFiles, which would be ok. Later in the thread, I proposed inlineExternalProperties, as suggested here, which I prefer because it matches the name of the objects in the array.

  "inlineExternalPropertyFiles", as we proposed in the ballot, is just wrong. There is an object in the spec called externalPropertyFile, but it is just a "locator" for an external file. Our new sarifLog property contains the contents of "external" files, not locators for them.

[kupsch]

In 3.10.3, the sentence describing the sarif sheme ("Such a URI..."), would be more specific as:

Such a URI uses the sarif scheme. The sarif URI scheme consists of only a scheme (with the value sarif) and a path component. The path component is interpreted as a JSON pointer [RFC6901] into the SARIF document containing the URI. The authority, query and fragment URI components are forbidden.

[kupsch]

The sarif scheme should be registered with IANA. If not on someone list of task already.

[ghost]

@kupsch @michaelcfanning

Jim, I added your proposed text change to the punch list in #342. I opened #343, "Register the sarif scheme with IANA", labled tracking-only (although if we get it registered before we become a standard, I'll mention it in the spec).

[ghost]

Approved in e-ballot #2.