add dynamic scope

[yelhamer]

Summary:

This scope will be part of the effort to add dynamic analysis capabilities to capa (project).

Place in the scope hierarchy:

The way I suggest the scope to be integrated in the current scope hierarchy is for it to be a parallel to the function scope, in the sense that its features would upstream into the file scope, but it wouldn't require any features from other scopes in order to do the dynamic scope matching.

This way, we will be able to extract only dynamic capabilities if requested by the user (in case they want to do dynamic analysis only), as well as making it simpler to integrate and maintain dynamic feature extractors into the existing capa code.

In short:
FILE ⊂ FUNCTION ⊂ BASIC BLOCK ⊂ INSTRUCTION
FILE ⊂ DYNAMIC

I am unsure however if the function scope should be a subset of the dynamic one. I think it shouldn't, but I think it would be better to make that decision when the to-be-added dynamic-features' list becomes clearer.

Syntax:

The proposed hierarchy placement would lead to a syntax that's something like the following:

rule:
  meta:
    ...
    scope: file
    ...
  features:
    - and:
      - dynamic:
        - or:
          - network-traffic: run-command
          - network-traffic: download-file
          ...
      - function:
        - or:
          - match: connect TCP socket
          ...

This would allow users to combine dynamic and code features to construct rules, as well as write purely dynamic or purely static capability rules.

Implementation:

in main.py

There are several modifications to be made in main.py (such as updating the argument parsing to give users the ability to run dynamic analysis only), however, I will try to highlight only the main parts necessary to the scope's implementation in this section.

First, we should add a find_dynamic_capabilities() function in main.py, this function would get passed the dynamic extractor, and is responsible for extracting solely dynamic features and returning them to the find_capabilities() function.

Second, we should update the the find_capabilities() function as to run both functions: find_code_capabilities() and find_dynamic_capabilities(), and then pass a set of features that includes function_and_lower features as well as dynamic features to the find_file_capabilities() function.

rule creation

The statement builder function should be modified as to account for the new scope, namely, it should implement checking to make sure that the passed-on rule does indeed follow the previously described hierarchy.

As for the RuleSet class, the RuleSet._index_rules_by_features.rec() should be updated as dynamic features get added to the dynamic scope. Namely, features that require network traffic parsing (for example) should probably be considered as hard features.

feature extraction

I suggest adding a parent DynamicExtractor class that is a child of the FeatureExtractor class, and that all future feature extractors (CAPE, VMRay, etc.) should implement; then, modify the get_extractor() function to return a list of FeatureExtractors rather than just one extractor, and implement logic in the find_capabilities() function (that's based on isinstance() checking) to pass the appropriate extractor to the corresponding function. For example, the find_capabilities() function would check if an extractor is an instance of the DynamicExtractor class, if so, it passes that extractor to the the find_dynamic_capabilities() function; else, it passes it to the find_code_capabilitites() function.

The good thing about this approach is that it would make it easier to adjust the code in the future to account for several dynamic extractors, in the case that a user might want to run their sample against several sandboxes or several instances of the same sandbox (different CAPE clusters for example).

[mr-tz]

There are many great points in here! Regarding the rule syntax:

rule:
  meta:
    ...
    scope: file
    ...
  features:
    - and:
      - dynamic:
          ...

How about scope: dynamic? The current suggestion feels like it would mask the true scope of a rule. Like I want to write a dynamic rule, but specify file instead.

Have you envisioned how to run capa here? This sounds like we want the binary and dynamic analysis results as input, such as capa.exe malware.exe_ sandbox-results.json or similar.

Potentially, the dynamic scope could entail all other scopes? Although, I'm not sure we really want to combine static and dynamic features in a rule.

Another thing to consider, it would be great if we could use existing rules on dynamic analysis without (much or any) modifications.

[mr-tz]

Another thing to consider, it would be great if we could use existing rules on dynamic analysis without (much or any) modifications.

And that leads to your proposal, so scratch most of my prior comment :) Leaving it here for documentation though.

What you suggest is great!

[williballenthin]

lets discuss the various designs here: #1517 before returning here with more concrete plans.

[mr-tz]

completed as part of v7