Logging improvements (SARIF, console sections, clean logs at startup.…

…..) (#1614) * Fix SARIF * typo * Clear report folder Fixes #1502 * Lint fix * changelog * [MegaLinter] Apply linters fixes * Clear report folder during tests quick build * README * Remove CI paths from SARIF logs * Manage sections in GHA & gitlab CI logs * Fix GHA section logs * typo * typo * Reorder logs * fix * cspell + lint fixes * [MegaLinter] Apply linters fixes * Doc + json schema * cspell * [MegaLinter] Apply linters fixes * Fix CLEAR_REPORT_FOLDER * More logs * Fix test classes about output folder * [MegaLinter] Apply linters fixes * Fix test classes after renaming sections * Fix sarif count * Initialize correctly output_sarif , also using can_output_sarif Co-authored-by: nvuillam <[email protected]>
oxsecurity · Jul 18, 2022 · e934a14 · e934a14
1 parent 457ade4
commit e934a14
Show file tree

Hide file tree

Showing 18 changed files with 192 additions and 59 deletions.
diff --git a/.cspell.json b/.cspell.json
@@ -161,6 +161,7 @@
         "KXXXXNXXX",
         "KXXXXXXXXXXK",
         "Klingenberg",
+        "Ksection",
         "Ktlint",
         "Kubernetes",
         "Ku\u010dera",

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -8,6 +8,13 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 
 Note: Can be used with `megalinter/megalinter@beta` in your GitHub Action mega-linter.yml file, or with `megalinter/megalinter:beta` docker image
 
+- Improve console logs by using collapsible sections in GitHub Actions and Gitlab CI (disable by defining `CONSOLE_REPORTER_SECTIONS: false`)
+- Define `CLEAR_REPORT_FOLDER=true` to empty report folder at the beginning of each run ([#1502](https://github.com/oxsecurity/megalinter/issues/1502))
+- Improve SARIF output
+  - Replace CI paths in logs
+  - Add missing required properties so SARIF is [valid](https://sarifweb.azurewebsites.net/Validation)
+  - Add MegaLinter information in SARIF linter runs
+
 - Linter versions upgrades
   - [eslint](https://eslint.org) from 8.19.0 to **8.20.0** on 2022-07-17
   - [markdownlint](https://github.com/DavidAnson/markdownlint) from 0.31.1 to **0.32.0** on 2022-07-17

diff --git a/README.md b/README.md
@@ -33,6 +33,8 @@
 MegaLinter is an **Open-Source** tool for **CI/CD workflows** that analyzes the **consistency of your code**, **IAC**, **configuration**, and **scripts** in your repository sources, to **ensure all your projects sources are clean and formatted** whatever IDE/toolbox is used by their developers, powered by [**OX security**](https://www.ox.security/).
 
 Supporting [**48** languages](#languages), [**21** formats](#formats), [**20** tooling formats](#tooling-formats) and **ready to use out of the box**, as a GitHub action or any CI system **highly configurable** and **free for all uses**.
+
+[**Upgrade to MegaLinter v6 !**](https://github.com/oxsecurity/megalinter/issues/1592)
 <!-- welcome-phrase-end -->
 
 <!-- online-doc-start -->
@@ -658,6 +660,7 @@ Configuration is assisted with auto-completion and validation in most commonly u
 |------------------------------------------------------------|------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
 | **ADDITIONAL_EXCLUDED_DIRECTORIES**                        | \[\]                                     | List of additional excluded directory basenames. They are excluded at any nested level.                                                                                                                    |
 | [**APPLY_FIXES**](#apply-fixes)                            | `none`                                   | Activates formatting and auto-fixing [(more info)](#apply-fixes)                                                                                                                                           |
+| **CLEAR_REPORT_FOLDER**                                    | `false`                                  | Flag to clear files from report folder (usually megalinter-reports) before starting the linting process                                                                                                    |
 | **DEFAULT_BRANCH**                                         | `HEAD`                                   | Deprecated: The name of the repository's default branch.                                                                                                                                                   |
 | **DEFAULT_WORKSPACE**                                      | `/tmp/lint`                              | The location containing files to lint if you are running locally.                                                                                                                                          |
 | **DISABLE_ERRORS**                                         | `false`                                  | Flag to have the linter complete with exit code 0 even if errors were detected.                                                                                                                            |

diff --git a/docs/reporters/ConsoleReporter.md b/docs/reporters/ConsoleReporter.md
@@ -10,7 +10,7 @@ Open GitHub action (or other CI tool) logs and look in MegaLinter step
 
 ## Configuration
 
-| Variable         | Description                    | Default value |
-|------------------|--------------------------------|---------------|
-| CONSOLE_REPORTER | Activates/deactivates reporter | true          |
-
+| Variable                  | Description                                     | Default value |
+|---------------------------|-------------------------------------------------|---------------|
+| CONSOLE_REPORTER          | Activates/deactivates reporter                  | true          |
+| CONSOLE_REPORTER_SECTIONS | Activates/deactivates sections for console logs | true          |
diff --git a/megalinter/Linter.py b/megalinter/Linter.py
@@ -144,7 +144,9 @@ def __init__(self, params=None, linter_config=None):
 
         self.is_active = params["default_linter_activation"]
         self.output_sarif = (
-            params["output_sarif"] if "output_sarif" in params else self.output_sarif
+            params["output_sarif"]
+            if "output_sarif" in params and self.can_output_sarif is True
+            else self.output_sarif
         )
         self.disable_errors_if_less_than = None
         self.disable_errors = (
@@ -1007,8 +1009,10 @@ def get_total_number_errors(self, stdout: str):
             except Exception as e:
                 total_errors = 1
                 logging.error(
-                    "Error while getting total errors from SARIF output.\nSARIF:"
+                    "Error while getting total errors from SARIF output.\nError:"
                     + str(e)
+                    + "\nstdout: "
+                    + stdout
                 )
         # Get number with a single regex.
         elif self.cli_lint_errors_count == "regex_number":

diff --git a/megalinter/MegaLinter.py b/megalinter/MegaLinter.py
@@ -8,6 +8,7 @@
 import logging
 import multiprocessing as mp
 import os
+import shutil
 import sys
 
 import chalk as c
@@ -26,6 +27,7 @@
     DEFAULT_REPORT_FOLDER_NAME,
     ML_DOC_URL,
 )
+from megalinter.utils_reporter import log_section_end, log_section_start
 from multiprocessing_logging import install_mp_handler
 
 
@@ -118,6 +120,7 @@ def __init__(self, params=None):
         self.compute_file_extensions()
         # Load MegaLinter reporters
         self.load_reporters()
+        logging.info(log_section_end("megalinter-init"))
 
     # Collect files, run linters on them and write reports
     def run(self):
@@ -130,6 +133,12 @@ def run(self):
             return
 
         # Collect files for each identified linter
+        logging.info(
+            log_section_start(
+                "megalinter-file-listing",
+                "MegaLinter now collects the files to analyse",
+            )
+        )
         self.collect_files()
 
         # Process linters serial or parallel according to configuration
@@ -677,6 +686,13 @@ def initialize_output(self):
                 self.report_folder = self.arg_output
         # Initialize output dir
         os.makedirs(self.report_folder, exist_ok=True)
+        # Clear report folder if requested
+        if config.get("CLEAR_REPORT_FOLDER", "false") == "true":
+            logging.info(
+                f"CLEAR_REPORT_FOLDER found: empty folder {self.report_folder}"
+            )
+            shutil.rmtree(self.report_folder, ignore_errors=True)
+            os.makedirs(self.report_folder, exist_ok=True)
 
     def initialize_logger(self):
         logging_level_key = config.get("LOG_LEVEL", "INFO").upper()
@@ -728,7 +744,7 @@ def initialize_logger(self):
     def display_header():
         # Header prints
         logging.info(utils.format_hyphens(""))
-        logging.info(utils.format_hyphens("MegaLinter"))
+        logging.info(utils.format_hyphens("MegaLinter, by OX Security"))
         logging.info(utils.format_hyphens(""))
         logging.info(
             " - Image Creation Date: " + config.get("BUILD_DATE", "No docker image")
@@ -743,6 +759,7 @@ def display_header():
         logging.info("The MegaLinter documentation can be found at:")
         logging.info(" - " + ML_DOC_URL)
         logging.info(utils.format_hyphens(""))
+        logging.info(log_section_start("megalinter-init", "MegaLinter initialization"))
         if os.environ.get("GITHUB_REPOSITORY", "") != "":
             logging.info(
                 "GITHUB_REPOSITORY: " + os.environ.get("GITHUB_REPOSITORY", "")

diff --git a/megalinter/descriptors/schemas/megalinter-configuration.jsonschema.json b/megalinter/descriptors/schemas/megalinter-configuration.jsonschema.json
@@ -1409,6 +1409,13 @@
       "title": "Activate console reporter",
       "type": "boolean"
     },
+    "CONSOLE_REPORTER_SECTIONS": {
+      "$id": "#/properties/CONSOLE_REPORTER_SECTIONS",
+      "default": true,
+      "description": "Define to false if you do not want logs to be organized in sections",
+      "title": "Activate console logs sections",
+      "type": "boolean"
+    }, 
     "COPYPASTE_FILTER_REGEX_EXCLUDE": {
       "$id": "#/properties/COPYPASTE_FILTER_REGEX_EXCLUDE",
       "title": "Excluding regex filter for COPYPASTE descriptor",

diff --git a/megalinter/linters/DustilockLinter.py b/megalinter/linters/DustilockLinter.py
@@ -71,24 +71,25 @@ def manage_sarif_output(self, return_stdout):
                             "driver": {
                                 "informationUri": "https://github.com/Checkmarx/dustilock",
                                 "name": "dustilock",
+                                "version": self.get_linter_version(),
                                 "rules": [
                                     {
                                         "id": "PACKAGE_JSON_ERROR",
-                                        "name": "package_json_error",
+                                        "name": "Error in package.json",
                                         "shortDescription": {
                                             "text": "Dependency injection in package.json"
                                         },
                                     },
                                     {
                                         "id": "PYTHON_REQUIREMENT_ERROR",
-                                        "name": "python_requirement_error",
+                                        "name": "Error in Python requirements",
                                         "shortDescription": {
                                             "text": "Dependency injection in python requirements.txt"
                                         },
                                     },
                                     {
                                         "id": "OTHER_ERROR",
-                                        "name": "other_error",
+                                        "name": "Other errors",
                                         "shortDescription": {"text": "Other error"},
                                     },
                                 ],

diff --git a/megalinter/reporters/ConsoleLinterReporter.py b/megalinter/reporters/ConsoleLinterReporter.py
@@ -7,6 +7,7 @@
 import chalk as c
 from megalinter import Reporter, config, utils
 from megalinter.constants import ML_DOC_URL
+from megalinter.utils_reporter import log_section_end, log_section_start
 
 mega_linter_version = config.get("BUILD_VERSION", "latest")
 DOCS_URL_DESCRIPTORS_ROOT = f"{ML_DOC_URL}/{mega_linter_version}/descriptors"
@@ -44,10 +45,45 @@ def produce_report(self):
                 f"{DOCS_URL_DESCRIPTORS_ROOT}/{self.master.descriptor_id.lower()}_"
                 f"{self.master.linter_name.lower().replace('-', '_')}"
             )
+        # Output linter status
+        base_phrase = f"Linted [{self.master.descriptor_id}] files with [{self.master.linter_name}]"
+        elapse = str(round(self.master.elapsed_time_s, 2)) + "s"
+        total_errors = str(self.master.total_number_errors)
+        if self.master.return_code == 0 and self.master.status == "success":
+            logging.info(
+                log_section_start(
+                    f"processed-{self.master.name}",
+                    c.green(f"✅ {base_phrase} successfully - ({elapse})"),
+                )
+            )
+        elif self.master.return_code == 0 and self.master.status != "success":
+            logging.warning(
+                log_section_start(
+                    f"processed-{self.master.name}",
+                    c.yellow(
+                        f"✅ {base_phrase}: Found {total_errors} non blocking error(s) - ({elapse})"
+                    ),
+                )
+            )
+        elif self.master.return_code != 0 and self.master.status != "success":
+            logging.error(
+                log_section_start(
+                    f"processed-{self.master.name}",
+                    c.red(
+                        f"❌ {base_phrase}: Found {total_errors} error(s) - ({elapse})"
+                    ),
+                )
+            )
+        else:
+            logging.error(
+                log_section_start(
+                    f"processed-{self.master.name}",
+                    f"❌ There is a MegaLinter issue, please report it: {self.master.return_code}"
+                    + " / {self.master.status}",
+                )
+            )
         # Linter header prints
         msg = [
-            "",
-            c.bold(f"### Processed [{self.master.descriptor_id}] files"),
             f"- Using [{self.master.linter_name} v{linter_version}] {linter_doc_url}",
         ]
         if self.master.descriptor_id != self.master.name:
@@ -86,23 +122,5 @@ def produce_report(self):
                 logging.error(f"--Error detail:\n{self.master.stdout}")
             elif self.report_type == "detailed":
                 logging.info(f"--Log detail:\n{self.master.stdout}")
-        # Output linter status
-        base_phrase = f"Linted [{self.master.descriptor_id}] files with [{self.master.linter_name}]"
-        elapse = str(round(self.master.elapsed_time_s, 2)) + "s"
-        total_errors = str(self.master.total_number_errors)
-        if self.master.return_code == 0 and self.master.status == "success":
-            logging.info(c.green(f"✅ {base_phrase} successfully - ({elapse})"))
-        elif self.master.return_code == 0 and self.master.status != "success":
-            logging.warning(
-                c.yellow(
-                    f"✅ {base_phrase}: Found {total_errors} non blocking error(s) - ({elapse})"
-                )
-            )
-        elif self.master.return_code != 0 and self.master.status != "success":
-            logging.error(
-                c.red(f"❌ {base_phrase}: Found {total_errors} error(s) - ({elapse})")
-            )
-        else:
-            logging.error(
-                f"❌ There is a MegaLinter issue, please report it: {self.master.return_code} / {self.master.status}"
-            )
+        # Close section
+        logging.info(log_section_end(f"processed-{self.master.name}"))
diff --git a/megalinter/reporters/ConsoleReporter.py b/megalinter/reporters/ConsoleReporter.py
@@ -10,6 +10,7 @@
 import terminaltables
 from megalinter import Reporter, config
 from megalinter.constants import DEFAULT_RELEASE, ML_DOC_URL, ML_REPO, ML_REPO_URL
+from megalinter.utils_reporter import log_section_end
 
 
 class ConsoleReporter(Reporter):
@@ -55,7 +56,7 @@ def initialize(self):
         logging.info("")
         for table_line in table.table.splitlines():
             logging.info(table_line)
-        logging.info("")
+        logging.info(log_section_end("megalinter-file-listing"))
 
     def produce_report(self):
         table_header = ["Descriptor", "Linter", "Mode", "Files", "Fixed", "Errors"]

diff --git a/megalinter/reporters/SarifReporter.py b/megalinter/reporters/SarifReporter.py
@@ -8,13 +8,15 @@
 import random
 from json.decoder import JSONDecodeError
 
-from megalinter import Reporter, config
+from megalinter import Linter, Reporter, config
 from megalinter.constants import (
     DEFAULT_SARIF_REPORT_FILE_NAME,
     DEFAULT_SARIF_SCHEMA_URI,
     DEFAULT_SARIF_VERSION,
     ML_DOC_URL,
 )
+from megalinter.utils import normalize_log_string
+from megalinter.utils_reporter import get_linter_doc_url
 
 
 class SarifReporter(Reporter):
@@ -86,7 +88,7 @@ def produce_report(self):
                         logging.error(str(e))
                 if load_ok is True:
                     # fix sarif file
-                    linter_sarif_obj = self.fix_sarif(linter_sarif_obj)
+                    linter_sarif_obj = self.fix_sarif(linter_sarif_obj, linter)
                     # append to global megalinter sarif run
                     sarif_obj["runs"] += linter_sarif_obj["runs"]
                     # Delete linter SARIF file if LOG_FILE=none
@@ -95,6 +97,8 @@ def produce_report(self):
                     ):
                         os.remove(linter.sarif_output_file)
         result_json = json.dumps(sarif_obj, sort_keys=True, indent=4)
+        # Remove workspace prefix from file names
+        result_json = normalize_log_string(result_json)
         # Write output file
         sarif_file_name = f"{self.report_folder}{os.path.sep}" + config.get(
             "SARIF_REPORTER_FILE_NAME", DEFAULT_SARIF_REPORT_FILE_NAME
@@ -126,18 +130,44 @@ def filter_fields(self, obj, fields_to_keep):
 
     # Some SARIF linter output contain errors (like references to line 0)
     # We must correct them so SARIF is valid
-    def fix_sarif(self, linter_sarif_obj):
+    def fix_sarif(self, linter_sarif_obj, linter: Linter):
         # browse runs
         if "runs" in linter_sarif_obj:
             for id_run, run in enumerate(linter_sarif_obj["runs"]):
+
+                # Add MegaLinter info
+                run_properties = run["properties"] if "properties" in run else {}
+                run_properties["megalinter"] = {
+                    "linterKey": linter.name,
+                    "docUrl": get_linter_doc_url(linter),
+                    "linterVersion": linter.get_linter_version(),
+                }
+                run["properties"] = run_properties
+
+                # fix missing informationUri
+                if (
+                    "tool" in run
+                    and "driver" in run["tool"]
+                    and "informationUri" not in run["tool"]["driver"]
+                ):
+                    run["tool"]["driver"]["informationUri"] = get_linter_doc_url(linter)
+
+                # fix missing version
+                if (
+                    "tool" in run
+                    and "driver" in run["tool"]
+                    and "version" not in run["tool"]["driver"]
+                ):
+                    run["tool"]["driver"]["version"] = linter.get_linter_version()
+
                 # fix duplicate rules property
                 if (
                     "tool" in run
                     and "driver" in run["tool"]
                     and "rules" in run["tool"]["driver"]
                 ):
                     rules = run["tool"]["driver"]["rules"]
-                    rules_updated = []
+                    rules_updated: list = []
                     for rule in rules:
                         # If duplicate id, update duplicate items ids with a random value
                         if "id" in rule and any(
@@ -151,6 +181,7 @@ def fix_sarif(self, linter_sarif_obj):
                             )
                         rules_updated += [rule]
                     run["tool"]["driver"]["rules"] = rules_updated
+
                 # fix results property
                 if "results" in run:
                     # browse run results
@@ -166,6 +197,8 @@ def fix_sarif(self, linter_sarif_obj):
                                     )
                                 result["locations"][id_location] = location
                         run["results"][id_result] = result
+
+                # Update run in full list
                 linter_sarif_obj["runs"][id_run] = run
         return linter_sarif_obj
 

diff --git a/megalinter/reporters/TextReporter.py b/megalinter/reporters/TextReporter.py
@@ -94,6 +94,6 @@ def produce_report(self):
         with open(text_file_name, "w", encoding="utf-8") as text_file:
             text_file_content = "\n".join(text_report_lines) + "\n"
             text_file.write(text_file_content)
-            logging.info(
+            logging.debug(
                 f"[Text Reporter] Generated {self.name} report: {text_file_name}"
             )