Merge branch 'main' into MCP39F511N

* main: (103 commits) Removing some noise debug BCC typos Bump actions/checkout from 3 to 4 (#450) Bump orjson from 3.9.7 to 3.9.9 (#506) Bump fastapi from 0.103.2 to 0.104.0 (#511) Added BCC functionality for email sending (#510) Refactors the system checkers (#507) Bump psutil from 5.9.5 to 5.9.6 (#508) FIXME disabled in Pylint Added simple HTML5 validation to request.html Network key is optional for HOG Clarified error message and clarified actual exception caught extends the install script to have a random password option (#504) Jobs.py now sends email when old jobs are present; Bugfixes in SQL query Flow process runtime bumped to 60+ minutes Sorting must be additionaly done by table to show indicator Repositories generated table also shall show sorting indicators Removing legacy datatable includes for energy-timeline Removed 'sortable' class where we are using datatables and not jQuery-Sort Bugifx: If Jobs object is not set do not access ...
green-coding-solutions · Oct 20, 2023 · a0e71e4 · a0e71e4
2 parents eebaa66 + 94a9df7
commit a0e71e4
Show file tree

Hide file tree

Showing 188 changed files with 5,360 additions and 2,461 deletions.
diff --git a/.github/actions/gmt-pytest/action.yml b/.github/actions/gmt-pytest/action.yml
@@ -9,10 +9,6 @@ inputs:
     description: 'The root directory of the gmt repository'
     required: false
     default: '.'
-  tests-directory:
-    description: 'The directory where to run the tests from'
-    required: false
-    default: './test'
   tests-command:
     description: 'The command to run the tests'
     required: false
@@ -24,26 +20,33 @@ inputs:
 runs:
   using: 'composite'
   steps:
-    - name: setup python
+    - name: setup_python
       uses: actions/setup-python@v4
       with:
         python-version: '3.10'
-        cache: 'pip'
-
-    - name: pip install
-      working-directory: ${{ inputs.gmt-directory }}
+
+    - id: python_cache
+      uses: actions/cache@v3
+      with:
+        path: venv
+        key: pip-${{ steps.setup_python.outputs.python-version }}-${{ hashFiles('requirements.txt') }}-${{ hashFiles('requirements-dev.txt') }}-${{ hashFiles('metric_providers/psu/energy/ac/xgboost/machine/model/requirements.txt') }}
+
+    - name: install script and packages
       shell: bash
+      working-directory: ${{ inputs.gmt-directory }}
       run: |
-        pip install -r requirements-dev.txt
-        pip install -r metric_providers/psu/energy/ac/xgboost/machine/model/requirements.txt
-
-    - name: Run Install / Setup scripts
+        ./install_linux.sh -p testpw -a http://api.green-coding.internal:9142 -m http://metrics.green-coding.internal:9142 -n -t
+        source venv/bin/activate
+        python3 -m pip install -r requirements-dev.txt
+        python3 -m pip install -r metric_providers/psu/energy/ac/xgboost/machine/model/requirements.txt
+      
+    - name: disable unneeded metric providers and run test setup script
       shell: bash
       working-directory: ${{ inputs.gmt-directory }}
       run: |
-        ./install_linux.sh -p testpw -a http://api.green-coding.internal:9142 -m http://metrics.green-coding.internal:9142 -n -t -w
+        source venv/bin/activate
         python3 disable_metric_providers.py ${{ inputs.metrics-to-turn-off }}
-        cd test && python3 setup-test-env.py --no-docker-build
+        cd tests && python3 setup-test-env.py --no-docker-build
 
     - name: Set up Docker Buildx
       id: buildx
@@ -63,30 +66,33 @@ runs:
 
     - name: Start Test container
       shell: bash
-      working-directory: ${{ inputs.gmt-directory }}/test
+      working-directory: ${{ inputs.gmt-directory }}/tests
       run: |
-        ./start-test-containers.sh -d
+        source ../venv/bin/activate && ./start-test-containers.sh -d
    
     - name: Sleep for 10 seconds
       run: sleep 10s
       shell: bash
-
+
+    # - name: Setup upterm session
+    #   uses: lhotari/action-upterm@v1
+
     - name: Run Tests
       shell: bash
-      working-directory: ${{ inputs.tests-directory }}
+      working-directory: ${{ inputs.gmt-directory }}/tests
       run: |
-        ${{ inputs.tests-command }} -rA | tee /tmp/test-results.txt
+        source ../venv/bin/activate
+        python3 -m ${{ inputs.tests-command }} -rA | tee /tmp/test-results.txt
 
     - name: Display Results
       shell: bash
       if: always()
-      working-directory: ${{ inputs.tests-directory }}
       run: |
         cat /tmp/test-results.txt | grep -oPz '(=*) short test summary(.*\n)*' >> $GITHUB_STEP_SUMMARY
 
     - name: Stop Containers
       shell: bash
       if: always()
-      working-directory: ${{ inputs.gmt-directory }}/test
+      working-directory: ${{ inputs.gmt-directory }}/tests
       run: |
         ./stop-test-containers.sh
diff --git a/.github/dependabot.yml b/.github/dependabot.yml
@@ -11,7 +11,12 @@ updates:
     schedule:
       interval: "daily"
   - package-ecosystem: "docker"
-    directory: "/"
+    directory: "/docker/"
+    target-branch: "main"
+    schedule:
+      interval: "weekly"  
+  - package-ecosystem: "docker"
+    directory: "/docker/auxiliary-containers/gcb_playwright/"
     target-branch: "main"
     schedule:
       interval: "weekly"

diff --git a/.github/workflows/build-and-push-containers.yml b/.github/workflows/build-and-push-containers.yml
@@ -0,0 +1,36 @@
+name: Build and Push Containers
+on:
+  pull_request:
+    types:
+      - closed
+    paths:
+      - 'docker/auxiliary-containers/**/Dockerfile'
+
+  workflow_dispatch:
+
+jobs:
+  build-and-push-containers:
+    if: github.event.pull_request.merged == true
+    runs-on: ubuntu-latest
+    permissions:
+      packages: write
+      contents: read
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+        ## This is needed for multi-architecture builds
+      - name: Set up QEMU
+        uses: docker/setup-qemu-action@v3
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v3
+
+      - name: Login to Docker Registry
+        uses: docker/login-action@v3
+        with:
+          username: ${{ secrets.DOCKERHUB_USERNAME }}
+          password: ${{ secrets.DOCKERHUB_TOKEN }}
+
+      - name: Build and Push Containers
+        run: bash ./docker/auxiliary-containers/build-containers.sh
diff --git a/.github/workflows/tests-bare-metal-main.yml b/.github/workflows/tests-bare-metal-main.yml
@@ -22,7 +22,7 @@ jobs:
 
       # - if: ${{ github.event_name == 'workflow_dispatch' || steps.check-date.outputs.should_run == 'true'}}
       - name: 'Checkout repository'
-        uses: actions/checkout@v3
+        uses: actions/checkout@v4
         with:
           ref: 'main'
           submodules: 'true'

diff --git a/.github/workflows/tests-eco-ci-energy-estimation.yaml b/.github/workflows/tests-eco-ci-energy-estimation.yaml
@@ -11,7 +11,7 @@ jobs:
       contents: read
     steps:
       - name: 'Checkout repository'
-        uses: actions/checkout@v3
+        uses: actions/checkout@v4
         with:
           ref: 'main'
           submodules: 'true'
@@ -30,7 +30,7 @@ jobs:
         name: 'Setup, Run, and Teardown Tests'
         uses: ./.github/actions/gmt-pytest
         with:
-         metrics-to-turn-off: '--categories RAPL Machine Sensors Debug CGroupV2 MacOS --providers CpuFrequencySysfsCoreProvider'
+         metrics-to-turn-off: '--categories RAPL Machine Sensors Debug CGroupV2 MacOS'
          github-token: ${{ secrets.GITHUB_TOKEN }}
 
       - name: Eco CI Energy Estimation - Get Measurement

diff --git a/.github/workflows/tests-vm-main.yml b/.github/workflows/tests-vm-main.yml
@@ -20,31 +20,31 @@ jobs:
 
       - if: ${{ github.event_name == 'workflow_dispatch' || steps.check-date.outputs.should_run == 'true'}}
         name: 'Checkout repository'
-        uses: actions/checkout@v3
+        uses: actions/checkout@v4
         with:
           ref: 'main'
           submodules: 'true'
 
       - name: Eco CI Energy Estimation - Initialize
-        uses: green-coding-berlin/eco-ci-energy-estimation@701b5f2f4ba601be587823cd0786f07cb6ae2ee6
+        uses: green-coding-berlin/eco-ci-energy-estimation@v2
         with:
           task: start-measurement
 
       - if: ${{ github.event_name == 'workflow_dispatch' || steps.check-date.outputs.should_run == 'true'}}
         name: 'Setup, Run, and Teardown Tests'
         uses: ./.github/actions/gmt-pytest
         with:
-         metrics-to-turn-off: '--categories RAPL Machine Sensors Debug CGroupV2 MacOS --providers CpuFrequencySysfsCoreProvider'
+         metrics-to-turn-off: '--categories RAPL Machine Sensors Debug CGroupV2 MacOS'
          github-token: ${{ secrets.GITHUB_TOKEN }}
 
       - name: Eco CI Energy Estimation - Get Measurement
-        uses: green-coding-berlin/eco-ci-energy-estimation@701b5f2f4ba601be587823cd0786f07cb6ae2ee6
+        uses: green-coding-berlin/eco-ci-energy-estimation@v2
         with:
           task: get-measurement
           branch: main
 
       - name: Eco CI Energy Estimation - End Measurement
-        uses: green-coding-berlin/eco-ci-energy-estimation@701b5f2f4ba601be587823cd0786f07cb6ae2ee6
+        uses: green-coding-berlin/eco-ci-energy-estimation@v2
         with:
           task: display-results
           branch: main
diff --git a/.github/workflows/tests-vm-pr.yml b/.github/workflows/tests-vm-pr.yml
@@ -12,7 +12,7 @@ jobs:
       pull-requests: write
     steps:
       - name: 'Checkout repository'
-        uses: actions/checkout@v3
+        uses: actions/checkout@v4
         with:
           ref: ${{ github.event.pull_request.head.ref }}
           submodules: 'true'
@@ -25,7 +25,7 @@ jobs:
       - name: 'Setup, Run, and Teardown Tests'
         uses: ./.github/actions/gmt-pytest
         with:
-         metrics-to-turn-off: '--categories RAPL Machine Sensors Debug CGroupV2 MacOS --providers CpuFrequencySysfsCoreProvider'
+         metrics-to-turn-off: '--categories RAPL Machine Sensors Debug CGroupV2 MacOS'
          github-token: ${{ secrets.GITHUB_TOKEN }}
 
       - name: Eco CI Energy Estimation - Get Measurement

diff --git a/.gitignore b/.gitignore
@@ -14,6 +14,6 @@ static-binary
 .pytest_cache
 test-compose.yml
 test-config.yml
-test/structure.sql
+tests/structure.sql
 tools/sgx_enable
 venv/
diff --git a/.pylintrc b/.pylintrc
@@ -25,9 +25,15 @@ disable=missing-function-docstring,
     too-many-branches,
     too-many-statements,
     too-many-arguments,
+    too-many-return-statements,
+    too-many-instance-attributes,
+    invalid-name,
+    wrong-import-position,
+    wrong-import-order,
+    ungrouped-imports,
+    fixme
 
 
-#    import-error
 
 [MASTER]
 ignore=env

diff --git a/README.md b/README.md
@@ -7,6 +7,8 @@
 
 The Green Metrics Tool is a developer tool is indented for measuring the energy consumption of software and doing life-cycle-analysis.
 
+One of it's key features is the detailed statistical dashboard and the measurement cluster for making reproducible measurements.
+
 It is designed to re-use existing infrastructure and testing files as much as possible to be easily integrateable into every software repository and create transparency around software energy consumption.
 
 It can orchestrate Docker containers according to a given specificaion in a `usage_scenario.yml` file.
@@ -27,15 +29,18 @@ as well as a web interface to view the measured metrics in some nice charts.
 # Frontend
 To see the frontend in action and get an idea of what kind of metrics the tool can collect and display go to out [Green Metrics Frontend](https://metrics.green-coding.berlin)
 
-
 # Documentation
 
 To see the the documentation and how to install and use the tool please go to [Green Metrics Tool Documentation](https://docs.green-coding.berlin)
 
-# Screenshots
+# Screenshots of Single Run View
 
-![Web Flow Demo with CPU measurement provider](https://www.green-coding.berlin/img/projects/gmt-screenshot-1.webp "Web Charts demo with docker stats provider instead of energy")
-> Web Flow Demo with CPU measurement provider
+![](https://www.green-coding.berlin/img/projects/gmt-screenshot-1.webp)
+![](https://www.green-coding.berlin/img/projects/gmt-screenshot-2.webp)
+![](https://www.green-coding.berlin/img/projects/gmt-screenshot-3.webp)
+![](https://www.green-coding.berlin/img/projects/gmt-screenshot-4.webp)
 
-![Web Flow Demo with energy measurement provider](https://www.green-coding.berlin/img/projects/gmt-screenshot-2.webp "Web Charts demo with docker stats provider instead of energy")
-> Web Flow Demo with energy measurement provider
+
+# Screenshots of Comparison View
+![](https://www.green-coding.berlin/img/projects/gmt-screenshot-5.webp)
+![](https://www.green-coding.berlin/img/projects/gmt-screenshot-6.webp)