Set GitHub Actions with test cases #46

jzelenjak · 2023-09-19T05:52:40Z

Closes #38

test

Add sudo to apt in the workflow

Add GitHub token to the workflow

Add gzipped alerts

Add extracting alerts to workflow

…r-sage

jzelenjak · 2023-09-21T13:05:25Z

.github/workflows/test.yml

+
+on:
+  pull_request:
+    branches-ignore:


Should ignore PR to the docker branch

jzelenjak · 2023-09-21T13:10:32Z

.github/workflows/test.yml

+          FF_REPO: "https://github.com/tudelft-cda-lab/FlexFringe"
+        shell: bash
+        run: |
+          gh release download latest -R $FF_REPO -p "flexfringe-x64-linux"  # Might have to be updated if FlexFringe decides to change the binary name


I have added comments like this to indicate the dependency on FlexFringe (also present in other parts of this file)

jzelenjak · 2023-09-21T13:15:03Z

.github/workflows/test.yml

+      - name: Run regression tests on CPTC-2017
+        env:
+          CHANGES_IDS: ${{ contains(github.event.pull_request.labels.*.name, 'changes-ids') }}  # With 'changes-ids' label state IDs will be removed from the AGs
+        if: '!contains(github.event.pull_request.labels.*.name, ''changes-ags'')'  # With 'changes-ags' label no regression tests will be run


So there are three cases:

Label changes-ags is present ⇒ don't run the regression tests

Label changes-ids is present ⇒ run with -i (removes IDs and skips stats-sinks-ags.sh, since the states cannot be matched due to different IDs)

Else, run normally

Something like this (tested on my fork):

jzelenjak · 2023-09-21T13:16:07Z

CONTRIBUTING.md

+
+Follow the same procedure as above, however **do add a label `changes-ags`**. This will skip the regression tests and only run the sink tests and the Python tests. Since there is no ground truth for the attack graphs, make sure that the changes to the attack graphs make sense. Please carefully describe them in the Pull Request description.
+
+### Changes to the IDs


The part related to the label changes-ids

jzelenjak · 2023-09-21T13:21:09Z

CONTRIBUTING.md

+
+If you want to add new test cases, feel free to do so. For Python tests, you can add them to the `tests.py` file. In addition, you can add the new tests to the GitHub Actions by modifying the `.github/workflows/test.yml` file. The tests, however, need first be approved, see the procedure above.
+
+### Updating the docker branch


Reminded about the existence of the docker branch

jzelenjak · 2023-09-21T13:22:16Z

README.md

+- Clone [FlexFringe repository](https://github.com/tudelft-cda-lab/FlexFringe).
+- Move `spdfa-config.ini` file to `FlexFringe/ini/` directory. Alternatively, you can set the `path_to_ini` variable in `sage.py` to `"./spdfa-config.ini"`.
+- In case you move the `FlexFringe/` directory to another location, update the function `flexfringe` in `model_learning.py` accordingly.
+- You can find the compressed alerts for the [Collegiate Penetration Testing Competition (CPTC)](https://cp.tc/research) and [Collegiate Cyber Defense Competition (CCDC)](https://github.com/FrankHassanabad/suricata-sample-data) datasets (taken from the linked sources) in the `alerts/` directory. To uncompress the alerts, run:


Referencing the alerts + command that uncompresses them. Does it look fine to you?

jzelenjak · 2023-09-21T13:27:25Z

test-scripts/test-ags.sh

+echo "------------"
+
+
+echo "Test 6: Checking sinks stats"


I have decided to skip this test when running with a -i option (i.e. removing state IDs). It is not possible to join the states in two attack graphs if the IDs are different (the script will behave as if they have been merged, as was the case when merging sinks)

jzelenjak · 2023-09-21T13:28:43Z

tests.py

+    total_tests += 1
+
+
+# TODO: has to be written


As you have once mentioned, I have left TODOs for the future

jzelenjak · 2023-09-21T13:32:21Z

tests.py

+from signatures.attack_stages import MicroAttackStage
+
+
+def run_episode_test(frequencies, expected, test_name="test"):


MicroAttackStage.DATA_EXFILTRATION and 'alert' are just placeholders. They are not used, but are needed for the code to run

jzelenjak · 2023-09-21T13:45:53Z

tests.py

+passed_tests = 0
+total_tests = 0
+
+# Test case 1: normal sequence


Do these test cases make sense to you?

jzelenjak and others added 30 commits September 9, 2023 15:01

Add method-level documentation for SAGE

8cb0756

Add test files for SAGE

879cc80

Merge pull request #1 from tudelft-cda-lab/38-test-cases-for-sage

d0655af

test

Add sudo to apt in the workflow

87e15cb

Merge pull request #3 from tudelft-cda-lab/38-test-cases-for-sage

8c4d313

Add sudo to apt in the workflow

Add GitHub token to the workflow

f470fa7

Merge pull request #4 from tudelft-cda-lab/38-test-cases-for-sage

c0fa41b

Add GitHub token to the workflow

Add gzipped alerts

7b63f39

Merge pull request #5 from tudelft-cda-lab/38-test-cases-for-sage

a82bebf

Add gzipped alerts

Add extracting alerts to workflow

1e9be55

Merge pull request #6 from tudelft-cda-lab/38-test-cases-for-sage

cdf968d

Add extracting alerts to workflow

Update test.yml

3812881

Update test.yml

3f682f0

Update test.yml

3087f7c

Update test.yml

1cbc84a

Update test.yml

2010b78

Update test.yml

f5f2ebc

Update test.yml

9868328

Update test.yml

09fc95f

Update test.yml

2a355db

Update test.yml

4c64bee

Update test.yml

ff1961f

Update test.yml

f6f4cea

Update test.yml

45b00d5

Update test.yml

7fc25f1

Update test.yml

c6e0370

Update test.yml

6964e6a

Update stats-ff.sh

f4b40dd

Update test-ags.sh

b3947b0

Update test.yml

2190853

jzelenjak added 9 commits September 19, 2023 17:37

Add CONTRIBUTING.md

e39de44

Merge branch '32-add-documentation-for-methods' into 38-test-cases-fo…

3cde828

…r-sage

Address some comments for the PR

2f05335

Add 'changes-ids' to the workflow

615b0e5

Add chmod back to the workflow file

368b0f7

Update test.yml

24684d6

Remove unnecessary command from test.yml

5aeac05

Fix missing alert windows at the end

b6831af

Update diff-ags.sh and test-ags.sh

4da578d

jzelenjak commented Sep 21, 2023

View reviewed changes

Mention the docker branch in CONTRIBUTING.md

30bd7ae

jzelenjak commented Sep 21, 2023

View reviewed changes

tests.py

total_tests += 1

# TODO: has to be written

Copy link

Collaborator Author

jzelenjak Sep 21, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As you have once mentioned, I have left TODOs for the future

jzelenjak commented Sep 21, 2023

View reviewed changes

jzelenjak added 2 commits September 21, 2023 15:33

Merge branch 'get-episodes-fix' into 38-test-cases-for-sage

4033751

Update tests.py

d6ce40c

jzelenjak commented Sep 21, 2023

View reviewed changes

tests.py

passed_tests = 0

total_tests = 0

# Test case 1: normal sequence

Copy link

Collaborator Author

jzelenjak Sep 21, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do these test cases make sense to you?

Remove blank lines

733ac86

jzelenjak added the changes-ags This PR changes the attack graphs (regression tests will be skipped) label Sep 21, 2023

Remove printing in ag_generation.py

93b3ea7

jzelenjak marked this pull request as ready for review September 21, 2023 14:32

jzelenjak requested a review from azqanadeem September 21, 2023 14:32

azqanadeem approved these changes Oct 2, 2023

View reviewed changes

azqanadeem merged commit 0325a81 into main Oct 2, 2023
1 check passed

jzelenjak deleted the 38-test-cases-for-sage branch October 7, 2023 12:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Set GitHub Actions with test cases #46

Set GitHub Actions with test cases #46

jzelenjak commented Sep 19, 2023

jzelenjak Sep 21, 2023 •

edited

Loading

jzelenjak Sep 21, 2023

jzelenjak Sep 21, 2023

jzelenjak Sep 21, 2023

jzelenjak Sep 21, 2023

jzelenjak Sep 21, 2023

jzelenjak Sep 21, 2023

jzelenjak Sep 21, 2023

jzelenjak Sep 21, 2023 •

edited

Loading

jzelenjak Sep 21, 2023


		Follow the same procedure as above, however do add a label `changes-ags`. This will skip the regression tests and only run the sink tests and the Python tests. Since there is no ground truth for the attack graphs, make sure that the changes to the attack graphs make sense. Please carefully describe them in the Pull Request description.

		### Changes to the IDs


		If you want to add new test cases, feel free to do so. For Python tests, you can add them to the `tests.py` file. In addition, you can add the new tests to the GitHub Actions by modifying the `.github/workflows/test.yml` file. The tests, however, need first be approved, see the procedure above.

		### Updating the docker branch

		from signatures.attack_stages import MicroAttackStage


		def run_episode_test(frequencies, expected, test_name="test"):

Set GitHub Actions with test cases #46

Set GitHub Actions with test cases #46

Conversation

jzelenjak commented Sep 19, 2023

jzelenjak Sep 21, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jzelenjak Sep 21, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jzelenjak Sep 21, 2023 •

edited

Loading

jzelenjak Sep 21, 2023 •

edited

Loading