Empty rolie

[JanHoefelmeyer]

Solves #336

Csaf checker explicitely filters out if the schema error was no entries and will give a warning for this error only.

Csaf aggregator creates "empty" ROLIE feeds if no entries are present.

Csaf Provider already has an option to create ROLIE feeds during setup. This is currently not mandatory, however, but would be possible to make mandatory if needed.

Csaf Downloader will find feeds with emptry entry, but will not download anything from them.

[tschmidtb51]

Please see my comments.

BTW: Shouldn't the change in the csaf_provider to generate the appropriated ROLIE feeds be part of this PR or do you want to have that separate?

[s-l-teichmann]

LGTM

[tschmidtb51]

1. The creation of empty ROLIE feeds when setting up a provider (using csaf_uploader -a create) is not implemented (but it would make the test much easier).
2. Using the aggregator with an empty ROLIE feed (e.g. create a provider, upload one document, delete all entries from the ROLIE feed) results in an error.

Happy to discuss the issues - maybe I misunderstood something

[mfd2007]

I can confirm that there is some problem with the aggregator. If the feed is empty, something doesn't work as expected:

csaf-feed-tlp-white.json.txt

2023/05/23 09:35:19 provider-metadata: https://acme.test/.well-known/csaf/provider-metadata.json
2023/05/23 09:35:19 AdvisoryFileProcessor.Process: Found 1 ROLIE feed(s).
....
2023/05/23 09:35:19 Feed URL: https://acme.test/.well-known/csaf/white/csaf-feed-tlp-white.json
....
2023/05/23 09:35:20 white: 0
....
2023/05/23 09:35:20 error: 'ACME' failed: open /data1/archive/ACME-2023-05-23-093519/white/changes.csv: no such file or directory

In that case the provider won't be listed in the aggregator.json. On the other hand, is this a real issue? Does an aggregator want to list an "empty" CSAF provider? Why should the aggregator do this?

According to the provider, I can't find the right option to create the ROLIE feed. For the different TLP-Level there won't be create any feeds.

[tschmidtb51]

Intended behavior:

• csaf_provider: create empty ROLIE feeds as configured during setup (csaf_uploader -a create)
• csaf_aggregator: create empty ROLIE feed where appropriate
• csaf_downloader: "ignore" empty ROLIE feeds
• csaf_checker: info (or warn?) on empty ROLIE feeds

[tschmidtb51]

I haven't located the bug in the aggregator why it is not creating a ROLIE feed with an empty feed array, but I'm happy to document that as it is consistent (doesn't show up in the aggregator.json either) - and leave that for now.

Documented in #377

[s-l-teichmann]

LGTM

[mfd2007]

LGTM

[tschmidtb51]

Please see my comments. Additional remarks:

1. I couldn't test the csaf_aggreagtorchange because List provider with empty feeds  #377. It fails as soon as you have a ROLIE feed with an empty entry array. See example error message: 'Example_Company_02' failed: open /var/www/aggregator-data/Example_Company_02-2023-06-23-222130/white/csaf-feed-tlp-white.json: no such file or directory.
2. There is still some confusion regarding the error message and its meaning in the csaf_checker.
3. The change in c495006 didn't fix the issue in the csaf_provider. The entry array is still empty when delivered.

[tschmidtb51]

@JanHoefelmeyer #357 (comment) still exists - the other two have been resolved with the last 3 commits.

There are also merge conflicts...

[tschmidtb51]

The merge conflict resolution seems to be gone wrong - now any domain requested fails. I haven't found the line number yet where the issue is. But basically, it reports

2023/06/30 20:43:10 Failed to find valid provider-metadata.json for domain domain.tld. Continuing with next domain.
{
  "version": "2.1.1-123-g30c2218",
  "date": "2023-06-30T18:43:02.567470825Z"
}

for any valid domain

[s-l-teichmann]

The merge conflict resolution seems to be gone wrong - now any domain requested fails. I haven't found the line number yet where the issue is. But basically, it reports

2023/06/30 20:43:10 Failed to find valid provider-metadata.json for domain domain.tld. Continuing with next domain.
{
  "version": "2.1.1-123-g30c2218",
  "date": "2023-06-30T18:43:02.567470825Z"
}

for any valid domain

@tschmidtb51 That was indeed a problem introduced by a faulty merge.
(Forgot to remove an else which should not be there)

I cleaned up the error handling in commit 89e69a7 .

[s-l-teichmann]

I was able to run ./csaf_checker -v -o bsi.json bsi.de successfully

[tschmidtb51]

LGTM and works

[tschmidtb51]

@s-l-teichmann Thank you for the fast fix!

[s-l-teichmann]

LGTM