Move audit logs to a dedicated logs directory

[watson]

Depends on #116282

The only new commit in this PR is the last one (13e27d6)

This PR adds a new folder to the root of the Kibana directory: logs.

The reason behind this change is to align more closely with where Elasticsearch stores logs. This was previously discussed here: #82578 (comment)

The new logs directory will currently only be used to store audit logs.

[azasypkin]

ACK: will review today

[azasypkin]

Everything looks good, but I'm wondering if we should commit /logs directory (empty dir with .gitignore inside or something like this)? Otherwise if I just try to enable audit logs locally Kibana crashes with Error: ENOENT: no such file or directory, open '.../kibana/logs/audit.log' forcing me to manually create logs folder.

[watson]

@elasticmachine merge upstream

[watson]

I'm wondering if we should commit /logs directory (empty dir with .gitignore inside or something like this)? Otherwise if I just try to enable audit logs locally Kibana crashes with Error: ENOENT: no such file or directory, open '.../kibana/logs/audit.log' forcing me to manually create logs folder.

We used to have an empty data directory checked into git as well, but as far as I understand we went away from that approach to instead create when accessing it if it wasn't there. I guess this is more stable in case a user manually deletes it. @spalger can probably weigh in on the original thoughts around this.

If that's the case, I think it would make sense to change the audit code to create it if it doesn't exist already as well. I actually thought it worked like that all ready since the tests didn't fail. But I guess the tests doesn't fully test that. So I will update this PR to improve that if @spalger agrees this is the way to go.

[watson]

The default settings uses a rolling log-file which in turn use this code to write the file to disk:

kibana/src/core/server/logging/appenders/rolling_file/rolling_file_manager.ts

Lines 43 to 46 in e8eb222

	this.outputStream = createWriteStream(this.filePath, {
	encoding: 'utf8',
	flags: 'a',
	});

So we can either update this code to create any missing directories first, though that would impact all rolling file appenders - which might be a nice thing, but is a bigger discussion.

Alternatively, we can update the audit setup code to create the dir, which seems like a smaller change.

[spalger]

We used to have an empty data directory checked into git as well, but as far as I understand we went away from that approach to instead create when accessing it if it wasn't there. I guess this is more stable in case a user manually deletes it. @spalger can probably weigh in on the original thoughts around this.

If I recall correctly we ran into a few issues with the .keep files not integrating with other processes... that said I don't exactly remember why we got rid of them. Maybe @tylersmalley knows.

So we can either update this code to create any missing directories first, though that would impact all rolling file appenders - which might be a nice thing, but is a bigger discussion.

Can you think of a reason this would be undesirable? I think we should do it across the board and include the appropriate release note.

[watson]

I've created a PR to show how this could be done: #117666

If we decide to go down this path, I think we want to land it as a separate PR and then wait for that to land before continuing with this one. So let's move the related to discussion to that PR.

[tylersmalley]

I don't know if we purposefully got rid of the data directory. Looks like it might have been accidentally removed in 7bac741. I am not opposed to trying it again if it simplifies things.

[watson]

While a little more complex, I think it's more stable if Kibana could bootstrap its required empty directories on boot. This makes it easier in case you on purpose or accidentally delete them while cleaning up. As far as I can tell the two directories in question here are data and the new logs. However, I don't want this to be a blocker for this PR, so I'll go ahead and add logs/.empty in this PR and then we can follow up with another PR (for instance #117666) where we make them "dynamic".

[cla-checker-service]

💚 CLA has been signed

[azasypkin]

ACK: will review today

[azasypkin]

LGTM, thanks!

[jbudz]

Can we add an '--exclude', usr/share/kibana/logs, a few lines up so we don't copy the .gitempty/folder over twice?

[watson]

Done 👍

[kibanamachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 0565f0f
• Storybooks Preview

Metrics [docs]

✅ unchanged

History

• 💚 Build #5355 succeeded 455ee51
• 💚 Build #4307 succeeded a3bae15
• 💚 Build #3554 succeeded bfc9049
• 💛 Build #2380 was flaky 13e27d6759b837e0eec27d53b045168038ed0158
• 💔 Build #2358 failed ba78c40de850cf90660f22fd77f3a56ae747ed0b

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @watson

[kibanamachine]

💚 Backport successful

Status	Branch	Result
✅	8.0

This backport PR will be merged automatically after passing CI.