Move umask from code to service files

[rdner]

Enhancement

What does this PR do?

Before this the umask value was hard-coded in libbeat. It caused
some confusion among the users since file permission configuration was
practically ignored by a beat on the level of the binary. It's been
decided that we move umask to the service files, so the distribution
is secured by default but it still allows the users to set the value
if they choose to.

Why is it important?

The issue started as a security concern here #14005

Then was addressed by adding a hard-coded umask for all files created by libbeat #14119

Later documented here #28347

We've got some feedback from our customers about confusion and lack of documentation. Examples can be found here #20584 (comment)

Checklist

• My code follows the style guidelines of this project
  - [ ] I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
  - [ ] I have made corresponding change to the default configuration files
  - [ ] I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

• systemd unit sets umask to 0027 for both beats and elastic agent
• launchd manifest contains umask property set to 0027 for both beats and elstic agent
• shell scripts that start both elastic agent and beats contain umask 0027 command

I made sure that all current permission defaults in beats are secure (set to 0600):

libbeat

• Output files

  • Config

    beats/libbeat/outputs/fileout/config.go

    Line 41 in cbdba8a

    Permissions: 0600,

  • Reference

    beats/libbeat/_meta/config/output-file.reference.yml.tmpl

    Line 33 in cbdba8a

    #permissions: 0600

  • Documentation https://github.com/elastic/beats/blob/cbdba8aa1645a896023e387b370b5b88960f3176/libbeat/outputs/fileout/docs/fileout.asciidoc#permissions

• Log files

  • Config

    beats/libbeat/logp/config.go

    Line 78 in cbdba8a

    Permissions: 0600,

  • Reference

    beats/libbeat/_meta/config/logging.reference.yml.tmpl

    Line 58 in cbdba8a

    #permissions: 0600

  • Documentation https://github.com/elastic/beats/blob/cbdba8aa1645a896023e387b370b5b88960f3176/libbeat/docs/loggingconfig.asciidoc#loggingfilespermissions

filebeat

• Registry
  • Config

    beats/filebeat/config/config.go

    Line 65 in cbdba8a

    Permissions: 0600,

  • Reference

    beats/filebeat/_meta/config/filebeat.global.reference.yml.tmpl

    Line 10 in cbdba8a

    #filebeat.registry.file_permissions: 0600

  • Documentation https://github.com/elastic/beats/blob/cbdba8aa1645a896023e387b370b5b88960f3176/filebeat/docs/filebeat-general-options.asciidoc#registryfile_permissions

How to test this PR locally

I configured filebeat to have a file output with 666 permissions.

For that one can use the following config section in the filebeat config file:

output.file:
   path: "/path/to/the/output/directory"
   filename: "output.log"
   permissions: 0666

Before this change this would results in a file with 640 permissions, after this change it's 644 which lets users to make files available to the world as they request.

After setting umask 027 prior to running the filebeat binary the output file is created with 640 as expected.

Related issues

• Closes [filebeat] default owner of output file  #25455

[mergify]

This pull request does not have a backport label. Could you fix it @rdner? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-01-14T10:00:22.605+0000

• Duration: 203 min 21 sec

• Commit: 52bfcc1

Test stats 🧪

Test	Results
Failed	0
Passed	47848
Skipped	4284
Total	52132

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[rdner]

/test

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b feature-move-umask-to-service-files upstream/feature-move-umask-to-service-files
git merge upstream/master
git push upstream feature-move-umask-to-service-files

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[rdner]

The failing e2e test does not seem to be related to the change:

[2022-01-14T14:21:40.576Z] ERRO[2022-01-14T14:21:40Z] Error executing command                       args="[--kubeconfig /tmp/test-3728211276/kubeconfig --namespace test-9e7ba835-424b-4811-aad9-4ef5808d7342 cp --no-preserve test-9e7ba835-424b-4811-aad9-4ef5808d7342/elastic-agent-kdb7r:/tmp/beats-events /tmp/test-1292283775/events]" baseDir=. command=kubectl env="map[]" error="exit status 1" stderr="error: unable to upgrade connection: container not found (\"elastic-agent\")\n"

So, I think it's safe to merge the PR.

[rdner]

Detailed testing steps:

1. Access a Linux machine
2. Go to the terminal and run umask 002
3. Confirm the current umask value with the command umask it should print 002
4. Create the following config file filebeat.yml for filebeat:

filebeat.inputs:
  - type: log
    paths:
      - "/home/user/input/input.log"

output.file:
  path: "/home/user/output_folder/"
  filename: "output_file.log"
  permissions: 0666

5. Make sure there is no file /home/user/input/input.log and /home/user/output_folder/ is empty.
6. Download the filebeat binary (artefact) to the current directory
7. Start filebeat with ./filebeat -e -c filebeat.yml in the same terminal session used in steps 2 and 3
8. Write a line to /home/user/input/input.log, for example using this command:

echo "some message" >> /home/user/input/input.log

9. You should see a new file in /home/user/output_folder/ with permissions 664
10. Stop filebeat
11. Remove all files in /home/user/output_folder/
12. Run umask 027
13. Repeat steps 7 and 8
14. You should see a new file in /home/user/output_folder/ with permissions 640

[dikshachauhan-qasource]

Hi @jlind23

This ticket has been validated and found working fine. Below are the detailed observation and steps followed:

Before setting umask 027

• Access your Linux machine.
• Go to Terminal and check current/default umask settings at machine, by hitting command 'umask' at terminal.
• Now download the filebeat artifact and make below proposed changes in your_filebeat.yml. Note Here, we are providing 666 permissions to output file.

filebeat.inputs:
- type: Filestream
  enabled: true
  paths:
       - /home/user/input_folder/*

output.file:
  path: "/home/user/output_folder/"
  filename: "output_file.log"
  permissions: 0666

• Now save the changes at filebeat.yml and hit terminal with below command:
  ./filebeat -e -c your_filebeat.yml
• Observe, filebeat service will start generating some log events.
• Now at another terminal instance, hit below command to generate log file.
  echo "message" >> output.log
• Go to the output.log file and confirm permissions available on this file.

Observation: output.log file is generated with 664 permissions.

After setting umask 027

• Stop filebeat.
• Delete the output.log file from output folder.
• Perform same steps mentioned above, however at step 2 , this time we will reset umask settings to 0027 using following command at terminal : 'umask 0027' .
  Note: we need to reset umask setting before starting the filebeat service.

Observation: output.log file is generated with 640 permissions.

Screenshot:
Before umask setting:

After umask setting:

@rdner Thanks for helping us out in validating same.

Thanks
QAS