Fix empty file edge case

[rdner]

What does this PR do?

It's possible that file scanning happens when the file was created but was not yet written to. In this case the size is 0.

We should not spawn any resources (e.g. harvesters) for such files until they actually have some content. We create events only when read something from a file, so having a harvester on an empty file is not useful.

Not handling this edge case also causes our tests to be flaky, sometimes an expected size does not match an actual size (0).

Why is it important?

To fix flaky tests and to make the code more efficient.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
  - [ ] I have made corresponding changes to the documentation
  - [ ] I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
  - [ ] I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

1. Run Filebeat with this configuration:

filebeat.inputs:
  - type: filestream
    id: my-filestream-id
    paths:
      - "/path/to/your/logs/*.log"
    prospector.scanner.check_interval: 5s
path.data: "/path/to/your/data"
logging:
  level: debug
output.console:
  enabled: true

Note the path placeholders you need to fix first.

2. Place an empty file (touch some.log) in /path/to/your/logs
3. Wait up to 5 seconds to see this log message in Filebeat's output:

{                                                                   
  "log.level": "warn",                                              
  "@timestamp": "2023-07-17T09:51:27.315+0200",                     
  "log.logger": "scanner",                                          
  "log.origin": {                                                   
    "file.name": "filestream/fswatch.go",                           
    "file.line": 369                                                
  },                                                                
  "message": "file \"/Users/rdner/Projects/es_confs/empty-files/logs/some.log\" has no content yet, skipping",                          
  "service.name": "filebeat",                                       
  "ecs.version": "1.6.0"                                            
}

4. Add a line to the file using the following command:

echo "line" >> some.log

you can see that Filebeat now detected the file, started ingestion and created an event:

{                                                                   
  "log.level": "debug",                                             
  "@timestamp": "2023-07-17T09:52:21.436+0200",                                                                                         
  "log.logger": "processors",                                       
  "log.origin": {                                                   
    "file.name": "processing/processors.go",                        
    "file.line": 213                                                
  },                                                                
  "message": "Publish event: {\n  \"@timestamp\": \"2023-07-17T07:52:21.436Z\",\n  \"@metadata\": {\n    \"beat\": \"filebeat\",\n    \"type\": \"_doc\",\n    \"version\": \"8.10.0\"\n  },\n  \"input\": {\n    \"type\": \"filestream\"\n  },\n  \"ecs\": {\n    \"version\": 
\"8.0.0\"\n  },\n  \"host\": {\n    \"name\": \"MacBook-Pro.localdomain\"\n  },\n  \"agent\": {\n    \"name\": \"MacBook-Pro.localdomain\",\n    \"type\": \"filebeat\",\n    \"version\": \"8.10.0\",\n    \"ephemeral_id\": \"97426136-07de-4b0a-99b4-bcc60a8c201a\",\n    \"id
\": \"7326a55b-6e54-470c-9557-7f2d0c5f0efc\"\n  },\n  \"log\": {\n    \"offset\": 0,\n    \"file\": {\n      \"path\": \"/Users/rdner/Projects/es_confs/empty-files/logs/some.log\"\n    }\n  },\n  \"message\": \"some\"\n}",
  "service.name": "filebeat",                                                                                                           
  "ecs.version": "1.6.0"                                                                                                                
} 

• Related Add new fingerprint file identity #35734
• Related [Filebeat] Implement a more stable file identifier #34419

[rdner]

filebeat-goIntegTest tests make sure that the external behaviour stayed the same, I double-checked.

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-07-18T06:56:43.955+0000

• Duration: 70 min 4 sec

Test stats 🧪

Test	Results
Failed	0
Passed	8058
Skipped	757
Total	8815

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[leehinman]

LGTM

[miksir]

my logs now full of spam "has no content yet, skipping" for all zero size logs. And it's on the "warning" level

[Alphayeeeet]

Maybe change the log level of that message to info or even debug to prevent spam? @Team

[rdner]

@miksir @Alphayeeeet sorry for the log spam, addressed in #38421