Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable warning/error logs from libraries in tests #22186

Conversation

findepi
Copy link
Member

@findepi findepi commented May 29, 2024

Warning and errors are rarely seen by engineers, but still it's better not to hide them, they may (now or in the future) contain some actionable information. When a library over-verbosely logs some warnings/errors, let's suppress these selectively.

Follows #19676

@cla-bot cla-bot bot added the cla-signed label May 29, 2024
@github-actions github-actions bot added hudi Hudi connector iceberg Iceberg connector delta-lake Delta Lake connector bigquery BigQuery connector mongodb MongoDB connector labels May 29, 2024
@findepi
Copy link
Member Author

findepi commented May 29, 2024

they may (now or in the future) contain some actionable information

I can't say that for sure for libraries we currently use in this project, but that's certainly my past experience.
Sometimes people use warnings in a manner similar to @Deprecated -- deprecation is suitable for staticaly compiled API elements, while warnings are used for things detected at runtime (like Trino's warning when ORDER BY gets ignored).

@findepi findepi force-pushed the findepi/enable-warning-error-logs-from-libraries-in-tests-3939ed branch 2 times, most recently from a923d29 to 5b90fa7 Compare May 29, 2024 10:35
findepi added 2 commits May 29, 2024 12:36
Warning and errors are rarely seen by engineers, but still it's better
not to hide them, they may (now or in the future) contain some
actionable information. When a library over-verbosely logs some
warnings/errors, let's suppress these selectively.
@findepi findepi force-pushed the findepi/enable-warning-error-logs-from-libraries-in-tests-3939ed branch from 5b90fa7 to 5e5b9c2 Compare May 29, 2024 10:36
@findepi findepi merged commit 29fcd50 into trinodb:master May 29, 2024
61 of 63 checks passed
@findepi findepi deleted the findepi/enable-warning-error-logs-from-libraries-in-tests-3939ed branch May 29, 2024 12:03
@github-actions github-actions bot added this to the 449 milestone May 29, 2024
Copy link
Member

@martint martint left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Libraries should not be reporting problems via logging. If there’s an actionable error, they should fail with an exception.

These logs were turned off intentionally, precisely because they were generating noise in the CI logs that’s not actionable and just makes it harder to spot real problems such as test failures.

@findepi
Copy link
Member Author

findepi commented May 29, 2024

Libraries should not be reporting problems via logging. If there’s an actionable error, they should fail with an exception.

I tend to agree. But I do not get to choose what libraries do and I have been burned in the past by ignoring warnings from a library. This was so important that I even had a test framework to watch out for new warning/errors and it turned helpful in future situations.

Agreeing that we may need to suppress certain overly verbose loggers. For example Iceberg library is a known offender and logs A LOT at INFO level. So if they also log a lot at warn/error level then either we help them fix this, or suppress such a misbehaving library.

@findepi findepi added test no-release-notes This pull request does not require release notes entry labels May 29, 2024
@losipiuk
Copy link
Member

losipiuk commented Jun 2, 2024

These logs were turned off intentionally, precisely because they were generating noise in the CI logs that’s not actionable and just makes it harder to spot real problems such as test failures.

@martint can you comment on the intention of turning off WARN/ERROR? I get why we do not want to have flooding of INFO logs from external libs but seeing errors in logs can point us to problems in our code. Even if not in an optimal way - it is good nonetheless.

I think this change is good and should have not been reverted.

@findepi
Copy link
Member Author

findepi commented Jun 6, 2024

Example useful warning logged by Iceberg library

WARN	Finalizer	org.apache.iceberg.aws.s3.S3FileIO	Unclosed S3FileIO instance created by:
	org.apache.iceberg.aws.s3.S3FileIO.<init>(S3FileIO.java:120)
	org.apache.iceberg.aws.s3.S3FileIO.<init>(S3FileIO.java:106)
        <caller>

i noticed this in some other application using Iceberg library. Wonder whether something like this gets logged when Trino uses Iceberg library.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bigquery BigQuery connector cla-signed delta-lake Delta Lake connector hudi Hudi connector iceberg Iceberg connector mongodb MongoDB connector no-release-notes This pull request does not require release notes entry test
Development

Successfully merging this pull request may close these issues.

4 participants