-
Notifications
You must be signed in to change notification settings - Fork 71
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Allow configuring a dedicated metrics logger #978
feat: Allow configuring a dedicated metrics logger #978
Conversation
8f74b18
to
deee553
Compare
Codecov Report
@@ Coverage Diff @@
## main #978 +/- ##
==========================================
+ Coverage 82.18% 83.00% +0.81%
==========================================
Files 37 39 +2
Lines 3621 3748 +127
Branches 625 629 +4
==========================================
+ Hits 2976 3111 +135
+ Misses 479 473 -6
+ Partials 166 164 -2
📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
9929edb
to
ae65f65
Compare
pre-commit.ci autofix |
3e5ad46
to
d505afb
Compare
26ac1bc
to
c6a8271
Compare
c6a8271
to
b88d8b5
Compare
a49bffb
to
f486643
Compare
f486643
to
131c498
Compare
Exciting!! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good! 🚀
Co-authored-by: Aaron ("AJ") Steers <[email protected]>
Summary
New metrics helpers
The most important difference: the previous implementation only emitted total records at the end of sync, while the new one emits the accumulated record count between regular intervals (60s by default). This allows, among other things, to plot record counts against time for a child and parent stream:
Takes inspiration from
singer-python
. The new classes actually output time-series data for record count, sync duration and HTTP request duration.The
Point
classThis dataclass has a custom
__str__()
method that renders the object as JSON, that means disabling metrics via log config (e.g. levelERROR
) results in not needing to perform the costly object->string conversion and makes testing metrics easier by capturing the raw object from theLogRecord
instance.Logging configuration
Since metrics are just special logs, I think the best way to configure them is via generic log configuration. So, the SDK now supports an
SINGER_SDK_LOG_CONFIG
env var that points to a logging config file in YAML format (same as meltano).📚 Documentation preview 📚: https://meltano-sdk--978.org.readthedocs.build/en/978/