Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Monitoring improvements #52

Merged
merged 16 commits into from
Jan 6, 2022
Merged

Monitoring improvements #52

merged 16 commits into from
Jan 6, 2022

Conversation

topliceanu
Copy link
Contributor

@topliceanu topliceanu commented Dec 17, 2021 •
edited
Loading

  • add tests for the Schema registry
  • refactor config
  • drop flags and keep only the env vars
  • add configurable poll interval and timeout for reading from solana and rdd
  • add test for encoding and empty state and transmission objects in avro + fixes
  • remove all keys from structures to get go's type system to help out
  • add base 58 encoding 'feed_state_account'
  • add exporter abstraction for kafka and prometheus.
  • remove dependency on core/logging Not atm
  • add module to follow weiwatcher changes on feeds and restart Doing this in a separate PR

@topliceanu topliceanu changed the base branch from master to feature/relay December 17, 2021 01:27
@krebernisak krebernisak force-pushed the monitoring-improvements branch from 826f164 to c9f9d3f Compare December 20, 2021 16:38
@krebernisak krebernisak changed the base branch from feature/relay to develop December 20, 2021 16:39
@krebernisak krebernisak force-pushed the monitoring-improvements branch from d840827 to a19db50 Compare December 20, 2021 19:22
@topliceanu topliceanu force-pushed the monitoring-improvements branch 2 times, most recently from e5ea247 to d8599d1 Compare January 4, 2022 15:14
@topliceanu topliceanu marked this pull request as ready for review January 4, 2022 16:13
@topliceanu topliceanu requested review from aalu1418, pedrovsg and a team and removed request for pedrovsg January 4, 2022 18:20
pedrovsg
pedrovsg reviewed Jan 5, 2022
View reviewed changes
pkg/monitoring/poller.go
data, err := s.reader.Read(ctx, s.account)
var data interface{}
var err error
func() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why this pattern?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So I can defer cancel() which ensures it will be executed even if Read panics!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Defer cancel should always run since you call it before the read. Don't understand the difference of wrapping into a func

pedrovsg
pedrovsg reviewed Jan 5, 2022
View reviewed changes
pkg/monitoring/mapping.go Outdated Show resolved Hide resolved
@topliceanu topliceanu removed the request for review from aalu1418 January 5, 2022 17:35
@topliceanu
expand the Schema interface and add tests for the SchemaRegistry
8c0bc2e
@topliceanu
refactor config throughout the project:
30e34f3 
-, move to a separate package
- remove flags and json, only using env vars
- add validation
- add feature flags, configurable poll interval and timout for solana
  and rdd
@topliceanu
add test for encoding an empty state and transmission structures as well
da99b01 
as fixes for the cases that don't work
@topliceanu
introduce the exporter abstraction, add tests for it and update the c…
2647d8b 
…odebase
@topliceanu
solve blocking test and add test to check leaked goroutines
c07138c
@topliceanu
split out the benchmark into the state and transmission paths
7bb9d65
@topliceanu
reuse timer in the poller to save some memory
50195d1
@topliceanu
Monitoring: base 58 encoding 'feed_state_account'
139da02
@topliceanu
fix poller test
8b956e5
@topliceanu
add github.com/mr-tron/base58
3a1695e
@topliceanu
refactor exporter files
f631274
@topliceanu
refactor the schema files
1c288a8
@topliceanu
fix bug where the wrong schema was associated with a topic
8f6f4a6
@topliceanu
cosmetic changes and log line tweaks
4f3a95a
@topliceanu
restor validator and flagging_threshold to preserve the schema's back…
78efcf5 
…wards compatibility
@topliceanu topliceanu force-pushed the monitoring-improvements branch from 5d31a06 to 78efcf5 Compare January 6, 2022 12:36
@topliceanu topliceanu requested a review from pedrovsg January 6, 2022 12:37
@topliceanu
Copy link
Contributor Author

@pedrovsg @johnnymugs I rebased on develop and this is now ready for another review

@topliceanu topliceanu requested a review from a team January 6, 2022 12:39
@topliceanu
Metrics: remove namespace, add feed_id (#80)
a7ef3f0 
* Remove namespace from metrics. Also add feed_id as a label to all
metrics

* fix bad rebase
@topliceanu topliceanu merged commit 601eaeb into develop Jan 6, 2022
@topliceanu topliceanu deleted the monitoring-improvements branch January 6, 2022 17:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@johnnymugs johnnymugs johnnymugs approved these changes

@pedrovsg pedrovsg Awaiting requested review from pedrovsg

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@topliceanu @johnnymugs @pedrovsg