-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Event property counter #7500
Merged
+286
−70
Merged
Event property counter #7500
Changes from 1 commit
Commits
Show all changes
56 commits
Select commit
Hold shift + click to select a range
598c213
create event property model
mariusandra b609261
add null
mariusandra 01307be
rename cache vars
mariusandra 37aad8c
update event properties table on ingestion
mariusandra ed359ea
match date formats
mariusandra 8f80f87
match date formats
mariusandra 17cb696
better string handling
mariusandra 304a740
property type can be null too
mariusandra 586c39b
pass event timestamp
mariusandra d738f9b
update property type later
mariusandra 4cbc1cc
perform all updates through a buffer object
mariusandra 1ad5949
move to EventPropertyCounter
mariusandra b9be65f
Merge branch 'master' into event-property-counter
mariusandra 2227052
fix migration
mariusandra 4a1da7c
improve flush last seen at job
mariusandra e64062a
flush job periodically + env
mariusandra 129532b
upsert all event properties in 1 query
mariusandra ba1b831
log to statsd
mariusandra 2c3c27d
enable property counter only if experimental mode enabled
mariusandra f69e68c
use now() instead of event timestamp
mariusandra 1c2a777
fix seconds
mariusandra 356d341
Merge branch 'master' into event-property-counter
mariusandra 6215919
add user/pass for default postgres
mariusandra 30ddc74
add tests
mariusandra 48842e1
use big integers
mariusandra 8e7429f
make query work with 50k props
mariusandra 7162157
processing events saves event properties
mariusandra 6434800
fix script
mariusandra 19c12e1
test date format detection
mariusandra 1ff7d94
default enabled
mariusandra 6486d70
only enable event property counter for specific teams
mariusandra 4895947
Merge branch 'master' into event-property-counter
mariusandra 4026f71
eslint fixes
mariusandra 12d3cc4
Merge branch 'master' into event-property-counter
mariusandra 07b20ee
fix logs double-sync noise in tests
mariusandra 9f16444
fix bigint test
mariusandra c1dc433
don't do tasks that make no sense
mariusandra 114fd2d
remove dead code
mariusandra 523fddd
simpler test setup
mariusandra 97fdd2d
different contraint name
mariusandra 7a5a8a0
refactor team manager
mariusandra a61884b
greatly simplify the system
mariusandra 20e893b
fetch cached event properties
mariusandra 3ce42cc
fix team manager and timestamps
mariusandra 0d42e92
add cached entry
mariusandra fbe080e
also don't cache event properties for teams that have it disabled
mariusandra 8e961b8
remove indexes that are not going to be used
mariusandra aa638e7
Merge branch 'master' into event-property-counter
mariusandra e7772ab
remove unused imports
mariusandra 8b01a16
blacked
mariusandra d070faa
remember event properties with a LRU cache
mariusandra f0f4f36
fix eslint
mariusandra a39253a
clean up the last bits
mariusandra cd463f4
move ONE_HOUR to constants
mariusandra 5db0f2d
use sane_repr
mariusandra c5469cd
fix typo
mariusandra File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
clean up the last bits
commit a39253ae8edb1e4bc6f344ad3697b067fdc71ca3
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It we could do this insert in a batch instead of per-property 🤔
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's pretty much what I was doing in the original implementation, leading to the deadlocks mentioned above. Even if we go from
ON CONFLICT DO UPDATE
toON CONFLICT DO NOTHING
, I'm going to have to implement code to retry the transactions if the block deadlocks, and all other complexities that come with it.It would much safer since we're no longer updating the rows that are already in the database, and not running all flush operations at precisely the same moment on every thread, but these deadlocks can't be avoided --> the same new event may come in 10 times in the same
millisecond
, be handled by 4 threads, which all insert the same properties at precisely the same moment.Thus, while a bit slower, I think ~20 simple inserts for each new event the first time it's seen will scale much better.