Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Move Feedback Ingestion to its own Topic / Consumer #66100

Closed
4 tasks done
Tracked by #64671
JoshFerge opened this issue Feb 29, 2024 · 5 comments
Closed
4 tasks done
Tracked by #64671

Move Feedback Ingestion to its own Topic / Consumer #66100

JoshFerge opened this issue Feb 29, 2024 · 5 comments
Assignees
@aliu39

Comments

@JoshFerge
Copy link
Member

JoshFerge commented Feb 29, 2024 •
edited by aliu39
Loading

Tasks
Preview Give feedback

Motivation: when errors pipeline has problems, we don't want to lose feedbacks. Additionally, we can add things like a dead letter queue, etc.
@JoshFerge JoshFerge mentioned this issue Feb 29, 2024
Closed
@cmanallen
Copy link
Member

@aliu-at-sentry You've been volunteered for this issue but don't worry! I'll pair with you and we can tackle it together. Let's plan to start on this on Monday. We can use our first standup as the jumping off point.

@aliu39 aliu39 mentioned this issue Mar 1, 2024
Closed
aliu39 added a commit to getsentry/sentry-kafka-schemas that referenced this issue Mar 1, 2024
@aliu39
Create ingest-feedbacks.yaml
766ba05 
https://getsentry.atlassian.net/browse/OPS-5317
getsentry/sentry#66100

Pipeline/cluster = replays, since replay team owns user feedback. We expect the volume to be too low to justify its own cluster.
@aliu39 aliu39 mentioned this issue Mar 1, 2024
Merged
aliu39 added a commit to getsentry/sentry-kafka-schemas that referenced this issue Mar 5, 2024
@aliu39
Create ingest-feedbacks.yaml
c500a24 
https://getsentry.atlassian.net/browse/OPS-5317
getsentry/sentry#66100

Pipeline/cluster = replays, since replay team owns user feedback. We expect the volume to be too low to justify its own cluster.
@aliu39 aliu39 mentioned this issue Mar 6, 2024
Merged
aliu39 added a commit to getsentry/sentry-kafka-schemas that referenced this issue Mar 7, 2024
@aliu39
Create ingest-feedbacks.yaml
b2366ed 
https://getsentry.atlassian.net/browse/OPS-5317
getsentry/sentry#66100

Pipeline/cluster = replays, since replay team owns user feedback. We expect the volume to be too low to justify its own cluster.
aliu39 added a commit to getsentry/sentry-kafka-schemas that referenced this issue Mar 12, 2024
@aliu39 @getsantry
feat(replay): Config new ingest-feedback-events topic (#227)
e74bfe4 
* Create ingest-feedbacks.yaml

https://getsentry.atlassian.net/browse/OPS-5317
getsentry/sentry#66100

Pipeline/cluster = replays, since replay team owns user feedback. We expect the volume to be too low to justify its own cluster.

* Update ingest-feedbacks description

* Fix ingest-feedbacks.yaml producers

+small change to descrip

* add new schema and pipeline + rename

- move to new 'user-feedback' pipeline
- add an initial schema generated from a widget feedback (captured by chrome dev tools) and https://github.com/glideapps/quicktype
- rename to 'ingest-feedback-events' since 'FeedbackEvent' seems to be the corresponding type in Django + SDK/typescript

* style(lint): Auto commit lint changes

* add topic to CODEOWNERS

* change schema ver to draft-07

* remove field titles and add schema title, remove owners-ingest

* Add react ex and make all optional except contexts:{feedback:{contact_email, message, name, source}}

- all fields have additionalProperties = true, making schema totally flexible and backward/forward-compatible
- rename examples folder

* style(lint): Auto commit lint changes

* style(lint): Auto commit lint changes

* modify schema to match sdk event type

* style(lint): Auto commit lint changes

* style(lint): Auto commit lint changes

---------

Co-authored-by: getsantry[bot] <66042841+getsantry[bot]@users.noreply.github.com>
aliu39 added a commit that referenced this issue Mar 13, 2024
@aliu39
feat(replay): add config for new topic ingest-feedback-events (#66484)
9ab40c3 
Linked to #66100 and
https://getsentry.atlassian.net/browse/OPS-5317

Dependent on:
[sentry-kafka-schemas
PR](getsentry/sentry-kafka-schemas#227)
@aliu39
Copy link
Member

aliu39 commented Mar 18, 2024

IMG_7242
Red is TODO

@JoshFerge
Copy link
Member Author

JoshFerge commented Mar 20, 2024 •
edited by aliu39
Loading

Relay TODO:

  • Add an option backed feature flag an option in sentry/getsentry, make sure it is available in relay
  • add a KafkaTopic to the kafka topic enum
  • Modify services/store.rs so that it produces ItemType::UserReportV2 to the new kafka topic. sidenote: Attachments should still go to the attachments topic
  • Separate attachments and feedback in the same envelope, send each to their respective topics. See issue above ^
    • Remove option flag once tested
  • Remove option flag once tested

@aliu39 aliu39 mentioned this issue Mar 27, 2024
Merged
@bruno-garcia bruno-garcia mentioned this issue Mar 27, 2024
Merged
@aliu39 aliu39 mentioned this issue Mar 27, 2024
Merged
aliu39 added a commit that referenced this issue Mar 28, 2024
@aliu39
feat(feedback): add option for ingest feedback topic (#67839)
7e3ded7 
relates to #66100
aliu39 added a commit to getsentry/relay that referenced this issue Apr 9, 2024
@aliu39
feat(feedback): produce to ingest-feedback-events topic (#3344)
01311a4 
Option PR:
depends on getsentry/sentry#67839 (MERGED)

Feature flag PRs (outdated/unused):
~getsentry/getsentry#13426 (REVERTED)~
~getsentry/sentry#67747 (REVERTED)~
~getsentry/sentry-options-automator#973
(CLOSED)~

Relates to getsentry/sentry#66100
This was referenced Apr 17, 2024
Closed
Closed
@aliu39
Copy link
Member

aliu39 commented Apr 19, 2024 •
edited
Loading

Metrics to track ingest in s4s:
https://p.datadoghq.com/sb/FH6-Y3-fdb1541238e21113f4febca249245966

the test project I used

@aliu39
Copy link
Member

aliu39 commented May 10, 2024

🎉🎉🎉🎉🎉

@aliu39 aliu39 closed this as completed May 10, 2024
This was referenced May 15, 2024
Closed
Closed
@github-actions github-actions bot locked and limited conversation to collaborators May 26, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@aliu39 aliu39

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@JoshFerge @cmanallen @aliu39