[HOLD for payment 2024-03-14] HIGH: Onyx updates from push notifications can get applied out of order

[arosiclair]

Note: Waiting for #34082 to build the troubleshooting tools

• Coming from this slack convo cc @iwiznia

Problem

When a user receives a report comment, we send them a push notification for it. The notification payload contains Onyx updates for the report and the new report action. We apply those onyx updates here and call ReconnectApp in the background using backgroundRefresh.

There is a chance this update gets applied out of order eg:

1. User A sends a message and edits it quickly
2. User B applies updates for the new message and edit from Pusher
3. User B receives a delayed push notification for the message and applies the update for the original un-edited message

Why is this important?

If this happens, the report and report action data will stay out of date until the next updates are received

Solution

Start including lastUpdateID and previousUpdateID in the notification payload and use those to apply the update like we do for reliable updates here so those updates always get applied in order.

Upwork Automation - Do Not Edit

• Upwork Job URL: https://www.upwork.com/jobs/~018a77e81664929acf
• Upwork Job ID: 1735753015692898304
• Last Price Increase: 2023-12-15

Issue Owner

Current Issue Owner: @JmillsExpensify

[melvin-bot]

Job added to Upwork: https://www.upwork.com/jobs/~018a77e81664929acf

[melvin-bot]

Triggered auto assignment to @JmillsExpensify (Bug), see https://stackoverflow.com/c/expensify/questions/14418 for more details.

[melvin-bot]

Bug0 Triage Checklist (Main S/O)

• This "bug" occurs on a supported platform (ensure Platforms in OP are ✅)
• This bug is not a duplicate report (check E/App issues and #expensify-bugs)
  • If it is, comment with a link to the original report, close the issue and add any novel details to the original issue instead
• This bug is reproducible using the reproduction steps in the OP. S/O
  • If the reproduction steps are clear and you're unable to reproduce the bug, check with the reporter and QA first, then close the issue.
  • If the reproduction steps aren't clear and you determine the correct steps, please update the OP.
• This issue is filled out as thoroughly and clearly as possible
  • Pay special attention to the title, results, platforms where the bug occurs, and if the bug happens on staging/production.
• I have reviewed and subscribed to the linked Slack conversation to ensure Slack/Github stay in sync

[melvin-bot]

Triggered auto assignment to Contributor Plus for review of internal employee PR - @allroundexperts (Internal)

[melvin-bot]

@JmillsExpensify, @arosiclair, @allroundexperts Whoops! This issue is 2 days overdue. Let's get this updated quick!

[arosiclair]

No progress on this yet. I'm heads down on Notification Polish and then OOO for a couple days. I should be able to get to this next week.

[mallenexpensify]

@allroundexperts can you review the PR?
#33371

I think you might not have been auto-assigned cuz of the formatting here? (reallly don't know, just trying to expedite a fix here)

Related to #33191 and https://expensify.slack.com/archives/C049HHMV9SM/p1703005526338499

[allroundexperts]

Sure, thanks for the reminder!

[arosiclair]

Realized I still had to handle whisper messages. Just finished handling that case. Now I have a test failing in CI but not locally 🙃. I'm going to reach out in #engi-chat on how to get that fixed. Figured it out! Auth and Web-E PR's are out for review. App is held until those are deployed.

[arosiclair]

• Auth PR is deployed
• Web-E PR is ready for review & merge
• App PR will be ready for testing once the Web-E is deployed

[iwiznia]

Nice, super excited for this to go out. Thanks for the hard work on it! ❤️

[arosiclair]

All PR's are now merged. Just waiting for the App PR to hit staging so I can QA. ETA is EOW.

[arosiclair]

QA on staging passed! This should be out on prod in the next deploy. ETA is today/tomorrow.

[melvin-bot]

The solution for this issue has been 🚀 deployed to production 🚀 in version 1.4.48-0 and is now subject to a 7-day regression period 📆. Here is the list of pull requests that resolve this issue:

• Apply onyx updates from push notifications reliably #36047

If no regressions arise, payment will be issued on 2024-03-14. 🎊

For reference, here are some details about the assignees on this issue:

• @allroundexperts requires payment through NewDot Manual Requests

[melvin-bot]

BugZero Checklist: The PR fixing this issue has been merged! The following checklist (instructions) will need to be completed before the issue can be closed:

• [@allroundexperts] The PR that introduced the bug has been identified. Link to the PR:
• [@allroundexperts] The offending PR has been commented on, pointing out the bug it caused and why, so the author and reviewers can learn from the mistake. Link to comment:
• [@allroundexperts] A discussion in #expensify-bugs has been started about whether any other steps should be taken (e.g. updating the PR review checklist) in order to catch this type of bug sooner. Link to discussion:
• [@allroundexperts] Determine if we should create a regression test for this bug.
• [@allroundexperts] If we decide to create a regression test for the bug, please propose the regression test steps to ensure the same bug will not reach production again.
• [@JmillsExpensify] Link the GH issue for creating/updating the regression test once above steps have been agreed upon:

[arosiclair]

This wasn't really a regression just new functionality for reliable updates.

[melvin-bot]

Payment Summary

Upwork Job

• Reviewer: @allroundexperts owed $500 via NewDot

BugZero Checklist (@JmillsExpensify)

• I have verified the correct assignees and roles are listed above and updated the neccesary manual offers
• I have verified that there are no duplicate or incorrect contracts on Upwork for this job (https://www.upwork.com/ab/applicants/1735753015692898304/hired)
• I have paid out the Upwork contracts or cancelled the ones that are incorrect
• I have verified the payment summary above is correct

[JmillsExpensify]

The payment summary above is correct, so @allroundexperts please add this one to your next round of Expensify requests. Closing in the meantime.

[joekaufmanexpensify]

Stepping in for payment summary for Jason. Only payment is $500 to @allroundexperts for PR review. Paid via NewDot.

[JmillsExpensify]

Thanks! $500 approved for @allroundexperts