Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature: Unfurl status links #4033

Merged
merged 32 commits into from
Oct 13, 2023
Merged

Conversation

igor-sirotin
Copy link
Collaborator

@igor-sirotin igor-sirotin commented Sep 15, 2023

closes #3762
desktop PR: status-im/status-desktop#12303

This PR contains #4110. Will remove before merging.

Description

This PR introduces support for unfurling Status contact/community/channel links.

Storing the data

As we discussed, status links are unfurled in a separate database column.

So UnfurlURLs will now return a UnfurlURLsResponse with both link types:

type UnfurlURLsResponse struct {
LinkPreviews []common.LinkPreview `json:"linkPreviews,omitempty"`
StatusLinkPreviews []common.StatusLinkPreview `json:"statusLinkPreviews,omitempty"`
}

And a separate field is added to Message:

UnfurledStatusLinks unfurled_status_links = 17;

Contained data

The preview itself contains more data than the original encoded URL.
We add more data (e.g. icons). if it's available locally in the database, this is required for the expected designs.

We could also fetch data from Waku, if it's not available locally, but that might require more time and eventually might fail Which would be a strange behaviour, having that there's some data simply encoded in the URL. We would need some special mechanism of "supplementing the preview":

  • in the first stage, decode the url and return it to the user immediately
  • at the same time start fetching from waku and if more info is fetched before the message is sent, add it to the preview
  • maybe add some special loading UI for this case?

This is much more to do, but it's a not a bad idea I think.

Minor updates to shared urls

@MishkaRogachev adding you as reviewer for this part

@status-im-auto
Copy link
Member

status-im-auto commented Sep 15, 2023

Jenkins Builds

Click to see older builds (144)
Commit #️⃣ Finished (UTC) Duration Platform Result
✖️ 87a293a #1 2023-09-15 09:47:02 ~2 min tests 📄log
✔️ 87a293a #1 2023-09-15 09:48:02 ~3 min linux 📦zip
✔️ 87a293a #1 2023-09-15 09:49:46 ~5 min android 📦aar
✔️ 87a293a #1 2023-09-15 09:50:57 ~6 min ios 📦zip
✖️ f34ec90 #2 2023-09-20 16:50:14 ~2 min tests 📄log
✔️ f34ec90 #2 2023-09-20 16:51:05 ~3 min linux 📦zip
✔️ f34ec90 #2 2023-09-20 16:52:16 ~4 min ios 📦zip
✔️ f34ec90 #2 2023-09-20 16:52:31 ~4 min android 📦aar
✔️ ee2c411 #3 2023-09-21 15:39:38 ~1 min linux 📦zip
✖️ ee2c411 #3 2023-09-21 15:40:37 ~2 min tests 📄log
✔️ ee2c411 #3 2023-09-21 15:42:13 ~3 min ios 📦zip
✔️ ee2c411 #3 2023-09-21 15:43:02 ~4 min android 📦aar
✖️ 185b8c3 #4 2023-09-25 15:34:58 ~1 min tests 📄log
✔️ 185b8c3 #4 2023-09-25 15:35:09 ~1 min linux 📦zip
✔️ 185b8c3 #4 2023-09-25 15:35:47 ~2 min android 📦aar
✔️ 185b8c3 #4 2023-09-25 15:42:36 ~8 min ios 📦zip
9942a95 #5 2023-09-26 10:53:18 ~18 sec ios 📄log
9942a95 #5 2023-09-26 10:53:18 ~19 sec android 📄log
9942a95 #5 2023-09-26 10:54:07 ~1 min linux 📄log
✖️ 9942a95 #5 2023-09-26 10:54:36 ~1 min tests 📄log
cfdfac5 #6 2023-09-26 11:21:03 ~13 sec linux 📄log
cfdfac5 #6 2023-09-26 11:21:10 ~18 sec ios 📄log
cfdfac5 #6 2023-09-26 11:21:11 ~19 sec android 📄log
✖️ cfdfac5 #6 2023-09-26 11:21:42 ~48 sec tests 📄log
f23866c #7 2023-09-26 15:42:32 ~12 sec linux 📄log
f23866c #7 2023-09-26 15:42:35 ~17 sec ios 📄log
✖️ f23866c #7 2023-09-26 15:43:13 ~53 sec tests 📄log
f23866c #7 2023-09-26 15:43:30 ~1 min android 📄log
10e5fcb #8 2023-09-27 13:19:24 ~18 sec android 📄log
10e5fcb #8 2023-09-27 13:19:25 ~14 sec linux 📄log
10e5fcb #8 2023-09-27 13:19:33 ~22 sec ios 📄log
✖️ 10e5fcb #8 2023-09-27 13:21:14 ~2 min tests 📄log
5567a3d #9 2023-09-28 17:45:57 ~13 sec linux 📄log
5567a3d #9 2023-09-28 17:46:02 ~18 sec android 📄log
5567a3d #9 2023-09-28 17:46:02 ~19 sec ios 📄log
✖️ 5567a3d #9 2023-09-28 17:46:40 ~57 sec tests 📄log
c7126c1 #10 2023-09-28 21:28:37 ~18 sec ios 📄log
c7126c1 #10 2023-09-28 21:28:38 ~18 sec android 📄log
✖️ c7126c1 #10 2023-09-28 21:29:17 ~58 sec tests 📄log
c7126c1 #10 2023-09-28 21:29:26 ~1 min linux 📄log
✖️ f8bab25 #11 2023-09-28 21:36:41 ~1 min tests 📄log
✔️ f8bab25 #11 2023-09-28 21:37:53 ~2 min linux 📦zip
✔️ f8bab25 #11 2023-09-28 21:39:34 ~4 min ios 📦zip
✔️ f8bab25 #11 2023-09-28 21:39:44 ~4 min android 📦aar
✖️ 6dba927 #12 2023-09-28 21:37:37 ~51 sec tests 📄log
✖️ 287ae89 #13 2023-09-28 21:38:43 ~1 min tests 📄log
✔️ 287ae89 #12 2023-09-28 21:39:26 ~1 min linux 📦zip
✔️ 287ae89 #12 2023-09-28 21:41:44 ~1 min android 📦aar
✔️ 287ae89 #12 2023-09-28 21:42:39 ~2 min ios 📦zip
✖️ 70ed74b #14 2023-09-29 16:46:11 ~1 min tests 📄log
✔️ 70ed74b #13 2023-09-29 16:46:32 ~1 min linux 📦zip
✔️ 70ed74b #13 2023-09-29 16:47:02 ~1 min android 📦aar
✔️ 70ed74b #13 2023-09-29 16:48:14 ~3 min ios 📦zip
✔️ 69545bc #14 2023-09-30 20:38:22 ~1 min linux 📦zip
✔️ 69545bc #14 2023-09-30 20:38:54 ~1 min android 📦aar
✔️ 69545bc #14 2023-09-30 20:40:12 ~3 min ios 📦zip
✖️ 69545bc #15 2023-09-30 20:52:10 ~15 min tests 📄log
✖️ 69545bc #16 2023-09-30 21:12:32 ~19 min tests 📄log
✔️ b463154 #15 2023-10-02 10:12:58 ~1 min linux 📦zip
✔️ b463154 #15 2023-10-02 10:13:32 ~2 min android 📦aar
✔️ b463154 #15 2023-10-02 10:14:38 ~3 min ios 📦zip
✖️ b463154 #17 2023-10-02 10:35:35 ~24 min tests 📄log
✔️ 2f45bdb #16 2023-10-02 10:15:04 ~1 min linux 📦zip
✔️ 2f45bdb #16 2023-10-02 10:15:17 ~1 min android 📦aar
✔️ 2f45bdb #16 2023-10-02 10:17:31 ~2 min ios 📦zip
✔️ 196cf80 #17 2023-10-02 10:16:43 ~1 min linux 📦zip
✔️ 196cf80 #17 2023-10-02 10:16:57 ~1 min android 📦aar
✔️ 196cf80 #17 2023-10-02 10:20:54 ~3 min ios 📦zip
✖️ 196cf80 #18 2023-10-02 10:50:56 ~15 min tests 📄log
✖️ 196cf80 #19 2023-10-02 11:17:42 ~23 min tests 📄log
✔️ 7a24d45 #18 2023-10-02 13:39:52 ~3 min linux 📦zip
✔️ 7a24d45 #18 2023-10-02 13:39:55 ~3 min ios 📦zip
✔️ 7a24d45 #18 2023-10-02 13:41:11 ~4 min android 📦aar
✖️ 7a24d45 #20 2023-10-02 14:03:23 ~26 min tests 📄log
✔️ f61e309 #19 2023-10-02 14:56:07 ~1 min android 📦aar
✔️ f61e309 #19 2023-10-02 14:56:52 ~2 min linux 📦zip
✔️ f61e309 #19 2023-10-02 14:57:34 ~2 min ios 📦zip
✖️ f61e309 #21 2023-10-02 15:19:39 ~25 min tests 📄log
✔️ 58f936a #20 2023-10-02 16:41:52 ~1 min linux 📦zip
✔️ 58f936a #20 2023-10-02 16:44:02 ~3 min ios 📦zip
✔️ 58f936a #20 2023-10-02 16:45:29 ~4 min android 📦aar
✖️ 58f936a #22 2023-10-02 17:06:58 ~26 min tests 📄log
✔️ c2f0645 #21 2023-10-02 21:21:44 ~1 min linux 📦zip
✔️ c2f0645 #21 2023-10-02 21:22:14 ~2 min android 📦aar
✔️ c2f0645 #21 2023-10-02 21:23:08 ~3 min ios 📦zip
✔️ c2f0645 #23 2023-10-02 21:50:45 ~30 min tests 📄log
✔️ 87729a2 #22 2023-10-04 16:32:35 ~1 min linux 📦zip
✔️ 87729a2 #22 2023-10-04 16:33:05 ~2 min android 📦aar
✔️ 87729a2 #22 2023-10-04 16:34:06 ~3 min ios 📦zip
✖️ 87729a2 #24 2023-10-04 16:58:37 ~27 min tests 📄log
✔️ b726e65 #23 2023-10-04 16:43:15 ~1 min linux 📦zip
✔️ b726e65 #23 2023-10-04 16:43:36 ~1 min android 📦aar
✔️ b726e65 #23 2023-10-04 16:45:11 ~3 min ios 📦zip
✖️ b726e65 #25 2023-10-04 16:59:54 ~1 min tests 📄log
✔️ a910e33 #24 2023-10-04 19:33:59 ~1 min linux 📦zip
✔️ a910e33 #24 2023-10-04 19:34:51 ~2 min android 📦aar
✔️ a910e33 #24 2023-10-04 19:36:06 ~3 min ios 📦zip
✖️ a910e33 #26 2023-10-04 19:59:40 ~26 min tests 📄log
✔️ d9fa6de #25 2023-10-05 09:55:41 ~1 min linux 📦zip
✔️ d9fa6de #25 2023-10-05 09:57:02 ~2 min ios 📦zip
✔️ d9fa6de #25 2023-10-05 09:58:33 ~4 min android 📦aar
✖️ d9fa6de #27 2023-10-05 10:23:44 ~29 min tests 📄log
✔️ d9fa6de #28 2023-10-05 11:33:55 ~29 min tests 📄log
✔️ 04468f7 #26 2023-10-05 14:59:05 ~1 min linux 📦zip
✔️ 04468f7 #26 2023-10-05 14:59:19 ~1 min android 📦aar
✖️ 04468f7 #29 2023-10-05 15:11:45 ~14 min tests 📄log
✖️ 04468f7 #30 2023-10-05 16:14:02 ~30 min tests 📄log
✖️ 04468f7 #31 2023-10-05 16:45:31 ~29 min tests 📄log
✔️ e3c47db #27 2023-10-06 12:30:21 ~1 min linux 📦zip
✔️ e3c47db #27 2023-10-06 12:30:36 ~1 min android 📦aar
✔️ e3c47db #27 2023-10-06 12:31:25 ~2 min ios 📦zip
✖️ e3c47db #32 2023-10-06 12:58:29 ~29 min tests 📄log
✔️ 1323dc4 #28 2023-10-06 15:04:28 ~1 min linux 📦zip
✔️ 1323dc4 #28 2023-10-06 15:04:48 ~1 min android 📦aar
✔️ 1323dc4 #28 2023-10-06 15:05:55 ~2 min ios 📦zip
✖️ 1323dc4 #33 2023-10-06 15:31:21 ~28 min tests 📄log
✔️ b6c8c66 #29 2023-10-06 15:55:36 ~1 min linux 📦zip
✔️ b6c8c66 #29 2023-10-06 15:56:02 ~2 min android 📦aar
✔️ b6c8c66 #29 2023-10-06 15:56:54 ~2 min ios 📦zip
✖️ b6c8c66 #34 2023-10-06 16:21:04 ~27 min tests 📄log
✔️ b6c8c66 #35 2023-10-06 16:55:19 ~29 min tests 📄log
✔️ e587069 #30 2023-10-09 10:48:16 ~1 min linux 📦zip
✔️ e587069 #30 2023-10-09 10:49:12 ~2 min android 📦aar
✔️ e587069 #30 2023-10-09 10:50:18 ~3 min ios 📦zip
✔️ e587069 #36 2023-10-09 11:16:39 ~29 min tests 📄log
✔️ 3a24ade #31 2023-10-09 11:36:42 ~2 min android 📦aar
✔️ 3a24ade #31 2023-10-09 11:37:17 ~2 min linux 📦zip
✔️ 3a24ade #31 2023-10-09 11:37:50 ~3 min ios 📦zip
✖️ 3a24ade #37 2023-10-09 12:03:38 ~29 min tests 📄log
✔️ 3a24ade #38 2023-10-09 12:36:11 ~29 min tests 📄log
✖️ 221594e #39 2023-10-11 12:44:41 ~1 min tests 📄log
✔️ 221594e #32 2023-10-11 12:45:07 ~1 min linux 📦zip
✔️ 221594e #32 2023-10-11 12:45:49 ~2 min android 📦aar
✔️ 221594e #32 2023-10-11 12:46:40 ~3 min ios 📦zip
✔️ 7c7992b #33 2023-10-11 13:07:14 ~1 min linux 📦zip
✔️ 7c7992b #33 2023-10-11 13:08:37 ~3 min ios 📦zip
✔️ 7c7992b #33 2023-10-11 13:10:11 ~4 min android 📦aar
✖️ 7c7992b #40 2023-10-11 13:21:29 ~15 min tests 📄log
✖️ 7c7992b #41 2023-10-11 13:46:18 ~14 min tests 📄log
✖️ 7c7992b #42 2023-10-11 18:56:14 ~23 min tests 📄log
✔️ 3911890 #34 2023-10-11 19:48:49 ~1 min linux 📦zip
✔️ 3911890 #34 2023-10-11 19:49:04 ~1 min android 📦aar
✔️ 3911890 #34 2023-10-11 19:50:35 ~3 min ios 📦zip
✔️ 3911890 #43 2023-10-11 20:18:20 ~30 min tests 📄log
Commit #️⃣ Finished (UTC) Duration Platform Result
✔️ 0c50f6a #35 2023-10-13 11:51:11 ~2 min android 📦aar
✔️ 0c50f6a #35 2023-10-13 11:51:59 ~3 min linux 📦zip
✖️ 0c50f6a #44 2023-10-13 11:52:07 ~3 min tests 📄log
✔️ 0c50f6a #35 2023-10-13 11:52:32 ~3 min ios 📦zip
✔️ 157e2de #36 2023-10-13 11:54:30 ~1 min linux 📦zip
✔️ 157e2de #36 2023-10-13 11:55:27 ~2 min android 📦aar
✔️ 157e2de #36 2023-10-13 11:56:34 ~3 min ios 📦zip
✔️ 157e2de #45 2023-10-13 12:24:59 ~31 min tests 📄log

@igor-sirotin igor-sirotin force-pushed the feat/issue-3762-status-unfurler branch 2 times, most recently from 9942a95 to cfdfac5 Compare September 26, 2023 11:20
@igor-sirotin igor-sirotin force-pushed the feat/issue-3762-status-unfurler branch from f8bab25 to 6dba927 Compare September 28, 2023 21:36
@igor-sirotin igor-sirotin added the E:New Unfurling API Implementation of the new unfurling API for all links label Sep 29, 2023
@igor-sirotin igor-sirotin marked this pull request as ready for review September 29, 2023 17:08
@igor-sirotin
Copy link
Collaborator Author

It's ready to review, I will fix the tests meanwhile. Probably it's linter issues only.

Copy link
Contributor

@MishkaRogachev MishkaRogachev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, Igor! Great work!

protocol/common/message_test.go Outdated Show resolved Hide resolved
protocol/linkpreview_unfurler_status.go Outdated Show resolved Hide resolved
protocol/linkpreview_unfurler_status.go Outdated Show resolved Hide resolved
protocol/linkpreview_unfurler_status.go Show resolved Hide resolved
protocol/linkpreview_unfurler_status.go Outdated Show resolved Hide resolved
@igor-sirotin
Copy link
Collaborator Author

igor-sirotin commented Oct 2, 2023

I missed to implement the message editing, which was kindly caught by TestEditMessageWithLinkPreviews.
I'm fixing that now, but that won't change the idea of PR, so I think it makes sense to start reviewing anyway.

UPD: fixed.

@igor-sirotin igor-sirotin force-pushed the feat/issue-3762-status-unfurler branch 2 times, most recently from 2f45bdb to 196cf80 Compare October 2, 2023 10:15
protocol/common/message.go Outdated Show resolved Hide resolved
protocol/common/message.go Outdated Show resolved Hide resolved
protocol/common/message.go Outdated Show resolved Hide resolved
protocol/linkpreview_unfurler_status.go Outdated Show resolved Hide resolved
protocol/linkpreview_unfurler_status.go Outdated Show resolved Hide resolved
server/handlers.go Outdated Show resolved Hide resolved
@@ -1157,7 +1157,7 @@ func (api *PublicAPI) GetTextURLs(text string) []string {
// be removed from the response.
//
// This endpoint expects the client to send URLs normalized by GetTextURLs.
func (api *PublicAPI) UnfurlURLs(urls []string) ([]common.LinkPreview, error) {
func (api *PublicAPI) UnfurlURLs(urls []string) (protocol.UnfurlURLsResponse, error) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a reminder that this is a breaking change for the mobile client. Before merging this PR it's important to create a mobile PR that is reviewed, tested, and ready to be merged before you merge your PR.

I can implement the changes in status-mobile for us since it's trivial, but it will need to go over QA, and that can take some time at the moment.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI: @igor-sirotin, I have a mobile branch with the fix for the new response UnfurlURLsResponse. I'll open the mobile PR as soon as this status-go PR is finished.

server/handlers.go Outdated Show resolved Hide resolved
@@ -0,0 +1,2 @@
ALTER TABLE user_messages ADD COLUMN unfurled_status_links BLOB;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just sharing, nothing to do now, but I still fear status links have far more reasons to change than the relatively stable API of normal unfurled links, which could mean painful migrations. But since the user_messages table should never have its data migrated (according to our past discussions) then things can get tricky in the future.

A part of me still thinks this more "fragile" blob should be captured in a separate table, with explicit columns, although the join could affect the performance (or maybe not).

Don't worry, I'm just rambling here.

Copy link
Collaborator Author

@igor-sirotin igor-sirotin Oct 4, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree.

Actually there's a good news that we're going to update it on receiver side anyway. Only IDs are required to fetch the data. So we could migrate it later to some simple table, where we have IDs as separate columns, but all other data as blob 🤔

protocol/messenger_linkpreview.go Outdated Show resolved Hide resolved
@igor-sirotin
Copy link
Collaborator Author

igor-sirotin commented Oct 4, 2023

@ilmotta, thank you very much for your review and detailed comments, I appreciate that ❤️
l I've addressed all of you comments, please, check it out.

I decided to refactor things on the way, therefore you'll find some new separate files.
I also took an opportunity to refactor handlers_test to be a Suite with test cases.

Finnaly, as discussed, I will update the unfurling logic in this PR.

@igor-sirotin igor-sirotin force-pushed the feat/issue-3762-status-unfurler branch from 87729a2 to b726e65 Compare October 4, 2023 16:41
@ilmotta
Copy link
Contributor

ilmotta commented Oct 5, 2023

@ilmotta, thank you very much for your review and detailed comments, I appreciate that ❤️ l I've addressed all of you comments, please, check it out.

I decided to refactor things on the way, therefore you'll find some new separate files. I also took an opportunity to refactor handlers_test to be a Suite with test cases.

Finnaly, as discussed, I will update the unfurling logic in this PR.

Thank you @igor-sirotin. I'm going to review today again.

@status-im-auto
Copy link
Member

✔️ status-go/prs/ios/PR-4033#26 🔹 ~3 min 39 sec 🔹 04468f7 🔹 📦 ios package

@igor-sirotin
Copy link
Collaborator Author

Pushed the change:

  • ignore URL data
  • fetch from waku if nothing's available locally

Copy link
Contributor

@MishkaRogachev MishkaRogachev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work!

protocol/common/message.go Show resolved Hide resolved
protocol/linkpreview_unfurler_status.go Outdated Show resolved Hide resolved
Copy link
Contributor

@ilmotta ilmotta left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work in this PR @igor-sirotin. And thank you for organizing meetings to openly discuss the features.

Since the code in this PR is relatively stable now, I'll move forward with the mobile PR today (Friday) and get it checked by QAs. The mobile QA team is quite busy at the moment. I'll keep you posted when it's approved so you can merge this one.

protocol/linkpreview_unfurler_status.go Outdated Show resolved Hide resolved
@pavloburykh
Copy link

@igor-sirotin hello! Could you please rebase current branch? We need it for proper testing of correspondent mobile PR status-im/status-mobile#17573 Thank you!

@igor-sirotin igor-sirotin force-pushed the feat/issue-3762-status-unfurler branch from e3c47db to 1323dc4 Compare October 6, 2023 15:02
@igor-sirotin
Copy link
Collaborator Author

@pavloburykh sure, done 👌

@pavloburykh
Copy link

@pavloburykh sure, done 👌

thank you @igor-sirotin!

@igor-sirotin igor-sirotin force-pushed the feat/issue-3762-status-unfurler branch from 3911890 to 0c50f6a Compare October 13, 2023 11:48
@igor-sirotin igor-sirotin force-pushed the feat/issue-3762-status-unfurler branch from 0c50f6a to 157e2de Compare October 13, 2023 11:52
@igor-sirotin igor-sirotin merged commit aded258 into develop Oct 13, 2023
2 checks passed
@igor-sirotin igor-sirotin deleted the feat/issue-3762-status-unfurler branch October 13, 2023 12:25
@igor-sirotin
Copy link
Collaborator Author

🎉

ilmotta added a commit to status-im/status-mobile that referenced this pull request Oct 13, 2023
…n status-go (#17573)

Adapt the JSON RPC response to the new shape returned by `wakuext_unfurlURLs` on
v0.170.0. The changes were introduced PR
status-im/status-go#4033. There are no behavioral
changes in the API as far as mobile is concerned at the moment.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
E:New Unfurling API Implementation of the new unfurling API for all links
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Unfurl Status deep links
6 participants