Search by sharing an image on iOS

[dessant]

A new feature was added to search with an image from the device by sharing it on iOS (1cae724). The image is passed from a Share Extension to Safari by first opening the https://search-by-image.localhost/share?id=<shareId> page for the extension to detect the event using the webNavigation API, verify if the one time ID matches with the one stored by the Share Extension, then redirect the tab to a local extension page that fetches the image using native messaging.

This solution for bridging the share event with the browser worked well, and requesting search-by-image.localhost did not cause any network requests. The problem is that around the time the feature was merged, Apple released Safari 15.4 with a regression that breaks mostly all event listeners related to tabs in the background page. Extensions can no longer reliably detect from non-persistent background pages if a new page is loaded or if a new tab is opened in the browser. I have submitted a bug report using Feedback Assistant.

https://feedbackassistant.apple.com/feedback/9960355

Safari 15.4 has introduced a regression in the web extension API on macOS, iOS and iPadOS. Opening a web page hosted by the extension in Safari, such as the settings of the extension, breaks `tabs` and `webNavigation` event listeners in the background page.

After the preferences page of the extension has been opened, new web requests in the current tab or any other tab will no longer trigger an event in the background page. Closing the preferences page or opening new tabs will not help, the extension must be reloaded to fix the issue.

The event listeners work as expected in Safari 15.2.

The code snippet below can be tested in a background page, list the following permissions in the extension manifest: tabs, webNavigation, <all_urls>.


browser.tabs.onUpdated.addListener(() => console.log('tabs.onUpdated event'));

browser.webNavigation.onCompleted.addListener(() =>
  console.log('webNavigation.onCompleted event')
);


Steps to reproduce:

- Install the extension in Safari 15.4 on macOS, allow access to all websites
- Go to Develop > Web Extension Background Pages and view the logs of the background page
- Open a web page in Safari to verify that the relevant events are logged
- Go to Safari > Preferences > Extensions and open the extension's preferences
- Visit a website in Safari and notice that event listeners are no longer called

[dessant]

The only reliable workaround I have found is to open a real web page in Safari when the user shares an image, and handle the event using a content script, so we'll have to use this method until the regression is fixed in Safari.

The https://searchbyimage.vapps.dev/share page was set up to act as a bridge during a share event, this URL serves an empty page that is cached by Cloudflare.

We open the https://searchbyimage.vapps.dev/share#<shareId> page, the ID is passed to Safari using an URI fragment so that these unique tokens are not sent to Cloudflare every time the user shares an image. The content script detects the event, verifies the ID and redirects the tab to a local extension page, at which point the image is fetched using native messaging.

[dessant]

In the future there won't be a need for such workarounds if extensions will be able to register themselves as share targets.

w3c/webextensions#132

[dessant]

The feature will be enabled only for iOS at the moment, more work is needed in the native app to support sharing on macOS. The Xcode project with the native app will be merged in the app/safari folder in the next few months when macOS support for sharing is implemented.

[dessant]