Protocol issues experienced while developing the new iOS SDK

[felix-schwarz]

This issue attempts to give a compact overview of protocol issues I ran into while implementing the ios-sdk.

Info endpoints

I found the number of endpoints to hit before being able to start a session in the iOS app unnecessarily long:

• .well-known/openid-configuration to detect OIDC
• status.php to detect redirections, version and maintenance mode
• user.php to retrieve the current user
• capabilities.php to retrieve the current set of capabilities

A single endpoint for fetching all of this information in one request would simplify this greatly. such an endpoint could be passed a list of info segments to include, like f.ex.: info.php?include=status,user,capabilities (or an extended status.php with that functionality).

Thumbnails

Clients currently have to request a thumbnail for a file to determine if a thumbnail is available. In a directory with many files for whose file types no thumbnails are available, this generates a lot of unnecessary requests.

What would help:

• a list of mime types (provided through f.ex. capabilities) for which the server supports thumbnail generation
• the DAV endpoint providing this info for every item with a custom tag (f.ex. oc:thumbnail-available)

The latter approach would have the benefit of keeping the logic for which item thumbnails are available in the server.

Related issue: owncloud/core#31267

Unnecessary large PROPFIND responses

PROPFINDs typically return two d:propstat tags for every item.

First, the part with information on the item:

<d:response>
    <d:href>/remote.php/dav/files/admin/</d:href>
    <d:propstat>
        <d:prop>
            <d:resourcetype>
                <d:collection/>
            </d:resourcetype>
            <d:getlastmodified>Fri, 23 Feb 2018 11:52:05 GMT</d:getlastmodified>
            <d:getetag>"5a9000658388d"</d:getetag>
            <d:quota-available-bytes>-3</d:quota-available-bytes>
            <d:quota-used-bytes>5812174</d:quota-used-bytes>
            <oc:size>5812174</oc:size>
            <oc:id>00000009ocre5kavbk8j</oc:id>
            <oc:permissions>RDNVCK</oc:permissions>
        </d:prop>
        <d:status>HTTP/1.1 200 OK</d:status>
    </d:propstat>

Then, second, the part with requested information not available for the item:

    <d:propstat>
        <d:prop>
            <d:creationdate/>
            <d:getcontentlength/>
            <d:displayname/>
            <d:getcontenttype/>
        </d:prop>
        <d:status>HTTP/1.1 404 Not Found</d:status>
    </d:propstat>

… and finally the closing tag:

</d:response>

The second part (on information not available) consumes bandwidth, CPU cycles, memory and power - all of which are very valuable on a mobile device - but provides no benefit at all.

It'd therefore be great to be able to omit the second/useless part. And if it needs to be there to conform with the WebDAV standard, provide an option to omit it.

WebDAV

Related to the aforementioned issue: I like and respect WebDAV (a lot!), but where it really falls short is in the expression of information in a compact format. Above XML, after all, just contains this information:

location: /remote.php/dav/files/admin/
type: collection
last-modified: Fri, 23 Feb 2018 11:52:05 GMT
Etag: "5a9000658388d"
quota-available: -3
quota-used: 5812174
size: 5812174
fileID: 00000009ocre5kavbk8j
permissions: RDNVCK

Even in HTTP-header-esque notation, it's already a lot less bytes. Now imagine what a binary format could achieve ([VID] is a VarInt encoding block type + length, [VIN] is a VarInt encoding a number):

[VID]/remote.php/dav/files/admin/[VID][VIN][VID][VIN][VID]"5a9000658388d"[VID][VIN][VID][VIN][VID][VIN][VID]00000009ocre5kavbk8j[VID][VIN]

Therefore, when working on a new architecture, I believe it could be good future-proofing to move away from WebDAV internally - and isolate everything related to WebDAV into a backend that performs the conversion between the native internals and WebDAV requests and responses.

With protocol-agnostic internals, a new backend providing access via new, more compact request and response formats would not be far away.

Sharing API

There's currently no way to detect changes to shares without requesting and comparing them in entirety.

It'd be great to have an ETag returned for Sharing API responses that only changes if the requested shares in the response change. And then, also support forIf-None-Match.

Change detection

In order to find changed items, it is currently necessary to check the ETag of the root folder, if it changed, fetch its contents, then find items with changed ETags among it … and repeat this until all changes have been discovered.

Some ideas on how this could be improved:

Sync Anchor

Delivered as part of the PROPFIND response. That SyncAnchor could then be used to make a subsequent request for a list of changes that occurred since that PROPFIND response. If a SyncAnchor is too old, a respective error would be returned.

Event subscription

Clients could subscribe to change events. Every change on the server would then generate an event and put it in the client's event queue.

While the client is connected, events could be delivered immediately (push).

If the client can't consume the events as fast as they are generated (f.ex. because it's not connected or only through a slow connection), and the number of events in the queue surpasses a limit (f.ex. 1000 events - or a time threshold), they get dropped and replaced with a single reload event, which would tell the client to perform old-style change detection by traversing the tree.

Alternatively, the subscription could expire at that point, and the client receive an error response when trying to resume the connection to the event subscription endpoint.

Metadata change propagation

Currently, not all changes to an item that are relevant to clients lead to a new ETag. Examples include a change in sharing of oc:favorite status.

For this reason, the new iOS app currently has to:

• always retrieve a full PROPFIND (depth 1) when the user navigates to a directory, to make sure sharing and favorite status of the items are up-to-date when presented to the user. The toll on bandwidth/CPU/memory/battery can be significant for folders with a lot of items. If all relevant changes were propagated to the ETag, a depth 0 PROPFIND would be sufficient to detect changes.
• poll the server for a list of favorites when the user enters a view presenting a list of all of them.
• poll the server for a full (=> also see Sharing API above) list of shares on a regular basis to keep file.

I understand that the ETag may, by definition and meaning, not be the right vehicle for propagating these changes. A MTag (or any other name) for tracking and propagating these metadata changes could also solve this issue.

File IDs / Local IDs

Implementing offline support in the new iOS app and SDK required finding a way to efficiently track an item from local generation to upload to file on the server.

Since the oc:id (File ID) of an item is generated by the server, it can't be known beforehand - and therefore can't be used at the beginning of the item's lifecycle.

The solution implemented in the new iOS SDK was to generate a unique ID locally and track the item by Local ID. This works well, but involves quite a bit of complexity to ensure that, once a file is on the server, it always get's the same Local ID attached.

A great simplification would be if it was possible for clients to provide a Client ID, which gets stored alongside the File ID and doesn't change for the lifetime of the item on the server.

And then, the option to have all APIs (especially PROPFIND and sharing) also include that Client ID in addition to the existing File ID in responses.

Client-generated IDs would be prefixed with an app-specific prefix, f.ex. ios:[UUID goes here].

In cases, where no Client ID was provided for an item, its Item ID would be returned instead.

Moved files

Moved files present a challenge as they often first get noticed by clients as missing from the folder they were previously in. Only to reappear in a different folder while changes are discovered.

In consequence, the client needs to keep the missing item around until it has finished change discovery and can't f.ex. remove local copies immediately.

A way to perform a PROPFIND on a File ID to determine its current whereabouts and status would be great to make this more efficient.

Atomic operations

Item operations – like uploads or folder creation – only return rudimentary information on the item operated upon.

While that information is sufficient to form subsequent requests, it's not sufficient for clients to generate a full-fledged local representation of that item with all metadata.

In consequence, after performing item operations, it's often necessary to perform a subsequent PROPFIND on the item to retrieve the entire set of metadata.

This is not a big issue, but costs (some) performance and there's - in theory - a small window between the operation finishing and the PROPFIND during which another client could replace, move or delete the item again.

If the response contained more metadata (requested f.ex. through a new HTTP request header listing the tags requested), the operation would be entirely atomic.

[labkode]

Hi @felix-schwarz, without going into the details yet of every topic you mentioned, have you considered to use GRPC natively instead of implementing "by hand" all the protocol logic that ownCloud currently uses? Being the transport HTTP/2 and backed by Protobufs the latency and payload size is reduced.

The GRPC SDK for the CS3APIs can be easily generated to ObjectiveC natively or to SWIFT by using https://github.com/grpc/grpc-swift.

From our point of view it would make sense that connection to future OCIS deployments will be done purely by GRPC and the connection to these current APIS (WebDAV, OCS) only for supporting the current ownCloud 10 deployments.

@butonic, @felixboehm what is your opinion on this topic?