Support for Direct File Upload via Presigned URLs/SAS Tokens

[mandm-pt]

Currently, when using the KM service, file uploads must be handled through a secure backend service or app. It is not feasible to upload files directly from a frontend or a publicly published app because API keys cannot be securely stored in public code.

To work around this, files must be uploaded to the backend, which then uploads them to the KM service. However, platforms like AWS and Azure Blob Storage offer Presigned URLs and SAS Tokens, allowing files to be securely uploaded directly to storage without routing through the backend.

Proposed Solution: Introduce a feature that enables direct file uploads to storage (e.g., AWS S3 or Azure Blobs) via presigned URLs or similar mechanisms. The workflow would be:

Retrieve a URL for direct upload.
Upload the file contents to the storage directly from the frontend.
Notify the KM service that the file is available for processing.

Checking the code I see 4 implementation of IDocumentStorage. Follows the way I see it working for 3 of them, for Mongo I am not sure how we could make it working.

• AzureBlobsStorage - Using GenerateSasUri from BlobContainerClient and return back the url
• SimpleFileStorage - Return back the directoyy path to upload the file
• AWSS3Storage - Using method GetPreSignedURL from AmazonS3Client and return back the url
• MongoDbAtlasStorage - Not sure what could be the solution. Perhaps creating an empty document and return the ID ? (But it doesn't make much sense since this storage doesn't support what I am proposing)

This feature would simplify the upload process, improve security, and reduce backend overhead by enabling direct uploads from the client-side while ensuring that KM is aware when a file is ready for processing.

[dluc]

hi @mandm-pt did you try the ImportWebPageAsync API, passing a link to the file, including SAS tokens?

graph TD
    A[Frontend] -->|send file to| B[Your backend]
    B -->|upload file to| C[Azure Blob]
    C -->|get Blob URL and SAS token from| D[Your backend]
    D -->|send Blob URL and SAS token to| E[KM]
    E -->|download file from| C
    E -->|extract content from file| F[KM]

Loading

[mandm-pt]

Hi @dluc thank you for your quick reply.
I tried the upload via URL. Actually I started to modify the code locally and then I noticed that method and tried. It does work fine but I end up with a new file "content.url" with the URL to the file, and the actual file beside.

That reflects on the RelevantSources property of KM response. In the SourceUrl, instead of the link to download the file and telling me that the answer comes from a PDF (for example), it gives me the previously SAS token used to upload the file (which should be invalid, depending how is generated).

Basically this work around works from the point of view of getting the content of the file processed, but the file that was directly uploaded to the storage is kind of disconnected and is treated as any regular URL existing on the internet and not like an uploaded file.

[mandm-pt]

Forgot to add the diagram with the steps that I am proposing.
Has the number to show the order of steps

graph TD
    A[Frontend] -->|1 - Get link to upload file| B[Your backend]
    A -->|2 - upload file to| C[Azure Blob]
    A -->|3 - Acknoledge file upload| B[Your backend]
    B -->|4 - Tells KM to start processing file| D[KM]
    C -->|5 - extract content from file that already exists in storage| D[KM]

Loading

Step 5 implies downloading the file, but assumes the upload was performed by KM and not a publicly accessible file

[dluc]

The important thing is that the upload works with the existing API.

The problem seems to be only with SourceUrl, ie about downloading the file.

Is that something you could handle in the backend replacing the SAS token with a new one?

[mandm-pt]

Well, as I said functionality works but it seems you are processing a file that is not yours (in my example, uploading from a frontend app).
Whatever solution I can make up, it will always be a bit hacky in my opinion, dealing with SAS token replaces. A quick and more robust solution might be, adding an extra param on the ImportWebPageAsync method, to indicate if we want to download the remote file and store it ourselves. With that you:

• Guarantee that you have always access to original content
• In case of files, you can pass the RelevantSources references, if you store the a copy of the original file

In the scenario that I was referring maybe the disadvantage would be ending up with 2 copies of the same file in storage, or downloading and overriding the file that you just downloaded (since the URL points to the same place where you will be saving the copy).