Companion OneDrive documentation - No Session/Body attached #3880

Kieran-Bacon · 2022-07-12T12:28:39Z

Hi,

I am looking to add onedrive support to our companion server and found that there is very little documentation about showing how to set-up onedrive/companion, and even less for the errors you can get during the process.

I think it would be very helpful to have a single page that runs through the minimal required set-up of onedrive (and the other providers) to get things to work. Even if these are links to more general documentation by those providers on how to set them up, it would be invaluable.

My current experience has been a headache:

The documentation https://uppy.io/docs/companion/ simply suggests that a key and secret key should be provided to companion. The example Kubernetes setup https://github.com/transloadit/uppy/blob/main/packages/%40uppy/companion/KUBERNETES.md suspiciously doesn't provide the Onedrive environment variables, and I imagine that even if they were, it would be something cute along the lines of "YOUR ONEDRIVE KEY" so you'd have no idea what the key actually was meant to look like/how long etc...

After registering my companion app on onedrive as a web app (and worrying that they seem to refer to everything as ID rather than KEY), I added the redirect URL and created a secret whose secret ID and value I've used for the ONEDRIVE_KEY and ONEDRIVE_SECRET. I've gotten the following output while trying to connect.

companion: 2022-07-12T11:22:03.395Z [info] companion.client.version uppy client version @uppy/provider-views=2.1.2
companion: 2022-07-12T11:22:03.403Z [debug] null No session/body attached to req object. Exiting dispatcher.

And the only issue that comes up is transloadit/uppy-server#60 which suggests that is a requirement for another package expression-session. Unfortunately, I can't find anything anywhere describing what that is or why it is an issue, and this issue is very old.

I can see that uppy is making a request to the companion server at the following address: https://{my companion hostname}/onedrive/connect?state={TOKEN}&uppyVersions=%40uppy%2Fprovider-views%3D2.1.2

I can see in the token that only the site origin is being passed, and there is no payload to the get request being made so I am really unsure where to go from here. I am not sure whether this is a client-side or onedrive authentication issue since I've set it up according to the documentation, but, the documentation doesn't cover what the expected payloads should look like.

Issues like the below would likely really benefit from some guidance for provider set-up just so that can isolate the problem faster.

The text was updated successfully, but these errors were encountered:

Murderlon · 2022-07-12T12:40:37Z

Hi, I'm not sure what went wrong. But we're redoing the docs (for instance for Companion: transloadit/uppy.io#5) and I agree this should be better explained. Would be nice if you could create a bullet list of questions of things that are unclear so I can tackle them in the docs.

In the meantime, perhaps @mifi knows whats going on.

Kieran-Bacon · 2022-07-12T13:33:17Z

That is good to hear! I feel like there are lots of small gotchas that can be very frustrating.

In general, something spelt out like https://filemanagerpro.io/article/how-can-i-get-my-microsoft-account-client-id-and-client-secret-key/ would likely be a great start. I have done the steps in the link above so the paranoid sets that there must be something subtle in the default settings that I'm missing.

Ultimately:

Is the application ID == client id == key from companions perspective?
Over defining the redirect URL, are any other routes required? is a signout URL required? (which it says it wants in azure)
Are there permissions/roles that need to be defined?
For the Implicit grant and hybrid flows sections, should Companion be using either access token or id token or both or none?
What account types need to be given access in the Supported account types section? I have opened it as broadly as I could.

Kieran-Bacon · 2022-07-13T12:19:54Z

So using the application id and including the express_session package has worked and I can now go through the authentication path, giving access to the app to read and list one drive. So setting up onedrive it is as simple as the guide I posted, no need to turn on the access/id token. Not using the express_sessions was pretty silly on my behalf, this is something I missed in the plugging into an existing express server code snippet

I also found the following issue that takes through another persons experience and their solution, which could be useful in the documentation.

Connect onedrive return bad request #2314

Unfortunately, this hasn't completed resolved the connection issues, companion produced the following log line when one of us tried to upload.

companion: 2022-07-13T10:51:53.670Z [debug] null Did not receive access token for provider onedrive
companion: 2022-07-13T10:51:53.671Z [debug] callback.oauth.resp {
  error: {
    error: 'invalid_client',
    error_description: "AADSTS650051: Using application 'desc-uploader' is currently not supported for your organization ***  because it is in an unmanaged state. An administrator needs to claim ownership of the company by DNS validation of *** before the application desc-uploader can be provisioned.\r\n" +
      'Trace ID: 81fa011e-ce1c-4797-982a-aa18b3c71800\r\n' +
      'Correlation ID: 6e0dba8e-48f4-465f-96ea-4d0f84e08df8\r\n' +
      'Timestamp: 2022-07-13 10:51:53Z'
  }
}

I personally don't see these error messages when I try to upload from a personal account or the organisation account hosting the app.

Though I do get the following

companion: 2022-07-13T10:57:35.232Z [debug] null Instantiating uploader.
companion: 2022-07-13T10:57:35.233Z [debug] null Starting download stream.
companion: 2022-07-13T10:57:35.669Z [debug] null Waiting for socket connection before beginning remote download/upload.
companion: 2022-07-13T10:57:35.669Z [debug] db551332 uploader.socket.wait waiting for socket connection
companion: 2022-07-13T10:57:36.013Z [info] companion.client.version uppy client version 1.0.0
companion: 2022-07-13T10:58:35.671Z [error]  Error: Timed out waiting for socket connection
    at Timeout.<anonymous> (/app/node_modules/@uppy/companion/lib/server/Uploader.js:370:32)
    at listOnTimeout (node:internal/timers:557:17)
    at processTimers (node:internal/timers:500:7)

It is very unclear why the timeout is occurring. I have set the COMPANION_CLIENT_SOCKET_CONNECT_TIMEOUT to 600000 (as its not clear if its seconds or microseconds, and its not listed in the options in the documentation page) as suggested in the following issue but, it's had no effect and the upload doesn't seem to have started in the first place. The test files I am working with are small in size.

"Timed out waiting for socket" when attempting to upload multiple large files via Companion #3640

Kieran-Bacon · 2022-07-15T08:58:01Z

Hi @Murderlon, sorry to be a bother. I've not been able to make any progress uploading files from one drive - do you have any idea of something I could try to get it working?

Murderlon · 2022-07-15T10:24:35Z

Hi, sorry to hear this is taking so long. I personally never worked with OneDrive (I don't even have an account) and these errors are not clear to me either. @mifi generally is most knowledgeable here but he is off.

Everything works on our example page and the only thing we do is deploy this Dockerfile and set COMPANION_ONEDRIVE_DOMAIN_VALIDATION, COMPANION_ONEDRIVE_KEY, and COMPANION_ONEDRIVE_SECRET in Heroku. We don't manually set the timeout (which should be milliseconds btw).

Are you on latest versions of Uppy and Companion?

Murderlon · 2022-07-15T10:37:15Z

Just saw this isn't documented and I'm not sure why 🙈

uppy/packages/@uppy/companion/src/standalone/index.js

Lines 175 to 195 in 43fb673

    
           // WARNING: This route is added in order to validate your app with OneDrive. 
        
           // Only set COMPANION_ONEDRIVE_DOMAIN_VALIDATION if you are sure that you are setting the 
        
           // correct value for COMPANION_ONEDRIVE_KEY (i.e application ID). If there's a slightest possiblilty 
        
           // that you might have mixed the values for COMPANION_ONEDRIVE_KEY and COMPANION_ONEDRIVE_SECRET, 
        
           // please DO NOT set any value for COMPANION_ONEDRIVE_DOMAIN_VALIDATION 
        
           if (process.env.COMPANION_ONEDRIVE_DOMAIN_VALIDATION === 'true' && process.env.COMPANION_ONEDRIVE_KEY) { 
        
             router.get('/.well-known/microsoft-identity-association.json', (req, res) => { 
        
               const content = JSON.stringify({ 
        
                 associatedApplications: [ 
        
                   { applicationId: process.env.COMPANION_ONEDRIVE_KEY }, 
        
                 ], 
        
               }) 
        
               res.header('Content-Length', `${Buffer.byteLength(content, 'utf8')}`) 
        
               // use writeHead to prevent 'charset' from being appended 
        
               // eslint-disable-next-line max-len 
        
               // https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-configure-publisher-domain#to-select-a-verified-domain 
        
               res.writeHead(200, { 'Content-Type': 'application/json' }) 
        
               res.write(content) 
        
               res.end() 
        
             }) 
        
           }

Kieran-Bacon · 2022-07-15T10:44:14Z

Amazing! this looks like we are on for a winner!

per your last message, I am in the middle of re-installing and building the companion server and I will add this in as well. I will let you know how that gets on.

BTW
the package.json looks

{

  "dependencies": {
    "@uppy/companion": "^3.6.0",
    "express-session": "^1.17.3"
  },

  "devDependencies": {
    "@tsconfig/node16": "^1.0.3",
    "@types/node": "^18.0.0",
    "browserify": "^17.0.0",
    "terser": "^5.14.1",
    "typescript": "^4.7.4"
  },
  }

and the companion server looks


import express from 'express'
import bodyParser from 'body-parser'
import companion from '@uppy/companion'
import express_session from 'express-session'

const app = express()

// Companion requires body-parser and express-session middleware.
// You can add it like this if you use those throughout your app.
//
// If you are using something else in your app, you can add these
// middlewares in the same subpath as Companion instead.
app.use(bodyParser.json())
app.use(express_session({
    secret: process.env.COMPANION_SECRET,
    cookie: { maxAge: 43200 }

}))

// Construct the s3_prefix for the data
var s3_prefix = process.env.S3_DATA_PREFIX || ''
if (s3_prefix[s3_prefix.length-1] != '/'){
    s3_prefix = s3_prefix + '/'
}

console.log("S3 Prefix: ", s3_prefix)

const options = {
    secret: process.env.COMPANION_SECRET,
    filePath: process.env.COMPANION_DATADIR,
    server: {
        protocol: 'https',
        host: process.env.COMPANION_DOMAIN
    },
    sendSelfEndpoint: process.env.COMPANION_DOMAIN,
    uploadUrls: [...],
    allowLocalUrls: true,
    providerOptions: {
        s3: {
            key: process.env.COMPANION_AWS_KEY,
            secret: process.env.COMPANION_AWS_SECRET,
            bucket: process.env.COMPANION_AWS_BUCKET,
            region: process.env.COMPANION_AWS_REGION,
            useAccelerateEndpoint: false, // default: false,
            expires: 3600, // default: 300 (5 minutes)
            acl: process.env.COMPANION_AWS_ACL, // default: public-read
            getKey: (req, filename, metadata) => {
                console.log(req, filename, metadata)
                var filepath = `${s3_prefix}${metadata.site_id}/${metadata.camera_id}/${filename}`
                console.log("File path: ", filepath)
                return filepath
            }
        },
        onedrive: {
            key: process.env.COMPANION_ONEDRIVE_KEY,
            secret: process.env.COMPANION_ONEDRIVE_SECRET
        }
    }
}

console.log('Options: ', options)

app.use('/', companion.app(options))

const server = app.listen(3020)
companion.socket(server)

Kieran-Bacon · 2022-07-15T11:00:05Z

Unfortunately, that doesn't seem to have done the trick. I've logged in to pod to confirm that the settings are there. I am still getting the following

companion: 2022-07-15T10:55:28.024Z [info] companion.client.version uppy client version @uppy/companion-client=2.2.1
companion: 2022-07-15T10:55:28.191Z [debug] null Instantiating uploader.
companion: 2022-07-15T10:55:28.194Z [debug] null Starting download stream.
companion: 2022-07-15T10:55:28.618Z [debug] null Waiting for socket connection before beginning remote download/upload.
companion: 2022-07-15T10:55:28.618Z [debug] 24687a6e uploader.socket.wait waiting for socket connection
companion: 2022-07-15T10:55:28.711Z [info] companion.client.version uppy client version 1.0.0
companion: 2022-07-15T10:56:28.621Z [error]  Error: Timed out waiting for socket connection
    at Timeout.<anonymous> (/app/node_modules/@uppy/companion/lib/server/Uploader.js:370:32)
    at listOnTimeout (node:internal/timers:557:17)
    at processTimers (node:internal/timers:500:7)

I did notice that going to the endpoint referenced in the snippet returns
Cannot GET /.well-known/microsoft-identity-association.json

Potentially this is the issue? and the headers never get set as a consequence of the get method failing?

Murderlon · 2022-07-15T11:17:57Z

I did notice that going to the endpoint referenced in the snippet returns
Cannot GET /.well-known/microsoft-identity-association.json

Interesting, we should look into this.

companion: 2022-07-15T10:55:28.711Z [info] companion.client.version uppy client version 1.0.0

Are you on latest Uppy or on v1?

Kieran-Bacon · 2022-07-15T11:27:07Z

I believe I have version 3.6.0 though I understand how that 1.0.0 looks suspicious

from the package-lock.json

@uppy/companion 3.6.0
node_modules/@uppy/companion 3.6.0
node_modules/@uppy/companion/node_modules/cookie 0.4.0
node_modules/@uppy/companion/node_modules/depd 2.0.0
node_modules/@uppy/companion/node_modules/express-session 1.17.1
node_modules/@uppy/companion/node_modules/safe-buffer 5.2.0

Murderlon · 2022-07-15T11:41:07Z

That is @uppy/companion, can you post the version of @uppy/core and @uppy/onedrive

Kieran-Bacon · 2022-07-15T11:45:06Z

Oh, I didn't realise that I had to install them - I've only installed @uppy/companion and those packaged where all the packages with 'uppy' in their path.

My bad - will install them now

Kieran-Bacon · 2022-07-15T11:58:05Z

sorry to report that the problem persists

"@uppy/companion": {
      "version": "3.7.0",
      "resolved": "https://registry.npmjs.org/@uppy/companion/-/companion-3.7.0.tgz",
      "integrity": "sha512-s2Lwlkd8VSiQcY880322Mfk9CJ6tylj9s/BIj7ZqYgP1R1uqAJuQ/3soflMmngrONRIgGje/2vVsWcoWj4Gz7Q==",
      "requires": {
        "@purest/providers": "1.0.1",
        "atob": "2.1.2",
        "aws-sdk": "^2.1038.0",
--
    "@uppy/companion-client": {
      "version": "2.2.1",
      "resolved": "https://registry.npmjs.org/@uppy/companion-client/-/companion-client-2.2.1.tgz",
      "integrity": "sha512-Y3E10NJLMfp/wjgthNhx3gJtT67fzFCPNPFwpNNRs5iJsW6PANhJ420eyMUFzfmEZ56ZzGYxr5pzJZx8YxHICQ==",
      "requires": {
        "@uppy/utils": "^4.1.0",
        "namespace-emitter": "^2.0.1"
      }
    },
    "@uppy/core": {
      "version": "2.3.1",
      "resolved": "https://registry.npmjs.org/@uppy/core/-/core-2.3.1.tgz",
      "integrity": "sha512-KV04X7ueYbYX1p37/i3QsoQSw8IDP8Yb+Bh9KNN0X2Vcun6K2VnNjhVtPmPXtyjDZooK7lVIqhRX8TZWcSfgSQ==",
      "requires": {
        "@transloadit/prettier-bytes": "0.0.7",
        "@uppy/store-default": "^2.1.0",
        "@uppy/utils": "^4.1.0",
        "lodash.throttle": "^4.1.1",
        "mime-match": "^1.0.2",
        "namespace-emitter": "^2.0.1",
        "nanoid": "^3.1.25",
        "preact": "^10.5.13"
--
    "@uppy/onedrive": {
      "version": "2.1.1",
      "resolved": "https://registry.npmjs.org/@uppy/onedrive/-/onedrive-2.1.1.tgz",
      "integrity": "sha512-2XMP3/v82raymJMRMIfGXamdnEC2zMDqMGN+VHNHwZ/emup0tWZHvf/QmtA0RvoChIBs21uKeuQsO4xmHFRoRQ==",
      "requires": {
        "@uppy/companion-client": "^2.2.0",
        "@uppy/provider-views": "^2.1.1",
        "@uppy/utils": "^4.1.0",
        "preact": "^10.5.13"
      }
    },

Murderlon · 2022-07-15T12:09:29Z

Can you confirm it doesn't work when you try to reach you Companion instance in this setup?

https://codesandbox.io/s/uppy-dashboard-xpxuhd?file=/src/index.js

Kieran-Bacon · 2022-07-15T12:26:19Z

I've given it a go with no luck

const UPLOADER = "s3-multipart";
const COMPANION_URL = "https://companion.desc.dev.ngenius.ai";

companion: 2022-07-15T12:22:26.841Z [info] companion.client.version uppy client version @uppy/companion-client=2.2.1
companion: 2022-07-15T12:22:27.004Z [debug] null Instantiating uploader.
companion: 2022-07-15T12:22:27.005Z [debug] null Starting download stream.
companion: 2022-07-15T12:22:27.518Z [debug] null Waiting for socket connection before beginning remote download/upload.
companion: 2022-07-15T12:22:27.519Z [debug] 7f808863 uploader.socket.wait waiting for socket connection
companion: 2022-07-15T12:22:27.606Z [info] companion.client.version uppy client version 1.0.0
companion: 2022-07-15T12:23:27.518Z [error]  Error: Timed out waiting for socket connection
    at Timeout.<anonymous> (/app/node_modules/@uppy/companion/lib/server/Uploader.js:370:32)
    at listOnTimeout (node:internal/timers:557:17)
    at processTimers (node:internal/timers:500:7)

mifi · 2022-07-16T04:25:11Z

Hi!

Are you running a single instance of companion or multiple?
Are you using something in front, like a reverse proxy?
Do you see any websocket request being sent from your browser in your developer tools when you start the upload? is it sending to the correct host/post (the companion server's?)
Do other providers like for example google drive work?

Could you try to create a new project just with companion and one-drive.

or checkout the uppy repo, and then try to run it in development mode, to see if that works. Follow steps from here:

https://uppy.io/docs/contributing.html

then create a .env with the needed content (see .env.example), then run:

yarn run dev:with-companion

This is just to make sure that it fails with the same error without any other variables, and to make sure all packages are up to date.

Kieran-Bacon · 2022-07-18T10:31:24Z

Hi,

Companion is running as a Kubernetes deployment with a single pod on EKS.
The deployment is sitting behind a classic load balancer (who is responsible for handling the SSL certificate)
When the upload is 'started', I can see two requests being made to the companion server address. The requests are: a 200 successful POST+preflight with the following data

Query Parameter

driveId: b!GLuy5kLKGUuZCP_5jwsaJbdC-jgYs9VFiJAq0tui--D_eSliqKaDQ6byKbTE999H

Payload

{"fileId":"01ZJEIQDESANF6PWMNTNCILRP2UJBD6Z5G","protocol":"s3-multipart","size":128,"metadata":{"name":"details.txt","type":"text/plain","organisation_id":"***","site_id":"***","camera_id":"***"}}

Response

{"token":"81bcc1f4-f3bf-4902-8c10-5aa1e7d81980"}

And then a 204 Options request with just the query data
Query Parameter

driveId: b!GLuy5kLKGUuZCP_5jwsaJbdC-jgYs9VFiJAq0tui--D_eSliqKaDQ6byKbTE999H

I have not yet tried the other providers, local uploads are working correctly.

I will try creating a new project and giving it a go, though I will not likely be today sorry.

mifi · 2022-07-24T06:29:08Z

Thanks. what about the websocket though?

In your companion log, after this line:

companion: 2022-07-24T06:25:37.477Z [debug] e8274265 uploader.socket.wait waiting for socket connection

...you're supposed to see:

companion: 2022-07-24T06:25:37.520Z [info] socket.connect connection received from e8274265-1ec4-4a94-a416-0b8da8f7769e
companion: 2022-07-24T06:25:37.521Z [debug] e8274265 uploader.socket.wait socket connection received

so it would be interesting to see whether the browser sends the websocket request and what happens to it. I suspect that it gets lost in the proxy.

in chrome after downloading a file it's supposed to look like this:

(the first request is the websocket connection)

Kieran-Bacon · 2022-09-21T13:38:23Z

Hi @mifi

I'm very sorry for not getting back to you - I haven't been able to focus until this week.

You were bang on the money - the websocket was not being established because the AWS classic ELB doesn't support websockets. Once updating the cluster to use Network load balancers everything started to work correctly.

Sorry again for the delay and thanks for your help!

Kieran-Bacon added Feature Triage labels Jul 12, 2022

Murderlon assigned mifi Jul 12, 2022

Murderlon added Question Companion The auth server (for Instagram, GDrive, etc) and upload proxy (for S3) 📖 Docs and removed Feature Triage labels Jul 12, 2022

Kieran-Bacon closed this as completed Sep 21, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Companion OneDrive documentation - No Session/Body attached #3880

Companion OneDrive documentation - No Session/Body attached #3880

Kieran-Bacon commented Jul 12, 2022

Murderlon commented Jul 12, 2022 •

edited

Loading

Kieran-Bacon commented Jul 12, 2022

Kieran-Bacon commented Jul 13, 2022

Kieran-Bacon commented Jul 15, 2022

Murderlon commented Jul 15, 2022 •

edited

Loading

Murderlon commented Jul 15, 2022

Kieran-Bacon commented Jul 15, 2022

Kieran-Bacon commented Jul 15, 2022 •

edited

Loading

Murderlon commented Jul 15, 2022

Kieran-Bacon commented Jul 15, 2022

Murderlon commented Jul 15, 2022

Kieran-Bacon commented Jul 15, 2022

Kieran-Bacon commented Jul 15, 2022

Murderlon commented Jul 15, 2022

Kieran-Bacon commented Jul 15, 2022 •

edited

Loading

mifi commented Jul 16, 2022

Kieran-Bacon commented Jul 18, 2022

mifi commented Jul 24, 2022

Kieran-Bacon commented Sep 21, 2022

Companion OneDrive documentation - No Session/Body attached #3880

Companion OneDrive documentation - No Session/Body attached #3880

Comments

Kieran-Bacon commented Jul 12, 2022

Murderlon commented Jul 12, 2022 • edited Loading

Kieran-Bacon commented Jul 12, 2022

Kieran-Bacon commented Jul 13, 2022

Kieran-Bacon commented Jul 15, 2022

Murderlon commented Jul 15, 2022 • edited Loading

Murderlon commented Jul 15, 2022

Kieran-Bacon commented Jul 15, 2022

Kieran-Bacon commented Jul 15, 2022 • edited Loading

Murderlon commented Jul 15, 2022

Kieran-Bacon commented Jul 15, 2022

Murderlon commented Jul 15, 2022

Kieran-Bacon commented Jul 15, 2022

Kieran-Bacon commented Jul 15, 2022

Murderlon commented Jul 15, 2022

Kieran-Bacon commented Jul 15, 2022 • edited Loading

mifi commented Jul 16, 2022

Kieran-Bacon commented Jul 18, 2022

mifi commented Jul 24, 2022

Kieran-Bacon commented Sep 21, 2022

Murderlon commented Jul 12, 2022 •

edited

Loading

Murderlon commented Jul 15, 2022 •

edited

Loading

Kieran-Bacon commented Jul 15, 2022 •

edited

Loading

Kieran-Bacon commented Jul 15, 2022 •

edited

Loading