Writing to resumable GCS writestream throws errors

[zbjornson]

Environment details

• OS: Win 10
• Node.js version: 6.10.3
• npm version: 4
• google-cloud-node version: 1.1.1

Steps to reproduce

var gcs = require("@google-cloud/storage")({projectId: "..."});
var bucket = gcs.bucket("mybucket");
var file = bucket.file("test.txt");
var ws = file.createWriteStream();
// then either:
require("fs").createReadStream("somefile").pipe(ws);
// or
ws.write("some text"); ws.end();

throws these errors:

> TypeError: Cannot read property 'length' of undefined                                                        
    at Upload.Duplexify._forward (node_modules\duplexify\index.js:161:76)                     
    at DestroyableTransform.onreadable (node_modules\duplexify\index.js:126:10)               
    at emitNone (events.js:86:13)                                                                              
    at DestroyableTransform.emit (events.js:185:7)                                                             
    at emitReadable_ (node_modules\through2\node_modules\readable-stream\lib\_stream_readable.js:434:10)                                                                                                     
    at emitReadable (node_modules\through2\node_modules\readable-stream\lib\_stream_readable.js:428:65)                                                                                                      
    at readableAddChunk (node_modules\through2\node_modules\readable-stream\lib\_stream_readable.js:224:35)                                                                                                  
    at DestroyableTransform.Readable.push node_modules\through2\node_modules\readable-stream\lib\_stream_readable.js:176:10)                                                                                
    at DestroyableTransform.Transform.push (node_modules\through2\node_modules\readable-stream\lib\_stream_transform.js:123:32)                                                                              
    at afterTransform (node_modules\through2\node_modules\readable-stream\lib\_stream_transform.js:79:51)                                                                                                    
> TypeError: Cannot read property 'crc32c' of undefined                                                        
    at Duplexify.<anonymous> (node_modules\@google-cloud\storage\src\file.js:1017:27)         
    at emitNone (events.js:86:13)                                                                              
    at Duplexify.emit (events.js:185:7)                                                                        
    at Upload.<anonymous> (node_modules\@google-cloud\storage\src\file.js:1888:11)            
    at emitNone (events.js:91:20)                                                                              
    at Upload.emit (events.js:185:7)                                                                           
    at finishMaybe (node_modules\readable-stream\lib\_stream_writable.js:485:14)              
    at afterWrite (node_modules\readable-stream\lib\_stream_writable.js:367:3)                
    at onwrite (node_modules\readable-stream\lib\_stream_writable.js:358:9)                   
    at Upload.WritableState.onwrite (node_modules\readable-stream\lib\_stream_writable.js:122:5)                                                                                                             

The error does not happen if the write stream is created with {resumable: false}. This is such a simple case to be failing that I feel like I must be doing something wrong...

[stephenplusplus]

This should all come down to these lines in gcs-resumable-upload:

• gcs-resumable-upload: request stream is almost over. We stop the user's stream from finishing until we get the "complete" event by calling cork(): https://github.com/stephenplusplus/gcs-resumable-upload/blob/227726dd7c77f232aea0af3bc6b901c0f1c64146/index.js#L162

• gcs-resumable-upload: No error. Looks good. Emit metadata: https://github.com/stephenplusplus/gcs-resumable-upload/blob/227726dd7c77f232aea0af3bc6b901c0f1c64146/index.js#L170

• google-cloud/storage: Set the metadata on the instance: https://github.com/GoogleCloudPlatform/google-cloud-node/blob/f2267e6264f1f0b3dd9b6c8a87a91e046e2f428a/packages/storage/src/file.js#L1885

• gcs-resumable-upload: Uncork the user's stream, so it can get the finish event: https://github.com/stephenplusplus/gcs-resumable-upload/blob/227726dd7c77f232aea0af3bc6b901c0f1c64146/index.js#L173

• google-cloud/storage: Get the finish event, and emit "complete": https://github.com/GoogleCloudPlatform/google-cloud-node/blob/f2267e6264f1f0b3dd9b6c8a87a91e046e2f428a/packages/storage/src/file.js#L1888

• google-cloud/storage: PROBLEM LINE from your stack trace Use the crc32c properties from the metadata object that should have been already assigned: https://github.com/GoogleCloudPlatform/google-cloud-node/blob/f2267e6264f1f0b3dd9b6c8a87a91e046e2f428a/packages/storage/src/file.js#L1017

I'm a little stuck. I've tried to reproduce, but can't. Maybe with the outline above, you can think of where this can fail?

[zbjornson]

Thanks for digging through that. This seems to be a fatal combination of allowed dependency versions. In a fresh directory, if I npm install @google-cloud/storage and then npm install [email protected], I get the errors. Version 3.5.0 works, so I think the storage module's package.json just needs to be updated. It still allows duplexify back to 3.2.0. Does that repro the problem for you?

[stephenplusplus]

Sounds like this commit could be addressing the underlying problem, which was fixed in 3.4.4: mafintosh/duplexify@acf662f

Thanks for discovering that-- impressive! PR sent: #2385

[zbjornson]

Thanks for the fast fix! :)