Fix multipart upload from stream with metadata

[petslane]

With current [email protected] when uploading a large file from a stream using putObject() and providing metadata will fail with error:

(node:15496) UnhandledPromiseRejectionWarning: S3Error: Metadata cannot be specified in this context.
    at Object.parseError (/app/node_modules/minio/dist/main/xml-parsers.js:79:11)
    at /app/node_modules/minio/dist/main/transformers.js:156:22
    at DestroyableTransform._flush (/app/node_modules/minio/dist/main/transformers.js:80:10)
    at DestroyableTransform.prefinish (/app/node_modules/readable-stream/lib/_stream_transform.js:129:10)
    at DestroyableTransform.emit (events.js:315:20)
    at prefinish (/app/node_modules/readable-stream/lib/_stream_writable.js:611:14)
    at finishMaybe (/app/node_modules/readable-stream/lib/_stream_writable.js:620:5)
    at endWritable (/app/node_modules/readable-stream/lib/_stream_writable.js:643:3)
    at DestroyableTransform.Writable.end (/app/node_modules/readable-stream/lib/_stream_writable.js:571:22)
    at IncomingMessage.onend (_stream_readable.js:684:10)

Same issue as #743 but I think it fixed fPutObject(). this issue fixes putObject().

Example code:

import minioClient from './minio/client';
import { S3_BUCKET_NAME } from './util/config';
import { createReadStream } from 'fs';

(async () => {
  const stream = createReadStream('/home/petslane/Desktop/VID_20181224_203254.mp4');
  await minioClient().putObject(
    S3_BUCKET_NAME,
    'test.mp4',
    stream,
    {
      'my-metadata': 'data',
    },
  );
})();

$ ./mc stat xxx/bucket/test.mp4
Name      : test.mp4
Date      : 2021-08-11 15:36:42 EEST 
Size      : 623 MiB 
ETag      : 57497455a24df914d1f438a18f817a05-2 
Type      : file 
Metadata  :
  Content-Type          : binary/octet-stream 
  X-Amz-Meta-My-Metadata: data 

refs #743 #744

[prakashsvmx]

@petslane thank you for the contribution.
Please add a reproduction steps and test data. Also please update the functional tests.

[petslane]

Updated example code - copied the wrong example.

Updated functional test - added metadata to multipart upload.

[prakashsvmx]

@petslane What was the file size used?
is this error only with AWS S3? with the specific version of (minio-js) [email protected]?

Also please add minio-server version if used against minio-server.

because i checked the master branch with 1 MB, 700 MB, 1GB, 3 GB, 6GB files with meta data .did not get any error.

Test code

 minioClient = new Minio.Client({
  endPoint: 'localhost',
  port: 9000,
  useSSL: false,
  accessKey: 'minio',
  secretKey: 'minio123'
})


const bucketName ="sph-my-bucket"
const objectName = '6GB.zip'//6.4GB actual
const filePath = "/home/prakash/Documents/MinIO/DataCollection/6GB.zip"


function  putObject (buckName, objectName,cb) {

  const metaData = {
    'my-metadata': 'data',
  }
  const stream = fs.createReadStream(filePath)
  minioClient.putObject(
    buckName,
    objectName,
    stream,
    metaData,
    cb
  )
}


putObject(bucketName, objectName)

~ $ mc stat local/sph-my-bucket/
Name      : 6GB.zip
Date      : 2021-08-12 16:54:51 IST 
Size      : 6.0 GiB 
ETag      : df7e5ff97c36beef8592778c575effc4-12 
Type      : file 
Metadata  :
  X-Amz-Meta-My-Metadata: data 
  Content-Type          : binary/octet-stream 

[petslane]

The file size I used was 623 MiB (so it's a multipart upload, but you can configure partSize client option to some smaller size to test multipart upload with smaller files).

There is no problem with minio server as it just ignores these metadata headers for part upload requests, but AWS S3 and others will throw a "Metadata cannot be specified in this context" error. With multipart upload, metadata must be provided only with upload init requests, upload parts must be without metadata.

[prakashsvmx]

@petslane We will test/investigate this with S3. thank you for the details.

[prakashsvmx]

Tested with AWS S3. Changes look good to me. 👍