insertMany: problems with errors & inserted documents handling

[NebSehemvi]

Do you want to request a feature or report a bug?
Bug, but maybe a feature
What is the current behavior?

I have Mongoose schema:

const QuestionType = new mongoose.Schema({
  code: { name: 'Type code', type: String, required: true, unique: true },
  text: { name: 'Description of type', type: String, required: true },
  questions: [{ type: mongoose.Types.ObjectId, ref: 'Question' }],
}, { timestamps: true });

And I have an API for inserting question types:

db[collection].insertMany(req.body, { ordered: false, rawResult: true }, (err, docs) => {
			if (err) return res.status(500).send({ ...handleError(err), writeErrors: err.writeErrors, docs: docs});
			if ('mongoose' in docs) {
				return res.status(500).json(docs);
			}
			return res.json(docs);			
});

And it's hard to describe in few sentences, what is wrong, because it seems to me that error handling is messed up.

For example, if my data is:
[{"code":"MEDIUM","text":"1111"},{"code":"test","text":"222"},{"code":"HARD","text":"2222"}]
Response is:
{"error":{"title":"BulkWriteError","code":11000,"message":"E11000 duplicate key error collection: game.questiontypes index: code_1 dup key: { code: \"MEDIUM\" }"},"writeErrors":[{"code":11000,"index":0,"errmsg":"E11000 duplicate key error collection: game.questiontypes index: code_1 dup key: { code: \"MEDIUM\" }","op":{"questions":[],"_id":"5eb19d28e646883e3459ce07","code":"MEDIUM","text":"1111","__v":0,"createdAt":"2020-05-05T17:06:48.409Z","updatedAt":"2020-05-05T17:06:48.409Z"}},{"code":11000,"index":2,"errmsg":"E11000 duplicate key error collection: game.questiontypes index: code_1 dup key: { code: \"HARD\" }","op":{"questions":[],"_id":"5eb19d28e646883e3459ce09","code":"HARD","text":"2222","__v":0,"createdAt":"2020-05-05T17:06:48.410Z","updatedAt":"2020-05-05T17:06:48.410Z"}}]}

Documents with MEDIUM and HARD code types exist, I will get errors for them, but I don't have any information about insterting test - no error, no data in docs callback parameter, but that one document was actually inserted.

Another option of strange behavior:
if my data is [{"code":"MEDIUM","text":"1111"},{"code":"test2","text":""}, I will get only one error - {"error":{"title":"BulkWriteError","code":11000,"message":"E11000 duplicate key error collection: game.questiontypes index: code_1 dup key: { code: \"MEDIUM\" }"}} though I expect for at least two errors - no text parameter for second array value was provided.

1. It looks like required fields validation runs only if every duplicate validation is correct
2. created documents aren't passed to the callback function if any 'unique' validation error has appeared for array of inserting documents.
3. WriteErrors appear on error object if number of errors is more than 1, though it would be nicer to place it if it's only one error either

If the current behavior is a bug, please provide the steps to reproduce.

1. To have an API something like:

app.post('/createMany/QuestionType', (req, res) => {
		db[collection].insertMany(req.body, { ordered: false, rawResult: true }, (err, docs) => {
			if (err) return res.status(500).send({ ...handleError(err), writeErrors: err.writeErrors, docs: docs });
			if ('mongoose' in docs) {
				return res.status(500).json(docs);
			}
			return res.json(docs);			
		});
});

2. Create at least 2 documents with it
3. Try to create next combinations:
   a) 1 duplicate, at least 1 new doc without required fields
   b) 2 duplicate, at least 1 new doc without required fields
   c) only new docs without required fields
   d) 1 duplicate, at least 1 new doc with required fields
4. Behold the apocalypse for validation & errors!

What is the expected behavior?

1. Unified error object? At least mongoose validation for required fields should always run and return errors for those documents which passes mongo check for unique validation but not passes other mongoose validations
2. Return of inserted documents should be correct (not the empty array, when in real case database is +n new valid documents)

What are the versions of Node.js, Mongoose and MongoDB you are using? Note that "latest" is not a version.

node: 13.1.0
mongoose: 5.9.11
mongodb: MongoDB 4.2.1 Community

[vkarpov15]

One important note to clarify: unique is not a validator. That has some important ramifications for how insertMany() error handling works: Mongoose executes all validators, and then sends the documents that pass validation to MongoDB, which is responsible for catching duplicates.

There are some improvements we can make here though:

1. You're write about writeErrors only appearing if there's more than 1 error. That's a problem that we should fix.
2. The ids of all inserted docs are available in err.result.result.insertedIds, so in order to get the documents that have been inserted, you need to go through that list and filter out any that errored out. We should add a property insertedDocs.

[NebSehemvi]

Thanks for the answer and the milestone!

I'll wait for release with insertMany() improvements, until then I'll stick with create().