Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mongoose 6.2.3 breaks behaviour used since 5.12.2 #11445

Closed
SK-FComputer opened this issue Feb 23, 2022 · 6 comments
Closed

Mongoose 6.2.3 breaks behaviour used since 5.12.2 #11445

SK-FComputer opened this issue Feb 23, 2022 · 6 comments
Labels
confirmed-bug We've confirmed this is a bug in Mongoose and will fix it.
Milestone

Comments

@SK-FComputer
Copy link

Do you want to request a feature or report a bug?
bug

What is the current behavior?

If the current behavior is a bug, please provide the steps to reproduce.
After updating to typegoose 9.7.0 and mongoose 6.2.3 (Works fine in 9.6.0 and 6.2.0) i get the error:

Error [MongoNotConnectedError]: MongoClient must be connected to perform this operation
    at getTopology (D:\project\node_modules\mongoose\node_modules\mongodb\src\utils.ts:451:9)
    at Db.createCollection (D:\project\node_modules\mongoose\node_modules\mongodb\src\db.ts:261:18)
    at NativeConnection.createCollection (D:\project\node_modules\mongoose\lib\connection.js:404:11)
    at NativeConnection.Connection.onOpen (D:\project\node_modules\mongoose\lib\connection.js:635:10)
    at NativeConnection.wireup (D:\project\node_modules\mongoose\lib\drivers\node-mongodb-native\connection.js:105:13)
    at Object.onceWrapper (events.js:519:28)
    at NativeConnection.emit (events.js:400:28)
    at NativeConnection.emit (domain.js:470:12)
    at NativeConnection.set (D:\project\node_modules\mongoose\lib\connection.js:126:12)
    at NativeConnection.set (D:\project\node_modules\mongoose\lib\connection.js:119:23)

I have a master process that starts workers with the library https://threads.js.org/.
Before updating i had to create a new connection (DB class) In the child worker, such that i could use the typegoose model functions.

Code Example

DB (other DB connections here too, but truncated for example):

import { connect } from 'mongoose';
import { config } from './config';


export default class DB {
	constructor(public entities: any[] = []) {
		config;
	}

	async init(
		createMongoDB: boolean = true,
		debug: boolean = false
	) {
		try {
			if (createMongoDB ?? true) {
				await connect(process.env.DB_MONGO_CONN);
				console.log('MongoDB Connected');
			}
		} catch (error) {
			console.error(error);
			throw error;
		}
	}
}

Master process:

const worker = await spawn(
			new Worker(workerPath, {
				resourceLimits: {
					maxOldGenerationSizeMb: 32_768,
				},
			}),
			{ timeout: 30_000 }
		);
const workerResult = await worker(data);

Child worker

import { connection } from 'mongoose';
import { expose } from 'threads/worker';
import DB from '../../database';
import { ITables } from '../../models';
import { ProductModel } from '../../odm';

async function processData(data: ITables['LAGKART']['data']) {
	try {
		await new DB().init(true);
		const updates = [];
		for (const val of data) {
			if (
				val.SYS_CHANGE_OPERATION === 'I' ||
				val.SYS_CHANGE_OPERATION === 'U'
			) {
				updates.push({
					updateOne: {
						filter: { lxbenummer: val.LXBENUMMER },
						update: val.data,
						upsert: true,
					},
				});
			} else {
				updates.push({
					deleteOne: {
						filter: { lxbenummer: val.LXBENUMMER },
					},
				});
			}
		}
		const update = await ProductModel.bulkWrite(updates);

		if (update.result.writeConcernErrors.length) {
			throw update.getWriteConcernError();
		} else if (update.result.writeErrors.length) {
			throw update.getWriteErrors();
		}
	} catch (error) {
		throw error;
	} finally {
		try {
			await connection.close();
		} catch (error) {
			console.error('Connection already closed!');
		}
	}
}

expose(processData);

ProductModel

import { getModelForClass, prop } from '@typegoose/typegoose';
import { connection, Types } from 'mongoose';
class Product {
	_id: Types.ObjectId;

	@prop({ required: true, select: false })
	dataset: string;
}

const productDb = connection.useDb('dbName', { useCache: true });

const ProductModel = getModelForClass(Product, {
	existingConnection: productDb,
});

export { Product, ProductModel };

tsconfig.json

{
	"compilerOptions": {
		"baseUrl": "./src",
		"outDir": "./dist",
		"allowJs": false,
		"sourceMap": true,
		"moduleResolution": "node",
		"removeComments": true,
		"target": "ES2020",
		"module": "CommonJS",
		"experimentalDecorators": true,
		"emitDecoratorMetadata": true,
		"noImplicitAny": true,
		"esModuleInterop": true
	},
	"include": ["./src/**/*"],
	"exclude": ["node_modules", "**/node_modules/*"]
}

What is the expected behavior?
I expect it to not throw an error. Worked since typegoose 8.0.0-beta.2 and mongoose 5.12.2. Only with latest versions this breaks,

I suspect it has to do with the import "order" since:
After trial and error i can see that using dynamic import instead of import { ProductModel } from '../../odm'; i don't get this error message, but this hasn't been a problem since start of project.
Last version of typegoose + mongoose breaks this behaviour and i can't seem to find out why.

What are the versions of Node.js, Mongoose and MongoDB you are using? Note that "latest" is not a version.

  • System: linux / windows
  • NodeJS: 14.17.6
  • Typescript: 4.5.5
  • Compiler / Transpiler: tsc / ts-node
  • Typegoose(NPM): 9.7.0
  • Typegoose(GIT): commithash
  • mongoose: 6.2.3
  • mongodb: 4.2.7
@SK-FComputer
Copy link
Author

Probably occurs because of this

So i guess as @hasezoey pointed out here (typegoose/typegoose#672 (comment)), there has been an update to useDb (ad01d5e).

Why this breaks old behaviour i don't understand, and my current only solution is using dynamic imports. This is not feasible in all situations..

@SK-FComputer
Copy link
Author

SK-FComputer commented Feb 24, 2022

IS NOT A TYPEGOOSE ISSUE, tested with [email protected] and mongoose 6.2.1 and confirmed to work.
Only upgrade of mongoose breaks this behaviour.

Minimal reproduceable example:
https://github.com/SK-FComputer/mongoose-useDb-error

In src/worker.ts if DB() init is NOT used:

MongoDB Connected
Master update:  {
  title: 'Master',
  productID: 1,
  _id: new ObjectId("62175c1a6139f021165f602e"),
  __v: 0
}
Error [MongooseError]: Operation `products.insertOne()` buffering timed out after 10000ms
    at Timeout.<anonymous> (C:\Users\User\Desktop\mongoose-error\node_modules\mongoose\lib\drivers\node-mongodb-native\collection.js:151:23)
    at listOnTimeout (internal/timers.js:557:17)
    at processTimers (internal/timers.js:500:7)
Closing MongoDB connection master..

In src/worker.ts if DB() init is used:

MongoDB Connected
Master update:  {
  title: 'Master',
  productID: 1,
  _id: new ObjectId("62175804f97d81c206ce2ab3"),
  __v: 0
}
MongoNotConnectedError: MongoClient must be connected to perform this operation
    at getTopology (C:\Users\User\Desktop\mongoose-error\node_modules\mongodb\src\utils.ts:451:9)
    at Db.createCollection (C:\Users\User\Desktop\mongoose-error\node_modules\mongodb\src\db.ts:261:18)
    at NativeConnection.createCollection (C:\Users\User\Desktop\mongoose-error\node_modules\mongoose\lib\connection.js:404:11)
    at NativeConnection.Connection.onOpen (C:\Users\User\Desktop\mongoose-error\node_modules\mongoose\lib\connection.js:635:10)
    at NativeConnection.wireup (C:\Users\User\Desktop\mongoose-error\node_modules\mongoose\lib\drivers\node-mongodb-native\connection.js:105:13)
    at Object.onceWrapper (events.js:519:28)
    at NativeConnection.emit (events.js:400:28)
    at NativeConnection.emit (domain.js:470:12)
    at NativeConnection.set (C:\Users\User\Desktop\mongoose-error\node_modules\mongoose\lib\connection.js:126:12)
    at NativeConnection.set (C:\Users\User\Desktop\mongoose-error\node_modules\mongoose\lib\connection.js:119:23)

@IslandRhythms IslandRhythms added the needs clarification This issue doesn't have enough information to be actionable. Close after 14 days of inactivity label Feb 25, 2022
@IslandRhythms
Copy link
Collaborator

What file do I need to run? sync-data or worker?

@SK-FComputer
Copy link
Author

@IslandRhythms sync-data.ts, should see output that it can write from master but not from worker
Also removing the DB().init() in worker, won't use connection from master.

@IslandRhythms IslandRhythms added confirmed-bug We've confirmed this is a bug in Mongoose and will fix it. and removed needs clarification This issue doesn't have enough information to be actionable. Close after 14 days of inactivity labels Feb 28, 2022
@vkarpov15 vkarpov15 modified the milestones: 6.2.6, 6.2.7 Mar 6, 2022
@vkarpov15
Copy link
Collaborator

Fix will be in v6.2.7. We haven't been able to come up with a test case that reproduces this issue in our test suite, but I confirmed that 5bdd836 fixes the issue in your repo 👍

@SK-FComputer
Copy link
Author

Thank you very much! @vkarpov15

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
confirmed-bug We've confirmed this is a bug in Mongoose and will fix it.
Projects
None yet
Development

No branches or pull requests

3 participants