Dynamic databases causing memory leak

[NajamShehzad]

Prerequisites

• I have written a descriptive issue title

Mongoose version

6.6.4

Node.js version

14.19.3

MongoDB version

5.0.13

Operating system

No response

Operating system version (i.e. 20.04, 11.3, 10)

Windows 11

Issue

I'm facing memory leak issue when using multiple databases
I have a use case in my application, for each user i'm create a separate databases
When user make the request I load its database and store its database connection in the server. First I was not saving the connection and calling useDb with each request but then it start eating more and more memory then I store the database connection in a map to reuse it.

The below function I use to get the database from the registeredDatabase Map

getCollection(_db: string, type: string){
  const database = this.registeredDatabase.get(_db);
  if (!database) {
    this.registerSingleDatabase(_db);
    const newDatabase = this.registeredDatabase.get(_db);
    return newDatabase.models[type];
  }
  return database.models[type];
}

And to register new database in the memory

registerSingleDatabase(_db: string) {
  this.registeredDatabase.set(_db, this.connection.useDb(_db, { useCache: true }));
  const currentDatabaseConnection = this.registeredDatabase.get(_db);
  if (currentDatabaseConnection) {
    Object.keys(TempModelName).forEach((key) => {
      currentDatabaseConnection.model(key, AllSchema[key]);
    });
  }
}

Is there any better way to handle dynamic databases with mongoose.
I try to debug and create multiple memory snapshots and it show Native connections that keeps inside the memory and Model objects that stay there even If I don't need to use them. Then eventually the Server dies because of low memory

[vkarpov15]

I can offer some high level suggestions, I'm not 100% certain how your app works, but I'll do my best to make some reasonable inferences.

Given that you're creating a new database per user, I can see why you would be experiencing constant memory growth. Connections stay open by default, you need to do a little extra work to allow GC to clean up unused connections automatically. You can use the following getConnection() function:

getCollection(_db: string, type: string){
  const database = this.connection.useDb(_db, {
    useCache: false, // <-- avoid caching to allow GC to clean up
    noListener: true // <-- makes `database` avoid registering event emitters on base connection, which allows GC to clean up
  })
  // Register models if necessary
  if (database.models[type] === undefined) {
    this.registerSingleDatabase(database);
  }
  return database.models[type];
}

Also, in order to allow GC to clean up unused connections, you should do database.deleteModel(/.*/) when you're done with database to remove any models. That should be sufficient to allow your database to get GC-ed.

We also advise using your own caching rather than useCache if you expect to have a large number of db handles - more than 10k or so. useCache works well when you're fine with keeping connections open for the entire server lifecycle, but if you're running out of memory and need to evict stale connections you shouldn't use useCache.

See following comments for more info:

• Add noListener option to useDb() to help memory usage in cases where the app is calling useDb() on every request #9961 (comment)
• Add noListener option to useDb() to help memory usage in cases where the app is calling useDb() on every request #9961 (comment)