how to have multiple stores under the same db ?

[ctf0]

atm i have something like

import { Store, set, get, del, clear } from 'idb-keyval'
const cacheStore = new Store(
    'Media_Manager', // db
    'cacheStore' // store
)
const infoStore = new Store(
    'Media_Manager', // db
    'infoStore' // store
)

but i keep getting the error

Uncaught (in promise) DOMException: Failed to execute 'transaction' on 'IDBDatabase': One of the specified object stores was not found.

and it points to

_withIDBStore(type, callback) {
        return this._dbp.then(db => new Promise((resolve, reject) => {
            const transaction = db.transaction(this.storeName, type); // <<< this line <<<
            transaction.oncomplete = () => resolve();
            transaction.onabort = transaction.onerror = () => reject(transaction.error);
            callback(transaction.objectStore(this.storeName));
        }));
    }

so it that possible or do i've to create new db for each store ?

[jakearchibald]

Ah yes. I messed this up. Will come up with a way to fix this.

[jakearchibald]

Right now, yes, you need to create a db for each store. That wasn't my intention.

[ctf0]

yeah its okay i understand, i found that the api is kinda a mess as u need to use an upgrade something function to add new object stores under the same db 😞

[rodgobbi]

@jakearchibald I used your other module, idb, to implement this behavior of multiple keyval stores inside a single DB, but I'm planning soon to leverage the idb-keyval implementation for that because it is much smaller and simpler. After reading this issue, I gotta ask, are you already working on that? If not, can I implement and open a PR?

[jakearchibald]

@rodgobbi please do! I had a stab at this in https://github.com/jakearchibald/idb-keyval/compare/issue-31, but Edge is either violating the event loop, or there's something I'm not understanding.

[rodgobbi]

@jakearchibald nice, I'll gladly move forward this lib. I'll spend time on it asap.
For the implementation which I have in mind, for me it's easier and much less verbose to pass an array of store names as arg and return the corresponding stores with the methods to access their data, thus you can access each store just like an async localStorage. So I'll implement towards this direction. Any thoughts are welcome.

[jakearchibald]

@rodgobbi I'm not sure I understand the proposal. Could you sketch out the usage of the API?

[rodgobbi]

That would be function idbFactory(dbName: string, storeNames: string[], dbVersion: number): Store[].
And Store would have get, set, del, clear and keys bound to the DB-Store returned from the factory.
You would use like:

const [store1, store2] = idbFactory('db1', ['store1', 'store2' ], 1);
store1.set(...); // and so on

[jakearchibald]

Hmm, why not just:

const store1 = new Store('db1', 'store1');
const store2 = new Store('db1', 'store2');

[rodgobbi]

Already tried that, the main problem that I found was that you need to use a greater version of the previous existing db to upgrade that way. So it needs to be like:

const store1 = new Store('db1', 'store1', 1);
const store2 = new Store('db1', 'store2', 2);

At a first glance it's ok, but keeping track of this version across a project can be dangerous, for example: someone may increment the first version number to 2 and forget about the second one, which will lead to the store2 not being created, mainly if this second store is defined somewhere else rather than close to the first one.
After analyzing these possible outcomes I wondered that indexedDP API kind of implicitly forces you to define the desired shape of your DB at one place.
These are my thoughts.
As a plus, currently in my project, with the DB definition in one place like I showed, the factory can erase previous unused stores, using passed in store names.

[jakearchibald]

the main problem that I found was that you need to use a greater version of the previous existing db to upgrade that way

I think your proposal has the same problem:

const [store1, store2] = idbFactory('db1', ['store1', 'store2' ], 1);
// Then somewhere in unrelated code:
const [books, albums] = idbFactory('db1', ['books', 'albums' ], 1);

I don't think this project can assume that a single db is defined in a single place.

The fix I wrote in https://github.com/jakearchibald/idb-keyval/compare/issue-31 should work according to the spec (but doesn't in Edge), but it is hacky. It could be that IDB isn't suited to what I'm trying to do here.

[rodgobbi]

You are right about the same problem.
One point that I actually remember now which made me implement this way is being able to define multiple stores in a row, without the need to increment the version. The other one is just to bind the methods to the store name, to write less code.
Anyway, I saw there in the solution that you increment the version automatically, which solves the main problem that I was having. I'll try to help on that solution.

[ctf0]

isnt it possible to check if a previous store were found under the same name, and if yes then get the version and increment it ?
so ex

const cacheStore = new Store(
    'Media_Manager', // db
    'cacheStore' // store
)

// b4 creating the new store, get the db name
// check if we already have something with that name
// if yes get its number and ++ for the new store
const infoStore = new Store(
    'Media_Manager', // db
    'infoStore' // store
)

[jakearchibald]

@ctf0 seems like you're talking about https://github.com/jakearchibald/idb-keyval/compare/issue-31.

[ctf0]

@jakearchibald any updates ?

[jakearchibald]

I haven't had time to dig into this further. Any ideas?

[ctf0]

@jakearchibald sorry, this is kinda over my head :(

[goofballLogic]

damn. just ran into this. wish I'd realised sooner.

[goofballLogic]

If I understand correctly, a possible algorithm for this might be:

1. new Store(dbName, storeName)
2. indexedDB.open(dbName)
3. Obtain version and objectStoreNames from opened database
4. if storeName is not in objectStoreNames, reopen the database with version version + 1
5. onupgradeneeded will automatically create the needed store

I'll put together a PR as an example

[b4stien]

Hey, there is a PR doing this already, still posting this here if it can be of any help. We have a custom implem to solve this problem which reads like:

import isEqual from "lodash/isEqual";
import difference from "lodash/difference";

/**
 * Custom idb-keyval store to enable multiple stores per IndexedDB
 * database and auto-reconnect when DB has closed.
 *
 * Slightly different solution than
 * https://github.com/jakearchibald/idb-keyval/pull/50 for the
 * auto-reconnect feature.
 */
export class CustomStore {
  readonly _dbName: string;
  readonly _storeName: string;
  readonly _knownEntityStores: string[];
  _openedDb: Promise<IDBDatabase> | null;

  // Type compat with idb-keyval `Store`
  readonly storeName: string = "UNUSED";
  readonly _dbp: Promise<IDBDatabase> = new Promise(() => {});

  constructor(dbName: string, storeName: string, knownEntityStores: string[]) {
    this._dbName = dbName;
    this._storeName = storeName;
    this._knownEntityStores = knownEntityStores;
    this._openedDb = null;
  }

  _makeDbPromise(): Promise<IDBDatabase> {
    if (this._openedDb === null) {
      this._openedDb = new Promise<IDBDatabase>((resolve, reject) => {
        const openreq = indexedDB.open(this._dbName);
        openreq.onerror = () => reject(openreq.error);
        openreq.onsuccess = () => resolve(openreq.result);
      }).then(openreqResult => {
        const currentStoreNames = new Set(Array.from(openreqResult.objectStoreNames));
        const wantedStoreNames = new Set(this._knownEntityStores);

        // That's the main part that allows us to have multiple store per
        // database.
        // While opening the database we compare the stores we have
        // inside with the stores we want inside an create/drop the
        // diff.
        if (!isEqual(wantedStoreNames, currentStoreNames)) {
          // We will upgrade our DB to this "next version"
          const nextVersion = openreqResult.version + 1;

          // We need to reopen the database with `nextVersion` in order
          // to hook ourselves in `onupgradeneeded`.
          openreqResult.close();

          return new Promise<IDBDatabase>((resolveNewDbp, rejectNewDbp) => {
            const newOpenreq = indexedDB.open(this._dbName, nextVersion);
            newOpenreq.onerror = () => rejectNewDbp(newOpenreq.error);
            newOpenreq.onsuccess = () => {
              // When the connection closes, we remove our ref to
              // `this._openDb`.
              newOpenreq.result.onclose = () => {
                this._openedDb = null;
              };
              resolveNewDbp(newOpenreq.result);
            };

            newOpenreq.onupgradeneeded = () => {
              difference(
                Array.from(currentStoreNames),
                Array.from(wantedStoreNames)
              ).forEach(name => newOpenreq.result.deleteObjectStore(name));
              difference(
                Array.from(wantedStoreNames),
                Array.from(currentStoreNames)
              ).forEach(name => newOpenreq.result.createObjectStore(name));
            };
          });
        }

        // Same as before, when the connection closes, we remove our
        // ref to `this._openedDb`.
        openreqResult.onclose = () => {
          this._openedDb = null;
        };
        return openreqResult;
      });
    }

    return this._openedDb;
  }

  _withIDBStore(
    type: IDBTransactionMode,
    callback: (store: IDBObjectStore) => void
  ): Promise<void> {
    return this._makeDbPromise().then(
      db =>
        new Promise<void>((resolve, reject) => {
          const transaction = db.transaction(this._storeName, type);
          transaction.oncomplete = () => resolve();
          transaction.onabort = transaction.onerror = () => reject(transaction.error);
          callback(transaction.objectStore(this._storeName));
        })
    );
  }
}

[jakearchibald]

fwiw, I'm thinking about removing the custom store stuff and getting folks to use https://github.com/jakearchibald/idb if they want a custom store.

Implementation is here: https://github.com/jakearchibald/idb#keyval-store

[v-python]

I have no problem with a separate idb per custom store, except it would be nice to have an interface for deleting the custom store. I looked at this and could understand it, I looked at idb, and I'd have to learn a lot more about IndexedDB (or something) to understand that. Definitely a bigger learning curve.

[TheNando]

Don't know if anyone is still dealing with this issue, but I have a small workaround for this issue in my app. I just ensure that the stores are created with the native syntax prior to creating the idb-keyval stores:

// Manually create stores if they don't exist
const request = indexedDB.open('myDatabase')

request.onupgradeneeded = function(event) {
  const db = event.target.result
  db.createObjectStore('myStore1')
  db.createObjectStore('myStore2')
  // etc. for any additional stores
}

// Now the lib stores will work as expected
const store1 = new Store('myDatabase', 'myStore1')
const store2 = new Store('myDatabase', 'myStore2')

[jakearchibald]

#80 is my current plan for the next major version.

[ctf0]

@jakearchibald does

import { set, createStore } from 'idb-keyval';
const store = createStore('custom-keyval-db', 'custom-keyval-store');
set('hello', 'world', store);

solve it or do we still need this ticket ?

[codegard1]

Don't know if anyone is still dealing with this issue, but I have a small workaround for this issue in my app. I just ensure that the stores are created with the native syntax prior to creating the idb-keyval stores:

// Manually create stores if they don't exist
const request = indexedDB.open('myDatabase')

request.onupgradeneeded = function(event) {
  const db = event.target.result
  db.createObjectStore('myStore1')
  db.createObjectStore('myStore2')
  // etc. for any additional stores
}

// Now the lib stores will work as expected
const store1 = new Store('myDatabase', 'myStore1')
const store2 = new Store('myDatabase', 'myStore2')

I tried this fix and it did not resolve the issue. I was still unable to access 'myStore2', even when this code block immediately precedes calls to idb-keyval. (using "idb-keyval": "^3.2.0")

[jakearchibald]

The new version reworks custom stores a little, so it's easier to add your own implementation. Although if you need to do that, you may find https://github.com/jakearchibald/idb/ easier.

[DibyodyutiMondal]

I had a need for really simple key-value store, but I also need some basic partitioning for the data. I did not care what the indexed-db database name would be, but I wanted the stores to be different.

Say for example: different keyval store per type of entity I wanted to store, where I can do basic read/write via the entity's id

Obviously, the problem with this was that of the indexed db version during upgrades.

The solution I came up was this very inelegant solution to add an object store to an indexed db database

1. Probe the indexed db database, and get the current version and object store names
2. What version should you actually use? -> If the current list of object store names includes your object store, then use (version you probed + 1), else use the same version number you probed.
3. Check if the version you should use is the same as the version you probed, otherwise perform an upgrade operation, and add the object store you are trying to add.
4. return a UseStore function that points to this object store (whether pre-existing or updated)

Following is my implementation for the same. (This implementation can be generalised further, allowing for different db names as well)

import { promisifyRequest } from 'idb-keyval';

export const IDB_KEYVAL = 'DB_NAME';


let keyValDB$ = promisifyRequest(indexedDB.open(IDB_KEYVAL));

export function createKeyvalStore(storeName: string) {
  const probe$ = keyValDB$
    .then(db => {
      db.addEventListener('versionchange', () => {
        db.close();
      });
      return db;
    });

  const version$ = probe$
    .then(db => {
      const stores = db.objectStoreNames;
      if (!stores.contains(storeName)) {
        return db.version + 1;
      }
      else {
        return db.version;
      }
    });

  const upgrade$ = (async () => {
    const version = await version$;
    const probe = await probe$;

    if (version !== probe.version) {
      const upgrade = indexedDB.open(IDB_KEYVAL, version);
      upgrade.addEventListener('upgradeneeded', () => {
        upgrade.result.createObjectStore(storeName);
      });
      return new Promise<void>((resolve) => {
        upgrade.addEventListener('success', () => {
          upgrade.result.close();
          resolve();
        });
      });
    }
  })();

  keyValDB$ = upgrade$.then(() => changeDb());

  return async <T>(txMode: IDBTransactionMode, callback: (store: IDBObjectStore) => T | PromiseLike<T>) => {
    const db = await keyValDB$;
    return callback(db.transaction(storeName, txMode).objectStore(storeName));
  };
}

function changeDb() {
  return promisifyRequest(indexedDB.open(IDB_KEYVAL));
}

@jakearchibald I would really like your comments/inputs on this

[jakearchibald]

You probably want https://www.npmjs.com/package/idb rather than this package, as what you're doing is more complex than what idb-keyval was designed for.

[DibyodyutiMondal]

But I wanted to avoid the full idb package.
The intent is to be able to use the simple 'get' and 'set' apis from idb-keyval same as I did before, with a better ability to organize the indexeddb instances and object stores.

The helper above basically helps me create a custom store that I can later pass to 'get' and 'set' functions in idb-keyval, without needing the full idb package, because my use cause does not require the complex querying and indexing.

Previously, the rule for idb-keyval was: Only 1 object store, and database name and object store must be the same.

With this helper function. that rule becomes: many object stores in a database, with any name you like, but all object stores in that database should be controlled/created by idb-keyval.

The reason that all object stores in the given database should be controlled by idb-keyval is that the helper function updates the version of the database in a non-deterministic way. This is a problem for the normal 'upgradeneeded' flow of changing object store schemas.
Since idb-keyval's schema is unlikely to ever change (only 'key' and 'value') columns, this is not a problem.