Storage

[nicolodavis]

Games are currently held in an in-memory object, but should be persisted in either a MongoDB or Firebase instance. Options for other backends should also be possible.

TODO:

• Mongo support
• Firebase support
• Postgres support

[w74]

Do we have a server for this already? Furthermore, we should consider creating a "library" of games people have made.

[nicolodavis]

All the server code lives in src/server. Games aren't persisted yet, but I plan to add support for various backends.

[nicolodavis]

I'll link to any games people make on the main README page once that happens. We haven't reached 1.0 yet so I would imagine that people will start using this more widely once the API is stable.

[saeidalidadi]

I think writing a connector class for the reducer would be a better idea, so people could add their own odm or orms.

[nicolodavis]

Yes, this is the idea I had too, to make it extensible so that we're not limited to just the backends that come supported out of the box.

[saeidalidadi]

If you have the usage signature or api of the connector I will be happy to write the tests of that requirements.

[nicolodavis]

I was just thinking of adding a DB section to the server constructor:

Server({
  game: ...
  db: {
    // optional initialization code
    init: function() { ... }

    // called whenever a Redux action is received by the server
    onAction: function(gamestate) { ... } 
  }
})

Provided connectors can be used like:

import { Mongo } from './db/mongo';

Server({
  game: ...
  db: Mongo(uri: "..."),
})

We could probably first just wrap the existing in-memory object into this signature and build up from there.

[saeidalidadi]

There is a use case which data could be on multiple stores like redis or mongo. does this solution works for that case?
I think for now this case could be ignored but I think using generators and calling them in a loop like that koa is doing would be a good idea to work with persist stores.

[nicolodavis]

Let's make db an array then:

db: [
  {onAction: }, {onAction: }, ...
]

where the onAction handlers are called in the order they are defined.

[saeidalidadi]

This is cool and has more benefits

[saeidalidadi]

Is it possible to use redux-saga middleware?

[nicolodavis]

We could definitely look into redux-saga, but I would prefer to start simple and only use it if necessary.

I've created a simple interface for storage with just get and set. We can add support for something like Mongo or Redis to start.

[saeidalidadi]

I think Mongo will be simple for start, as Redis is a key-value and working with arrays and objects needs more handlings

[w74]

I do think that Mongo would be a bit better for board games as each individual player's "possessions" aren't always similar.

[saeidalidadi]

It doesn't need any database for test. just should be added an interface to let users of framework pass their own async actions in order to fetch state from db.

There are challenges like:

1. Redux store is not a persist one.
2. The reducers are the same for client and server.

[nicolodavis]

@saeidalidadi Would you be interested in adding Mongo support?

It should be as simple as just implementing another db class that talks to Mongo instead of the in-memory Map. Since this code runs only on the server, there shouldn't be a concern that the reducers are the same on the client and server.

[saeidalidadi]

It would be a pleasure for me to run this task. 👍

[jcgertig]

Any updates on the progress of this. It is a bit disconcerting because at the moment if we redeploy our app we lose game state.

[nicolodavis]

@jcgertig I haven't heard back from Saeid in a while. Perhaps he's busy. Would you like to take this over? Feel free to add support for your favorite storage system.

[nicolodavis]

I've added support for overriding the db implementation in 2721ad4. This should allow people that want to persist their games to hook it up to any storage solution that they want. I'll start working on adding Mongo when I get the time. In the meantime, PR's welcome :)

[saeidalidadi]

Hi @nicolodavis, as you said I am busy, but I find a solution based on this answer from Dan Abramov for persisting the state on action event, I try to get free and I will add this asap if you are agree.

Server({
  games,
  db : { 
   set: async setById (gameId, gameState) {
       // user query
       return result;
    }, 
   get: async getById (gameId) {
       // user query
       return gameState;
   }
})

[nicolodavis]

@saeidalidadi You mean a set / get that talks to Mongo, or are you referring to something else? Not sure I follow.

[saeidalidadi]

They could talk to all the remote stores and the developers would be able to add their own implementation for the gameId and also have more customization. setter and getter functions should be called inside framework. so the framework does'n know anything about the remote store.

[nicolodavis]

That's what I thought I implemented in 2721ad4. Is your proposal different?

[saeidalidadi]

They are about the same, so I will add a working code for this feature.

[nicolodavis]

Ok, I'll take a look when you send a PR. I'm still not exactly sure what your proposal is, but I'm sure it will make more sense when I look at the code :)

[saeidalidadi]

I have implemented the functionalities but have issues with tests in spy functions of sync suite. I played examples and everything is ok as I see. I think it would be better to make the PR and talk about the issues.

[philihp]

Can you throw up a branch? I'd love to check it out and see this progress.

Aside: How do y'all feel about Firebase for this?

[nicolodavis]

Firebase sounds good to me!

[saeidalidadi]

I have crated a simple implementation to persist state inside db in 8473137.
after an agreement about the idea I will change the tests and will come back with a green check.

[nicolodavis]

The MongoDB connector is implemented in 003fe46.

Need to add some error handling.

[rwforest]

@nicolodavis any plan /ETA on Firebase?

[nicolodavis]

Nobody is working on Firebase at the moment. Looking for contributors.

[bennygenel]

@nicolodavis I would like to add my contribution by working on Firebase.

I created a similar class to Mongo for Firebase/Firestore. Is this something you would like to add to the project?

/**
 * Firebase RDT/Firestore connector.
 */
export class Firebase {
  /**
   * Creates a new Firebase connector object.
   */
  constructor({ config, dbname, cacheSize, mockFirebase }) {
    if (cacheSize === undefined) cacheSize = 1000;
    if (dbname === undefined) dbname = 'bgio';
    // TODO: better handling for possible errors
    if (config === undefined) config = {};

    this.client = firebase;
    // Default engine is Firestore
    this.engine = config.engine === 'RTD' ? config.engine : 'Firestore';
    this.apiKey = config.apiKey;
    this.authDomain = config.authDomain;
    this.databaseURL = config.databaseURL;
    this.projectId = config.projectId;
    this.dbname = dbname;
    this.cache = new LRU({ max: cacheSize });
  }
  /**
   * Connect to the instance.
   */
  async connect() {
    var config = {
      apiKey: this.apiKey,
      authDomain: this.authDomain,
      databaseURL: this.databaseURL,
      projectId: this.projectId,
    };
    this.client.initializeApp(config);
    this.db =
      this.engine === 'Firestore'
        ? this.client.firestore()
        : this.client.database();
    return;
  }
  /**
   * Write the game state.
   * @param {string} id - The game id.
   * @param {object} store - A game state to persist.
   */
  async set(id, state) {
    const cacheValue = this.cache.get(id);
    if (cacheValue && cacheValue._stateID >= state._stateID) {
      return;
    }

    this.cache.set(id, state);

    const col =
      this.engine === 'RTD'
        ? this.db.ref(id)
        : this.db.collection(this.dbname).doc(id);
    delete state._id;
    await col.set(state);

    return;
  }

  /**
   * Read the game state.
   * @param {string} id - The game id.
   * @returns {object} - A game state, or undefined
   *                     if no game is found with this id.
   */
  async get(id) {
    let cacheValue = this.cache.get(id);
    if (cacheValue !== undefined) {
      return cacheValue;
    }

    let col, doc, data;
    if (this.engine === 'RTD') {
      col = this.db.ref(id);
      data = await col.once('value');
      doc = data.val();
    } else {
      col = this.db.collection(this.dbname).doc(id);
      data = await col.get();
      doc = data.data();
    }

    let oldStateID = 0;
    cacheValue = this.cache.get(id);
    /* istanbul ignore next line */
    if (cacheValue !== undefined) {
      /* istanbul ignore next line */
      oldStateID = cacheValue._stateID;
    }

    let newStateID = -1;
    if (doc) {
      newStateID = doc._stateID;
    }

    // Update the cache, but only if the read
    // value is newer than the value already in it.
    // A race condition might overwrite the
    // cache with an older value, so we need this.
    if (newStateID >= oldStateID) {
      this.cache.set(id, doc);
    }

    return doc;
  }

  /**
   * Check if a particular game exists.
   * @param {string} id - The game id.
   * @returns {boolean} - True if a game with this id exists.
   */
  async has(id) {
    const cacheValue = this.cache.get(id);
    if (cacheValue !== undefined) {
      return true;
    }

    let col, data, exists;
    if (this.engine === 'RTD') {
      col = this.db.ref(id);
      data = await col.once('value');
      exists = data.exists();
    } else {
      col = this.db.collection(this.dbname).doc(id);
      data = await col.get();
      exists = data.exists;
    }

    return exists;
  }
}

[nicolodavis]

@bennygenel that would be great!

[bennygenel]

@nicolodavis I'm planing for SQL integration as we talked but I have one question in mind.

Package dependencies are getting pretty big in my opinion. For someone just going to use mongodb having all other db packages downloaded is really unnecessary. After sequelize integration there is going to be 6 packages in total added to the project just for databases.

Is there something we can do about this?

[nicolodavis]

@bennygenel Yes, we can just follow the same package splitting approach that we do for the other stuff.

We can split the server side stuff into:
boardgame.io/server
boardgame.io/mongo
boardgame.io/firebase
etc.

import { Server } from 'boardgame.io/server';
import { Mongo } from 'boardgame.io/mongo';

const server = Server({
  games: [...],
  db: new Mongo(...),
});

package.json will still contain all the dependencies, but the user will not pull in the firebase dependency into their code if they just use mongo (for example).

I think this will involve getting rid of the environment variable approach of initializing the backends, though.

[pociej]

@nicolodavis i just tried mongo integration and i see it creates separate collection per each single game instance. Also i have no control on names of those collections. Is there some setup i missed or its intended to be like this?

[nicolodavis]

One collection per game instance was how it was originally designed. We can change it to a single collection (with an additional column for the game instance) and also add some customization over collection name.

The Mongo connector is a very small file that you can modify locally and use as a custom DB connector for now: https://github.com/nicolodavis/boardgame.io/blob/master/src/server/db/mongo.js

If you have a good experience using a single collection, I'll be happy to accept a PR and merge that in to the current implementation.

[pociej]

@nicolodavis i will think about collections structures and back to you.
Anyways i see something probably buggy. So if i understand well collection name is of the format gameName:gameId, which by default becomes default:....... Problem is that when i set some name of my game like :
MyGame = new Game({ name : 'awesomeGame'}) i don't see collection created in my mongo anymore.

[skdhayal]

Guys, any updates on mongodb connector ?

[nicolodavis]

@skdhayal Nobody is working on it at the moment.

[delucis]

@skdhayal In case you’re interested in helping out on a Mongo connector, here are some links.

• Source code for the old Mongo DB implementation, which might be a helpful basis for future work.
• Tests for that connector
• The base class new DB implementations should extend

At the moment we’re encouraging community maintainers to create connectors as separate packages as we don’t have the bandwidth to test and maintain connectors for every DB and I’m happy to provide guidance if you think this is something you’d like to work on. Some of the other community connectors might be a useful reference here, e.g. connectors for Firestore or Postgres.

[pociej]

@skdhayal i wanted to work on it soon. Let me know please if you are going to implement it, or maybe there is some work we can share.

[HydraOrc]

Chris asked to share my work on MongoDB Storage, so basically I just used the postgresql library code and edited it to work with Mongo.
P.S. I had to edit the fetch function, because it was actually written with mistakes

import { Async } from 'boardgame.io/internal';
import { emptyFunc } from 'defaults';
import { Game } from 'collections';

export class MongoStore extends Async {
  connect = emptyFunc

  /**
  * Create a new game.
  *
  * This might just need to call setState and setMetadata in
  * most implementations.
  *
  * However, it exists as a separate call so that the
  * implementation can provision things differently when
  * a game is created.  For example, it might stow away the
  * initial game state in a separate field for easier retrieval.
  */
  async createGame(_id, {
    initialState,
    metadata: { gameName, players, setupData, gameover, nextRoomID, unlisted },
  }) {
    await Game.collection.insertOne({
      _id,
      gameName,
      players,
      setupData,
      gameover,
      nextRoomID,
      unlisted,
      initialState,
      state: initialState,
      log: [],
    });
  }

  /**
  * Update the game state.
  *
  * If passed a deltalog array, setState should append its contents to the
  * existing log for this game.
  */
  async setState(_id, state, deltalog) {
    let _a;
    // 1. get previous state
    const game = await Game.collection.findOne({ _id });

    const previousState = game === null || game === void 0 ? void 0 : game.state;

    // 2. check if given state is newer than previous, otherwise skip
    if (!previousState || previousState._stateID < state._stateID) {
      await Game.collection.updateOne({ _id }, {
        $set: {
        // 3. set new state
          state,
          // 4. append deltalog to log if provided
          log: [...((_a = game === null || game === void 0 ? void 0 : game.log) !== null && _a !== void 0 ? _a : []), ...(deltalog !== null && deltalog !== void 0 ? deltalog : [])],
        },
      }, { upsert: true });
    }
  }

  /**
  * Update the game metadata.
  */
  async setMetadata(_id, {
    gameName,
    players,
    setupData,
    gameover,
    nextRoomID,
    unlisted,
  }) {
    await Game.collection.updateOne({ _id }, {
      $set: {
        gameName,
        players,
        setupData,
        gameover,
        nextRoomID,
        unlisted,
      },
    }, { upsert: true });
  }

  /**
  * Fetch the game state.
  */
  async fetch(_id, { state, log, metadata, initialState }) {
    const game = await Game.collection.findOne({ _id });

    const result = {};

    if (game) {
      const { gameName, gameover, nextRoomID, players, setupData, unlisted } = game;

      if (metadata && players) {
        result.metadata = {
          gameName,
          players,
          setupData,
          gameover,
          nextRoomID,
          unlisted,
        };
      }

      if (initialState) {
        result.initialState = game.initialState;
      }

      if (state) {
        result.state = game.state;
      }

      if (log) {
        result.log = game.log;
      }
    }

    return result;
  }

  /**
  * Return all games.
  */
  async listGames(opts) {
    let query;

    if (opts === null || opts === void 0 ? void 0 : opts.gameName) {
      query = { gameName: opts.gameName };
    }

    const games = await Game.collection.find(query, {
      _id: 1,
    });

    return games.map((game) => game.id);
  }
}

[skdhayal]

@skdhayal i wanted to work on it soon. Let me know please if you are going to implement it, or maybe there is some work we can share.

sorry @pociej - i am not working on it at all. Just busy with my own projects. Thanks

[delucis]

Closing this is as storage is handled by third-party adapters.