[cygnus][bug] Persistence Error Mongo Authentication and side effect

[srrspace]

I have detected a persistence error due to an exception of mongo authentication and I am facing to a strange side effect.

We have a mongodb installed configured to enable the security features.

We manage at least 2 kind of devices/sensors data coming from the devices through the iotagent-json (http/mqtt) that send the data to the Orion Context broker correctly (OCB) which notify cygnus with the corresponding data.

Cygnus is subscribed correctly to receive notifications from this 2 data set corresponding to their service domain.

Theoretically cygnus must create automatically the databases based on their "Fiware-Service" and create automatically the collections based on their "Fiware-ServicePath" without to take previously specific actions on the mongodb (i.e. not to create the databases manually beforehand).

A specific mongo user is created in the database "admin" (i.e. a cygnus user credential) with the correct rights and privileges.

At the beginning the corresponding database and collection doesn't not exist in the mongodb, as cygnus should create automatically the databases and the collections.

Cygnus receive the data but at the moment to store the data a "CygnusPersistenceError. -, Exception authenticating MongoCredential{mechanism=SCRAM-SHA-1,..." is raised.

IMHO all is due because cygnus try to create the mongo client with the credential of the mentioned DB that don't exist at the beginning and/or even if the DB existed and was created manually the user in this DB doesn't exist.

It is not viable to create beforehand manually the DB ("sth_") and the collections ("sth_") and their common user in each database (cygnus username/password) for each service domain and each kind of possible data.

Mongo need an "authSource" option parameter, in addition to the username and password in the agent configuration, to authenticate a user to create and manage all the databases and collections (inserts, find,...), having a common user in a specific (user) DB which allows to authenticate correctly this mongo user for all service domain and their database and collections.

The solution is to add a new agent configuration parameter "auth_source" (cygnus-ngsi.sinks.mongo-sink.mongo_auth_source) which will be used in the function MongoBackendImpl(...) (cygnus-common - MongoBackendImpl.java) and in the function getDatabase(...)

In the cygnus-ngsi, file NGSIMongoBaseSink.java a new variable "mongoAuthSource" is managed from "mongo_auth_source" context string.

For another hand there is a strange behavior due to how the mongo client with credential is created in the function getDatabase(...). There is a kind of optimization which try to reuse the same MongoClient object with credential for the subsequent call for different database/collection which also raise exception related to database/collection as persistence errors and the corresponding authentication error.

In the original piece of code, there is:

if (client == null) {
    if (mongoUsername.length() != 0) {
        MongoCredential credential = MongoCredential.createCredential(mongoUsername, dbName,
                mongoPassword.toCharArray());
        client = new MongoClient(servers, Arrays.asList(credential));
    } else {
        client = new MongoClient(servers);
    } // if else
} // if

The MongoClient with credential object is created one time with the first current dbName received in parameter at one moment and this same object is used by other databases.
If there is no credential, i.e. no authentication, it is ok, the client is created and reused without errors but if there is a need for authentication mechanism then there are exceptions raised and in debug we can see a mixes of database and collection each other.

Even if it was possible to authenticate in one database for one "Fiware-Service", the credential is using the current database (at first time) to create the object MongoCredential which is used to create a MongoClient which is also used to access and manage other different database but raise exception and mixes database with collection of the previous calls and subsequent calls.

And depending when and what is the first data/message received, i.e. for one kind of data message just after cygnus started, then one data set could be stored and the subsequent coming from other kind data will raise exceptions (persistence and authentication errors) and vice-versa.

To avoid this side effect, maybe it could be possible to create, use and close/destroy the MongoClient object systematically (i.e. createDatabase(...) creating a new MongoClient - createCollection(...) - insertContextDataRaw(...) - and close the previous MongoClient) or to manage a kind of mongo client object pool, but if a common authentication source database parameter is used this side effect could disappear.

I think that solving the authentication using "auth_source" will solve the two problems, i.e. the persistence error due to mongo authentication and the side effect of this mongo client optimization.

Currently I am working on this issue to correct this problem, in my local repository, then when I finish, I will test it in my FIWARE instance and I will create a pull request.

Attached an extract of cygnus.log:

extract-cygnus-log-persistence-error-mongo-authentication.txt

[oliveiradrodrigo]

I'm on running into de same issue. Need to run MongoDB with authentication but Cygnus can not connect since he uses his default database sth_default to authenticate the user. The default database for authentication usually is the admin database.

Could you please put the auth_source feature as soon as possible? :(

[fgalan]

Fixed by PR #1851