loopbackio / loopback-connector-mongodb Goto Github PK

Few ids are converted to hex numbers when saved to mongodb.

The id "line-by-line" is interpreted as ObjectId("6c696e652d62792d6c696e65") in mongo and
for id "line-stream" the mongo id is "line-stream"

What do you think to use the plural field to represent our collection name, instead of the model name?

https://github.com/strongloop/loopback-connector-mongodb/blob/master/lib/mongodb.js#L176

thanks

If I try to use $inc in the data field of update method I get "The dollar ($) prefixed field '$inc' in '$inc' is not valid for storage.". When I dug deeper I found that the only operation that update method is supporting is "$set" (see mongodb.js:643:43). Am I wrong? Is there no way of incrementing fields using this connector?

Upgrade to the latest node-mongodb-native version (2+)

ISSUE is how result of type array is inserted. See the value of result under "2. What was inserted finally." to understand the issue.

1. My input:
   { "cvsName" : "cvs_OrderVolume_StrongLoopTry",
   "database" : "pst",
   "description" : "Orders Previous Year to Date by Month",
   "detailedDescription" : "",
   "displayRank" : 89,
   "graphTitle" : "Previous Year to Date",
   "graphType" : "line",
   "linearResult" : ["2012-01", "2012-02"],
   "link" : [],
   "name" : "OrderVolumePreviousYear2DateByMonth",
   "pageId" : "newCustomerByMonth",
   "query" : "this is a complex select query which I cant print here.",
   "result" : [22644, 21752]
   }
2. What was inserted finally:
   {
   "cvsName" : "cvs_OrderVolume_StrongLoopTry",
   "database" : "pst",
   "description" : "Orders Previous Year to Date by Month",
   "detailedDescription" : "",
   "displayRank" : 89,
   "graphTitle" : "Previous Year to Date",
   "graphType" : "line",
   "linearResult" : [["2012-01"], ["2012-02"]],
   "link" : "",
   "name" : "OrderVolumePreviousYear2DateByMonth",
   "pageId" : "newCustomerByMonth",
   "query" : "this is a complex select query which I cant print here.",
   "result" : [[null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, "SO MANY MORE NULL THAT I EXCEEDED THE MAX OF 65535 characters. lol"]],
   "_id" : ObjectId("53150cc8cd26342c05e3d72b")
   }
3. My model defined in model.json
   "devil": {
   "properties": {
   "cvsName": {
   "type": "String"
   },
   "database": {
   "type": "String"
   },
   "description": {
   "type": "String"
   },
   "detailedDescription": {
   "type": "String"
   },
   "displayRank": {
   "type": "Number"
   },
   "graphTitle": {
   "type": "String"
   },
   "graphType": {
   "type": "String"
   },
   "linearResult": {
   "type": "Array"
   },
   "link": {
   "type": "String"
   },
   "name": {
   "type": "String"
   },
   "pageId": {
   "type": "String"
   },
   "query": {
   "type": "String"
   },
   "result": {
   "type": "Array"
   }
   },
   "public": true,
   "dataSource": "mongoDev",
   "plural": "devils"
   }

When using has_many relation its better to have count of related records or aggregation option

https://github.com/strongloop/loopback-connector-mongodb/blob/master/lib/mongodb.js#L727

            options = {background: true};
            if(p.unique) {
-              options.unqiue = true;
+              options.unique = true;
            }

#70 adds '$inc', '$max', '$min', '$mul', '$rename', '$setOnInsert', '$set', '$unset' operators. It falls back to $set if no operator is used.

See also #72

When storing entries into the database using composite IDs, it appears that one of the composite IDs is used as the internal _id property.

This is creating an issue with having composite IDs used in a many-to-many relationship manner.

This is accomplished by building a model entity with a foreign key to two other model entities, along with a 3rd ID as its own. It appears to use the first item of the composite id definition as defined by the LDL is used for the _id.

I upgraded to loopback-connector-mongodb 1.6.0 and I am getting the following error after calling updateOrCreate several times on the same doc object. I do not get this error if I revert only loopback-connector-mongodb back to 1.4.5. We're using mongo version 2.6.7.

 Assertion: 10307:Client Error: bad object in message: bson length doesn't match what we found in object with unknown _id

I don't have an isolated reproducible case yet as that would take me a full day most likely, but I wanted to post it to let you know.

Here is the verbose MongoDB log.

23T16:36:16.669-0500 [conn26] command scm_ops_tst.$cmd {
    findandmodify : "JobInstanceLog",
    query : {
        _id : ObjectId('54c2bed00f992bf8214288f1')
    },
    sort : {
        _id : 1
    },
    new : true,
    remove : false,
    upsert : true,
    update : {
        $set : {
            jobInstanceId : "512610624747220410000",
            jobId : "5474e123550d9b34260f05c7",
            jobName : "Pricing",
            startTime : new Date(1422048976768),
            hostname : "temp",
            logs: [...edited out...]
        }
    }
} keyUpdates:0 numYields:0 locks(micros) w:2575 reslen:8889 2ms
2015-01-23T16:36:17.144-0500 [conn26] Assertion: 10307:Client Error: bad object in message: bson length doesn't match what we found in object with unknown _id
2015-01-23T16:36:19.753-0500 [conn26] mongod.exe    ...\src\mongo\util\stacktrace.cpp(169)                           mongo::printStackTrace+0x43
2015-01-23T16:36:19.753-0500 [conn26] mongod.exe    ...\src\mongo\util\log.cpp(127)                                  mongo::logContext+0x9c
2015-01-23T16:36:19.753-0500 [conn26] mongod.exe    ...\src\mongo\util\assert_util.cpp(183)                          mongo::msgasserted+0xfb
2015-01-23T16:36:19.753-0500 [conn26] mongod.exe    ...\src\mongo\util\assert_util.cpp(174)                          mongo::msgasserted+0x13
2015-01-23T16:36:19.753-0500 [conn26] mongod.exe    ...\src\mongo\db\dbmessage.cpp(116)                              mongo::DbMessage::nextJsObj+0x11f
2015-01-23T16:36:19.753-0500 [conn26] mongod.exe    ...\src\mongo\db\dbmessage.h(213)                                mongo::QueryMessage::QueryMessage+0x74
2015-01-23T16:36:19.753-0500 [conn26] mongod.exe    ...\src\mongo\db\instance.cpp(253)                               mongo::receivedQuery+0xcc
2015-01-23T16:36:19.753-0500 [conn26] mongod.exe    ...\src\mongo\db\instance.cpp(437)                               mongo::assembleResponse+0x2f4
2015-01-23T16:36:19.753-0500 [conn26] mongod.exe    ...\src\mongo\db\db.cpp(203)                                     mongo::MyMessageHandler::process+0x111
2015-01-23T16:36:19.753-0500 [conn26] mongod.exe    ...\src\mongo\util\net\message_server_port.cpp(210)              mongo::PortMessageServer::handleIncomingMsg+0x671
2015-01-23T16:36:19.753-0500 [conn26] mongod.exe    ...\src\third_party\boost\libs\thread\src\win32\thread.cpp(185)  boost::`anonymous namespace'::thread_start_function+0x21
2015-01-23T16:36:19.753-0500 [conn26] mongod.exe    f:\dd\vctools\crt_bld\self_64_amd64\crt\src\threadex.c(314)      _callthreadstartex+0x17
2015-01-23T16:36:19.753-0500 [conn26] mongod.exe    f:\dd\vctools\crt_bld\self_64_amd64\crt\src\threadex.c(292)      _threadstartex+0x7f
2015-01-23T16:36:19.753-0500 [conn26] kernel32.dll                                                                   BaseThreadInitThunk+0xd
2015-01-23T16:36:19.753-0500 [conn26] 
2015-01-23T16:36:19.753-0500 [conn26] warning: DBException thrown :: caused by :: 10307 Client Error: bad object in message: bson length doesn't match what we found in object with unknown _id
2015-01-23T16:36:21.245-0500 [conn26] mongod.exe    ...\src\mongo\util\stacktrace.cpp(169)                           mongo::printStackTrace+0x43
2015-01-23T16:36:21.245-0500 [conn26] mongod.exe    ...\src\mongo\util\assert_util.cpp(66)                           mongo::DBException::traceIfNeeded+0xec
2015-01-23T16:36:21.245-0500 [conn26] mongod.exe    ...\src\mongo\util\assert_util.cpp(183)                          mongo::msgasserted+0x1a7
2015-01-23T16:36:21.245-0500 [conn26] mongod.exe    ...\src\mongo\util\assert_util.cpp(174)                          mongo::msgasserted+0x13
2015-01-23T16:36:21.245-0500 [conn26] mongod.exe    ...\src\mongo\db\dbmessage.cpp(116)                              mongo::DbMessage::nextJsObj+0x11f
2015-01-23T16:36:21.245-0500 [conn26] mongod.exe    ...\src\mongo\db\dbmessage.h(213)                                mongo::QueryMessage::QueryMessage+0x74
2015-01-23T16:36:21.245-0500 [conn26] mongod.exe    ...\src\mongo\db\instance.cpp(253)                               mongo::receivedQuery+0xcc
2015-01-23T16:36:21.245-0500 [conn26] mongod.exe    ...\src\mongo\db\instance.cpp(437)                               mongo::assembleResponse+0x2f4
2015-01-23T16:36:21.245-0500 [conn26] mongod.exe    ...\src\mongo\db\db.cpp(203)                                     mongo::MyMessageHandler::process+0x111
2015-01-23T16:36:21.245-0500 [conn26] mongod.exe    ...\src\mongo\util\net\message_server_port.cpp(210)              mongo::PortMessageServer::handleIncomingMsg+0x671
2015-01-23T16:36:21.245-0500 [conn26] mongod.exe    ...\src\third_party\boost\libs\thread\src\win32\thread.cpp(185)  boost::`anonymous namespace'::thread_start_function+0x21
2015-01-23T16:36:21.245-0500 [conn26] mongod.exe    f:\dd\vctools\crt_bld\self_64_amd64\crt\src\threadex.c(314)      _callthreadstartex+0x17
2015-01-23T16:36:21.246-0500 [conn26] mongod.exe    f:\dd\vctools\crt_bld\self_64_amd64\crt\src\threadex.c(292)      _threadstartex+0x7f
2015-01-23T16:36:21.246-0500 [conn26] kernel32.dll                                                                   BaseThreadInitThunk+0xd
2015-01-23T16:36:21.246-0500 [conn26] 
2015-01-23T16:36:21.246-0500 [conn26] AssertionException handling request, closing client connection: 10307 Client Error: bad object in message: bson length doesn't match what we found in object with unknown _id

Note: loopback-connector-mongodb dependency changes:

loopback-connector-mongodb 1.6.0 = mongodb 2.0.14 (Breaks)
loopback-connector-mongodb 1.4.5 = mongodb ~1.4.7 (Works)

When comparing the json sent, 1.4.5(LEFT) and 1.6.0 (RIGHT)

Only the 1,0 and true, false appear to be different which I would guess should not matter.

Also, because of this error, #81 was noticed as well.

Adding reference to #77 as I assume it's related.

Hi,

I am occasionally receiving "Connection fails: { [MongoError: no valid seed servers in list] name: 'MongoError', message: 'no valid seed servers in list' } " when instances bootup (app instances and mongo cluster are in different datacentres). Mongo is hosted at Compose.io

I suspect it's a timeout on the initial connection.

It would be nice to be able to give connection arguments directly to the driver through config.json

Pleas advise and many thanks!

Got the latest build and ran npm test. I am seeing some make errors posted below.

Build error

./node_modules/.bin/mocha -G --timeout 10000 test/.test.js
/bin/sh: ./node_modules/.bin/mocha: No such file or directory
make: ** [test] Error 127
npm ERR! Test failed. See above for more details.

If a sort order is not specified a default one by _id is used. This may impact the performance of some queries where a sort is neither needed nor specified upfront. For example, using model.findOne results in the MongoDB connector calling db.collection.find(query).sort({_id:1}).limit(1)
If the sort part is skipped the result is returned almost immediately using the appropriate indices. However, having the sort in there causes the entire query to slow down. In my case there is only one record matching the given criteria and the sort is not needed.

Would you consider making the sort order explicit?

Hello,

Unfortunately, I could not find a single example of how change collection name for models and base models (for example, AccessTokens).

Can you resolve my problem? Thanks.

When using a mongo db datasource and trying to query a model by using the and operator it fails when using "near" as well.

Query:
{"where": {"and": [{"location": {"near": "52.335421, 13.096018"}},{"name": {"like": "aus"}}]}}

Error:
"message": "Can't canonicalize query: BadValue not handled: $near", "stack": "MongoError: Can't canonicalize query: BadValue not handled: $near\n at Object.toError (/home/codio/workspace/node_modules/loopback-connector-mongodb/node_modules/mongodb/lib/mongodb/utils.js:114:11)\n at /home/codio/workspace/node_modules/loopback-connector-mongodb/node_modules/mongodb/lib/mongodb/cursor.js:689:54\n at Cursor.close (/home/codio/workspace/node_modules/loopback-connector-mongodb/node_modules/mongodb/lib/mongodb/cursor.js:972:5)\n at commandHandler (/home/codio/workspace/node_modules/loopback-connector-mongodb/node_modules/mongodb/lib/mongodb/cursor.js:689:21)\n at /home/codio/workspace/node_modules/loopback-connector-mongodb/node_modules/mongodb/lib/mongodb/db.js:1847:9\n at Server.Base._callHandler (/home/codio/workspace/node_modules/loopback-connector-mongodb/node_modules/mongodb/lib/mongodb/connection/base.js:445:41)\n at /home/codio/workspace/node_modules/loopback-connector-mongodb/node_modules/mongodb/lib/mongodb/connection/server.js:478:18\n at MongoReply.parseBody (/home/codio/workspace/node_modules/loopback-connector-mongodb/node_modules/mongodb/lib/mongodb/responses/mongo_reply.js:68:5)\n at null.<anonymous> (/home/codio/workspace/node_modules/loopback-connector-mongodb/node_modules/mongodb/lib/mongodb/connection/server.js:436:20)\n at EventEmitter.emit (events.js:95:17)"

Switching the model to in memory this query works just fine.

Attempting to grab a large data set from mongodb via REST API [ get : http://localhost:3000/api/someModel ] - should return a roughly 8mb string of JSON. This causes an error in mongodb connection pool, for which I'm not sure why --

{
  "error": {
    "name": "Error",
    "status": 500,
    "message": "parseError occured",
    "stack": "Error: parseError occured\n    at null.<anonymous> (/Users/Eric/Code/spry/vbox/projects/loopback/poleApp/server/node_modules/loopback-connector-mongodb/node_modules/mongodb/lib/mongodb/connection/connection_pool.js:192:34)\n    at EventEmitter.emit (events.js:98:17)\n    at Socket.<anonymous> (/Users/Eric/Code/spry/vbox/projects/loopback/poleApp/server/node_modules/loopback-connector-mongodb/node_modules/mongodb/lib/mongodb/connection/connection.js:410:20)\n    at Socket.EventEmitter.emit (events.js:95:17)\n    at Socket.<anonymous> (_stream_readable.js:745:14)\n    at Socket.EventEmitter.emit (events.js:92:17)\n    at emitReadable_ (_stream_readable.js:407:10)\n    at emitReadable (_stream_readable.js:403:5)\n    at readableAddChunk (_stream_readable.js:165:9)\n    at Socket.Readable.push (_stream_readable.js:127:10)"
  }
}

The problem is the node.js server stays responsive, but I can no longer query the mongodb -- there's actually no error posted to the console. Any ideas?

loopback-connector-mongodb version: "1.3.0",
Loopback-connector version : 1.0.0

With trackChanges enabled, I post new content to my model and the server dies shortly after with MongoError: insertDocument :: caused by :: 11000 E11000 duplicate key error index: mydb.mymodel-change.$_id_ dup key: { : "XXXXXX" }; When I restart the server, I get the same error. Is there a step I am missing after enabling trachChanges

Related:
strongloop/loopback#566

I can't use $push, $addToSet operator in update method ? please help me...

Hi,

When trying to update the details to db(mongodb), I am getting error.

/Users/username/JavaScriptWorkSpace/myproject/node_modules/loopback-connector-mongodb/node_modules/mongodb/lib/mongodb/connection/base.js:245
throw message;
^
TypeError: Object function ModelConstructor(data, options) {
if (!(this instanceof ModelConstructor)) {
return new ModelConstructor(data, options);
}
if (ModelClass.settings.unresolved) {
throw new Error('Model ' + ModelClass.modelName + ' is not defined.');
}
ModelBaseClass.apply(this, arguments);
} has no method 'updateAttributes'

at /Users/username/JavaScriptWorkSpace/myproject/models/UserWithPrivileges.js:49:24

Pasting the code I have wrote;

var loopback = require('loopback');
var app = require('../app');
var privilegeModel=app.models.privileges;
var Orginaluser=app.models.User;

privilegeModel.updateUserWithPrivilege = function (UserDetail,include, fn) {
var usersId=UserDetail.user_id;
Orginaluser.findOne({where: {"user_id": usersId}}, function(err, Orginaluser) {

    privilegeModel.updateAttributes({privilege_user_management: 'C,R,E'}, function(err, privilegeModel){
        console.log(privilegeModel);
    });
});

Please help let me know whether i am doing anything wrong here.

I upgraded to version 1.7.0 and a lot of my queries are not working. When an undefined value is passed to a condition, this condition is just skipped.

For example, if I want to get a post with authorId:1 and groupId:undefined, the result is that I get all of the post for author with id 1, instead of bringing the posts for this author that do not have a groupId defined.

var group;

Post.findOne({
        where: {
          and: [
            {groupId: group},
            {authorId: ctx.req.accessToken.userId}
          ]
        }}, function (err, post) { ...

makes sense?

looking at http://docs.strongloop.com/display/public/LB/Database+discovery+API, it seems that mongo does not have discovery capabilites.

whilst I can understand that mongo is a document-orientated db, would it not be possible to runt hrough the collections (thereby giving you the models) and sampling say 10 records to grab the top-level properties as a helpful start ?

Or is there something like this already ?

Hi there,
I made a POST call to rest with jQuery like this:

var obj = {};
obj.isFree = false;
obj.yob = 1981;
obj.name = "PlayerName";
$.post("/api/players?access_token="+getAccessToken(), obj, "json").done(function(result){
    ...
}).fail(function(){
    ...
});

but when I go through my DB the isFree field is set to true.

Here's my model:

{
  "name": "player",
  "plural": "players",
  "base": "PersistedModel",
  "idInjection": true,
  "properties": {
    "name": {
      "type": "string",
      "required": true
    },
    "yob": {
      "type": "number"
    },
    "photo": {
      "type": "string"
    },
    "isFree": {
      "type": "boolean"
    }
  },
  "validations": [],
  "relations": {
    "playerRole": {
      "type": "hasOne",
      "model": "playerRole",
      "foreignKey": ""
    }
  },
  "acls": [
    {
      "accessType": "*",
      "principalType": "ROLE",
      "principalId": "$everyone",
      "permission": "DENY"
    },
    {
      "accessType": "*",
      "principalType": "ROLE",
      "principalId": "$authenticated",
      "permission": "ALLOW"
    }
  ],
  "methods": []
}

Here the validation of the model in player.js:

module.exports = function(Player) {
    Player.validatesUniquenessOf('name', { scopedTo: ['yob'] });
};

Here the final result my DB mongo:

{
    "_id" : ObjectId("54d0e2f3074fbc9c97a6606f"),
    "name" : "PlayerName",
    "yob" : 1981,
    "photo" : null,
    "isFree" : true
}

I think this could be a bug in mongo connector.
Anyone with this issue?

It's a waste of bandwidth, and non-standard behavior (which probably is the result of BSON handling null, whereas JSON doesn't). Aparently NULL values are indeed inserted into the MongoDB documents.

An option to disable this somehow (on the input as well as output side of things) seems to be in order.

When the connection string points to a port where nobody is listening on, the ping method never returns. A message is logged instead:

Connection fails:  [Error: failed to connect to [localhost:4]]
It will be retried for the next request.

Test cases (the first one passes, the second one fails):

// test/mongodb.test.js
  describe('.ping(cb)', function() {
    it('should return true for valid connection', function(done) {
      db.ping(done);
    });

    it('should return descriptive error on ECONNREFUSED', function(done) {
      var ds = getDataSource({
        host: 'localhost',
        port: 4 // unassigned by IANA
      });
      ds.ping(function(err) {
        (!!err).should.be.true;
        err.code.should.equal('ECONNREFUSED');
        done();
      });
    });
  });

// patch for test/init.js
-global.getDataSource = global.getSchema = function () {
-  var db = new DataSource(require('../'), config);
+global.getDataSource = global.getSchema = function (customConfig) {
+  var db = new DataSource(require('../'), customConfig || config);

See also strongloop/strong-arc#135.

@raymondfeng

I get the idea behind doing so is to allow querying by _id, but the problem it's causing is any string of length 12 or 24 will be converted too, even worser if those strings matches ObjectID format, they will be converted to ObjectID type when they should remain String type.

In short, this kind of "forced" conversion is kind of a surprise and smell like a hack?

} else if (spec === 'inq') {
        query[k] = { $in: cond.map(function (x) {
          if ('string' !== typeof x) return x;
          return ObjectID(x);
        })};
      } 

See full code here:
http://goo.gl/Xwhlj4

For a model as follows:

"employee": {
  "properties": {
    "name": {"type": "String"},
    "testValue": {"type": "Buffer"}
  }
}

1. When the ‘employee’ is first created, the response is original object with the returned id from MongoDB.

{
  "name": "X",
  "testValue": [
    42
  ],
  "id": "533adc6528543b2e59d1657a”
}

The test value is the Buffer.

1. MongoDB stores testValue as Binary:

{
  "name": "X",
  "testValue": { "$binary" : "Kg==", "$type" : "0" },
  "_id": { "$oid" : "533adc6528543b2e59d1657a" }
}

1. When the query is run, MongoDB returns testValue as Binary. But it is not converted back to Buffer. And Binary.toJSON() returns the base64 encoded value.

The upgrade from 1.4.5 -> 1.5.0 prevents the creation of models that include JSON Schema documents that contain an embedded $ref. For example (simplified),

{
   "name": "example",
   "type": "object",
   "properties": {
     "billing_address": { "$ref": "/definitions/address" },
     "shipping_address": {
       "allOf": [
         { "$ref": "/definitions/address" },
         { "properties":
           { "type": { "enum": [ "residential", "business" ] } },
           "required": ["type"]
         }
       ]
     }
   }
}

Attempting to create a Model with such a property included produces the error:

{ [MongoError: key $ref must not start with '$']
  name: 'MongoError',
  message: 'key $ref must not start with \'$\'' }

Is this something wrong with loopback-connector-mongodb? Is it produced by a deprecation in the mongo 2.0 driver?

When attempting to set the id type to be Number, the MyModel.find method fails to find by id.

Try configuration:

{
  "name": "MyModel",
  "properties": {
    "id": {
      "type": "Number",
      "required": true,
      "id": true
    }
  }
}

I have noticed that the objects will be saved in MongoDB with an _id of type number (not String or ObjectID).

This causes all of my queries with updateAll (I suspect not the only method affected) to fail.

For example...

MongoDB.prototype.all(model, filter, callback)

If the filter parameter passed into the all method is { "username" : "Terekhov" }, then by the time it gets to

MongoDB.prototype.updateAll(model, where, data, cb)

...the where object looks like this:

{
    "username": "Terekhov",
    order:
    [
        "id"
    ]
}

My solution was to replace the use of filter.order with a tempOrder variable, like so:

  var order = {};
  var tempOrder = '';
  if (!tempOrder) {
    var idNames = this.idNames(model);
    if (idNames && idNames.length) {
      tempOrder = idNames;
    }
  }
  if (tempOrder) {
    var keys = tempOrder;
    if (typeof keys === 'string') {
      keys = keys.split(',');
    }
    for (var index = 0, len = keys.length; index < len; index++) {
      var m = keys[index].match(/\s+(A|DE)SC$/);
      var key = keys[index];
      key = key.replace(/\s+(A|DE)SC$/, '').trim();
      if(key === idName) {
        key = '_id';
      }
      if (m && m[1] === 'DE') {
        order[key] = -1;
      } else {
        order[key] = 1;
      }
    }
  } else {
    // order by _id by default
    order = {_id: 1};
  }
  cursor.sort(order);

However, it's entirely possible I just don't understand the reason to include the order property in the query object.

1. If this is so, what is the reason and how can I avoid it polluting my searches in updateAll?
2. If not, thoughts on the above fix, seem reasonable?

@raymondfeng seems like a serious bug: any in-code models I explicitly declared as dataSource: 'memory' are still being turned into MongoDB collections and items are inserted there instead of memory.

I'm unable to update an existing document in my mongodb. From what I have been able to read about so far it looks like a conflict between mongodb storing _id as an object conflicting with me passing a string value but I'm not sure how to change that. Should I have an explicit '_id' property in my model definition?

I have tried: Event.upsert, Event.updateOrCreate, and Event.save and they all fail

I also tried removing the '_id' property from the model before 'PUT'ing it back to the server:

delete saveEventObj._id;

but that ended up creating a copy of the original document with null for both 'id' and '_id' properties in the new value forcing me to delete the instance manually.

My model code looks like this:

  "event": {
    "properties": {
      "date": {
        "type": "date"
      },
      "type": {
        "type": "string"
      },
      "nearestCity": {
        "type": "string"
      },
      "stateProv": {
        "type": "string"
      },
      "country": {
        "type": "string"
      },
      "blurb": {
        "type": "string"
      },
      "notes": {
        "type": "string"
      },
      "createdDate": {
        "type": "date"
      },
      "lastUpdate": {
        "type": "date"
      },
      "material": {
        "type": "string"
      },
      "tags": {
        "type": "array"
      }
    },
    "public": true,
    "dataSource": "mongodb",
    "plural": "events"
  },

I'm also using the loopback-angular auto-generated lb-services code and calling:

          Event.upsert(saveEventObj,
              function(response){
                console.log('Saved the event: ' + JSON.stringify(response));
              },
              function(response){
                console.log('bad update event: ' + JSON.stringify(response));
              }
            );

my browser console has the following:

PUT http://0.0.0.0:3001/api/events?id=52e8a43c36354025bd0c1144 500 (Internal Server Error)

bad update event: {"data":{"error":{"name":"MongoError","status":500,"message":"Mod on _id not allowed","lastErrorObject":{"err":"Mod on _id not allowed","code":10148,"n":0,"connectionId":204,"ok":1},"ok":0,"errmsg":"Mod on _id not allowed","stack":"MongoError: Mod on _id not allowed\n    
at Object.toError (/Users/seanbrookes/_stacks/projects/oilwreck/node_modules/loopback-connector-mongodb/node_modules/mongodb/lib/mongodb/utils.js:110:11)\n    
at /Users/seanbrookes/_stacks/projects/oilwreck/node_modules/loopback-connector-mongodb/node_modules/mongodb/lib/mongodb/db.js:1097:29\n    
at /Users/seanbrookes/_stacks/projects/oilwreck/node_modules/loopback-connector-mongodb/node_modules/mongodb/lib/mongodb/db.js:1806:9\n    
at Server.Base._callHandler (/Users/seanbrookes/_stacks/projects/oilwreck/node_modules/loopback-connector-mongodb/node_modules/mongodb/lib/mongodb/connection/base.js:442:41)\n   
 at /Users/seanbrookes/_stacks/projects/oilwreck/node_modules/loopback-connector-mongodb/node_modules/mongodb/lib/mongodb/connection/server.js:485:18\n    
at MongoReply.parseBody (/Users/seanbrookes/_stacks/projects/oilwreck/node_modules/loopback-connector-mongodb/node_modules/mongodb/lib/mongodb/responses/mongo_reply.js:68:5)\n    
at null.<anonymous> (/Users/seanbrookes/_stacks/projects/oilwreck/node_modules/loopback-connector-mongodb/node_modules/mongodb/lib/mongodb/connection/server.js:443:20)\n    
at EventEmitter.emit (events.js:95:17)\n    at null.<anonymous> (/Users/seanbrookes/_stacks/projects/oilwreck/node_modules/loopback-connector-mongodb/node_modules/mongodb/lib/mongodb/connection/connection_pool.js:191:13)\n   
 at EventEmitter.emit (events.js:98:17)"}},
"status":500,"config":{"transformRequest":[null],"transformResponse":[null],"url":"/api/events","method":"PUT",
"data":{"date":"2013-10-17T07:00:00.000Z","type":"train","nearestCity":"Sexsmith","stateProv":"Alberta","country":"Canada","blurb":"<p> Residents in the northwestern Alberta town of Sexsmith were forced from their homes after four CN rail cars carrying anhydrous ammonia left the rails. The cars remained upright and there were no leaks.</p>",
"notes":null,"createdDate":null,"lastUpdate":1391530836262,"material":null,
"tags":[],
"id":"52e8a43c36354025bd0c1144","_id":"52e8a43c36354025bd0c1144",
"$$hashKey":"006"},"params":{"id":"52e8a43c36354025bd0c1144"},"headers":{"Accept":"application/json, text/plain, */*","Content-Type":"application/json;charset=utf-8"}}} 

How to save file to gridfs as below for example?
Thank you very much.

def save(inputStream, contentType, originalFilename) {
        def inputFile = gridFS.createFile(inputStream)
        String extension = originalFilename.split('\\.')[-1]
        String gridFSFilename=inputFile.id.toString() + '.' + extension
        inputFile.setFilename(gridFSFilename)
        inputFile.save()
        return inputFile.filename
    }

We should have a way of passing certain config option directly to the underlying native mongodb driver. I am specifically looking for ReadPreference config option.

Hi massive, i'm pretty new in nodejs, so probably i just missed something.

There is model defined:

  "name": "landingUser",
  "base": "PersistedModel",
  "strict": false,
  "idInjection": false,
  "properties": {
    "email": {
      "type": "string",
      "required": true,
      "index": true
    },
    "created": {
      "type": "date"
    }
  },
...

I use AngularJS on front end to save record into mongo:

                var apiResponse = LandingUser.create({email: $scope.user.email});
                apiResponse.$promise.then(
                    function(value, responseHeaders) { // if success

And i want to save current time into database (like default=datetime.utcnow in SQLAlchemy) so a added this hook into model:

LandingUser.definition.properties.created.default = Date.now;

Date.now or Timestamp it does not matter, in both cases see:

{ "_id" : ObjectId("54d65c97be1c481a79552dd1"), "email" : "[email protected]", "created" : 1423334551689 }
{ "_id" : ObjectId("54d65f4abe1c481a79552dd2"), "email" : "[email protected]", "created" : 1423335242802 }
{ "_id" : ObjectId("54d65fb2be1c481a79552dd3"), "email" : "[email protected]", "created" : 1423335346493 }
{ "_id" : ObjectId("54d6600cbe1c481a79552dd4"), "email" : "[email protected]", "created" : 1423335436174 }

It's not a dates it's just a numbers. I tried to google but with little success. After i even checked MongoDB driver docs (https://www.npmjs.com/package/mongodb#data-types) and tried to resolve using this:

    var mongo = require('mongodb');
    ....

    LandingUser.beforeCreate = function(next, data)
    {
        data.created = new mongo.Timestamp();
        next();
    };

Miracle happened i got ISODate() which is correct native representation of dates:

{ "_id" : ObjectId("54d670db32ad6cc826e610da"), "email" : "[email protected]", "created" : ISODate("1999-12-31T17:00:00Z") }

Is it a bug or feature? Can loopback-connector-mongodb serialize objects to native mongodb format or i should handle this manually?

For some reason, the include.test.js is commented out, and it turns out that some of these tests fail.

Also, common.batch.js doesn't run all tests, manipulation.test.js is missing for example (but will run fine when added).

Hi,

I have a strongloop app working pefectly on local even connecting to a remote mongo instance
But when I deploy on heroku or modulus.io I get ECONNREFUSED error doesnt matter where is the mongo instance,

Somebody have experience on StrongLoop on Heroku or Modulus.io?

Thank in advance!

Antonio

This causes all of my queries with (for example) updateAll to fail.

For example...

MongoDB.prototype.all(model, filter, callback)

If the filter parameter passed into the all method is { "username" : "Terekhov" }, then by the time it gets to

MongoDB.prototype.updateAll(model, where, data, cb)

...the object looks like this:

{
    "username": "Terekhov",
    order:
    [
        "id"
    ]
}

My solution was to replace the use of filter.order with a tempOrder variable, like so:

  var order = {};
  var tempOrder = '';
  if (!tempOrder) {
    var idNames = this.idNames(model);
    if (idNames && idNames.length) {
      tempOrder = idNames;
    }
  }
  if (tempOrder) {
    var keys = tempOrder;
    if (typeof keys === 'string') {
      keys = keys.split(',');
    }
    for (var index = 0, len = keys.length; index < len; index++) {
      var m = keys[index].match(/\s+(A|DE)SC$/);
      var key = keys[index];
      key = key.replace(/\s+(A|DE)SC$/, '').trim();
      if(key === idName) {
        key = '_id';
      }
      if (m && m[1] === 'DE') {
        order[key] = -1;
      } else {
        order[key] = 1;
      }
    }
  } else {
    // order by _id by default
    order = {_id: 1};
  }
  cursor.sort(order);

However, it's entirely possible I just don't understand the reason to include the order property in the query object.

1. If this is so, what is the reason and how can I avoid it polluting my searches in updateAll?
2. If not, thoughts on the above fix, seem reasonable?

which is the properly way to run queries like: GROUP BY , or find DISTINCT documents based on a key in a document's collection through Mongodb connector, also mongodb has $agregation and $mapReduce methods, please help me with this. Thank for answers.

Using a mongodb datasource, mongodb connector and angular sdk, queries using 'inq' and 'nin' fail with a RangeError message. Queries without these filters work fine. An example query:

MyModel.find({ filter: { where: { myField: { inq: ['val1', 'val2', 'val3']}}}}).$promise

Fails with the error message:

{"error": {"name":"RangeError", "status":500,"message":"attempt to write outside buffer bounds", "stack":"RangeError: attempt to write outside buffer bounds\n at Buffer.write (buffer.js:371:11)\n at packElement (/home/rmd/workspace/qb/node_modules/loopback-connector-mongodb/node_modules/mongodb/node_modules/bson/lib/bson/bson.js:877:60)\n at serializeObject (/home/rmd/workspace/qb/node_modules/loopback-connector-mongodb/node_modules/mongodb/node_modules/bson/lib/bson/bson.js:377:15)\n at Function.serializeWithBufferAndIndex (/home/rmd/workspace/qb/node_modules/loopback-connector-mongodb/node_modules/mongodb/node_modules/bson/lib/bson/bson.js:343:10)\n at BSON.serializeWithBufferAndIndex (/home/rmd/workspace/qb/node_modules/loopback-connector-mongodb/node_modules/mongodb/node_modules/bson/lib/bson/bson.js:1556:15)\n at QueryCommand.toBinary (/home/rmd/workspace/qb/node_modules/loopback-connector-mongodb/node_modules/mongodb/lib/mongodb/commands/query_command.js:236:35)\n at Connection.write (/home/rmd/workspace/qb/node_modules/loopback-connector-mongodb/node_modules/mongodb/lib/mongodb/connection/connection.js:265:37)\n at __executeQueryCommand (/home/rmd/workspace/qb/node_modules/loopback-connector-mongodb/node_modules/mongodb/lib/mongodb/db.js:1721:16)\n at Db._executeQueryCommand (/home/rmd/workspace/qb/node_modules/loopback-connector-mongodb/node_modules/mongodb/lib/mongodb/db.js:1871:7)\n at Cursor.nextObject (/home/rmd/workspace/qb/node_modules/loopback-connector-mongodb/node_modules/mongodb/lib/mongodb/cursor.js:749:13)\n at Cursor.toArray (/home/rmd/workspace/qb/node_modules/loopback-connector-mongodb/node_modules/mongodb/lib/mongodb/cursor.js:168:10)"}
}

Subsequent queries result in the connection being closed.

How to connect to mongos in mongodb sharding .

When using default values in a model and creating/upserting a model instance without specifying these values, the mongodb connector stores the document as-is, and does not fetch the default values to use it with $setOnInsert.
http://docs.mongodb.org/manual/reference/operator/update/setOnInsert/#up._S_setOnInsert

Exemple:
My model defined as:

{
  "name": "Device",
  "plural": "Devices",
  "base": "PersistedModel",
  "properties": {
    "deviceId": {
      "type": "string",
      "id": true,
      "required": true
    },
    "culture": {
      "type": "string"
    },
    "phoneNumber": {
      "type": "string"
    },
    "imei": {
      "type": "string"
    },
    "createdAt": {
      "type": "date"
    },
    "updatedAt": {
      "type": "date"
    }
  }
}

And using default value for createdAt with:

Device.definition.rawProperties.createdAt.default =
Device.definition.properties.createdAt.default = function() {
    return new Date();
};

And creating a model instance with:

this.upsert({
            'deviceId': 'PHONE_UNIQUE_ID',
            'culture': 'EN_UK',
            'imei': 'UNIQUE_IMEI',
            'phoneNumber': '+44940395039',
            'updatedAt': new Date()
        });

The resulting data in the mongodb database is:

{
    "_id": "PHONE_UNIQUE_ID",
    "culture": "EN_UK",
    "phoneNumber": "+44940395039",
    "imei": "UNIQUE_IMEI",
    "updatedAt": {
        "$date": "2014-11-26T13:53:43.783Z"
    }
}

As you can see, the createdAt value is missing. If I setup the memory connector, createdAt is here as expected.

I'd like to use aggregate but I have to convert queries like { "upload_time": {"gt": "2014-12-01T00:00:00.000Z"}} to { "upload_time": {$gt: new Date("2014-12-01T00:00:00.000Z")} for $match.
I found MongoDB.prototype.buildWhere function, how could I use it?
Thank you very much.

//json
{
    "acn":"jason",
    "email":"[email protected]",
    "where": {
        "and": [
            { "upload_time": {"gt": "2014-12-01T00:00:00.000Z"}}
        ]
    }
}
//numessage.js
var numessageCollection = Numessage.getDataSource().connector.collection(Numessage.modelName);
numessageCollection.aggregate([
                {$match: { $and:[ { key: { $in: siteIds } }, { "upload_time": {$gt: new Date("2014-12-01T00:00:00.000Z")}} ] } },
                {$group: { _id: {owner:"$owner"}, total: { $sum: 1 } } }
          ], function(err, groupByRecords) {
                ...
            });

It doesn't because it isn't fully correctly recursive. Yet nested specs are needed to query embedded documents. They almost work, with workarounds.

Workaround is at top level to use LoopBack syntax, at lower levels MongoDB syntax. Solves many cases.

Suggested low cost fix: Allow MongoDB syntax at any level.

Example given here. To achieve MongoDB getting

find({"permissions":{$elemMatch:{"user":{$in:["one","other"]},"code":4}}})

had to have a where like

{"permissions":{elemMatch:{"user":{$in:["one","other"]},"code":4}}}

To be clear what is wrong with this picture: Had to have $in because inq wouldn't be replaced. But couldn't make it both $in and $elemMatch because that would have become $$elemMatch.

LoopBack syntax

{"permissions":{elemMatch:{"user":{inq:["one","other"]},"code":4}}}

would result in MongoDB dysfunctional

{"permissions":{$elemMatch:{"user":{inq:["one","other"]},"code":4}}}

A simple fix I'm going to send a pull request for will allow a "pure MongoDB" syntax. Such as

{"permissions":{$elemMatch:{"user":{$in:["one","other"]},"code":4}}}

Seems much easier to implement than complete correct handling.

Some may appreciate the ability to "think and write MongoDB queries" all the way.

Hi, how to use mongodb query with logic 'or' in LoopBack Model? Thanks

Hi,

I have created a model class using the command .

slc lb model privileges -i --data-source mongo --connecor mongodb

Then I have edited the models.json file as shown below.

"privileges": {
"options": {
"relations": {
"user": {
"type": "belongsTo",
"model": "user",
"foreignKey": "user_id"
}
}
},
"properties": {
"user_id": {"type": "string"},
"privilege_user_management": { "type": "string"},
"privilege_country_management": {"type": "string"},
"privilege_menu_management": { "type": "string"}
},
"public": true,
"dataSource": "mongo"

}

I am able to insert data to db using user API as well as privileges API. But I am getting wrong response for filter result.

http://0.0.0.0:3000/api/privileges/findOne?filter={%22user_id%22:%22536b5c0b6e2af02903dd0697%22}

{
"user_id": "536b526a808f9cd902aae749",
"privilege_user_management": "C,R,U,D",
"privilege_country_management": "C,R,U,D",
"privilege_menu_management": "C,R,U,D",
"id": "536b56c56e2af02903dd0694"

}

The response getting back from API corresponds to the first entry in the privilege table. I am getting same result for all the user_id in DB.

Please let me know, whether I am doing something wrong? Do I need to do anything else to implement this method?

I have not written any logic for the model class I have created.

I've combed through the documentation, and searched everywhere for an answer to no avail. My question is how would I define the following model in model.json? Specifically how would I define the emails object property? I basically want to create detailed schemas for my MongoDB models to ensure consistent data. Any help is appreciated.

 var Customer = dataSource.createModel('customer', {seq: {type: Number, id: true}, name: String, emails: [{label: String, email: String}], age: Number});

When a model property defined in the model schema with the value undefined is persisted to MongoDB, the value is converted to null.

This is inconsistent with other connectors like MySQL, as they preserve undefined property values as `undefined.

The fix will IMO break backwards compatibility, thus it will need a new major version.

Test cases for test/mongodb.test.js to reproduce the problem. We may want to move these tests to the juggler's shared suite run by all connectors:

it('create should preserve `undefined` fields', function(done) {
  PostWithStringId.create(
    { title: 'a-title', content: undefined },
    function(err, created) {
      if (err) return done(err);

      created.toObject().should.eql({
        id: created.id,
        title: 'a-title',
        content: undefined
      });

      PostWithStringId.findById(created.id, function(err, found) {
        if (err) return done(err);
        found.toObject().should.eql({
          id: created.id,
          title: 'a-title',
          content: undefined
        });
        done();
      });
    });
});

it('updateOrCreate should preserve `undefined` fields', function(done) {
  PostWithStringId.create(
    { title: 'a-title', content: undefined },
    function(err, instance) {
      if (err) return done(err);
      instance.toObject().should.eql({
        id: instance.id,
        title: 'a-title',
        content: undefined
      });

      PostWithStringId.updateOrCreate(
        { id: instance.id, title: 'updated title' },
        function(err, updated) {
          if (err) return done(err);
          updated.toObject().should.eql({
            id: instance.id,
            title: 'updated title',
            content: undefined
          });
          done();
        });
    });
});

@raymondfeng You have mentioned that we should strip undefined properties from the data. AFAIK, juggler's lib/dao.js is already doing that, see the method removeUndefined in lib/utils.js. This makes the problem even more puzzling.

Hi, This is how i am defining my mongo connection details:- in my loopback config file,
My mongo is a replica set.

"mongo": {
"connector": "mongodb",
"url": "mongodb://90.40.26.46,90.40.26.47/ugc-prod?readPreference=secondary"
},

Now I want to add the "write concern" true here, can anyone help me how to accomplish that?

On line 382 of mongodb.js, result could be null if err is returned in callback from mongodb.

this.collection(model).findAndModify({
    _id: oid
  }, [
    ['_id', 'asc']
  ], data, {upsert: true, new: true}, function (err, result) {
    if (self.debug) {
      debug('updateOrCreate.callback', model, id, err, result);
    }
    var object = result.value;
    if (!err && !object) {
      // No result
      err = 'No ' + model + ' found for id ' + id;
    }
    self.setIdValue(model, object, id);
    object && idName !== '_id' && delete object._id;
    callback && callback(err, object);
  });

Hi,

I am facing issue in updating db using updateattributes.

Getting id related error.

Orginaluser.prototype.updateAttributes(UserDetail.user_details, function(err, theOrginaluser){

});

/Users/company/JavaScriptWorkSpace/myproject/node_modules/loopback-connector-mongodb/node_modules/mongodb/lib/mongodb/connection/base.js:245
throw message;
^
TypeError: Cannot set property 'user_id' of undefined
at /Users/company/JavaScriptWorkSpace/myproject/node_modules/loopback-datasource-juggler/lib/dao.js:888:37

Please help as soon as possible.

https://groups.google.com/forum/#!searchin/loopbackjs/updateattribute/loopbackjs/f2yx26ZBHGI/Cl7RZBoMvHYJ