turistforeningen / node-mongo-querystring

Query builder for accepting URL query parameters into your MongoDB queries. Safe and feature rich. Supports most of MongoDB's query operators such as $eq, $gt, $lt, $ne, $in, $nin, $exists, $regex, geospatial queries such as bbox and near, as well as your own custom query business logic!

MongoDB QueryString Parser

Accept MongoDB query parameters through URI queries safe and easy. This is useful when building an API and accepting various user specificed queries.

Features

• Aliased query parameters
• Blacklisted query parameters
• Whitelisted query parameters
• Basic operators
  • $eq
  • $gt
  • $gte
  • $lt
  • $lte
  • $ne
  • $in
  • $nin
  • $exists
  • $regex
• Parse string integers and floats to numbers
• Parse string boolean to ture/false booleans

operation	query string	query object
equal	?foo=bar	{ foo: "bar" }
unequal	?foo=!bar	{ foo: { $ne: "bar" }}
exists	?foo=	{ foo: { $exists: true }}
not exists	?foo=!	{ foo: { $exists: false }}
greater than	?foo=>10	{ foo: { $gt: 10 }}
less than	?foo=<10	{ foo: { $lt: 10 }}
greater than or equal to	?foo=>=10	{ foo: { $gte: 10 }}
less than or equal to	?foo=<=10	{ foo: { $lte: 10 }}
starts with	?foo=^bar	{ foo: { $regex: "^bar", $options: "i" }}
ends with	?foo=$bar	{ foo: { $regex: "bar$", $options: "i" }}
contains	?foo=~bar	{ foo: { $regex: "bar", $options: "i" }}
in array	?foo[]=bar&foo[]=baz	{ foo: { $in: ['bar', 'baz'] }}
not in array	?foo[]=!bar&foo[]=!baz	{ foo: { $nin: ['bar', 'baz'] }}

• Geospatial operators
  • $geoWithin (polygon)
  • $near (point)

operation	query string	query object
bbox	?bbox=0,1,2,3	{ geojson: { $geoWithin: { $geometry: { … } } } }
near	?near=0,1	{ geojson: { $near: { $geometry: { … } } } }
near (max distance)	?near=0,1,2	{ geojson: { $near: { …, $maxDistance: 2 } } }
near (max & min distance)	?near=0,1,2,3	{ geojson: { $near: { …, $minDistance: 3 } } }

• Custom query functions
  • after (date)
  • before (date)
  • between (date|date)

operation	query string	query object
after	?after=2014-01-01	{ endret: { $gte: "2014-01-01T00:00:00.000Z" } }
after	?after=1388534400	{ endret: { $gte: "2014-01-01T00:00:00.000Z" } }
before	?before=2014-01-01	{ endret: { $lt: "2014-01-01T00:00:00.000Z" } }
before	?before=1388534400	{ endret: { $lt: "2014-01-01T00:00:00.000Z" } }
between	?between=2014-01-01|2015-01-01	{ endret: { $gte: "2014-01-01T00:00:00.000Z", $lt: "2015-01-01T00:00:00.000Z" } }
between	?between=1388534400|1420088400	{ endret: { $gte: "2014-01-01T00:00:00.000Z", $lt: "2015-01-01T00:00:00.000Z" } }

Install

npm install mongo-querystring --save

API

var MongoQS = require('mongo-querystring');

new MongoQS(object options)

• Array ops - list of supported operators (default: ['!', '^', '$', '~', '>', '<', '$in'])
• object alias - query param aliases (default: {})
• object blacklist - blacklisted query params (default: {})
• object whitelist - whitelisted query params (default: {})
• object custom - custom query params (default: {})
• object string - string parsing
  • boolean toBoolean - parse "true", "false" string to booleans (default: true)
  • boolean toNumber - parse string integer and float values to numbers (default: true)
• regexp keyRegex - allowed key names (default: /^[a-zæøå0-9-_.]+$/i)
• regexp arrRegex - allowed array key names (default: /^[a-zæøå0-9-_.]+(\[\])?$/i)

Bult in custom queries

• bbox - bounding box geostatial query
• near - proximity geostatial query
• after - modified since query

var qs = new MongoQS({
  custom: {
    bbox: 'geojson',        // your geometry field
    near: 'geojson',        // your geometry field
    after: 'updated_on'     // your last modified field
  }
});

Define custom queries

Custom queries are on the folling form; you define the URL query parameter name that your users will be using and a function which takes the result query object and the value for query parameter.

var qs = new MongoQS({
  custom: {
    urlQueryParamName: function(query, input) {
      // do some processing of input value
      // add your queries to the query object
      query['someField'] = input;
      query['someOtherFiled'] = 'some value';
    }
  }
});

qs.parse(object params)

Params is an object with URI query params and their values. Ex. req.params if you are working with ExpressJS.

var query = qs.parse(req.params);

mongo.collection('mycol').find(query, field).toArray(function(err, documents) {
  // matching documents
});

Collaborators

Individuals making significant and valuable contributions are made Collaborators and given commit-access to the project. These individuals are identified by the existing Collaborators and their addition as Collaborators is discussed as a pull request to this project's README.md.

Note: If you make a significant contribution and are not considered for commit-access log an issue or contact one of the Collaborators directly.

• Andy Klimczak - @andyklimczak
• Hans Kristian Flaatten - @Starefossen
• Edward Knowles - @eknowles

MIT Licensed

node-mongo-querystring's Issues

valRegex improperly referenced

node-mongo-querystring/index.js

Line 184 in 9e3554c

ret.value = org.replace(this.valReqex, '');

• should be this.valRegex not this.valReqex
• but then this will replace all valid characters with '', instead of invalid chars
• and is also overwritten for ^ and $ a few lines below
  • this should be ret.value = `^${ret.value}`; instead
• an option like escapeRegex: true might be more appropriate here
  • or escapeRegexChars: /[\\^$.*+?()[\]{}|]/g, where use s.replace(this.escapeRegexChars, '\\$&') to escape (source: lodash _.escapeRegex)

Is there a way to query by ObjectID?

The node mongo driver supports querying by ObjectID. Is there a way to do this using mongo-querystring? If not, would this be hard to implement?

Add support for min and max distance for `near` query

equal to

Currently there isn't a way to use $gte. Need to find a url save symbol for equal.

near query parameters

When querying near leaving out the $minDistance parameters it fails.
near=12.1234,13.1345,2000
also when using a float for the $maxDistance it fails for me
near=12.1234,13.1345,2000.1,0

'[]' stripped from key names, array Regex fails, breaking $in & $nin

As the title states, currently with in an Express environment (using qs as a query parser), the trailing brackets are being stripped from the key names. This causes the regex to fail, as it is looking for foo[] and receiving foo instead. So both $in and $nin are currently broken.

accents or tildes query url, it does not work

Hello, thank you very much for this plugin, it has facilitated me a lot of the searches of my web application.

For me everything works very well, except in the searches of accents or tildes.

I have trouble using a search without tilde or accent.

Here are some examples:

with accent
url.com/fichas?txt_nombredescriptivo=~cerámica

results 10
with capital letters
url.com/fichas?txt_nombredescriptivo=~Cerámica

results 10
without accent
url.com/fichas?txt_nombredescriptivo=~ceramica

results 0

Since the mongodb console works for me without the accents, but from the url query no.

Thank you for your time and dedication. Greetings. :)
And excuse my English, :(

Filtering string fields will not work when filter value consists of numbers

qs.parse({"foo":">1"}) results in {"foo":{"$gt":1}} and not {"foo":{"$gt":"1"}} where the '1' is converted to a Number field automatically. If the foo field is a string field in the database than this search query will find nothing.

I haven't tested it but this will probably also be true for booleans. So you probably can't filter a text field on the word 'true'.

There should be a way to force the creation of a string filter for values that could also be a number or a boolean.

Better examples

Better examples are needed, preferably a complete scenario.

before, after, between should use date object instead of date string

In the mongodb shell I get no results if I try to pass a query such as the following: { endret: { $gte: "2014-01-01T00:00:00.000Z" } }. However, this does work for me: { endret: { $gte: new ISODate("2014-01-01T00:00:00.000Z") } }. This also works from javascript: { endret: { $gte: new Date("2014-01-01T00:00:00.000Z") } }.

Is this an error or am I missing something? maybe something changed in newer mongodb versions?

problem accents/diacritics

Hi, I'm new to programming, and I still can not find the solution to the next problem of accent searches, they could help me or give me an orientation. Thank you very much.

Hello, thank you very much for this plugin, it has facilitated me a lot of the searches of my web application.

For me everything works very well, except in the searches of accents or tildes.

I have trouble using a search without tilde or accent.

Here are some examples:

with accent
url.com/fichas?txt_nombredescriptivo=~cerámica

results 10
with capital letters
url.com/fichas?txt_nombredescriptivo=~Cerámica

results 10
without accent
url.com/fichas?txt_nombredescriptivo=~ceramica

results 0

Since the mongodb console works for me without the accents, but from the url query no.

Thank you for your time and dedication. Greetings. :)
And excuse my English, :(

Custom array query

I want to create a custom query for an array. This is my structure:

{
  _id: 1234,
  assigned: {
    users: [{
      _id: 1111
    }]
  }
}

I expect that for this query

?assigned[]=1111&assigned[]=2222

I would write a function

function assigned(query, input) { // where input is [1111,2222]
  query['assigned.users._id'] = { $in: input }
}

Unable to filter certain strings and decimal values

I've pulled down the sample code to test out the filtering for a monitor resolution api. The only change I've made to the provided sample code in examples/app.js is the qs variable

app.js

...
var qs = new MongoQS({
  custom: {
    //bbox: 'geojson',
    //near: 'geojson',
    exists: 'hdmi',
    equal: 'resolution'
  }
});
...

Database:

[
  {
    "series": "meh-monitor",
    "hdmi": false,
    "hdmi_version": "",
    "vga": true,
    "resolution": "1366x768"
  },
  {
    "series": "ok-monitor",
    "hdmi": true,
    "hdmi_version": "1.0",
    "vga": true,
    "resolution": "1920x1080"  
   },
  {
    "series": "good-monitor",
    "hdmi": true,
    "hdmi_version": "1.2",
    "vga": false,
    "resolution": "2560x1440"
  }  
]

Working Sample URLs:

• localhost:3000/api/places works returns all objects
• localhost:3000/api/places?hdmi=true, works returns objects with hdmi being true
• localhost:3000/api/places?series=ok-monitor, works and returns only the status "ok-monitor"

Non working sample URLs:

• localhost:3000/api/places?resolution=2560x1440
• localhost:3000/api/places?hdmi_version=1.2, do want to note running this the qs variable had equal: 'hdmi_version'

Provide Type Definitions

We're using TypeScript with our NodeJS server, so it would be great if there were some type definitions to go with this library (which works great!).

So far we're just using the following in our typings.d.ts to work around this atm:

declare module 'mongo-querystring' {
  export = class MongoQS {
    constructor(options: any)
    public parse (params: any): any
  }
}

Providing an index.d.ts file in the root directory of this project would be the most basic way of doing this. Publishing something to @types/node-mongo-querystring would be the more official way.

Thank you for considering this.

Want to encapsulate the query string into a single URL parameter

Hello - great project.

My requirements are slightly different to the default behaviour of node-mongo-querystring.

Given a URL like /?mongoquery=name=simon^location=London&page=2&rows=10

I would like to access my Mongo database query just from the mongoquery URL parameter.

I considered using node-mongo-querystring and base64 encoding the query so that it could be encapsulated, but that kills URL readability. Is there a good solution you can think of?

Why discard Booleans in parse?

The following test will fail, since it totally disallow the non-string values (stated in test.js line 520)

it('return boolean as boolean', () => {
  query = mqs.parse({
    foo: true,
    bar: false,
  });
  assert.deepEqual(query, {
    foo: true,
    bar: false,
  });
});

But here is my use case, I'm using fastify and utilized their integrated ajv as my validator. My ajv was configured to do a type coercion, hence, if I pass the following query ?foo=true&bar=false the returned value from validator are boolean values.

{
  foo: true,
  bar: false
}

If I pass this to mqs.parse() it returns an empty object since mqs totally disallow bool values.

Turning off the coercion is not an option for me. And converting it back to string is a dirty hack. I guess it might be good if this will allow by mqs itself.

I can send a PR and test, but I want to know the importance of discarding the boolean values in test.js line 520, if there's any.

Escaping operators

Hi, I've been playing a little with your lib but I found no way of escaping operators that occur in the first part of the values. For example, what would be the querystring for the following output:
{"foo":"!baz"}.

I've tried:
?foo=!baz => {"foo":{"$ne":"baz"}} (duh)
?foo=%21baz => {"foo":{"$ne":"baz"}} (ExpressJS resolves %21 before your lib sees it)
?foo=%27!baz%27 => {"foo":"'!baz'"} same problem, resolved by ExpressJS

But even without ExpressJS in between % escaping it doesn't seem to work.

P.S. Like another poster I would also like to combine the whole expression into one q=... parameter, otherwise it would cause conflicts with other meaningful parameters like page, limit etc.

Correct way to update and handle input values in custom function

First of all, awesome library so thank you. I'm struggling just a bit however and likely doing something dumb.

Part of my final query will need to do an $elemMatch. I didn't see this option in the docs so I figured custom function. The final structure of this piece of the mongo query should be:
place_amentities: {'$elemMatch': {lounge: 1, restrooms: 1}}

The params in the URL would be ?pa=lounge,restrooms

Pretty straightforward custom function:
pa: function(query, input) { const amentitiesArr = input.split(',').map(a => a +':'+1); query['place_amentities'] = { $elemMatch: amentitiesArr }; }

The issue I'm having is that final output looks like this:
place_amentities: { '$elemMatch': [ 'lounge:1', 'restrooms:1'] }

Instead of this:
place_amentities: { '$elemMatch': { lounge: 1, restrooms: 1} }

Everything seems to be a string. The core library doesn't do that though, everything is created exactly as Mongo would expect it, so I'm sure I'm just screwing up something simple in the custom function.

Thanks for any help you can provide. I know this isn't an "issue" but maybe others have run into the same question.

Parsing String 'null'

How to parse key=null?
Want to get null not 'null'.

Add support for $geoIntersects query

URL param:

?intersects=lon,lat

Should give this query:

​geojson: {
  $geoIntersects: {
    $geometry: {
      type: "Point" , coordinates: [ lat, lon ]
    }
  }
}

Not and return

Firstly thanks for this great library!

However the only operators I think are missing are the ability to control was it returned, I use the syntax "not" to exclude or "return" to include what fields are returned within document from mongo,

Use new ES6 / ES2015 syntax

Using $gt and $lt together

Hi,

thanks a lot for your effort on this library - extremely useful!

I'm having a problem, though, when using it to parse query params sent to a hapijs server and trying to specify a range of values for a parameter, such as:

url?count=>10&count=<100

parsed query comes out as:

count: { '$in': [ '>10', '<100' ] } }

The expected result would, naturally, be:

count: {$gt: 10, $lt: 100}

Do you have any suggestions on how to work around this, or is this something you would consider supporting?

Thanks!

Pagination

Hey,

Any thoughts on how to implement a custom page/limit filter? Near as I can tell this itsn't handled in a normal query which gets built, but are additional operators find().limit().skip()

Hexadecimal strings are coerced into (incorrect) numbers

Hi there,

First off: great work.

We've bumped into a bit of an issue when trying to pass long hexadecimal strings. When passing these, mongo-querystring will coerce this into a number. Example: 0xfbede9b15fbeec9e1579653ac3af52b0a54ec2a3725150c64e681264cb18cbf6 gets converted to 0, which is clearly not the intention of the caller.

Ideally (but this is debatable), I believe values in hexadecimal notation should/could be passed on as strings instead of numbers. Most of the time in Mongoland, this kind of value is stored as a string (eg a hash of some sorts).

Questions that remain could be:

• How do we detect hexadecimal notation? (is 0x enough or do we want to go more in-depth?)
• What if we want this to convert to binary instead (some store this kind of value as binary instead, but I'd like to think this is a rare case)
• Should we strip the 0x or not? (guess this could be a user option)

Perhaps start a brief debate on this, and I'd be happy to provide a PR once you settle on a potential approach [if any, of course]? I'm also aware number parsing can be disabled altogether, but we have a need for parsing to numbers in other cases.

PS: for now, although horribly inefficient and not robust since it can result in false positives, this can be worked around by using the ~ operator.

Can't make OR query for multiple parameters

Var object=[{id1:1,id2:3},{id1:4,id2:8}]

Collection name is 'logindata'
Query: ?id1=1 OR id2:9

So I get one record : {id1:1,id2:3}

Please help me out

Thanks

Custom near query not working

I'm having a problem with the custom near query again, using v4.4.1
If I specify a name for the query like so:

custom: {
    …
    near: 'loc'
}

It throws an error from Mongo:

error processing query: ns=echoesfullstack-dev.collections limit=10Tree: $and    pub_status == \"public\"
    GEONEAR  field=(query, point) => {
  const pointArr = point.split(',').map(p => parseFloat(p, 10));

  if (pointArr.length >= 2) {
    if (!isNaN(pointArr.reduce((a, b) => a + b))) {
      const max = pointArr[2];
      const min = pointArr[3];

      query[field] = {
        $near: {
          $geometry: {
            type: 'Point',
            coordinates: pointArr.splice(0, 2),
          },
        },
      };

      if (!isNaN(max)) {
        query[field].$near.$maxDistance = max;

        if (!isNaN(min)) {
          query[field].$near.$minDistance = min;
        }
      }
    }
  }
} maxdist=1.79769e+308 isNearSphere=0
Sort: {}
Proj: {}
 planner returned error: unable to find index for $geoNear query

It seems to be returning the entire function as a query. Is that expected behaviour?

I manually overrode the query with :

if (query.near) {
    let pointArr = req.query.near.split(',').map(p => parseFloat(p, 10));
    let n = {
      $near: {
        $geometry: {
          type: 'Point',
          coordinates: pointArr.splice(0, 2),
        }
      }
    };
    // maybe we have $maxDistance
    if (pointArr.length)
      n.$near.$maxDistance = parseFloat(pointArr.shift());
    delete query.near
    query.loc = n;
  }

and it now works fine for me.