arunoda / meteor-cluster

Smarter way to run cluster of meteor nodes

This is an archived project

Are you looking for our newest Meteor Cluster project?

Meteor Cluster is now retired & Meteor's Collection implementation implementation with Oplog support can be used to run a cluster of Meteor nodes.

meteor-cluster

Smarter way to run cluster of meteor nodes

See: Meteor Cluster - Introduction & How it works article

Meteor is not a toy anymore. People keen to build enterprise apps on top of the meteor. So people need to run cluster of meteor nodes for several reasons.

But when running cluster of nodes, meteor is not realtime anymore(between nodes). It get synced but takes few seconds.

Here comes the solution - meteor-cluster

Installation

just run mrt add cluster

Redis

meteor-cluster uses redis as the communicate channel between nodes. It uses pub/sub functionality of redis. So you need to have redis-server running.

If you are new to redis, read this guide

Configurations

meteor-cluster needs to know

• how to connect with redis
• what collections it needs to sync

It just a two lines of configuration. Add following inside you server code.

Meteor.startup(function() {
    Meteor.Cluster.init(); //assumes you are running redis-server locally
    Meteor.Cluster.sync(Posts, Notifications, Comments); //these are my collections
});

How to scale

Just fire up new nodes and it just works.

Tests

• start redis-server before testing
• mrt test-packages ./

API

Meteor.Cluster.init(redisConfig)

Initialize and connect to redis.

• redisConfig - null or {port: 6337, host: "localhost", auth: null, db: null}
• alternatively you can pass redis config via CLUSTER_URL env variable
• CLUSTER_URL formart: redis://redis:<password>@<host>:<port>

Meteor.Cluster.sync(collections...)

Sync given set of collection between nodes

• collections... - Collections defined as list of arguments

meteor-cluster's Issues

Amazon ElastiCache

Does this work with Amazon's ElastiCache? I don't think ElastiCache works with password. What's the best way to configure this?

License?

@arunoda I was wondering what is the license for your code... Thanks

run a meteor app in two separate servers

i am trying to run a meteor app in two separate servers, with a separate mongodb each, and just pointing meteor cluster of one of the nodes to the redis of the other. it is NOT syncing ok !?? what am i missing ?

server1
Meteor.Cluster.init(); // redis server is running at port xxxx

at server2
Meteor.Cluster.init({port: xxxx , host: "server1", auth: null, db: null});

404s from /cluster-ddp

Hi, I'm using meteor-cluster in the following scenario:

• AWS
• 2 instances running meteor-cluster as normal app-servers (e.g. not balancers)
• Deployment using mup
• ELB
• SSL termination on ELB
• DISABLE_WEBSOCKETS (since ELB+SSL)
• The site itself (if you want to test) is https://gormim.com

Most of the time things work well, but since I have an alert on ELB looking for 4xx I noticed that every once in awhile the calls to POST /cluster-ddp are 404ing

Here's just one screenshot of the 404s on my ELB.
http://cl.ly/image/17083P0j2x07
And I recorded the access logs, so here's an example request that 404s

2015-02-28T17:41:57.787697Z web 66.249.78.7:60042 172.31.38.90:80 0.00012 0.003284 0.000023 404 404 417 26 "POST https://gormim.com:443/cluster-ddp/f4a987ea4fac27bdc2f00df9dd01a691e23f29b8/web/352/etyjoyb4/xhr_send HTTP/1.1"

Most of the /cluster-ddp calls end with 200 (or 204), but occasionally they 404.
Here are just a few examples or 2xx lines:

2015-02-28T17:41:18.573685Z web 109.64.50.27:53781 172.31.12.222:80 0.000073 30.001697 0.000045 200 200 0 24 "POST https://gormim.com:443/cluster-ddp/f4a987ea4fac27bdc2f00df9dd01a691e23f29b8/web/607/lqhhw6rh/xhr HTTP/1.1"
2015-02-28T17:41:49.567741Z web 109.64.50.27:53761 172.31.12.222:80 0.000116 0.004414 0.000032 204 204 22 0 "POST https://gormim.com:443/cluster-ddp/f4a987ea4fac27bdc2f00df9dd01a691e23f29b8/web/607/lqhhw6rh/xhr_send HTTP/1.1"

As far as I can tell, this does not interfere in the sense that users are bothered by it, but it does make monitoring really hard (b/c it's hard to say what's a real error and that's just 404 from /cluster-ddp).

I wonder if you had seen this before or could guide me towards a solution?

Thanks!

Cluster Redirects POST Request

Observed Behaviour

We have a cluster of 3 servers, and we have a static url (with iron:router). We make a POST request and sometimes it seems that cluster is redirecting the request to another instance, making the POST request fail.

@arunoda is there anyway to solve this?

Thanks a lot for your work!

Not working in latest 0.6.5 meteor

I get this error after upgrading:

W20130815-05:53:23.000(3)? (STDERR) /home/manfred/.meteor/tools/4010e5731d/lib/node_modules/fibers/future.js:173
W20130815-05:53:23.001(3)? (STDERR) throw(ex);
W20130815-05:53:23.002(3)? (STDERR) ^
W20130815-05:53:23.006(3)? (STDERR) ReferenceError: Random is not defined
W20130815-05:53:23.006(3)? (STDERR) at new Cluster (packages/cluster/lib/cluster.js:3)
W20130815-05:53:23.006(3)? (STDERR) at Package (packages/cluster/lib/cluster.js:133)
W20130815-05:53:23.006(3)? (STDERR) at packages/cluster.js:233:4
W20130815-05:53:23.007(3)? (STDERR) at packages/cluster.js:240:3
W20130815-05:53:23.007(3)? (STDERR) at mains (/home/manfred/Dropbox/Projects/heroico/.meteor/local/build/programs/server/boot.js:153:10)
W20130815-05:53:23.007(3)? (STDERR) at Array.forEach (native)
W20130815-05:53:23.007(3)? (STDERR) at Function..each..forEach (/home/manfred/.meteor/tools/4010e5731d/lib/node_modules/underscore/underscore.js:79:11)
W20130815-05:53:23.008(3)? (STDERR) at /home/manfred/Dropbox/Projects/heroico/.meteor/local/build/programs/server/boot.js:80:5

Updates and delay time in a shared database cluster

I am writing an application in two parts with the first part (Heavy back end work) in typescript and node.js and the front end client work in meteor.js. The first part will write directly to a mongo database that my second part will read and use, when i set up meteor-cluster with a hosted redis instance and i define all the collections i would like to sync. How long does it take from when my first part of the application writes to a shared cluster of mongo databases until my meteor.js reads and displays the data to the user. I understand that meteor will read the db every 10 seconds but i am thinking this will allow it to be faster ?

Potential for using DDP.connect instead of Redis

With 0.6.5, DDP is supported for server to server connections. Is it feasible to use this instead of redis for event propagation across the cluster?

Meteor.connect has been renamed to DDP.connect and is now fully supported on the server. Server-to-server DDP connections use websockets, and can be used for both method calls and subscriptions.

From https://github.com/meteor/meteor/blob/devel/History.md