HA Solution about mcrouter

[pavlo-fb]

AllSyncRoute returns the worst reply, so if one server is down mcrouter returns an error (because it didn't manage to deliver request to both servers). If you don't care about errors that occur on write path (set and delete operations), use AllAsyncRoute instead of AllSyncRoute. Mcrouter will not wait for reply from memcached and will always return "NOT_STORED" for set and "NOT_FOUND" for delete.
Hope it helps.

[liyufu]

if there is any config i can study for? like (#9)
if there is any solution for blow Scene.
if one server is down,the mcrouter can write the data to other running memcached server.

[pavlo-fb]

if there is any config i can study for? like (#9)

Your config is pretty much the same as in #9 .

if there is any solution for blow Scene.

I'm not sure what is "blow Scene", sorry.

if one server is down,the mcrouter can write the data to other running memcached server.

With your current config mcrouter writes data to both memcached servers. You are probably confused with reply from mcrouter (in case one memcached is down, mcrouter returns SERVER_ERROR with your current config).
The reason is AllSyncRoute. It returns the worst reply received from both memcached servers (i.e. when one server is down, mcrouter returns SERVER_ERROR). This error doesn't mean that request to other memcached server failed or wasn't sent.
If you don't want to see errors from mcrouter when one memcached is down, you can replace AllSyncRoute with AllAsyncRoute. In this case mcrouter will ignore replies from memcached and will return NOT_STORED for set/NOT_FOUND for delete.

3:if all the memached server is all in the same datacenter,if there is only one pools is enough?

In general, if you just want to replicate data to multiple memcached servers, you don't need multiple pools. But it doesn't mean your config is worse if you use 2 pools. If you still want to use one pool, check example in https://github.com/facebook/mcrouter/wiki/Replicated-pools-setup

[liyufu]

Thanks for the reply! A few follow up questions. :-)

1 ： In some cases,the type FailoverRoute is only used between multiple pools.

2：The Sharded Pools is often used between one pool with multiple memcached servers or between multiple pools ?

3：I want to config the cache server like this picture。
While the mc1(memecached1) is down, the all mcrouter can send the requests(get\set\delete) to the mc2(memecached2)，the mc1 and mc2 server data is Replicated ,how to config it.

my config

config1

{
  "pools": {
    "cachea": {
      "servers":["172.16.200.98:11211","172.16.200.99:11211"]
    }
  },
 "route": {
    "type": "OperationSelectorRoute",
    "operation_policies": {
      "delete": "AllSyncRoute|Pool|cachea",
      "get": "AllFastestRoute|Pool|cachea",
      "set": "AllSyncRoute|Pool|cachea"
    }
 }
}

config2

{
    "pools": {
        "cache1": {
            "servers": [
                "172.16.200.98:11211"
            ]
        },
        "cache2": {
            "servers": [
                "172.16.200.99:11211"
            ]
        }
    },
    "routes": [
        {
            "route": {
                "type": "OperationSelectorRoute",
                "default_policy": "PoolRoute|cache1",
                "delete": {
                    "type": "AllSyncRoute",
                    "children": [
                        "PoolRoute|cache1",
                        "PoolRoute|cache2"
                    ]
                },
                "set": {
                    "type": "AllSyncRoute",
                    "children": [
                        "PoolRoute|cache1",
                        "PoolRoute|cache2"
                    ]
                },
                "get": {
                    "type": "AllFastestRoute",
                    "children": [
                        "PoolRoute|cache1",
                        "PoolRoute|cache2"
                    ]
                }
            }
        },
        {
            "route": {
                "type": "FailoverRoute",
                "children": [
                    "cache1",
                    "cache2"
                ]
            }
        }
    ]
}

[pavlo-fb]

Sorry for the delay :(

1. You can use FailoverRoute both to failover between pools and to failover between servers of one pool.
   Failover from pool A to pool B:

{
  "type": "FailoverRoute",
  "children": [ "PoolRoute|A", "PoolRoute|B" ]
}

Failover from server 0 to server 1, 2, ... n of pool A:

"FailoverRoute|Pool|A"

or

{
  "type": "FailoverRoute",
  "children": "Pool|A"
}

2. The same here, you can choose a route based on key hash. It works both with entire pools and servers of one pool (choosing servers of one pool is more useful option in my opinion).
   Sharded pool:

{
  "type": "HashRoute",
  "children": "Pool|A"
}

or, shorter:

"HashRoute|Pool|A"

almost the same, but it also adds AsynclogRoute in front:

"PoolRoute|A"

To choose different pools based on a key hash:

{
  "type": "HashRoute",
  "children": [ "PoolRoute|A", "PoolRoute|B" ]
}

3. First config is correct. The only drawback is it always sends get requests to both servers, so may produce more traffic than is actually needed. Here is a bit "fixed" version:

{
  "pools": {
    "cachea": {
      "servers":["172.16.200.98:11211","172.16.200.99:11211"]
    }
  },
 "route": {
    "type": "OperationSelectorRoute",
    "operation_policies": {
      "delete": "AllSyncRoute|Pool|cachea",
      // the difference is here: FailoverRoute instead of FastestRoute
      "get": "FailoverRoute|Pool|cachea",
      "set": "AllSyncRoute|Pool|cachea"
    }
 }
}

In this "fixed" version all get requests are sent to "172.16.200.98:11211". But if it is down, gets will go to "172.16.200.99:11211".

The second config you proposed is broken, here is a version that is equivalent to the first one you proposed:

{
    "pools": {
        "cache1": {
            "servers": [
                "172.16.200.98:11211"
            ]
        },
        "cache2": {
            "servers": [
                "172.16.200.99:11211"
            ]
        }
    },
    "route": {
        "type": "OperationSelectorRoute",
        "default_policy": "PoolRoute|cache1",
        "delete": {
            "type": "AllSyncRoute",
            "children": [
                "PoolRoute|cache1",
                "PoolRoute|cache2"
            ]
        },
        "set": {
            "type": "AllSyncRoute",
            "children": [
                "PoolRoute|cache1",
                "PoolRoute|cache2"
            ]
        },
        "get": {
            /**
             * the same here: change to `FailoverRoute` if you want to send get requests to cache1, 
             * but if it is down, send them to cache2
             */
            "type": "AllFastestRoute",
            "children": [
                "PoolRoute|cache1",
                "PoolRoute|cache2"
            ]
        }
    }
}

[alikhtarov]

Closing due to inactivity, feel free to re-open if more questions come up.

[vengit123]

Hi Following is my config file.
{
"pools": {
"cache1": {
"servers": [
"172.19.19.43:11311"
]
},
"cache2": {
"servers": [
"172.19.19.48:11311"
]
}
},
"route": {
"type": "OperationSelectorRoute",
"operation_policies": {
"set": {
"type": "AllAsyncRoute",
"children": [ "PoolRoute|cache1", "PoolRoute|cache2" ]
}
},
"default_policy": "PoolRoute|cache1"
}
}

My expectation is as follows:

1. Any add operation should add records into both the servers, when both mc servers i.e., "cache1 & cache2" are available (OR) should be added to any available server.

But, as per my observation "set" operation_policy is not getting hit. Instead, it is always taking default_policy and adding record only in cache1 eventhough cache2 is up and running.

Can any one help? What is wrong in the configuration? I am using mcrouter-2.0.0.0.

Thanks in-advance.
Venkat.

[vengit123]

This got solved after adding "add" policy in "operation_policies";
"add": {
"type": "AllAsyncRoute",
"children": [ "PoolRoute|cache1", "PoolRoute|cache2" ]
}