Retry failed requests in FastRouter about uwsgi

[unbit]

the rawrouter has already a retry subsystem, do you have a way to stress-test it to see if it is whorty to port it on fastrouter and httprouter too ?

[prymitive]

I did a little test:

# starting rawrouter
uwsgi --rawrouter :2200 -M --rawrouter-to 127.0.0.1:2201 --rawrouter-to 127.0.0.1:2202 --rawrouter-stats :2203 --rawrouter-max-retries 2

# starting 2 nodes
uwsgi --master --http :2201 --wsgi-file tests/staticfile.py
uwsgi --master --http :2202 --wsgi-file tests/staticfile.py

# ab run
ab -i -r -n 100000 http://localhost:2200/
Concurrency Level:      1
Time taken for tests:   12.202 seconds
Complete requests:      3459
Failed requests:        1
   (Connect: 0, Receive: 0, Length: 1, Exceptions: 0)
Write errors:           0

During ab run I have stopped second node, rawrouter logged this:

rr_instance_read(): Connection reset by peer [plugins/rawrouter/rawrouter.c line 114]
[uwsgi-rawrouter] unable to connect() to node "127.0.0.1:2201": Connection refused
[uwsgi-rawrouter] :2200 => marking 127.0.0.1:2201 as failed

But I still have 1 failed request. Am I doing it right?

[unbit]

the first error means the backend has been shutdown during a transfer, so the rawrouter already sent some data to the client, so it is not possibile to retry. The second one (Connection refused) should be the first error triggering a retry

[prymitive]

I just pressed ctr+c on one node, shouldn't it close gracefully after request is done?
I'm nor running ab with any concurrency so there should be no requests in backlog that could get killed.

[unbit]

gateways/router do not have graceful reload concept, they get destroyed as soon as the master trigger the shutdown/reload

[prymitive]

Would it be better if I tested it using --rawrouter-subscription-server? If so than what format for --subscribe-to should I use? Obviously I can't put domain there, and if I just use --subscribe-to localhost:port nothing happens, with --subscribe-to localhost:port:xxx I'm getting sendto(): Permission denied [core/protocol.c line 338]

[prymitive]

Can You help me with the question above?

[unbit]

Sorry for the late, regarding subscription on rawrouter it takes the form of ip:port (the tuple on which the rawrouter is bound).

[unbit]

Fallback will be much better in the next router-related commit (the one in which i hope SPDY support will be available too ;). I think i still need a couple of days...

[unbit]

you should be able to test the fallback system with current code

[prymitive]

Current master is broken:

cc1: warnings being treated as errors
core/protocol.c: In function ‘uwsgi_proto_check_15’:
core/protocol.c:660: error: ordered comparison of pointer with integer zero

[unbit]

just fixed it

[prymitive]

I still can't make rawrouter to work with subscriptions:

lukasz.mierzwa@sudoku-admin:~/uwsgi$ ./uwsgi --rawrouter :2200 -M --rawrouter-subscription-server 127.0.0.1:2100 --rawrouter-stats :2203 --rawrouter-max-retries 2 --rawrouter-use-cluster
*** Starting uWSGI 1.5-dev-744fed6 (64bit) on [Thu Jan 24 20:45:03 2013] ***
compiled with version: 4.4.3 on 24 January 2013 20:43:28
os: Linux-3.0.0-23-virtual #39~lucid1-Ubuntu SMP Fri Jul 20 19:37:29 UTC 2012
nodename: sudoku-admin
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 4
current working directory: /home/lukasz.mierzwa/uwsgi
detected binary path: /home/lukasz.mierzwa/uwsgi/uwsgi
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
uWSGI rawrouter bound on :2200 fd 3
uWSGI rawrouter subscription server bound on 127.0.0.1:2100 fd 4
Python version: 2.6.5 (r265:79063, Oct  1 2012, 22:16:31)  [GCC 4.4.3]
*** Python threads support is disabled. You can enable it with --enable-threads ***
Python main interpreter initialized at 0x224b790
*** Operational MODE: no-workers ***
spawned uWSGI master process (pid: 3188)
spawned uWSGI rawrouter 1 (pid: 3189)
*** rawrouter stats server enabled on :2203 fd: 9 ***

lukasz.mierzwa@sudoku-admin:~/uwsgi$ ./uwsgi --master -s 172.16.200.58:2201 --wsgi-file tests/staticfile.py --subscribe-to 127.0.0.1:2100
*** Starting uWSGI 1.5-dev-744fed6 (64bit) on [Thu Jan 24 20:45:18 2013] ***
compiled with version: 4.4.3 on 24 January 2013 20:43:28
os: Linux-3.0.0-23-virtual #39~lucid1-Ubuntu SMP Fri Jul 20 19:37:29 UTC 2012
nodename: sudoku-admin
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 4
current working directory: /home/lukasz.mierzwa/uwsgi
detected binary path: /home/lukasz.mierzwa/uwsgi/uwsgi
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
uwsgi socket 0 bound to TCP address 172.16.200.58:2201 fd 3
Python version: 2.6.5 (r265:79063, Oct  1 2012, 22:16:31)  [GCC 4.4.3]
*** Python threads support is disabled. You can enable it with --enable-threads ***
Python main interpreter initialized at 0x98dba0
your server socket listen backlog is limited to 100 connections
mapped 145024 bytes (141 KB) for 1 cores
*** Operational MODE: single process ***
WSGI app 0 (mountpoint='') ready in 0 seconds on interpreter 0x98dba0 pid: 3192 (default app)
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 3192)
spawned uWSGI worker 1 (pid: 3193, cores: 1)
subscribing to 127.0.0.1:2100

lukasz.mierzwa@sudoku-admin:~/uwsgi$ nc localhost 2203
{
        "version":"1.5-dev-744fed6",
        "pid":3189,
        "uid":10062,
        "gid":100,
        "cwd":"/home/lukasz.mierzwa/uwsgi",
        "active_sessions":0,
        "rawrouter":[
":2200",
"127.0.0.1:2100"
        ],
        "subscriptions":[

        ],
        "cheap":0
}

I wonder why rawrouter subscription socket is listed in rawrouter key in stats output.
Does this work for you?

I'll push 1.5 builds to dev cluster to check how it reacts with restarting nodes.

[unbit]

--subscribe-to 127.0.0.1:2100:ip:2200

the address of the rawrouter is the key. As you can bind the rawrouter to unlimited interfaces you can have a very rudimentary virtualhosting.

[prymitive]

FastRouter was improved a lot in 1.5 and during testing I did not had any failed requests other than those that occurred when node died/reloaded in the middle of request handling. Since we can't have retry in such case than I think I can close this issue.