hyperfastcgi stop working but not crash about hyperfastcgi

[antonheryanto]

Forgot to mention.
in old mono fastcgi got this warning/error Duplicate name, HTTP_COOKIE, encountered. Overwriting existing value (didn't know where its came). which might related #42 but as see the fix only applied to branch v0.3_stable

[xplicit]

What is the libevent version on your operating system? A socket error (0x21) occurred on fd 54 is not a normal error message which should not appear in logs (while Remote host disconnected from fd 49 can appear when client closes the page before request is processed).

Do you have requests which take large amount of time? In this case you should add fastcgi_read_timeout with large value (in seconds) to not break upstream connection, while request is processed.

Can you try run your app in docker instance? https://hub.docker.com/r/xplicit/hyperfastcgi/
It helps to isolate OS/configuration issues and check if problem exists in known working configuration.

Anyway if after these changes you still have issues with native listener, you can switch to ManagedListener with Combined (managed + native) or Managed transport. With your number of requests (100-300 rps) you should not get downgrading in performance. Sample configs are available in samples directory.

#16 workaround I tried on Ubuntu and did not check if it works on other operating systems.

You are lucky if mono-server-fastcgi just crashed after some working time. In my experience it tried to eat all existing memory than hang with 100% of CPU usage and stopped to process any requests from the clients. That's why I started to write my own implementation of fastcgi server.

[antonheryanto]

Thank you very much for your quick response, appreciate
following are the libevent information when hyperfastcgi start:

libev.c:461: Listen():        libevent version: 2.0.22-stable
libev.c:475: Listen():        libevent is using epoll for events.

the application is e-learning platform which handle lots of files upload with max upload size 300MB with students uploading from slow connection may explain lot of timeout, as your suggestion i have set fastcgi_read_timeout 600; will report later effect on when traffic is high as on my timezone now early in morning UTC+8

will try later running the apps on docker, now seem not an options because of complexity with legacy application.

i have try using ManagedListener but still crash quite often, but will try again later after inspect the effect on fastcgi_read_timeout and reduced possiblility of following errors happen:

fcgi-transport.c:399: parse_params():        Can't find app!

how to prevent its happen, as its cause headache 😅 as the process running but application state is broken. lately the hyperfastcgi running quite stable and longer, but when that error happen from user perspective the application is crash but process is running, but the only cure to the problem is restart the service. so how to prevent this ever happen or lets hyperfastcgi restart when this errors happen. The error not a 5xx error, is it trigger 404 to nginx?. a part from this errors, the hyperfastcgi perform really well on cpu and memory usage. appreciate your time, thank very much

[antonheryanto]

further diagnose on socket timeout, from nginx log seem the user try upload file more then 300mb sometimes over 1gb event already state max upload 300mb, but nginx didn't prevent the request to happen until user finish send upload and pass it to fastcgi which might cause socket timeout.

i have reconfigure nginx to prevent it happen based on http://stackoverflow.com/questions/4947107/nginx-upload-client-max-body-size-issue answer

 client_body_in_file_only clean;
 client_body_buffer_size 32k;
 client_max_body_size 300m;
 sendfile on; 
 send_timeout 300s;

will diagnose the effect

[antonheryanto]

its happen again

hyperfastcgi4[22289]: host-list.c:55: unregister_host():  app:80:/:/srv/www/htdocs/myApp/MyApp
fcgi-transport.c:399: parse_params():  Can't find app! HOST='myapp.com' port=80 path='/'

and backup service in nginx upstream not triggered to take place as the hyperfastcgi just return not found error and actually broken or crash

upstream hf {
  server 127.0.0.1:9001;
  server unix:/tmp/mf.sock backup;
  keepalive 30;
}

when restarted the service this error found

mono[17070]: g_hash_table_lookup: assertion 'hash_table != NULL' failed
mono[17070]: g_hash_table_insert_internal: assertion 'hash_table != NULL' failed
mono[17070]: g_hash_table_lookup: assertion 'hash_table != NULL' failed
mono[17070]: g_hash_table_insert_internal: assertion 'hash_table != NULL' failed
mono[17070]: g_hash_table_lookup: assertion 'hash_table != NULL' failed

any chance this happen because i have nested folder which handle by php-fpm fastcgi ?
btw to instruct hyperfastcgi to return error or exception when unregister_host() happen so backup service can take over. i thing is preferable as reduce possible gotcha handle kind of this error

[antonheryanto]

and also got nested aspx page for tinymce filemanager plugin which nested inside mvc folder, are these contributes to possible problem?

[xplicit]

Do you have some processes which send KILL signal to hyperfastcgi? Maybe some cron job which do this on regular basis?

If not, this possible could happen if web application in domain is crashed or recycled. I did not check yet, but this workaround might work (add at the end of OnHostUnload method) https://github.com/xplicit/HyperFastCgi/blob/master/srcHyperFastCgi/Transports/NativeTransport.cs#L19

 if (!isShutdown) 
        RegisterHost (host.VHost, host.VPort, host.VPath, host.Path);

I will check this case of AppDomain recycling more detaily on weekend, but if workaround mentioned above does not work for you as is you can temporary add Environment.Exit(0); in OnHostUnload to kill the whole process when domain is unloaded.

[antonheryanto]

Thank for your response,
I did have cronjob which restart hyperfastcgi service which run at 4 am (once a day) to make sure if hyperfastcgi hang/unload service is restart so it can properly worked again (which i did because prossible unload happen). but the unload/unregister happen mostly on peak hours mostly twice a day now which i think not related to that cron job which restart systemd service.

i have patch the NativeTransport OnHostUnload

protected override void OnHostUnload (IApplicationHost host, bool isShutdown)
{
    Logger.Write (LogLevel.Debug, "unregister Host when domain has problem");
    UnregisterHost (
        host.VHost,
        host.VPort,
        host.VPath
    );
    if (!isShutdown) {
         RegisterHost (host.VHost, host.VPort, host.VPath, host.Path);
         Logger.Write (LogLevel.Debug, "Register Host when not not shutdown");
     } else {
         Logger.Write (LogLevel.Debug, "Exit process when shutdown");
         Environment.Exit(0);
     }
}

will see the effect

[xplicit]

This commit should fix the issue. Could you check with new build?

06d416c

[antonheryanto]

Thanks for the quick fix,
previous workarounds are successful as when unregister_host happen its re-register again and exit process never triggered.

I have pull and build the fix and will report the outcome.

[antonheryanto]

after 24+ hours monitoring, i can confirm the host unregister issue solved as when its happen hyperfastcgi re-register again as shown in logs below:

Debug   AppHostBase.OnDomainUnload
Debug   Calling HostUnload event handler, num events=2
Debug   Unloading ApplicationHost domain, isShutdown=False
parse_params():        Can't find app!
parse_params():        Can't find app!
parse_params():        Can't find app!
parse_params():        Can't find app!
parse_params():        Can't find app!
parse_params():        Can't find app!
Debug   Register native transport
Debug   Configuring ApplicationHost
Debug   Configured host in domain 1de0ba32, id=2
unregister_host()
register_host()

so i close this issues, thank you very much for your quick response