Server open-files issue about prairielearn

[mwest1066]

My old notes on this issue:

The error appears as stdout logs from the main server.js in the form:

Error: EMFILE, open 'tests/tangNorm/testOverview.html'

This seems to be caused by having too many open files?

Perhaps this is due to persistent TCP connections from dead clients. E.g., netstat and lsof -i show old TCP connections still in ESTABLISHED state, but actually dead (due to client death, network failure, etc.).

The default TCP keepalive settings on Linux are:

[root@prairielearn2 ~]# cat /proc/sys/net/ipv4/tcp_keepalive_time 
7200
[root@prairielearn2 ~]# cat /proc/sys/net/ipv4/tcp_keepalive_intvl 
75
[root@prairielearn2 ~]# cat /proc/sys/net/ipv4/tcp_keepalive_probes 
9

This means that the server will wait for 7200 s (2 h) and then will send 9 probes at intervals of 75 s before force-closing the connection. Clearly this could result in many zombie connections.

The thinking would then be that express.js is keeping disk files open due to sendfile() streaming behavior. It's not clear whether this is actually true.

The maximum number of open files is:

[root@prairielearn2 ~]# ulimit -n
1024

This can be increased by changing /etc/security/limits.conf and adding:

* hard nofile 10000

It may also be then necessary to use ulimit to increase the actual limit in the shell. Also, it's possible that setting a soft limit in limits.conf will have the same effect (need to set both hard and soft?). It may also be necessary to edit /etc/pam.d/common-session and add:

session required pam_limits.so

If we want to fix the zombie TCP connection issue in Node then we could possibly use socket.setTimeout() or socket.setKeepAlive().

Note that sockets and files are both file descriptors, and ulimit only modifies the file descriptor limit.

To replicate on OS X, open two shells, one for the server and one for the client.

1. In the server shell, decrease the max number of open files:

   ulimit -n 210
   

2. Start the server with this lower ulimit.

3. In the client shell, increase the max number of open files (not strictly needed here):

   launchctl limit maxfiles 1024 10000
   ulimit -n 1024
   

4. Start the loadtest.js client, with it set to do 180 connectOpen() calls (just open connections, don't write to them or close them), and tryTransaction() doing only appLoad() with:

   MAX_ACTIVE_REQS = 30
   DELAY = 10
   

On OS X, apparently we can also do:

• Check open file limits:

  $ sysctl -a | grep files
  kern.maxfiles = 12288
  kern.maxfilesperproc = 10240
  

• Increase open file limits:

  $ sudo sysctl -w kern.maxfiles=12288
  $ sudo sysctl -w kern.maxfilesperproc=10240
  

• Increase ulimit:

  $ ulimit -n 10240
  

• Check open socket limits:

  $ sysctl -a | grep somax
  kern.ipc.somaxconn: 2048
  

• Increase open socket limits:

  $ sudo sysctl -w kern.ipc.somaxconn=2048
  

From: http://b.oldhu.com/2012/07/19/increase-tcp-max-connections-on-mac-os-x/

It max also be necessary to use the -S flag for ulimit, to set the soft limit:

ulimit -S -n 1024

To permanently change limits on OS X, edit (or create) /etc/launchd.conf to include the line:

maxfiles    16384          32768

Also edit (or create) /etc/sysctl.conf to include:

kern.maxfiles=20480

Apparently it is the case that on OS X you need to do all three of:

1. Increase kernel limits with sysctl
2. Increase launchd limits with launchctl
3. Increase ulimit limits with unlimit -n

ulimit applies only to the current shell, they other two are global.

It's not clear what order 1 and 2 need to be done in, or whether it makes a difference.

[mwest1066]

We haven't seen this bug in over two years, so I'm assuming it was a NodeJS bug that was fixed at some point.