Saturday, January 30, 2010

Too Many Open Files

Problem description:

Engineer reported error where the module failed to response to incoming request. Log file shows that the module is unable to open additional connection to handle the load. Error picked up from Linux - Too Many Open Files (The OS need to write into a file in order to establish a new connection [I think so])

Solution:

Run a command to identify the amount of file open limit on the server.

[SERVER]ulimit -n

Default value = 1024

Increasing it to 8192, but here come the headache, since it is a live platform, the changes has to be made without restarting the server.

_________________________________________________________________________

@@ Lets assume that we need to increase the ulimit for root user @@
_________________________________________________________________________

[SERVER]vi /etc/security/limits.conf

Add this onto the file and save it:



#@student        hard    nproc           20
#@faculty        soft    nproc           20
#@faculty        hard    nproc           50
#ftp             hard    nproc           0
#@student        -       maxlogins       4
root -       nofile          8192                     < ---- This one!


[SERVER]vi /etc/profile

Add this onto the file and save it:


USERLIM=root
if [ $(id -un) = $USERLIM ]
then
ulimit -n 8192
fi


[SERVER]source /etc/profile
_________________________________________________________________________

logout from the terminal (This will not take effect on the terminal that you opened while the ulimit is still in it's pervious value)

login to the terminal again

[SERVER]ulimit-n


it should be showing that new value now. Here you go, increase ulimit without rebooting the server.


Result:


Reported module are able to handle the load during peak hour.

_________________________________________________________________________