Liferay, Java, Linux, and Too Many Files

Liferay is a big application, made up of thousands of files. That's not the kind of typical, everyday application that your system is used to dealing with.

Everywhere I've worked (including Liferay), I've made a point of advocating to the IT department that a developer needs both a different kind of machine and a different kind of configuration than the regular worker. Thankfully, that's well understood here, and I've had the freedom to pretty much configure things the way I would be able to work best.

The advent of desktop Linux, however, has been both a boon and a detriment for the community that uses it. Linux on the desktop is now much easier to use and to install, but that comes at a price: many times the configuration that is shipped by default is only optimal for the typical desktop user. For developers, like other commercial, proprietary operating systems, there are OS settings that need to be tweaked. Sometimes you know what to do right away after the install, and sometimes you run into an issue that causes you to both learn a little more about your OS of choice and to come up with one more setting that needs to be customized when you build your new box.

Recently, I've been doing a lot of training slides. This activity tends to happen in fits and spurts: I spend a bunch of time trying to get something to work, and then in a burst, I create a whole bunch of slides about it. This lets me do stuff that I haven't necessarily done before, which generally means I run into a bunch of problems I haven't had before.

This week, I've been profiling Liferay and comparing the results of different configurations. I've been working specifically with the Netbeans profiler, for two reasons: 1) It's free and open source--which means we're free to use it in training, and 2) I've never used it before. So I wanted to see if it would work for the scenario I've got going. No guarantees here--I'm still working on it. :-)

Anyway, I kept having Netbeans crash (actually it wouldn't crash--it would just become unresponsive). Checking my .netbeans/6.1/var/log/messages.log file, I found that it was unable to open any more files because of a "Too many open files" exception. At first, I thought this was a Netbeans issue, because I never had this problem with Eclipse, but everything I found online pointed to an issue with the way the OS was configured.

I use Kubuntu, which is definitely a desktop OS. By default, Kubuntu (and presumably, Ubuntu) is configured to allow 1024 open files per shell session. Liferay easily exceeds this number, and this is the problem Netbeans was running into. I found several things you can do to fix this, though, and it all has to do with tweaking the number of files that are allowed to be opened by a user on Linux.

First, take a look at your /etc/sysctl.conf file. In that file is a property called fs.file-max. This should be set to 200000 or above. On Kubuntu, this was already set properly. If yours isn't, go ahead and change it, but this won't solve the problem by itself.

Next, there's a setting for shell level file limits. You can check what your current one is by using the command ulimit -n. Mine was 1024, and that was the problem.

To change this setting, you need to edit the file /etc/security/limits.conf. Add the following lines to the file:

* soft nofile 5000
* hard nofile 5000


This basically changes the setting from 1024 files to 5000 files for all users. I had no idea how much I was going over the limit, so this seemed like a good setting. Save the file and then reboot your machine, and everything should be fixed.

So if you find yourself with this error, it's likely because Liferay is a very large project, and your default settings are simply not sufficient to handle it. Linux is by design tweakable, so it's not so hard to fix.

Blogs
You don't actually need to reboot - just log out & log in.
Also, csh users use:
"limit descriptors"
to see the number of open file descriptors allowed