How to monitor Liferay (Tomcat) remotely through firewalls using VisualVM

 

  I'm on site these last days and one of my tasks was to execute some load testing and monitor a Liferay instance hosted in a network accessed only via SSH port. No other port except 22 and 80 was open. So after almost 2 days struggling with several different configurations I was able to monitor a remote instance of Liferay through JMX+RMI over SSH tunneling. I had to put together several pieces of information to get this working, let's see the steps.

  I will split this post in two configuration sections for JMX server (Tomcat) and JMX client (VisualVM).

 

  JMX server

  First of all you need to drop in the catalina-jmx-remote.jar to TOMCAT_HOME/lib directory of Tomcat. This will add to the classpath the listener org.apache.catalina.mbeans.JmxRemoteLifecycleListener responsible for enable the JMX+RMI over SSH tunneling. More info about this listener can be found at JMX Remote Lifecycle Listener. This jar may be found under extras directory of download binary area of Tomcat. Only versions newer or equal to 6.0.24 has it available. Your second step is configure the listener on server.xml add the following snippet under the <Server> tag (around line 35).

  

    <Listener className="org.apache.catalina.mbeans.JmxRemoteLifecycleListener" rmiRegistryPortPlatform="10001" rmiServerPortPlatform="10002" useLocalPorts="true" />

 

  The third and last step on the server side is set new parameters to JVM as following to enable JMX remote. Set them on the setenv.(sh|bat) of your Tomcat installation

 

    JAVA_OPTS="$JAVA_OPTS -Djava.rmi.server.hostname=localhost -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"

 

  JMX client

  Again your first step is drop in the jar catalina-jmx-remote.jar into the VISUALVM_HOME/platform/lib. This is a required step otherwise you gonna get a ClassNotFoundException for JmxRemoteLifecycleListener. After that you must start a SSH connection to your server with two tunnel enabled, you can use the command below

 

  ssh -vN -L10001:localhost:10001 -L10002:localhost:10002 [e-mail]

 

  Then you can fire a VisualVM and add a reference to your remote server under the Remote section. Finally you should open a connection using this connection string service:jmx:rmi://localhost:10002/jndi/rmi://localhost:10001/jmxrmi and voilà! We are done and ready to start the real work!

 

  

  Other considerations

 

  If you are trying to monitor a cluster of machines be aware that the property  java.rmi.server.hostname=localhost will break the cache replication.

  Maybe your are wondering about the reference of localhost in almost all commands instead of the server's names, right? Don't worry this is the right way beacuse we are using SSH tunnelling.

  

  Some folks would like to work with jconsole. Server configuration are identical. On the client side you gonna fire jconsole with the following parameter -J-Djava.class.path=$JAVA_HOME/lib/jconsole.jar:$JAVA_HOME/lib/tools.jar:catalina-jmx-remote.jar to be able to connect. 

  If you have connection problems I would suggest you to use jconsole to debug. You can use the follwoing command line and logging.properties file to enable debugging

 

jconsole -J-Djava.util.logging.config.file=logging.properties -J-Djava.class.path=$JAVA_HOME/lib/jconsole.jar:$JAVA_HOME/lib/tools.jar:catalina-jmx-remote.jar

 

// logging.properties file to activate JMX traces.

// Use this file on the java command line:

//     java -Djava.util.logging.conig.file=<logging.properties>

//

handlers= java.util.logging.ConsoleHandler

.level=INFO

 

java.util.logging.FileHandler.pattern = %h/java%u.log

java.util.logging.FileHandler.limit = 50000

java.util.logging.FileHandler.count = 1

java.util.logging.FileHandler.formatter = java.util.logging.XMLFormatter

 

java.util.logging.ConsoleHandler.level = FINEST

java.util.logging.ConsoleHandler.formatter = java.util.logging.SimpleFormatter

 

// Use FINER or FINEST for javax.management.remote.level - FINEST is

// very verbose...

//

javax.management.level=FINEST

javax.management.remote.level=FINER

 

  Abraço!

[Ray Augé]

Awesome! Thanks Thiago!

[Thiago Leão Moreira]

If you don't like fancy GUI you can still use the good and old CLI to monitor your Liferay. Check it out how at http://www.wolfe.id.au/2011/10/16/monitoring-the-openjdk-from-the-cli/

[Marcellus Tavares]

Nice post baron!

[Jonas Yuan]

Nice! Thanks Thiago!

[Ram Y]

Good and useful information. Thanks.

[Nilesh Gundecha]

Great piece of work Thiago. Thanks.

[Daniel Sanz]

Thaks Thiago. Nice post!

[himanshu jain]

nice

[Warner Pinero]

Hi. Thanks for that info.
I just want to ask where in server.xml specifically you insert the JmxRemoteLifecycleListener listener? There are many xml tags in server.xml and I'm not sure where to insert it.

Thanks for the help.

-Warner

[Thiago Leão Moreira]

Hey Warner,

Around the line 35, there is a list of listeners already declared.

[Warner Pinero]

Thanks Thiago. How about connecting using jconsole? I've tried jconsole but it didn't worked. But visualvm did.

[Thiago Leão Moreira]

Have you tried the configuration at the end of my post to jconsole? It works fine!

[Manish Kumar Jaiswal]

Hi Thiago,

I am trying to set profiler for a remote server ,Now in our scenerio we have
My local windows machine for profiling --> cloud ( gateway firewall --->apache----> Tomcat) .

1) After doing all configuration it says "start a SSH connection to your server with two tunnel enabled,
you can use the command below
ssh -vN -L10001:localhost:10001 -L10002:localhost:10002 [e-mail]"
where should I use this command ? Does in your scenerio your local machine is linux and you are using this command there
,or you are using this command at final linux server having tomcat.

2) "Finally you should open a connection using this connection string service:jmx:rmi://localhost:10002/jndi/rmi://localhost:10001/jmxrmi " I hope this will be automatically open when we fire remote
or do you mean to add these lines somewhere

Regards
Manish

[Manish Kumar Jaiswal]

Thanks Thiago ,
I did it finally ..But I have a question ..This has connected to the server for monitoring purpose can we also profile this way cause my purpose is to do some memory profiling ..if yes how to enable it.

Thanks
manish

[Thiago Leão Moreira]

Nope! It's just to monitoring.

[chris kabore]

great post! thanks.

[Johann Kneringer]

Unfortunatly I have problems implementing this solution.

Listener is set on the remote tomcat...

Tunneling to my remote server is also successfull...

But If I want to open the service in VisualVM i get following error:

"Cannot connect to service:jmx:rmi://localhost:10002/jndi/rmi://localhost:10001/jmxrmi using service:jmx:rmi://localhost:10002/jndi/rmi://localhost:10001/jmxrmi"

The catalina-jmx-remote.jar is placed on the remote tomcat and in my platform/lib folder of VisualVm.

There is no firewall set up.

Some tips would be very great!

regards

[Thiago Leão Moreira]

Hey Johann, try to use the jconsole, It is better to debug with it.

[Johann Kneringer]

thank you very much...it works with JConsole using

jconsole service:jmx:rmi://localhost:10002/jndi/rmi://localhost:10001/jmxrmi -J-Djava.util.logging.config.file=/usr/lib/jvm/jdk1.6.0_35/lib/logging.properties -J-Djava.class.path=/usr/lib/jvm/jdk1.6.0_35/lib/jconsole.jar:/usr/lib/jvm/jdk1.6.0_35ib/tools.jar:/usr/lib/jvm/jdk1.6.0_35/lib/catalina-jmx-remote.jar;

[Ilya Ilya]

It's just awesome!!
Everything start to work from first try!)))
I register on this site just for say thanks to you.
Thank you man, i appreciate you very much!

[Marco Ferretti]

Hi,
I am struggling with trying to connect to a remove liferay 6.2 instance the way you have provided.
I have followed step by step the guide but I am only getting a "cannot connect to service:jmx:rmi://localhost:10002/jndi/rmi://localhost:10001/jmxrmi using service:jmx:rmi://localhost:10002/jndi/rmi://localhost:10001/jmxrmi" . I tried jconsole too. It first tells me that it cannot connect securely, asks to fallback in a non secure connection and then fails.
I have tried adding the -Djava.rmi.server.hostname=127.0.0.1 parameter too but no luck.
On the tomcat logs I can clearly see that the ports are configured :
Nov 05, 2015 10:02:47 AM org.apache.catalina.mbeans.JmxRemoteLifecycleListener createServer
INFO: The JMX Remote Listener has configured the registry on port 10001 and the server on port 10002 for the Platform server

On my local machine I am opening a tunnel as follows :
ssh marco@<myserver> -L10001:127.0.0.1:10001 -L10002:127.0.0.1:10002 -vN

on the ssh logs I can see the connection going TO the remote server :
debug1: Connection to port 10001 forwarding to 127.0.0.1 port 10001 requested.
debug1: channel 5: new [direct-tcpip]

but I can't see the other port being connected.

Can you please help ?

[Thiago Leão Moreira]

Hey Marco, were you able to use jconsole with logging configured? It is the recommended way to debug problems.