Setting up Remote Staging for a community (or organization) in Liferay is easy: On the staging machine go to "Manage Settings, Staging", select "Remote Live" as staging type and provide the address, port and remote group id (the community/organization that you want to publish to).
On the production machine (where you want to publish to), you need to configure
axis.servlet.host.allowed in portal-ext.properties to allow the staging machine to access web services on the production machine.
Often problems arise when you want to check the "Use a Secure Network Connection" as well, e.g. use https to publish to your remote system. However, setting up a secure connection is easy when you know a few properties of encrypted connections. I'm going to explain the pitfalls of encrypted server-to-server communication in this post.
What's in this for you?
I'm going to discuss the two aspects of https that you need to know to understand the “why” of the following step-by-step instruction
I'm giving brief instruction on how to configure tomcat to accept https connections with a self-signed certificate (you'll find these in more details in other places on the internet, but as it's required for ssl-staging it's required to be done first)
Then I'm explaining the steps for configuring another server to connect to the first one in order to do remote staging over https.
Finally I'm giving some debugging info – typical things that can go wrong.
What problems does https solve?
The best known feature of https is that it provides an encrypted channel for the communication - nobody can eavesdrop on the communication. The lesser known but equally important feature is certified identity of the server: The client can be reasonably sure that it's indeed talking to the expected server, not to some Man-In-The-Middle attacker, intercepting the connection before it has been established. Obviously there would not much value in encrypting the communication when you can't be sure you're sending your data to the expected destination.
This server identity feature can be the reason for some trouble setting up Remote Staging through https as it involves an im- or explicit trust relationship between both machines.
(Far more detail about SSL, TLS, https and encryption in general is available all over the web; I've learned the most in the SecurityNow podcast, see episodes 30-37, 181, 183, 195 and probably more)
How does this trust work? You probably have heard about certificates and seen your browser protest when it believes that it doesn't trust a certificate. Why is this? Your browser contains a bunch of "trustworthy" certificate authorities that can "sign" server keys. They typically validate that the key-owner is also the owner of the domain. If you have such a trustworthy certificate, you have less work with the setup of the connection (but you're typically paying these companies for their services)
For internal connections, debugging and test setups, it's somewhat common to use self-signed certificates. These are obviously not trusted by your browser, so in a browser you get a severe looking warning. (if you want to try it: My own server at https://www.olafkock.de uses a self-signed certificate - at least at the time I'm writing this post)
With the interaction that a browser inherently provides (typically there's a user involved that can interact), it's easy to manually trust the certificate when the connection is being established.
However, when a server process like your staging server is connecting to a remote-live server that uses a self-signed certificate, there's no user sitting in front of the server process to manually add the trust relationship while the connection is being established. Consequentially the connection can not be established. This is source of some confusion and what we'll work on in the next section.
To setup a scenario for this article, I'm going to use two machines: staging.example.com and production.example.com .
We will be creating content on staging.example.com and publish this remotely through https to production.example.com. In order to do this, we first need to set up https on production.example.com. Note that this is the only server that needs to be available through https for purposes of this post.
Setting up https on production.example.com
There's a lot of documentation available for setting up https with your webserver of choice. I'm going quickly over this by providing the steps to enable https on a tomcat installation on production.example.com: We typically provide the key and the self-signed certificate in a custom keystore to tomcat. This contains the private key that must be well guarded: Everybody knowing this key is able to decrypt the traffic. The only place where it makes sense outside of your tomcat installation is in your backup.
When we have created the custom keystore we'll make it known to tomcat's connector on production.example.com. For this task we're using java's keytool:
production$ keytool -genkey -alias production.example.com -keyalg RSA -keystore /opt/liferay/production-keystore Enter keystore password: changeit Re-enter new password: changeit What is your first and last name? [Unknown]: production.example.com What is the name of your organizational unit? [Unknown]: whatever you like What is the name of your organization? [Unknown]: give the name of your organization here What is the name of your City or Locality? [Unknown]: give the name of your location here What is the name of your State or Province [Unknown]: provide it here What is the two-letter country-code for this unit? > de
With this you have generated a private key that can be used in tomcat. Note that the server name is actually in the field for your “first and last name”. You use this by adding a connector in tomcat's server.xml like this:
<Connector port="8443" protocol="HTTP/1.1" SSLEnabled="true" maxThreads="150" scheme="https" secure="true" clientAuth="false" sslProtocol="TLS" keystoreFile="/opt/liferay/production-keystore" keystorePass="changeit" />
(You'll find a predefined, commented, entry similar to this in tomcat's default server.xml)
Start tomcat (liferay) and make sure that it answers on https://production.example.com:8443. When you first access the URL with your browser, you will see that the browser doesn't trust your certificate – it's self-signed and the browser lets you go through some hoops to accept that you indeed trust it.
Note that the keystore now contains the private key – you better guard that file and keep it from being accessed by anybody. However, you definitely want to back it up.
While we're still working on production.example.com, make sure to have the portal-ext.properties configuration from the introduction in place:
tunnel.servlet.hosts.allowed=127.0.0.1,SERVER_IP,10.0.8.15 tunnel.servlet.https.required=true axis.servlet.hosts.allowed=127.0.0.1,SERVER_IP,10.0.8.15 axis.servlet.https.required=true
and remember to replace 10.0.8.15 with the actual IP of staging.example.com
Establishing trust of staging.example.com to production.example.com
We will use this setup to create a server-to-server connection from staging.example.com to production.example.com. In order to do this, we need to establish a trust relationship. During the setup of production.example.com, we created a separate keystore and referenced it from tomcat's configuration file server.xml. The keystore contains the private key as well as the (self signed) certificate.
When we want to connect from staging.example.com we only need public information from production.example.com – there's no need to copy over the full keystore. To illustrate that I'm going to use the https connection to access production.example.com and retrieve the certificate this way. The openssl tools can do that (available on every linux distribution and probably also on the other platforms):
staging$ openssl s_client -connect production.example.com:8443 CONNECTED(00000003) depth=0 /C=DE/ST=Hessen/L=Eschborn/O=Liferay GmbH/OU=Liferay GmbH/CN=production.example.com verify error:num=18:self signed certificate verify return:1 depth=0 /C=DE/ST=Hessen/L=Eschborn/O=Liferay GmbH/OU=Liferay GmbH/CN=production.example.com verify return:1 --- Certificate chain 0 s:/C=DE/ST=Hessen/L=Eschborn/O=Liferay GmbH/OU=Liferay GmbH/CN=production.example.com i:/C=DE/ST=Hessen/L=Eschborn/O=Liferay GmbH/OU=Liferay GmbH/CN=production.example.com --- Server certificate -----BEGIN CERTIFICATE----- MIICeTCCAeKgAwIBAgIETaiUJjANBgkqhkiG9w0BAQUFADCBgDELMAkGA1UEBhMC REUxDzANBgNVBAgTBkhlc3NlbjERMA8GA1UEBxMIRXNjaGJvcm4xFTATBgNVBAoT DExpZmVyYXkgR21iSDEVMBMGA1UECxMMTGlmZXJheSBHbWJIMR8wHQYDVQQDExZw cm9kdWN0aW9uLmV4YW1wbGUuY29tMB4XDTExMDQxNTE4NTMyNloXDTExMDcxNDE4 NTMyNlowgYAxCzAJBgNVBAYTAkRFMQ8wDQYDVQQIEwZIZXNzZW4xETAPBgNVBAcT CEVzY2hib3JuMRUwEwYDVQQKEwxMaWZlcmF5IEdtYkgxFTATBgNVBAsTDExpZmVy YXkgR21iSDEfMB0GA1UEAxMWcHJvZHVjdGlvbi5leGFtcGxlLmNvbTCBnzANBgkq hkiG9w0BAQEFAAOBjQAwgYkCgYEAqeWzhKHuDGI34JSbc0ccrFolTb+THN8SeX1x CVZirQ8zLqkKayQke0MEH3ZMPNXM6hKslVAOj4NNr+AhHc/BNw8qNvFRNCWFQ07f aAr+223bPc4tVnii+8xuYFjtc1vAEpagT8W79ebUxv7iRObherQPdtxhzm3MYfR4 /7BI44UCAwEAATANBgkqhkiG9w0BAQUFAAOBgQAaPRMRI9u7cxeiWuo7tiWpPMws 57pxgBumUyyxNpg9ij5QJeSrEBlV9HSsuzPuE+2F/x0Fo7U73tgVi7BfeAOiaVWy qpnJwdfODWknGni12u1o2+xxAL9i5nVtcDjTu2MjRU8Zlg5tWntZD74kKjMvCY4M aJAp7Gr+Pyh/qvLAYQ== -----END CERTIFICATE----- subject=/C=DE/ST=Hessen/L=Eschborn/O=Liferay GmbH/OU=Liferay GmbH/CN=production.example.com issuer=/C=DE/ST=Hessen/L=Eschborn/O=Liferay GmbH/OU=Liferay GmbH/CN=production.example.com --- No client certificate CA names sent --- SSL handshake has read 1216 bytes and written 253 bytes --- New, TLSv1/SSLv3, Cipher is EDH-RSA-DES-CBC3-SHA Server public key is 1024 bit Secure Renegotiation IS supported Compression: NONE Expansion: NONE SSL-Session: Protocol : TLSv1 Cipher : EDH-RSA-DES-CBC3-SHA Session-ID: 4DA8B14EA359D0AAE85D021E35942BE622A599032D107E0E8E42BE372C3B8FA4 Session-ID-ctx: Master-Key: 88AA956B660E031A631AC0A394D04F8516A988CE314F9CC6A5511E8C92879CA2DE2A9BF14E98BAA80778B8BD0CE3750D Key-Arg : None Start Time: 1302901070 Timeout : 300 (sec) Verify return code: 18 (self signed certificate) ---
The bold part in the above output is interesting. This is the public certificate that the server provides to clients to prove its identity. This is what we'll have to register as “trusted” on our client machine, staging.example.com. Copy this block (including the BEGIN CERTIFICATE and END CERTIFICATE lines into a new file. I'll name this publisher-cert.txt.
In order to be trusted by the running server process, this certificate needs to be imported into the JRE's keystore of the JRE that actually runs the server (staging.example.com), which you'll typically find in $JRE_HOME/lib/security/cacerts. This time it can't be “just added” like we did above with the keystore that we referenced from tomcat's server.xml. As we're now possibly dealing with some system-wide installation of java, you may need to gain root or administrative permissions to do this:
staging $ sudo keytool -import -alias production.example.com -keystore $JRE_HOME/lib/security/cacerts -file publisher-cert.txt Enter keystore password: Owner: CN=production.example.com, OU=Liferay, O=Liferay GmbH, L=Eschborn, ST=Hessen, C=DE Issuer: CN=production.example.com, OU=Liferay, O=Liferay GmbH, L=Eschborn, ST=Hessen, C=DE Serial number: 4dab0790 Valid from: Sun Apr 17 17:30:24 CEST 2011 until: Sat Jul 16 17:30:24 CEST 2011 Certificate fingerprints: MD5: 1C:CD:8E:31:B1:41:01:CE:DF:DC:69:32:AF:D2:59:41 SHA1: 90:E8:F2:3B:6C:79:58:A0:A3:DA:B2:89:C6:9D:51:16:6E:F3:F9:A9 Signature algorithm name: SHA1withRSA Version: 3 Trust this certificate? [no]: yes Certificate was added to keystore
With this, finally, our JRE on staging.example.com trusts the certificate used for production.example.com to be legitimate and we can finally create a server-to-server connection. Ok, might require a restart, this time for staging.example.com, but we're finally where we want to be.
If remote publishing still doesn't work, these are the most common things to check:
Allow access to webservices
In portal-ext.properties on production.example.com you need to allow staging.example.com to access the webservice used for publishing (tunnel.servlet.hosts.allowed, axis.servlet.hosts.allowed). You can easily check this by starting a browser on staging and point it to https://production.example.com:8443/tunnel-web/axis . If this shows a “403 access denied” message, the server will not be able to connect. If you have only shell access to your server, w3m, lynx or links are text-mode browsers that work well for this test.
Certificate name mismatch
If you have a certificate for example.com, but your server is available as (and you connect to) production.example.com, the host name and the certificate do not match. They have to match in order to get the trust relationship between the two computers established.
If you want to configure your settings differently in test and production environments, but still be able to use the same or similar host names, install the same trust relationships, you might want to look up wildcard certificates. These certify the identity of *.example.com, so you can reuse it for various setups.
if this seems to be configured correctly, you might have IPV4 vs. IPV6 issues: If production.example.com resolves as an IPV6 address, you need to allow staging's IPV6 address in portal-ext.properties, not the IPV4 address. Or configure your JVM to prefer the IPV4 stack over the IPV6 one: -Djava.net.preferIPv4Stack=true
if your servers are in different networks, make sure that the firewall between these networks allows access on the port that you need. Especially if you have nonstandard ports, like tomcat's 8443. If you follow the proceedings I've named above, you will not have these problems, because you've already connected to the port you need with the openssl tool to obtain the certificate.
If you have an apache or some other device handling the https there are more complexities to take into account: Tomcat needs to know that you're connecting through https – when you just use a reverse proxy to connect Apache to tomcat, this information is typically lost. As far as I know, ajp provides all required information to tomcat while http typically doesn't – but please test for yourself, don't take my words for granted.
NAT or multiple network interfaces
Maybe the staging connection doesn't originate from staging.example.com – or doesn't appear so – e.g. when there's NAT. This is easy to detect: Connect with a browser from staging.example.com to https://production.example.com/tunnel-web/axis. When you get a 403 error message, you'll see what IP is denied access. This is what you need to allow in tunnel.servlet.hosts.allowed and axis.servlet.hosts.allowed
https setup, trust
something can go wrong configuring the trust relationship between the servers involved. As you know, there's no way to “ad hoc” accept some certificate as trusted just as you can do in your browser. Make sure that the JVM uses the keystore you imported the certificate to. Also, make sure that the host name that you use match the name in the certificate and the one that you trust.
If you're working with portal instances, you need to use an endpoint (e.g. virtual host) within the target instance as the publishing target.
What else can go wrong?
If you find other things that (can) go wrong please post them in the comments in order to make this post a bit better.
My day-to-day job involves configuring this from time to time – but I'm not a cryptography expert. This means that I probably have used words like “certificate”, “key” in a somewhat unprecise way. If you can point out where I should have been more precise, please comment and I'll change it. (I know that SSL is supposed to be TLS nowadays, but it's an old habit and everybody knows what I'm talking about)
The initial settings and tests were done together with Nitharsan Manoharan and Randall Hidajat. Both did the dirty work of configuring and debugging the infrastructure. Thanks for the cooperation.