Broken Control Panel and Performance after upgrade to Liferay 6.2

[Alexander Lukashov, modificado hace 7 años.]

Hi!

I have upgraded from Lifereay 6.1. to Liferay 6.2. and now I have got follow issues:

1. Control Panel Issue.
Control Panel looks like totally broken. It seems that system couldn't load proper CSS for it.

2. CKEditor Issue
When I open Control Panel, I keep getting error that CKEDITOR is not defined.

3. Performance Issues. Pages are loading very slow if compare to Liferay 6.1. I keep getting 503 error and had to restart Liferay several times per day. JVM and Catalina Args have been copied from the Liferay 6.1. Machine has the same memory capabilities, as those, where I have Liferay 6.1. running.

Thanks in advance, waiting for your advices.

[Juan Gonzalez, modificado hace 7 años.]

Can you review server logs and paste here errors, if any?

About performance, it's true that Liferay 6.2 requires a bit more of JVM heap comparing to 6.1. Which settings of -Xmx and XX:MaxPermSize do you have now?

Thanks.

[Alexander Lukashov, modificado hace 7 años.]

Here are CATALINA_OPTS from the setenv.sh

CATALINA_OPTS="$CATALINA_OPTS -Dfile.encoding=UTF8 -Dorg.apache.catalina.loader.WebappClassLoader.ENABLE_CLEAR_REFERENCES=false -Duser.timezone=GMT"
CATALINA_OPTS="$CATALINA_OPTS -Duser.language=en -Duser.region=US"
CATALINA_OPTS="$CATALINA_OPTS -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XXarallelGCThreads=4 -XX:+CMSCompactWhenClearAllSoftRefs -XX:CMSInitiatingOccupancyFraction=85 -XX:+CMSScavengeBeforeRemark -XX:+CMSConcurrentMTEnabled -XXarallelCMSThreads=2"
CATALINA_OPTS="$CATALINA_OPTS -server -XX:NewSize=700m -XX:MaxNewSize=700m -Xms2048m -Xmx2048m -XX:MaxPermSize=784m -XX:SurvivorRatio=6 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=15"

About the system logs. Here is one of the errors, which I have mentioned:

java.lang.IllegalStateException: this writer hit an OutOfMemoryError; cannot commit
at org.apache.lucene.index.IndexWriter.prepareCommit(IndexWriter.java:3399)
at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3524)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3506)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3490)
at com.liferay.portal.search.lucene.IndexAccessorImpl._doCommit(IndexAccessorImpl.java:411)
at com.liferay.portal.search.lucene.IndexAccessorImpl._commit(IndexAccessorImpl.java:367)
at com.liferay.portal.search.lucene.IndexAccessorImpl._write(IndexAccessorImpl.java:560)
at com.liferay.portal.search.lucene.IndexAccessorImpl.updateDocument(IndexAccessorImpl.java:324)
at com.liferay.portal.search.lucene.LuceneHelperImpl.updateDocument(LuceneHelperImpl.java:793)
at com.liferay.portal.search.lucene.LuceneHelperUtil.updateDocument(LuceneHelperUtil.java:428)
at com.liferay.portal.search.lucene.LuceneIndexWriter.updateDocument(LuceneIndexWriter.java:118)
at com.liferay.portal.search.lucene.LuceneIndexWriter.updateDocuments(LuceneIndexWriter.java:138)
at sun.reflect.GeneratedMethodAccessor982.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.liferay.portal.kernel.messaging.proxy.ProxyRequest.execute(ProxyRequest.java:85)
at com.liferay.portal.kernel.messaging.proxy.ProxyMessageListener.receive(ProxyMessageListener.java:51)
at com.liferay.portal.kernel.messaging.InvokerMessageListener.receive(InvokerMessageListener.java:72)
at com.liferay.portal.kernel.messaging.ParallelDestination$1.run(ParallelDestination.java:71)
at com.liferay.portal.kernel.concurrent.ThreadPoolExecutor$WorkerTask._runTask(ThreadPoolExecutor.java:682)
at com.liferay.portal.kernel.concurrent.ThreadPoolExecutor$WorkerTask.run(ThreadPoolExecutor.java:593)

And in most cases it causes to Java Heap Space.

[Olaf Kock, modificado hace 7 años.]

In that case you'll need to raise your memory that's available to the JVM - see Juan's comment. I recommend to set the minimum and maximum to the same value (as you already did). Try raising them to "-Xms3072m -Xmx3072m" - if you still have memory problems, you might have a memory leak somewhere. Performance tuning results for 6.1 aren't necessarily valid for 6.2 as you can see.

You might also check if there are unused plugins that are deployed - if you undeploy them, you're saving some memory.

And make sure that the server has enough RAM to not run into the use of virtual memory, which would kill your performance again.

[Alexander Lukashov, modificado hace 7 años.]

Thanks for the answer. I have set up provided values, will check further performance.

And what about broken Control Panel? Which direction should I look to find out the resolution?

[Juan Gonzalez, modificado hace 7 años.]

And what about broken Control Panel? Which direction should I look to find out the resolution?

This can be caused by multiple issues. For example, those OOM could have caused the css files to not have being minified properly.

Which app server are you using?

[Alexander Lukashov, modificado hace 7 años.]

I'm using tomcat-7.0.62.

[Juan Gonzalez, modificado hace 7 años.]

Ok, can you try:

* Shutdown server.
* Apply those improvements Olaf said in JVM.
* Remove <TOMCAT>/temp files and directories
* Remove <TOMCAT>/work files and directories
* Startup server.

Attach here the startup server logs so we can see if there are something suspicious.

Thanks.

[Alexander Lukashov, modificado hace 7 años.]

Here are logs attached.

[Juan Gonzalez, modificado hace 7 años.]

I didn't see anything "too" bad that explains the control panel problem you're having now.

Does it work right now?

I saw you have some custom themes. If you have some custom one for control panel, then that would be a good bug candidate.

Another thing you should check for sure is the S3 amazon config, as without that you can't use Documents & Media.

[Alexander Lukashov, modificado hace 7 años.]

No, it's doesn't work after restart.

These customs themes are for site, not for Control Panel. Control Panel have default control panel theme. I have checked that in previous version.

Errors with S3 appeared for Liferay 6.1. as well, it's a staging environment, currently we are testing the possibility of upgrade from Liferay 6.1

[Juan Gonzalez, modificado hace 7 años.]

I guess you cleaned browser cache after upgrade to 6.2 right?

Can you open browser network console and paste here any error (404 etc) when accessing Control Panel?

Additionally, are you accessing Liferay from an external host (non-localhost), right?

Thanks.

[Alexander Lukashov, modificado hace 7 años.]

After clearing cache I have got 500 and then 503 error with description that server temporarily is not available now. But there was no any log entry in the log describing the reason.

After all I have got follow error:

15:28:50,135 WARN [C3P0PooledConnectionPoolManager[identityToken->j8wD3ocx]-AdminTaskTimer][ThreadPoolAsynchronousRunner:275] com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@6b9da4ab -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!


Yes, I'm accessing liferay from the external host, it's deploying not in local host.

[Juan Gonzalez, modificado hace 7 años.]

Ok, more questions:

* Is you server public? I mean, just after starting it up, people can access it or you restrict the access?
* Can you attach your portal-ext.properties?
* Did you increase your DB connection pool to see if it improves?

I saw you're using Liferay EE, so probably best solution would be to open a LESA ticket in your case.

[Alexander Lukashov, modificado hace 7 años.]

Hi!

Here is my answers:

1) It's public, but restricted with apache server, since it's staging site, so only development team could access it.
2) I have attached portal-ext file
3) I've tried, but it didn't helped. All current settings are in attached file.


Thanks in advance.

[Alexander Lukashov, modificado hace 7 años.]

Hi!

Now performance issue has been fixed, but Control Panel and CKEditor are still boroken with Styles.

As I have mentioned Liferay 6.2 uses new version of Bootstrap. Could it be connected with it and how could I set up proper version for Control Panel?

Thanks!

[Juan Gonzalez, modificado hace 7 años.]

Hi Alexander.

Can you enable this filter, then clean work and temp directories, and restart server?

com.liferay.portal.servlet.filters.strip.StripFilter=true

BTW, I think you can safely remove this property

control.panel.layout.regular.theme.id=controlpanel

just in case it's related with the issue.

Then clean browser cache and try to access again.

Thanks.

[Alexander Lukashov, modificado hace 7 años.]

Hi, Juan!

I've done requested changes, but it didn't help.

Control Panel looks the same. Attached part of screen shot.

Thanks in advance.

[Alexander Lukashov, modificado hace 7 años.]

Hi, Juan!

Have you got a chance to take a look on the broken panel?

Thanks in advance

[Juan Gonzalez, modificado hace 7 años.]

Didn't have time.

Are there any errors when looking into browser network logs? I mean 404 or 500 errors.

Additionally, any errors in server logs after latest changes?

[Alexander Lukashov, modificado hace 7 años.]

Hi!

There is not error messages in tge log file.

The only errors, which we have got are in browser.
I've attached screenshot.

Thanks in advance.

[Alexander Lukashov, modificado hace 7 años.]

Hi!

I've done additional things:

• restored DB and run upgrade process again;
• downloaded and install the latest available patch for Liferay;
  
• disabled minifier in portal-ext.properties

theme.css.fast.load=false
javascript.fast.load=false
minifier.enabled=false

Before starting cleared work and temp folder, cleared browser cache. But the issue with Control Panel is the same - it looks broken, part of functionality is not enabled. Also CKEditor is not reachable and keep getting errors about it in Browser Console.

Could you check this and point what could be done else? There is no error message on startup, performance is much better. This is very critical for us to resolve this issue.

Thanks in advance.

[Arun Das, modificado hace 7 años.]

Hi,
Since you are using Apache to front tomcat, please add the following to your portal-ext.properties file(hope your are using port 80 for apache).
web.server.http.port=80
and restart liferay.

Arun

[Juan Gonzalez, modificado hace 7 años.]

My best guess at this point is about your apache frontend web server.

I see you are accessing through https, probably there are problems when passing request from Apache to Liferay.

Accessing Liferay server without Apache in plain http request could help in discard this.

[Arun Das, modificado hace 7 años.]

Hi,
I didn't notice https. Thanks Juan for pointing it out. So you should add the following if you are using https
web.server.https.port=443
web.server.protocol=https


Arun

[Alexander Lukashov, modificado hace 7 años.]

Hi!

Thank you for all your efforts and help. Finally I have found the reason of this issue. There was a file in the custom hook - top_head.jsp, and it has caused to this issue.

This thread could be closed, since all issues have been resolved.

Thanks!

[Juan Gonzalez, modificado hace 7 años.]

Alexander Lukashov:

Hi!

Thank you for all your efforts and help. Finally I have found the reason of this issue. There was a file in the custom hook - top_head.jsp, and it has caused to this issue.

This thread could be closed, since all issues have been resolved.

Thanks!

Ouch! It isn't usually a good idea to hook that file, and indeed it's one of the first things to look at when upgrading, just checking the hooks.

Ok, after all I think other things were improved too, glad you found the issue.