Develop Reference

Running portlets on Liferay on 1Gig Server - Performance Issue

We have a couple of custom portlet applications running inside Liferay Portal.
The solution is installed on client’s computer which is entry-level (RAM <= 1 Giga). Due to red tape, it is rather unlikely the client switches to higher-end computers in the short term.
The issue is that the applications are very slow.
What are the hints to optimize Liferay configuration (or optimize the portlet application) so we are able to run decently on entry-level computers?
Or is it a good move to switch the portlets to lighter Portlets Containers alternatives such as Apache Pluto or GateIn?
Or running a portal like Liferay on entry-level computers is not an option? And we should consider porting the existing portlets to separate standard Java Web Applications so to achieve better performance?

Compare the price of tuning, minimizing the footprint and measuring the result to the price of just 1 more Gigabyte of RAM - which you might not even be able to purchase in this size any more.
Then compare the price for porting from a portal environment into Java Web Applications: You can't even be sure that this will result in a lower footprint, as you'll have to redo quite a bit of functionality that Liferay provides out of the box. Identity Management for example. Content Management as another one. This will take time (equaling money) that might be better spent with just a new server.
For ~40€/month you can get a hosted server, including network connectivity, power and even support, that is way more capable of serving an application like this than a server the size of a Raspberry Pi (<40€ total, I've seen Raspberry Pi hosting for less than 40€ per year).
I don't know what you mean with "Red Tape", but I'd say you're definitely going for the wrong target. While there is a point to tune Liferay, I'd not go for this kind of optimization.
You're not mentioning the version you're using - with that hardware I'm assuming that it's an ancient version. Back before the current version, Liferay was largely monolithic. While you can configure quite a bit (cache, deactivate some functionality) they'll not bring drastic advantages. The current version has been modularized and you can remove components that you don't use, lowering the footprint - however, it's not been built for that size of infrastructure.
And when you're running the portal on that kind of hardware, you're not running the database and an extra webserver on the same box as well, right? This would be the first thing to change: Minimize everything that's running outside of Liferay on the same OS/Box.

Performance Management Tool for 2 or more applications

I am working on an enterprise application which earlier used MyEclipse tool for Java / Java EE development, EJB 2.1 and WAS 7.0, recently we migrated to EJB 3.1, Websphere 8.5.5 and Eclipse Kepler. Now, we have noticed the performance of the application has increased and the screens load faster.
Now the problem that I am facing is to compare the earlier application with the one we've upgraded to and identify those areas which has led to speed up of the application. There are no performance metrics recorded for this application till date so I don't have anything to compare with.
All I am thinking was to have the Pre-Upgraded application deployed on a box and the Post-Upgrade application on the other box and record the time of load of all the screens. Now, this is not as subtle as being thought, so would like to know from you guys if there are any tools or strategies to compare two working applications and give performance metrics based on EJB methods time, JSP load time, Business Logic time, Database operations which gives true benefit analysis of the upgrade.
Also, do you guys think upgrade of application server and Integrated Development Environment (Eclipse Kepler) might have contributed to this speed?

If you still have both environments (WAS 7 and WAS 8.5.5) and some load scripts, I'd suggest to use PMI (Performance monitoring infrastructure) in WAS. You can enable metrics that interests you, set data to be saved to the log and run tests on both environments. Then you will be able to see gathered metrics for both environments.
The other option could be free WebSphere Application Server Performance Tuning Toolkit, which can be used to gather the performance data. Available either as standalone (older version) or as plugin to IBM Support Assistant (ISA).
Could upgrade of application server and Integrated
Developement Environment (Eclipse Kepler) might have contributed to
this speed?
Sure. The WAS 8.5.5 is in general faster than v7.0. For example it by default is using genCon garbage collection policy, which in most cases is more efficient that optthroughput.
The dev environment has no impact on application runtime performance, but maybe it is more responsive during development and thats why you have the 'faster' feeling.

I think what you need is benchmarking of both the versions of application and then compare both to see improvement.
for comparing both versions follow below approach,
Deploy both versions on exactly similar hw to create difference instance of 2 versions
Identify workflows/scenarios in which you found improvement and scenarios which are important for your application(mostly used/heavy/important for client etc.)
Carry out performance test/load test on those scenarios on both versions
Measure response time for all pages as well as system metrics i.e. cpu,memory,paging,disk etc.
Based on both versions results, carry out analysis and compare the both the versions.
If required carry out performance tuning and optimization round to improve the results.
This was about strategy.
For tools,
Check Ganglia,munin,graphite,carbon,sar,perfmon,nmon for system metrics (if its a cluster then RRD tools like ganglia,munin are better and if its a single box instance then sar for linux will do and on windows perfmon will do.)
For Load testing, JMeter is better option but you have quite enough funding then go for loadrunner,neoload,rational performance tester and for cloud, try blazemeter
For J2EE level analysis, IBM health center is available (according to me very inefficient to use), JProfiler, yourkit, jvisualvm are available
For WAS, Performance Monitoring Infra. is available with standard options it has low overhead but if you increase the logging counters and levels and it has huge performance impact.
I hope things are clear now :)

Java Webapp Performance Issues

I have a Web Application, Made entirely with Java. The Webapp doesn't use any Graphical / Model Framework, instead, the webapp uses The Model-View Controller. It's made only with Servlet specification (Servlet ver. 2.4).
The webapp it's developed since 2001, and it's very complex. Initially, was built for work with Tomcat 4.x/5.x. Actually, runs on Tomcat 6.x. But, we still having memory Leaks.
In Depth, the specifications of The Webapp can resumed as:
Uses Servlet v. 2.4 Specification.
It doesn't use Any Framework
It doesn't use JavaEE (Not EJB)
It's based on JavaSE (With Servlets)
Works Only on IE 6+ (Because of it's age)
Infrastructure Specification
Actually, the webapp works in three environments:
First
IBM Server (I don't remember exactly the model)
Intel Xeon 2.4 Ghz
32GB RAM
1TB HDD
Tomcat (Version 6) is configured to use 8GB of RAM
Second
Dell Server
Intel Xeon 2.0Ghz
4GB RAM
500GB HDD
Tomcat (Version 5.5) is configured to use 1.5GB of RAM
Third
Dell Server
Amd Opteron 1214 2.20Ghz
4GB RAM
320GB HDD
Tomcat (Version 6) is Configured to use 1.5GB of RAM
Database specification
The webapp uses SQL Server 2008 R2 Express Edition as a DBMS, except for the user of the first server-specification, that uses SQL Server 2008 R2 Standard Edition. For the connection pools, the app uses Apache DBCP.
Problem
Well, it has very serious performance issues. The webapp slow down continually, and, many times Denies the Service. The only way to recover the app is restarting The Apache Tomcat Service.
During a performance Audit, i've found several programming issues (Like database connections that never closes, excesive use of Vector collection [instead of ArrayList]).
I want to know how can improve the performance for the app, which applications can help me to monitoring the Tomcat performance and the Webapp Memory usage.
All suggestions are gladly accepted.

You could also try stagemonitor. It is an open source performance monitoring library. It records request response times, JVM metrics, request details including a call stack (profile) of the called methods during the request and more. Because of the low overhead, you can also use it in production.
The tuning procedure would be the following.
Identify slow requests with the Request Dashboard
Analyze the stack trace of the request with the Request Detail Dashboard to find out about slow methods
Dive into your code and try to optimize those slow methods
You can also correlate some metrics like the throughput or number of sessions with the response time or cpu usage
Analyze the heap with the JVM Memory Dashboard
Note: I am the developer of stagemonitor.

I would start with some tools that can help you profiling the application. Since you are developing webapp start with Lambda Probe and Java melody.
The first step is to determine the conditions under which the app starts to behave oddly. Ask yourself few questions:
Do performance issues arise right after applications starts, or overtime?
Do performance issues are correlated to quantity of client requests?
What is the real performance problem - high load on the server or lack of memory (note that they are related, so check which one starts first)
Are there any background processes which are performing some massive operations? Are they scheduled to run at some particular time period?
Try to find some clues before going deep into code. It will help you to narrow down possible causes.
As Joshua Bloch has stated in his book entitled "Effective Java" - performance issues are rarely the effect of some minor mistakes in source code (although, of course, misuse of Java constructs can lead to disaster). Usually the cause is bad system (API) architecture.
The last suggestion based on my experience - try not to think that high memory consumption is something bad. Tomcat will use as much memory as operating system and JVM will let him (not more than max settings) and just when it needs more - Tomcat will perform garbage collection. So a typical (proper!) graph of memory consumption looks like a saw. If you are dealing with memory leak, then the graph will be increasing constantly, but indefinitely. This is the most often misunderstood of memory leaks, so keep it in mind.
To be honest - we cannot help you much further. Those are just pointers, now you will have to make extensive research to figure out the cause :)

The general solution is to use a profiler e.g. YourKit, with a realistic workload which reproduces the problem.
What I do first is a CPU only profile, a memory only profile and finally a CPU & Memory profile on at once (I then look at the CPU profile results)
YourKit can also monitor your high level operations such a Java EE resources and JDBC connections. I haven't tried these as I don't use them. ;)
It can be a good idea to improve the efficiency even if its not the cause of the problem as it will reduce the amount of "noise" in these profiles and make your issues more obvious.
You could try increasing the amount of memory available but a suspect it will just delay the problem.

Ok. So I have seen huge Java applications run lesser configurations. You should try to do the following -
First connect a Profiler to your application and see which part of your application takes the most time. You can use JProfiler or Eclipse MAT ( I personally prefer JProfiler). Also try to take a look at the objects taking the most memory. This will help you narrow down to the parts which you need to rewrite to improve the performance.
Once you have taken a look at the memory leaks update your application to use 64bit JDK(assuming it already does not do so)
Take a look at your JVM arguments and optimize them.

You can try the open source tool Webapp Watcher in order to identify where in the code is the performance issue.
You have first to add a filter in the webapp (as explained here) in order to record metrics, and then import the logs in the WAW Analyzer tool and follow the steps described in the doc to know where is the potential performance issue in the code.

Use of JMX and How to use for existing applications

We have distributed web application developed few years back on JDK 5.
How JMX will help this application?
1) Will it help me to monitor performance (Memory, CPU and Network & Disk IO)?
2) If so then application is deployed in multiple servers,how can I monitor in one single dashboard?
3) Do I have to make any new code changes to the existing application or can I monitor without code changes?
4) What else can we do apart from performance monitoring? Because name is Management Extensions (MX) what can we manage and how? It seems primary objective is not monitoring, it is management?
I couldn’t get much about JMX from the tutorial provided in Oracle website http://docs.oracle.com/javase/tutorial/jmx/overview/why.html. It seems some business representative prepared this tutorial not a techie.
Can anyone help me in understanding this JMX stuff, your efforts are sincerely appreciated.
Thanks,
Your Friend.

JMX is a standardized way of getting information out of a running system and to invoke operations on it. The JVM gives you a set of MBeans through which you can access runtime data like memory consumption, GC stats and some more data. You can also invoke a number of operations. Your app server will also give you a number of MBeans which you can use to control the server and installed applications.
Yes, it will give you some performance data. See http://docs.oracle.com/javase/1.5.0/docs/guide/management/overview.html for a list of resources.
This depends on your application server. If it support clustered environments, it will probably give you a dashboard.
If you are fine with the standard JVM and app server MBeans, you won't need do any changes. But you may want to write your own MBean to give specific application status and statistics or ways to control your application. See http://docs.oracle.com/javase/tutorial/jmx/mbeans/standard.html
Well, you can do anything you like in a standardized way.
Take a look at jconsole (included in the JDK) to see what JMX offers for you!

Scalability of a single server for running a Java Web application

I want to gain more insight regarding the scale of workload a single-server Java Web application deployed to a single Tomcat instance can handle. In particular, let's pretend that I am developing a Wiki application that has a similar usage pattern like Wikipedia. How many simultaneous requests can my server handle reliably before going out of memory or show signs of excess stress if I deploy it on a machine with the following configuration:
4-Core high-end Intel Xeon CPU
8GB RAM
2 HDDs in RAID-1 (No SSDs, no PCIe based Solid State storages)
RedHat or Centos Linux (64-bit)
Java 6 (64-bit)
MySQL 5.1 / InnoDB
Also let's assume that the MySQL DB is installed on the same machine as Tomcat and that all the Wiki data are stored inside the DB. Furthermore, let's pretend that the Java application is built on top of the following stack:
SpringMVC for the front-end
Hibernate/JPA for persistence
Spring for DI and Security, etc.
If you haven't used the exact configuration but have experience in evaluating the scalability of a similar architecture, I would be very interested in hearing about that as well.
Thanks in advance.
EDIT: I think I have not articulated my question properly. I mark the answer with the most up votes as the best answer and I'll rewrite my question in the community wiki area. In short, I just wanted to learn about your experiences on the scale of workload your Java application has been able to handle on one physical server as well as some description regarding the type and architecture of the application itself.

You will need to use group of tools :
Loadtesting Tool - JMeter can be used.
Monitoring Tool - This tool will be used to monitor various numbers of resources load. There are Lot paid as well as free ones. Jprofiler,visualvm,etc
Collection and reporting tool. (Not used any tool)
With above tools you can find optimal value. I would approach it in following way.
will get to know what should be ratio of pages being accessed. What are background processes and their frequency.
Configure my JMeter accordingly (for ratios) , and monitor performance for load applied ( time to serve page ...can be done in JMeter), monitor other resources using Monitor tool. Also check count of error ratio. (NOTE: you need to decide upon what error ratio is not acceptable.)
Keep increasing Load step by step and keep writting various numbers of interest till server fails completely.
You can decide upon optimal value based on many criterias, Low error rate, Max serving time etc.
JMeter supports lot of ways to apply load.

To be honest, it's almost impossible to say. There's probably about 3 ways (of the top of my head to build such a system) and each would have fairly different performance characteristics. You best bet is to build and test.
Firstly try to get some idea of what the estimated volumes you'll have and the latency constraints that you'll need to meet.
Come up with a basic architecture and implement a thin slice end to end through the system (ideally the most common use case). Use a load testing tool like (Grinder or Apache JMeter) to inject load and start measuring the performance. If the performance is acceptable - be conservative your simple implementation will likely include less functionality and be faster than the full system - continue building the system and testing to make sure you don't introduce a major performance bottleneck. If not come up with a different design.
If your code is reasonable the bottleneck will likely be the database and somewhere in the region 100s of db ops per second. If that is insufficient then you may need to think about caching.

Definitely take a look at Spring Insight for performance monitoring and analysis.

English Wikipedia has 14GB data. A 8GB mem cache would have very high hit/miss ratio, and I think harddisk read would be well within its capacity. Therefore, the app is most likely network bound.
English Wikipedia has about 3000 page views per second. It is possible that tomcat can handle the load by careful tuning, and the network has enough throughput to server the traffic.
So the entire wikipedia site can be hosted on one moderate machine? Probably not. Just an idea.
-
http://stats.wikimedia.org/EN/TablesWikipediaEN.htm
http://stats.wikimedia.org/EN/TablesPageViewsMonthly.htm

Tomcat doesn't allow for spreading over multiple machines. If you really are concerned about scalability, you must consider what to do when your application outgrows a single machine.