I have a performance problem(Hi-CPU) in my server application. But the issue only appears when the server has been running for a long time(a week on average). I tried to use Netbeans built-in profiler to diagnose the problem, but after running for a few hours it simply stops responding.
Is there any way to perform long term monitoring of the application? Thanks.
UPDATE:This is NOT a web application
You should take a look at Visual VM if you havent done so already (if you're on JDK 6 update 7 or above). I'm sure its gonna be quite comforting since its an "adaptation" of Netbeans profiler into Java core.
I have used perf4j in the past for performance logging to find bottlenecks
I wrote a performance logger tool that samples metrics such as cpu, memory and threads count and logs them using log4j. By comparing the performance logs to your application logs, you can easlly detect performance patterns and profile your application. Enjoy!
I don't know much about profiling tools out there, so I am not going to recommend a profiling tool.
Instead, what I would do is use AspectJ to inject logging (Log4j) statements into my code. If you are aware of certain areas that you know you want monitored, then just inject the logging at those specific pointcuts. Otherwise, if you want to look at everything, then you can just generalize your pointcuts and inject all over.
Collect the logs for a week and then look for what might be behaving 'irregularly'. These log files will probably be pretty lengthy depending on what you decide to log, so you may need to use some file parsing to collect information like:
Compare these numbers from the beginning of the week with those when your performance dives. If you are getting noticeable performance hits, then I imagine these will be pretty obviously reflected in log files of this nature.
This may be a bit more work than running an out of the box profiling tool, but is pretty flexible and may be a good learning experience if you haven't done much with AspectJ before.
There is a commercial offering, that you could consider DynaTrace, from Compuware. I do not work for, and have never worked for or with Compuware, however my current organisation is currently considering using Dynatrace in our production environment.
Depending on the type of Java application that you're running, whether it is a desktop-based, or server-based, installation would be adding a jar to the classpath (plus adding a network appliance to your network). Running the application would cause instrumentation data to be sent to the Dynatrace servers, where you can retrieve it, and even trace individual user paths and transactions through the entire system (irrespective of the technology used).
Dynatrace is something I would envisage you left running at all times, rather than just when you wanted to diagnose a particular problem. Also, Dynatrace is not cheap - you're probably looking at a 5-figure cost per server, however from what I've seen, it appears to work almost like a permanently running debugger, but without too much of a performance penalty (<5%).
I don't know exactly what type of application you're running, but ours is a Java EE based web application, interfacing to some of our back-end servers.
HTH
