Develop Reference

Facing a strange memory "leak" with Java 1.6

I'm trying to understand where is used server memory.
When I look to the memory usage on the system via top I've got :
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6091 oper 20 0 2721m 1.4g 9180 S 0 11.4 6:13.42 java
10854 oper 20 0 2186m 1.1g 5104 S 1 9.1 114:52.15 java
9350 oper 20 0 2293m 971m 4892 S 0 8.0 40:15.68 java
9286 oper 20 0 2082m 800m 4852 S 0 6.6 31:31.44 java
10506 oper 20 0 1936m 711m 4900 S 0 5.9 49:09.64 java
8965 oper 20 0 1918m 680m 5076 S 0 5.6 106:53.10 java
All those process are tomcats 6.0.20 running on JVM 1.6.0_26
As we can see in the top report, one process is using 1.4GO an other 1.1GO... so much more than expected.
When I open JConsole on the first process I can see cumulated memory (the Heap and non heap memory) is arround 200Mo, on the second 385 Mo, the third 235Mo.
So my question is where is the unvisible memory ?
Top - JConsole =
1.4G - 200M = 1.2G
1.1G - 385M = 715M
971M - 235M = 736M
800M - 173M = 627M
Does any one have an idea ?
Thanks a lot.

Why FileNotFoundException is thrown while it exists on linux

This is the first time i have encounter such problem with file access by Java on linux. The problem is just like the header says - FileNotFoundException is thrown when file actually exists. Moreover application with same configuration (props.txt file) runs like it should on windows.
Let me provide a little bit of console output
datasu#dedi2392:~/netcrawler/dkpto$ ls -l
total 20
-rwxrw-rw- 1 datasu datasu 114 Aug 7 15:53 autoupdate
drwxr-xr-x 4 datasu datasu 4096 Aug 8 11:57 data
drwxr-xr-x 2 datasu datasu 4096 Aug 8 11:57 log
-rw-rw-rw- 1 datasu datasu 32 Aug 8 12:44 props.txt
-rwxrw-rw- 1 datasu datasu 126 Aug 8 12:55 propsUpdate
datasu#dedi2392:~/netcrawler/dkpto$ ./propsUpdate
Parent: /usr/home/datasu/netcrawler/dkpto
1# -> propsUpdate
2# -> autoupdate
3# -> props.txt
4# -> data
5# -> log
(No such file or directory)ava.io.FileNotFoundException: /usr/home/datasu/netcrawler/dkpto/props.txt
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.(Unknown Source)
at netcrawler.Autoupdater.readProperties(Autoupdater.java:71)
at netcrawler.Autoupdater.start(Autoupdater.java:54)
at netcrawler.Autoupdater.main(Autoupdater.java:47)
datasu#dedi2392:~/netcrawler/dkpto$ java -version
java version "1.6.0_45"
Java(TM) SE Runtime Environment (build 1.6.0_45-b06)
Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01, mixed mode)
datasu#dedi2392:~/netcrawler/dkpto$
and here is Java code responsible for generating that output (at least after calling ./propsUpdate)
private void readProperties(String args) throws FileNotFoundException, IOException {
System.out.println("Parent: " + new File(args).getAbsoluteFile().getParentFile().getAbsolutePath());
CommonTools.PrintArray(new File(args).getAbsoluteFile().getParentFile().list());
properties.load(new FileInputStream(new File(args).getAbsoluteFile())); // this line throws the exception
stageNumber = Integer.parseInt(properties.getProperty(PROP_STAGE_NUMBER_KEY, "0"));
}
So why the props.txt file is not found when it is actually there ?

The string "args" probably has a nonprinting character at the end, like a space. You could use String.trim() to remove such characters before using that variable.

Is your home folder really this path?
/usr/home/datasu
/home/datasu is where it normally is on linux.
Also, try changing that line to this:
properties.load(new FileInputStream(new File(args));
If you're calling that as ./propsUpdate ./props.txt that will work from the current working directory.

How to analyze the performance of a Java Class without actually running it?

We need to implement an application for evaluating results of an online programming challenge. The users will implement the programming challenge and compile their source through a web interface. We are supposed to compile the submitted sources on the fly and present some statistics of the program like expected memory consumption and possible performance indicators of the sources. Does anybody know how can we gather memory consumption and performance indicators of the program statically from the sources?

While you could possibly do static analysis of the source to infer performance characteristics, I suspect it would be far simpler to just run a JUnit test suite over the code.
If you can present your challenge as a code stub or interface, you should be able to create a suitable JUnit suite which validates correctness and tests performance.
Granted, JUnit may not be the best way of running performance tests but you can likely bend it to the task. Alternatively you could look at JMeter or something similar.

Found something very useful. I am not sure if this is what I am looking for. I am yet to analyse the results. But this is quite interesting.
We can gather some performance statistics using the HPROF profiler agent shipped with the JDK release. The good thing is that it can be run during the compilation to produce some interesting statistics on the code beign compiled. Following are some samples. More details can be found at http://download.oracle.com/javase/7/docs/webnotes/tsg/TSG-VM/html/tooldescr.html#gbluz
$ javac -J-agentlib:hprof=heap=sites Hello.java
SITES BEGIN (ordered by live bytes) Wed Oct 4 13:13:42 2006
percent live alloc'ed stack class
rank self accum bytes objs bytes objs trace name
1 44.13% 44.13% 1117360 13967 1117360 13967 301926 java.util.zip.ZipEntry
2 8.83% 52.95% 223472 13967 223472 13967 301927 com.sun.tools.javac.util.List
3 5.18% 58.13% 131088 1 131088 1 300996 byte[]
4 5.18% 63.31% 131088 1 131088 1 300995 com.sun.tools.javac.util.Name[]
$ javac -J-agentlib:hprof=heap=dump Hello.java
HEAP DUMP BEGIN (39793 objects, 2628264 bytes) Wed Oct 4 13:54:03 2006
ROOT 50000114 (kind=<thread>, id=200002, trace=300000)
ROOT 50000006 (kind=<JNI global ref>, id=8, trace=300000)
ROOT 50008c6f (kind=<Java stack>, thread=200000, frame=5)
:
CLS 50000006 (name=java.lang.annotation.Annotation, trace=300000)
loader 90000001
OBJ 50000114 (sz=96, trace=300001, class=java.lang.Thread#50000106)
name 50000116
group 50008c6c
contextClassLoader 50008c53
inheritedAccessControlContext 50008c79
blockerLock 50000115
OBJ 50008c6c (sz=48, trace=300000, class=java.lang.ThreadGroup#50000068)
name 50008c7d
threads 50008c7c
groups 50008c7b
ARR 50008c6f (sz=16, trace=300000, nelems=1,
elem type=java.lang.String[]#5000008e)
[0] 500007a5
CLS 5000008e (name=java.lang.String[], trace=300000)
super 50000012
loader 90000001
:
HEAP DUMP END
$ javac -J-agentlib:hprof=cpu=times Hello.java
CPU TIME (ms) BEGIN (total = 2082665289) Wed oct 4 13:43:42 2006
rank self accum count trace method
1 3.70% 3.70% 1 311243 com.sun.tools.javac.Main.compile
2 3.64% 7.34% 1 311242 com.sun.tools.javac.main.Main.compile
3 3.64% 10.97% 1 311241 com.sun.tools.javac.main.Main.compile
4 3.11% 14.08% 1 311173 com.sun.tools.javac.main.JavaCompiler.compile
5 2.54% 16.62% 8 306183 com.sun.tools.javac.jvm.ClassReader.listAll
6 2.53% 19.15% 36 306182 com.sun.tools.javac.jvm.ClassReader.list
7 2.03% 21.18% 1 307195 com.sun.tools.javac.comp.Enter.main
8 2.03% 23.21% 1 307194 com.sun.tools.javac.comp.Enter.complete
9 1.68% 24.90% 1 306392 com.sun.tools.javac.comp.Enter.classEnter
10 1.68% 26.58% 1 306388 com.sun.tools.javac.comp.Enter.classEnter
...
CPU TIME (ms) END

Failed to install WWW::HtmlUnit in Windows box

I'm trying to install WWW::HTMLUnit on Windows 7. There're step that I run through:
Install Inline::Java 0.53
Install WWW::HTMLUnit 0.15
At step 2, after nmake, I type nmake test to test module but it failed. Here's output:
C:\nmake test
Microsoft (R) Program Maintenance Utility Version 9.00.30729.01
Copyright (C) Microsoft Corporation. All rights reserved.
C:\Perl\bin\perl.exe "-MExtUtils::Command::MM" "-e" "test_harness(0, 'blib\lib', 'blib\arch')" t/*.t
t/00_basic...........
t/00_basic...........NOK 1/1# Failed test 'use WWW::HtmlUnit;'
# at t/00_basic.t line 9.
# Tried to use 'WWW::HtmlUnit'.
# Error: Class com.gargoylesoftware.htmlunit.WebClient not found at C:/Perl/site/lib/Inline/Java.pm line 619
# BEGIN failed--compilation aborted at (eval 4) line 2, <GEN7> line 4.
# Looks like you failed 1 test of 1.
t/00_basic...........dubious
Test returned status 1 (wstat 256, 0x100)
DIED. FAILED test 1
Failed 1/1 tests, 0.00% okay
t/01_hello...........Class com.gargoylesoftware.htmlunit.WebClient not found at C:/Perl/site/lib/Inline/Java.pm line 619
BEGIN failed--compilation aborted at t/01_hello.t line 4, <GEN7> line 4.
t/01_hello...........dubious
Test returned status 26 (wstat 6656, 0x1a00)
t/02_hello_sweet.....dubious
Test returned status 19 (wstat 4864, 0x1300)
t/03_clickhandler....Class com.gargoylesoftware.htmlunit.WebClient not found at C:/Perl/site/lib/Inline/Java.pm line 619
BEGIN failed--compilation aborted at t/03_clickhandler.t line 6, <GEN7> line 4.
t/03_clickhandler....dubious
Test returned status 29 (wstat 7424, 0x1d00)
DIED. FAILED tests 1-8
Failed 8/8 tests, 0.00% okay
Failed Test Stat Wstat Total Fail List of Failed
-------------------------------------------------------------------------------
t/00_basic.t 1 256 1 1 1
t/01_hello.t 26 6656 ?? ?? ??
t/02_hello_sweet.t 19 4864 ?? ?? ??
t/03_clickhandler.t 29 7424 8 16 1-8
Failed 4/4 test scripts. 9/9 subtests failed.
Files=4, Tests=9, 3 wallclock secs ( 0.00 cusr + 0.00 csys = 0.00 CPU)
Failed 4/4 test programs. 9/9 subtests failed.
NMAKE : fatal error U1077: 'C:\Perl\bin\perl.exe' : return code '0x1d'
Stop.
From above log, I could see that:
class Error: com.gargoylesoftware.htmlunit.WebClient could not be found.
I have no idea that I missed anything.
Any help would be appreciated.
Thanks.
Minh.

I found it.
There's different between path in Unix and Windows system. Unix uses ':' for a delimiter but Windows uses ';'. So what I've done is that open HTMLUnit.pm and change all of ':' to ';'.
With HTMLUnit version 0.15 I made changes at these lines below:
Line 78:
return join ';', map { "$jar_path/$_" } qw( # return join ':', map { "$jar_path/$_" } qw(
Line 127:
$custom_jars = join(';', #{$parameters{'jars'}}); # $custom_jars = join(':', #{$parameters{'jars'}});
Line 148:
CLASSPATH => collect_default_jars() . ";" . $custom_jars, # CLASSPATH => collect_default_jars() . ":" . $custom_jars,
And it works like a magic.

(it wouldn't let me comment on an existing answer)
I see your answer about ':' vs ';'. I'll try to include a fix in the next WWW::HtmlUnit release (I am the author of the perl bindings).

Java: why are multiple objects showing up with runhprof output?

I was curious about the runhprof output? I am mainly concerned about the memory section. It looks like there are multiple entries of the same class. Why would that be.
Is there a way to get hprof to print how much memory a particular class(the instances of that class) take up in memory. One value for each class.
Also, what tools do you use beside 'hat' to analyze the output?
I ran the java command with jvm arg:
-Xrunhprof:heap=sites,depth=4,format=a,file=prof/hprof_dump.txt
Here is brief snippet of the output. Some classes are listed multiple times in the output.
SITES BEGIN (ordered by live bytes) Tue Jul 28 19:33:41 2009
percent live alloc'ed stack class
rank self accum bytes objs bytes objs trace name
1 29.75% 29.75% 700080 43755 576000016 36000001 307483 java.lang.Double
2 7.13% 36.88% 167840 5245 370432 11576 300993 clojure.lang.PersistentHashMap$LeafNode
3 2.09% 38.98% 49296 2054 60048 2502 301295 clojure.lang.Symbol
4 2.09% 41.07% 49200 3 49200 3 301071 char[]
5 1.33% 42.40% 31344 1306 68088 2837 300998 clojure.lang.PersistentHashMap$BitmapIndexedNode
6 1.10% 43.50% 25800 645 25800 645 301050 clojure.lang.Var
7 1.05% 44.54% 24624 3 24624 3 301069 byte[]
8 0.86% 45.40% 20184 841 49608 2067 301003 clojure.lang.PersistentHashMap$INode[]
9 0.78% 46.18% 18304 572 58720 1835 301308 clojure.lang.PersistentList
10 0.75% 46.93% 17568 549 17568 549 308832 java.lang.String[]
11 0.70% 47.62% 16416 2 16416 2 301036 byte[]

Eclipse Memory Analyzer is excellent. Loads the dump file up very very quickly, produces lots of nice reports about the heapdump, lets you query the dump for objects/classes using a SQL-like language. Love it.