I've installed weblogic 10.3.3, configured simple domain with default configuration. And after starting weblogic I cannot use it via admin console because it is starting too slow (~10 mins). It seems that admin console application is being deployed incorrectly. Even when I enable 'staging' mode in AdminConsole deployment options, it does not help. Does anybody have a solution for this problem? It really bugs me out.
Configuration properties:
Server: Dell PowerEdge r410 server (Six-core Intel® Xeon® x64 CPU, 8GB RAM)
OS: Ubuntu Maverick 10.10 x86_64
Weblogic: 10.3.3 x64 (used wls1033_generic.jar file for installation)
Java: 1.6.0_17_i586 (tried with different jdk's including x64, but it does not work either)
Turns out weblogic uses random number generator during startup. Because of the bug in java it reads random bits from /dev/random. There are almost no problems with /dev/random except that it is extremely slow. It takes sometimes 10 minutes or more to generate one number. Simple solution exists – using /dev/urandom instead. It is not that good as /dev/random, but at least it is instant.
Java somehow maps /dev/urandom file to /dev/random. That’s why default settings in $JAVA_HOME/jre/lib/security/java.security are useless, they just do not make any sense.
Problem fix is very simple – adding string export JAVA_OPTIONS="-Djava.security.egd=file:/dev/./urandom" to the /etc/bash.bashrc file. Usage of /dev/./urandom instead of simple /dev/urandom is another hack. JVM does not understand the option value otherwise.
Be aware of this problem if you try to setup weblogic under UNIX-based OS.
Without looking at the startup logs we could only speculate. Weblogic typically is a beast. It typically takes anywhere fromo 6-8 minutes to start up for me on my development machines, but that seems like a nice server for it to take 10 minutes for you.
The Admin Console should not be this slow, nor should it take 6-8 minute to start a server or deploy the Admin Console. The deployment of the admin console on my machine takes less than 30 seconds, and startup is under 60 seconds. Startup time for WebLogic is a common complaint, but chances are that something is misconfigured if there is a very long startup or admin console deployment time.
Related
We have an (Eclipse RCP) application of 90MB with 139 self-signed jars which starts in 8s without Web Start and in 10s in an older version of Java 7. We configured Java to not use the browser proxy, i.e. deployment.proxy.type=0.
With each update of Oracle's Java startup performance drops. It takes more and more time to fully start up:
7u60/7u65/8u25: 13s (starts after 5s of web start processing)
7u75: 23s
8u31: 20s
8u40: 29s
8u51/8u60ea: 32s
What can I do to solve this issue?
From the trace/logs I can see that it is very probable that this slowdown is completely due to validating the cached jars taking much more time. Note that this question is similar but doesn't provide the following details:
Diagnosis:
When cached, the update check runs in only 0.5s (server returns "304 Not
Modified"), but even with a full download it takes only a few seconds on the gigabit network.
After the update check, for each jar XXX there is a log entry:
validating cached jar XXX.jar
When this is done, com.sun.javaws.Main is started after which the same validation seems to happen again and takes about the same amount of time, then the application starts.
The time spent in validating the cached jars seems to correspond to
the extra time required before the application starts.
The web start splash screen always shows for about 2 seconds corresponding to the update check and is then hidden. Then after almost 20 seconds the Java console finally appears and my application actually starts.
During the delay, jp2launcher.exe uses about 16% processor time on a quad core with hyperthreading (8 logical cores). So it looks like it is fully using one of the logical processors.
What I have already tried but did not make any difference:
clearing web start cache (countless times)
configure deployment.properties to disable certificate revocation check (as well as blacklist.check and validation.ocsp, validation.crl)
running offline
using the version download protocol
add to site exceptions list
check web server logs for problems. None found, update check runs in about 500ms for all 138 jars.
use another web server
checked certificate expiration date 17 feb 2016
validated my jnlp with JaNela and found no serious issues
create a deployment rule set to allow the application to run unsigned in order to speed up validation. This should be possible and looks like a promising way to solve this, but I could not get it working. See also my answer on this post.
configured Java to "do not start the console"
Detail: some weird behavior on 7u60
In 7u60 the application is started after about 5 seconds, after which the Java console APPEARS to be doing the jar validation in the background while the application is already started. HOWEVER the .log file reports that the application gets started AFTER all the validations are done. It reports this as 25 seconds and then shows the first System.out of my application which actually happened after only 5 seconds or so. It also shows the jar update check with the server taking 10 times as long as reported by the server. So I guess this is an issue with the logging framework lagging behind! Haven't seen this on 8u51.
Not an answer per se (yet), but I found that Java 8u25 when tracing is enabled, only generate a single trace file. 8u51 generates two files, one from the JVM used to update the application and other to run it. This is new (two JVM startups) and I think is related to the new setting for using native Windows sandbox capabilities. The problem is that it shouldn't have to validate the signatures again on the second JVM. The separation on two JVM instances always happen no matter if the setting for using native sandbox is disabled (the default).
I reported a regression bug, will edit the answer if I get an answer from Oracle.
Note: Java 8u31 still runs everything on one JVM but have the same doubled startup time the question stated.
I have searched and searched and did not help me much hence posting the new question.
Platform
Ubuntu 11.10 server 64 bit
JVM 1.7.0_03
Tomcat 7
There is nothing special in the configuration - front end server is apache using ajp connector. Tomcat runs as ubuntu service.
On our server, tomcat7 is dying and could not figure out the reason. I have checked all the log files (syslog, catalina.out, even auth.log) to see if there is something getting logged.
As per top command server still has around 4gb of memory free and cpu usage is averaging around 35% most of the times.
In order to isolate the problem, is there any way to get the exit status code of tomcat process that terminated?
I read some reports where jvm logging error log in case of jvm crash. I am not seeing it either.
It seems like I need to set ulimit to get the core dump, but not sure how to do it for tomcat service or is the setting valid for all the users.
It seems like I need to set ulimit to get the core dump, but not sure how to do it for tomcat service or is the setting valid for all the users.
One way to do that without interfering with anything else would be to add a ulimit command to the catalina.sh script. (It is a bit hacky ... but it sounds like you are at the point where hackiness might give happiness.)
We just started a project in ADF, Oracle's Java EE framework. Usually we develop in Tomcat, then deploy into the production WebLogic servers.
But we realized ADF requires a WebLogic server installed locally to develop in Eclipse/JDeveloper. This is really heavyweight, even configured as dev, it is killing our machines, and taking a long time to deploy.
So, are there any configuration parameters we could tweak so WebLogic takes as few resources as possible? Are there any alternatives we could use?
We are mostly concerned about memory (it is taking a wonderful 800MB,) and startup time (~2 minutes)
On the memory consumption issue, you might want to try setting the memory parameters of the JVM used by your WebLogic server. Log in to your WL Web Admin Console and go to Environment/Servers/[your server]/Configuration/Server Start and, on the "Arguments", setting something like -Xms256m -Xmx256m will set your JVM's initial (Xms) and maximum (Xmx) heap size to 256 megabytes. You will want to play around with these numbers and find the best values for your environment. But please be aware that your Eclipse instance might be consuming a lot of memory as well.
Regarding the startup time, although a bit larger than I would expect, they seem OK. This problem is very frequent, and I don't think you will be able to definitely solve it. WebLogic has much more features than Tomcat, and this reflects in other characteristics of the environment (like startup time). You might find some useful tips here, though: Speed up Weblogic Server startup times
There is no restriction to use a local server for development. You can always go for a shared server and deploy your projects for testing.
I agree with the previous post. If memory is a concern then you may install a stand alone WLS and deploy to it from an EAR. Make sure for the stand alone WLS you install the Application Development Runtime Libraries as by default WLS does not have domains enabled for ADF - http://www.oracle.com/technetwork/developer-tools/adf/downloads/index.html
When you install WLS, make sure you install it in development mode (which is setting the embedded WLS has too)
Beside of this, there is not much you can do to start WLS withlimited functionality.
Frank
Short description of my problem: I start up Tomcat with my deployed Wicket application. When I want to shut down tomcat I get this error message:
Error occurred during initialization of VM
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:640)
at java.lang.ref.Reference.<clinit>(Reference.java:145)
I am running the following setup:
Ubuntu Linux: 10.04 (lucid) with a 2.6.18-028stab094.3 kernel
Java Version: "1.6.0_26" Java HotSpot(TM) 64-Bit Server VM
Tomcat Version: 7.0.23
jvm_args: -Xms512m -Xmx512m -XX:MaxPermSize=205m (these are added via CATALINA_OPTS, nothing else)
Wicket 1.5.1
Tomcat is configured with two virtual hosts on subdomains with ModProxy
My application is deployed as ROOT.war in the appbase directory (it makes no difference if I deploy one or both applications)
'''No application deployed does not result in OOM on shutdown''', unless I mess around with the jvm args
The size of the war is about 500k, all libraries are deployed in tomcat/common/lib (directory which I added to common.loader in conf/catalina.properties)
ulimit -u -> unlimited
When I check the Tomcat manager app it says the following about the JVM memory:
Free memory: 470.70 MB Total memory: 490.68 MB Max memory: 490.68 MB
(http connector) Max threads: 200 Current thread count: 6 Current thread busy: 1
'top' or 'free -m' is similar:
Mem: 2097152k total, 1326772k used, 770380k free, 0k buffers
20029 myuser 18 0 805m 240m 11m S 0 11.7 0:19.24 java
I tried to start jmap to get a dump of the heap, it also fails with an OutOfMemoryError. Actually as long as one or both of my applications are deployed any other java process fails with the same OOM Error (see top).
The problem occurs while the application is deployed. So something is seriously wrong with it. However the application is actually running smoothly for quite a while. But I have seen OOMs in the application as well, so I don't trust the calm.
My application is using a custom filter class? Could that be it?
For completeness (hopefully), here's the list of libraries in my common/lib:
activation-1.1.jar
antlr-2.7.6.jar
antlr-runtime-3.3.jar
asm-3.1.jar
asm-commons-3.1.jar
asm-tree-3.1.jar
c3p0-0.9.1.1.jar
commons-collections-3.1.jar
commons-email-1.2.jar
dependencies-provided.tgz
dom4j-1.6.1.jar
ejb3-persistence-1.0.2.GA.jar
geronimo-annotation_1.0_spec-1.1.1.jar
geronimo-jaspic_1.0_spec-1.0.jar
geronimo-jta_1.1_spec-1.1.1.jar
hibernate-annotations-3.4.0.GA.jar
hibernate-commons-annotations-3.1.0.GA.jar
hibernate-core-3.3.0.SP1.jar
hibernate-entitymanager-3.4.0.GA.jar
hibernate-search-3.1.0.GA.jar
javassist-3.4.GA.jar
joda-time-1.6.2.jar
jta-1.1.jar
log4j-1.2.16.jar
lombok-0.9.3.jar
lucene-core-2.4.0.jar
mail-1.4.1.jar
mysql-connector-java-5.1.14.jar
persistence-api-1.0.jar
quartz-2.1.1.jar
servlet-api-2.5.jar
slf4j-api-1.6.1.jar
slf4j-log4j12-1.6.1.jar
stringtemplate-4.0.2.jar
wicket-auth-roles-1.5.1.jar
wicket-core-1.5.1.jar
wicket-datetime-1.5.1.jar
wicket-extensions-1.5.1.jar
wicket-request-1.5.1.jar
wicket-util-1.5.1.jar
xml-apis-1.0.b2.jar
I appreciate any hint or even speculation that gives me additional ideas what to try.
Update: I tested some more and found that this behaviour only occurs while one or both of my applications are deployed. The behaviour does not occur on "empty" tomcat (that was a mistake on my part messing with jvm args)
Update2: I am currently experimenting trying to reproduce this behaviour in a virtual box, I want to debug this with a profiler. I am still not convinved that it should be impossible to run my setup on 2GB RAM.
Update3 (10/01/12): I am trying to run jenkins instead of my own application. Same behaviour, so it is definitely server configuration issues. Jenkins jobs fail when maven is called, so I need not even try the shutdown hack suggested below because I need a second java process while running Jenkins. It was suggested to me that because this is a Virtual Server ulimits may be imposed from outside and I would not be able to see them. I think I'll ask a new question regarding this. Thx all.
Update4 (02/05/12): see below for the answer that contains the hint. I'll clarify some more up here: I am now 95% sure that the errors occur because I am reaching my thread limit. However because this is a virtual server the method described below would not work to check this value because it is not visible with ulimit, that was what was confusing me and only today I found out that this is the "numproc" value that I can see in the Parallels Power Panel that I can log into for my virtual server. There were Resource Alerts for numproc but I did not see those either until just now. The value has a hard limit of 96 which I cannot change of course. The current value of numproc corresponds to the number of processes I see with "top" after toggling "H" to see threads. I had a very hard time finding this because this numproc value is hidden deep inside the panel. Sadly 96 is a rather low number if you want to run a tomcat with apache and mysql. I am also very sad that I cannot even find this value in the small print of my hosting contract and it is rather relevant to my application I dare say. So I guess I'll need a server upgrade.
Thanks all for your helpful answers in the end everyone helped me a bit to find out what the problem was.
The tomcat shutdown procedure consits of sending an command/word via a tcp port to the running tomcat VM. This port is configured in the server.xml (if I remember corretly, writting on my phone right now). So far so good.
Unfortunately, the shutdown script does this by starting a 2. VM using the same java options used for the tomcat. Your system simply has not enough memory for this.
As a sollution you could write your own stop script using telnet or something.
I could help with later if needed.
Hope that helps.
Viele grüsse Bert
Seems you have too many threads open.
Use this command :
ulimit -u
What is the result ?
Should be something like :
max user processes (-u) 100
If this is correct, you can edit this file :
/etc/security/limits.conf
and the the following modifications :
#<domain> <type> <item> <value>
user soft nproc 10000
user hard nproc 10000
You can probably survive for a while like this. All you need to do is kill the tomcat process whenever you need to restart it. It is not a nice approach, but the main concern is that your application runs correctly.
It seems to me though, that on the long run, you might need to order a hosting plan with more RAM available.
I was having a similar problem with a tomcat installation just last week. I managed to fix it by giving tomcat a smaller heap. Something like this:
export CATALINA_OPTS=”-Xms256m -Xmx512m”
Before starting Tomcat may help. In the meantime you'll have to kill it the old fashioned way, with a kill -9 ;)
EDIT: you could also take look here, it appears tomcat automatically creates a bunch of "spare" threads, but you can limit those as well as your max thread count in the config. Hope it helps.
HI All,
I am working on WAS 7 and i am using windows 7 operating system and IE9 so can anyone suggest me what kind of good PC configuration i needed because my PC is getting hanged when i started worked on this... so PC Configuration will be like RAM ? processor ? like that...
Look at this for the bare minimum requirements.
http://www-01.ibm.com/support/docview.wss?rs=180&uid=swg27012421
Ensure that the memory is far higher than what is stated in this document. 1 GB is certainly not adequate.
It also depends on what you are running on this box (besides WebSphere App Server)
How many instances (or servers) of WAS are you running on it. Have about 1.5 GB (not referring to the JVM Heap here but the whole java process) per server instance as a ball park to have a reasonable performance.
HTH
Manglu