Problem:
I have an J2EE application that is running very slow for a given task (to generate some files) using a specific data set. I decided to run the visualvm profiler over the application to understand the nature of the problem. The application is running over Widlfly 10 using the latest Java 8.
To avoid configuration complexity, both profiler (visualvm) and Wildfly server are running locally.
The first problem met was the JVMTI error 62, that was resolved using the -Xverify:none as jvm parameter.
The second problem (the blocking one) is that if I run the profiler when the application is deployed and running, the Wildfly deployment scanner throws Exceptions like this one:
13:19:04,476 INFO [stdout] (*** Profiler Agent Communication Thread) Profiler Agent: Waiting for connection on port 5140 (Protocol version: 15)
13:19:04,517 INFO [stdout] (*** Profiler Agent Communication Thread) Profiler Agent: Established connection with the tool
13:19:04,847 INFO [stdout] (*** Profiler Agent Communication Thread) Profiler Agent: Local accelerated session
13:19:15,984 WARN [org.jboss.as.ejb3.timer] (EJB default - 6) WFLYEJB0043: A previous execution of timer [id=e6a6ea46-6c2a-466d-b767-cccc8d5021a9 timedObjectId=ecs.ecs.BatchCoordinator auto-timer?:false persistent?:false timerService=org.jboss.as.ejb3.timerservice.TimerServiceImpl#142b16ca initialExpiration=Sun Aug 05 13:17:05 COT 2018 intervalDuration(in milli sec)=5000 nextExpiration=Sun Aug 05 13:19:15 COT 2018 timerState=IN_TIMEOUT info=null] is still in progress, skipping this overlapping scheduled execution at: Sun Aug 05 13:19:15 COT 2018.
13:19:16,025 ERROR [org.jboss.as.controller.management-operation] (DeploymentScanner-threads - 1) WFLYCTL0013: Operation ("read-resource") failed - address: ([("deployment" => "my_application.war")]): java.lang.NoClassDefFoundError: org/jboss/as/controller/AttributeDefinition$$Lambda$157
at org.jboss.as.controller.AttributeDefinition$$Lambda$157/1799183343.get$Lambda(Unknown Source)
at org.jboss.as.controller.AttributeDefinition.getNoTextDescription(AttributeDefinition.java:872)
at org.jboss.as.controller.AttributeDefinition.addResourceAttributeDescription(AttributeDefinition.java:697)
at org.jboss.as.controller.ListAttributeDefinition.addResourceAttributeDescription(ListAttributeDefinition.java:158)
at org.jboss.as.controller.descriptions.DefaultResourceDescriptionProvider.getModelDescription(DefaultResourceDescriptionProvider.java:129)
at org.jboss.as.controller.operations.global.ReadResourceHandler.doExecuteInternal(ReadResourceHandler.java:370)
at org.jboss.as.controller.operations.global.ReadResourceHandler.doExecute(ReadResourceHandler.java:173)
at org.jboss.as.controller.operations.global.GlobalOperationHandlers$AbstractMultiTargetHandler.execute(GlobalOperationHandlers.java:263)
at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:890)
at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:659)
at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:370)
at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1329)
at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:400)
at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:208)
at org.jboss.as.controller.ModelControllerImpl$3.execute(ModelControllerImpl.java:667)
at org.jboss.as.controller.ModelControllerImpl$3.execute(ModelControllerImpl.java:657)
at org.jboss.as.server.deployment.scanner.DefaultDeploymentOperations.getDeploymentsStatus(DefaultDeploymentOperations.java:76)
at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService$ScanContext.<init>(FileSystemDeploymentService.java:1622)
at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService$ScanContext.<init>(FileSystemDeploymentService.java:1571)
at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService.scan(FileSystemDeploymentService.java:572)
at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService.scan(FileSystemDeploymentService.java:491)
at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService$DeploymentScanRunnable.run(FileSystemDeploymentService.java:250)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
at org.jboss.threads.JBossThread.run(JBossThread.java:320)
Caused by: java.lang.ClassNotFoundException: org.jboss.as.controller.AttributeDefinition$$Lambda$157 from [Module "org.jboss.as.controller:main" from local module loader #4cc77c2e (finder: local module finder #7a7b0070 (roots: D:\jbossas\wildfly-10.1.0.Final-ecs\modules,D:\jbossas\wildfly-10.1.0.Final-ecs\modules\system\layers\base))]
at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:198)
at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:363)
at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:351)
at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:93)
... 30 more
If I run the profiler without any deployed application, VisualVM's profiler starts without problems; however, if I tried to deploy my application under this state/operation mode, the wildfly scanner never starts and the application is never deployed.
So, What I'm doing wrong??
Finally, I put to work the visualVM Profiler... These are the steps that you need to do in order to execute a Wildfly 10 CPU profile using VisualVM (both executed locally!!):
Download the latest VisualVM (In my case was the version 1.4.1).
Download JDK10 ... You will need the JDK10 in order to run the VisualVM only. This step is required because of this bug (this was the bug that I was getting before when profiling the application)
Change the visualVM configuration file (<visual-vm>/etc/visualvm.conf) to point to your JDK10 installation...
Go to <visual-vm>/profiler/lib and copy the library: jfluid-server.jar into: <wildfly>/standalone/lib/ext ... This library is required to overcome the following error: java.lang.NoClassDefFoundError: org/netbeans/lib/profiler/server/ProfilerRuntimeCPUFullInstr (More about this error in here and here)
Next, modify the wildfly startup script to include the package org.netbeans.lib.profiler.server in the SystemProperty: jboss.modules.system.pkgs. Example: -Djboss.modules.system.pkgs=org.jboss.byteman,org.netbeans.lib.profiler.server. (According to this link, this property forces the wildfly to locate such package from any classloader)
Start the visualVM.
Start the wildfly server (including your application EAR/WAR/JAR application)... NOTE: I was running the wildfly server from Eclipse using JBoss Tools; under this configuration, my application (WAR) was in exploded mode!
Go to visualVM, open the org.jboss.modules.Main process. Then go to the profiler tab. Modify the target Profile classes and Outgoing calls and put the Classes that you want to monitor/profile (NOTE: For now, the winner configuration for me was: Profile classes: my.package.** and Include outgoing calls: my.package.**)
Hit the CPU Button, wait for the VisualVM to instrument your target classes and then invoke the code that is running slow; later, analyze the results...
Related
We are unable to successfully startup Wildfly 18 when the 1.8 Java Corretto JDK is configured with the latest BC FIPS libraries to support FIPS compliant AWS EC2 Linux based environments.
bc-fips-1.0.2.1.jar (also tried with bc-fips-1-0-2.jar)
bctls-fips-1.0.12.2.jar
The error appears to be related to the unsuccessful initialization of the Wildlfy SSL context during startup. When the request to get context data is being issued - the BCJSSE provider does not appear have performed the context data initialization - and a null value for the context data results in the exception being thown.
Enabling FINEST level logging for both Wildfly and the BouncyCastle libraries does not provide any trace level information to confirm context initialization prior to when the error condition is reported.
--
2022-01-24 14:46:52,518 FINE [org.bouncycastle.jsse.provider.PropertyUtils] (MSC service thread 1-2) Boolean system property [org.bouncycastle.jsse.trustManager.checkEKU] defaulted to: true
2022-01-24 14:46:52,639 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-5) MSC000001: Failed to start service org.wildfly.security.ssl-context.httpsSSLContext: org.jboss.msc.service.StartException in service org.wildfly.security.ssl-context.httpsSSLContext: Failed to start service
at org.jboss.msc.service.ServiceControllerImpl$StartTask.execute(ServiceControllerImpl.java:1731)
at org.jboss.msc.service.ServiceControllerImpl$ControllerTask.run(ServiceControllerImpl.java:1559)
at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: SSLContext has not been initialized.
at org.bouncycastle.jsse.provider.ProvSSLContextSpi.getContextData(ProvSSLContextSpi.java:877)
at org.bouncycastle.jsse.provider.ProvSSLContextSpi.engineGetServerSessionContext(ProvSSLContextSpi.java:821)
at javax.net.ssl.SSLContext.getServerSessionContext(SSLContext.java:386)
at org.wildfly.security.ssl.SSLContextBuilder.lambda$build$0(SSLContextBuilder.java:340)
at org.wildfly.security.OneTimeSecurityFactory.create(OneTimeSecurityFactory.java:53)
at org.wildfly.extension.elytron.SSLDefinitions$6.lambda$getValueSupplier$1(SSLDefinitions.java:1173)
at org.wildfly.extension.elytron.TrivialService.start(TrivialService.java:53)
at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1739)
at org.jboss.msc.service.ServiceControllerImpl$StartTask.execute(ServiceControllerImpl.java:1701)
... 6 more
--
The JDK security providers are configured as following:
security.provider.1 = org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider
security.provider.2 = sun.security.provider.Sun
security.provider.3 = org.bouncycastle.jsse.provider.BouncyCastleJsseProvider fips:BCFIPS
security.provider.4 = sun.security.rsa.SunRsaSign
security.provider.5 = sun.security.ec.SunEC
security.provider.6 = com.sun.crypto.provider.SunJCE
security.provider.7 = sun.security.jgss.SunProvider
security.provider.8 = com.sun.security.sasl.Provider
security.provider.9 = org.jcp.xml.dsig.internal.dom.XMLDSigRI
Also - with the same JDK provider settings and standalone.xml settings in place - Wildlfy startup is successful when the JDK is configured with an older set of BCFIPS jars.
bc-fips-1.0.1.jar
bctls-fips-1.0.8.jar
Any help or suggestions on why the Wildfly 18 server is no longer able to successfully initialize the HTTPS service SSL context during startup, when the JDK is configured with the newest set of BouncyCastle FIPS jars would be very much appreciated.
Thank you,
-Jayesh
There is a fix to the problem of initializing the SSL context with Wildfly 18, when using updated BC FIPS libraries.
The fix is available in Wildfly 21.
https://github.com/wildfly-security/wildfly-elytron/commit/3d7375f5f61526a7b8c4046878adc68e6777b308
I have installed SoapUI 5.5.0 in rhel 7.8 server and trying to open it via MobaXterm on windows 10 local desktop.
When i run soapui.sh, soapui window opens up and display below messages on server,
#./soapui.sh
================================
=
= SOAPUI_HOME = /opt/SoapUI-5.5.0
=
================================
Configuring log4j from [/opt/SoapUI-5.5.0/bin/soapui-log4j.xml]
16:52:38,534 INFO [DefaultSoapUICore] Creating new settings at [/root/soapui-settings.xml]
16:52:39,328 INFO [PluginManager] 0 plugins loaded in 2 ms
16:52:39,328 INFO [DefaultSoapUICore] All plugins loaded
Configuring log4j from [/opt/SoapUI-5.5.0/bin/soapui-log4j.xml]
16:52:39,556 INFO [DefaultSoapUICore] Creating new settings at [/root/soapui-settings.xml]
16:52:39,601 INFO [PluginManager] 0 plugins loaded in 11 ms
16:52:39,601 INFO [DefaultSoapUICore] All plugins loaded
16:52:39,844 INFO [SoapUI] Used java version: 1.8.0_221
It opens another window called as 'Stay Tuned!'. As soon as i click on 'Yes, I want to know' or 'skip' button it gives below error on linux server and soapui window disappears.
Exception in thread "Thread-12" sun.reflect.generics.reflectiveObjects.NotImplementedException
at com.smartbear.analytics.impl.SoapUIOSMixpanelProvider.trackError(SoapUIOSMixpanelProvider.java:53)
at com.smartbear.analytics.AnalyticsManager$1.run(AnalyticsManager.java:55)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
libGL error: unable to load driver: swrast_dri.so
libGL error: failed to load driver: swrast
Prism-ES2 Error : GL_VERSION (major.minor) = 1.4
java.lang.UnsupportedOperationException: Internal Error
at com.sun.glass.ui.gtk.GtkApplication.lambda$new$5(GtkApplication.java:158)
at java.security.AccessController.doPrivileged(Native Method)
at com.sun.glass.ui.gtk.GtkApplication.<init>(GtkApplication.java:140)
at com.sun.glass.ui.gtk.GtkPlatformFactory.createApplication(GtkPlatformFactory.java:41)
at com.sun.glass.ui.Application.run(Application.java:146)
at com.sun.javafx.tk.quantum.QuantumToolkit.startup(QuantumToolkit.java:278)
at com.sun.javafx.application.PlatformImpl.startup(PlatformImpl.java:211)
at javafx.embed.swing.JFXPanel.lambda$initFx$2(JFXPanel.java:232)
at java.lang.Thread.run(Thread.java:748)
Its a vm on vmware platform and have driver as below.
00:0f.0 VGA compatible controller: VMware SVGA II Adapter (prog-if 00 [VGA controller])
Subsystem: VMware SVGA II Adapter
Flags: bus master, medium devsel, latency 64, IRQ 16
I/O ports at 1070 [size=16]
Memory at e8000000 (32-bit, prefetchable) [size=128M]
Memory at fe000000 (32-bit, non-prefetchable) [size=8M]
[virtual] Expansion ROM at c0400000 [disabled] [size=32K]
Capabilities: [40] Vendor Specific Information: Len=00 <?>
Capabilities: [44] PCI Advanced Features
Kernel driver in use: vmwgfx
Kernel modules: vmwgfx
I have below libgl related packages installed on server.
# rpm -qa |grep -i libGL
libglvnd-1.0.1-0.8.git5baa1e5.el7.x86_64
libglvnd-opengl-1.0.1-0.8.git5baa1e5.el7.x86_64
mesa-libGL-18.3.4-10.el7.x86_64
mesa-libglapi-18.3.4-10.el7.x86_64
libglvnd-glx-1.0.1-0.8.git5baa1e5.el7.x86_64
libglvnd-egl-1.0.1-0.8.git5baa1e5.el7.x86_64
libglvnd-gles-1.0.1-0.8.git5baa1e5.el7.x86_64
libglvnd-core-devel-1.0.1-0.8.git5baa1e5.el7.x86_64
mesa-libGLU-devel-9.0.0-4.el7.x86_64
libglvnd-devel-1.0.1-0.8.git5baa1e5.el7.x86_64
mesa-libGL-devel-18.3.4-10.el7.x86_64
mesa-libGLU-9.0.0-4.el7.x86_64
Can you please let me know how to fix this issue?
Thanks,
The below line in /opt/SoapUI-5.5.0/bin/soapui.sh was uncommented to disable the browser component. After his, the SoapUI was no longer crashing as before.
#uncomment to disable browser component
JAVA_OPTS="$JAVA_OPTS -Dsoapui.browser.disabled=true"
Been digging on this for a while. I reviewed multiple articles on this issue. This one was the closest:
Tomcat 8 on CentOS 7 does not start as service (but it starts manually ....)
The difference being that I am running Tomcat 9.0.33. Here are the particulars:
java version "1.8.0_121"\
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)\
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)\
Tomcat 9.0.33
NAME="CentOS Linux"\
VERSION="7 (Core)"\
ID="centos"\
ID_LIKE="rhel fedora"\
VERSION_ID="7"\
PRETTY_NAME="CentOS Linux 7 (Core)"\
ANSI_COLOR="0;31"\
CPE_NAME="cpe:/o:centos:centos:7"\
HOME_URL="https://www.centos.org/"\
BUG_REPORT_URL="https://bugs.centos.org/"\
CENTOS_MANTISBT_PROJECT="CentOS-7"\
CENTOS_MANTISBT_PROJECT_VERSION="7"\
REDHAT_SUPPORT_PRODUCT="centos"\
REDHAT_SUPPORT_PRODUCT_VERSION="7"\
As a side note, everything was starting normally with no issues until recently. As far as I know there haven't been any major changes to the environment. But, when I ran the "systemctl restart" command recently, the startup began to fail. There are 5 instances of Tomcat 9.0.33 running at different ports and paths and those have not changed. I have not restarted two of the instance (afraid they won't start) the other three flat out won't start. Details below:
Systemd unit file for tomcat\
[Unit]\
Description=Apache Tomcat Web Application Container in Liferay 7.32 TEST for UAT\
After=syslog.target network.target
[Service]\
Type=forking
Environment=JAVA_HOME=/opt/jdk1.8.0_121/jre\
Environment=CATALINA_PID=/opt/liferay/uatapi/liferay-ce-portal-7.3.2-ga3/tomcat-9.0.33/temp/tomcat.pid\
Environment=CATALINA_HOME=/opt/liferay/uatapi/liferay-ce-portal-7.3.2-ga3/tomcat-9.0.33\
Environment=CATALINA_BASE=/opt/liferay/uatapi/liferay-ce-portal-7.3.2-ga3/tomcat-9.0.33\
Environment='CATALINA_OPTS=-Xms1024m -Xmx2048m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=20 -XX:ParallelGCThreads=8 -server -Xdebug -Xrunjdwp:transport=dt_socket,address=5000,server=y,suspend=n'\
Environment='JAVA_OPTS=-Djava.awt.headless=true -Djava.security.egd=file:/dev/./urandom -Duser.timezone=GMT -Dfile.encoding=UTF-8'
ExecStart=/opt/liferay/uatapi/liferay-ce-portal-7.3.2-ga3/tomcat-9.0.33/bin/startup.sh\
ExecStop=/bin/kill -15 $MAINPID
User=tomcat\
Group=tomcat\
UMask=0007
[Install]\
WantedBy=multi-user.target\
Results when running systemctl start liferayuat
● liferayuat.service - Apache Tomcat Web Application Container in Liferay 7.32 TEST for UAT\
Loaded: loaded (/etc/systemd/system/liferayuat.service; enabled; vendor preset: disabled)\
Active: failed (Result: exit-code) since Sat 2020-12-05 08:44:08 CST; 3s ago\
Process: 10891 ExecStop=/bin/kill -15 $MAINPID (code=exited, status=1/FAILURE)\
Process: 10851 ExecStart=/opt/liferay/uatapi/liferay-ce-portal-7.3.2-ga3/tomcat-9.0.33/bin/startup.sh \(code=exited, status=0/SUCCESS)\
Main PID: 10861 (code=exited, status=0/SUCCESS)
Dec 05 08:44:08 systemd[1]: Starting Apache Tomcat Web Application Container in Liferay 7.32 TEST for UAT...\
Dec 05 08:44:08 startup.sh[10851]: Existing PID file found during start.\
Dec 05 08:44:08 startup.sh[10851]: Removing/clearing stale PID file.\
Dec 05 08:44:08 startup.sh[10851]: Tomcat started.\
Dec 05 08:44:08 systemd[1]: Started Apache Tomcat Web Application Container in Liferay 7.32 TEST for UAT.\
Dec 05 08:44:08 systemd[1]: liferayuat.service: control process exited, code=exited status=1\
Dec 05 08:44:08 systemd[1]: Unit liferayuat.service entered failed state.\
Dec 05 08:44:08 systemd[1]: liferayuat.service failed.
Then the ONLY thing in catalina.out:
Listening for transport dt_socket at address: 5000\
java.lang.ClassNotFoundException: org.apache.catalina.startup.Catalina\
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)\
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)\
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)\
at org.apache.catalina.startup.Bootstrap.init(Bootstrap.java:261)\
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:443)\
So, when I start the instance with systemctl start it will fail. But if I run this command (as root...) then it will start:
/opt/liferay/uatapi/liferay-ce-portal-7.3.2-ga3/tomcat-9.0.33/bin/startup.sh
If I run that full commmand AS tomcat it doesn't start with the same error. So, it appears that the issue is permissions. The tomcat user and group are owners of all files and folders. But, somehow, the tomcat user either doesn't have permissions or the path gets jacked up so that the class files can't be found. I followed the suggestions in the article I referenced above but the changes had no impact.
I tripped across one article on SELINX that seemed to point to an issue there. This are the SELINUX settings:
SELinux status: enabled
SELinuxfs mount: /sys/fs/selinux
SELinux root directory: /etc/selinux
Loaded policy name: targeted
Current mode: permissive
Mode from config file: permissive
Policy MLS status: enabled
Policy deny_unknown status: allowed
Max kernel policy version: 31\
The workaround to keep the instances running is just to manually start them but what is causing systemctl start NOT to work? I suspect permissions but I sure as heck can't see why since everything is owned by tomcat:tomcat
So, this is self-inflicted as most "mysteries" are. I still cannot account for some of the differences I see when looking into SELinux contexts between the instances but the REAL cause was subtle (to me). Permissions on the {tomcat root}/lib and {tomcat root}/lib/ext had no execute permissions. That may have been due to a jar that was added recently and then needed to be updated by owner and permissions. In any case, the original issue resulted in many trial and error attempts to fix it which complicated matters further.
I discovered the solution by doing a folder by folder, file by file comparison between working and non-working instances. Apparently the new jar and the owner/permission changes were applied to all but the production version.
Thanks for the suggestions.
I have a java web project working fine on my local glassfish server. I need to deploy it to a remote glassfish server using netbeans so I setup the server on an AWS instance of ubuntu and successfully connected it with netbeans.
I am able to see the resources under the server in netbeans services pane but when am trying to deploy the project, its failing with following logs:
Running Log:
Deploying on GlassFish Server (AWS)
profile mode: false
debug mode: false
force redeploy: true
Distributing /xxxx/xxx/yyyy/target/yyyy-1.0.war
GlassFish Server (AWS), deploy, null, false
IDE Log:
INFO [glassfish]: Requested Entity: public id = -//GlassFish.org//DTD GlassFish Application Server 3.1 Resource Definitions//EN, system id = http://glassfish.org/dtds/glassfish-resources_1_5.dtd
INFO [null]: Last record repeated again.
INFO [org.netbeans.modules.glassfish.tooling.admin.RunnerHttpDeploy]: IO exception caught in handleSend() method:
java.io.IOException: Error writing request body to server
at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.checkError(HttpURLConnection.java:3518)
at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3501)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
at java.util.zip.DeflaterOutputStream.deflate(DeflaterOutputStream.java:253)
at java.util.zip.ZipOutputStream.closeEntry(ZipOutputStream.java:255)
at java.util.zip.ZipOutputStream.finish(ZipOutputStream.java:360)
at java.util.zip.DeflaterOutputStream.close(DeflaterOutputStream.java:238)
at java.util.zip.ZipOutputStream.close(ZipOutputStream.java:377)
[catch] at org.netbeans.modules.glassfish.tooling.admin.RunnerHttpDeploy.handleSend(RunnerHttpDeploy.java:267)
at org.netbeans.modules.glassfish.tooling.admin.Runner.handleHTTPConnection(Runner.java:828)
at org.netbeans.modules.glassfish.tooling.admin.Runner.call(Runner.java:939)
at org.netbeans.modules.glassfish.tooling.admin.Runner.call(Runner.java:73)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
INFO [org.netbeans.modules.j2ee.deployment.devmodules.api.Deployment]
The module has not been deployed.
See the server log for details.
[catch] at org.netbeans.modules.j2ee.deployment.devmodules.api.Deployment.deploy(Deployment.java:259)
at org.netbeans.modules.maven.j2ee.execution.DeploymentHelper.perform(DeploymentHelper.java:208)
at org.netbeans.modules.maven.j2ee.execution.ExecutionChecker.executionResult(ExecutionChecker.java:93)
at org.netbeans.modules.maven.execute.MavenCommandLineExecutor.run(MavenCommandLineExecutor.java:315)
at org.netbeans.core.execution.RunClassThread.run(RunClassThread.java:153)
Server Log:
-- nothing --
The IDE is telling me to check the server logs for more details but server does not have any logs (I cleared the logfile before deploying the project, so there is no server stating info but it is running fine as I am able to access it from the browser and cmdline).
One more things, though the version of both the servers are same, the remote server only have web profile installed whereas the local is the installation done along with netbeans.
The config details are:
Project - Maven web app
Netbeans 8.2
Glassfish 4.1.1
Remote OS: Ubuntu 16 LTS
Local OS: Fedora 25
Please ask for any more info required.
Thanks.
i had the /var disk full on my debian server and neo4j stopped as expected. I freed up the space on the disk but the neo4j server does not start throwing the error given below in the logs. I don't have any java or neo4j process running on the server that i may kill.
Things were stable with my neo4j setup for the past 6 months with around 1000 nodes. I am little novice on the java side so please let me know if i have missed out on anything basic.
Output on command line: service neo4j-service restart
Restarting Neo4j Graph Database: neo4jWARNING: Max 1024 open files
allowed, minimum of 40 000 recommended. See the Neo4j manual. WARNING!
You are using an unsupported Java runtime.
* Please use Oracle(R) Java(TM) 7 to run Neo4j Server. Download "Java Platform (JDK) 7" from:
http://www.oracle.com/technetwork/java/javase/downloads/index.html
* Please see http://docs.neo4j.org/ for Neo4j Server installation instructions. Using additional JVM arguments: -server
-XX:+DisableExplicitGC -Dorg.neo4j.server.properties=conf/neo4j-server.properties -Djava.util.logging.config.file=conf/logging.properties -Dlog4j.configuration=file:conf/log4j.properties -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled Starting Neo4j Server...WARNING: not changing user process [4733]... waiting for
server to be ready..... Failed to start within 120 seconds. Neo4j
Server may have failed to start, please check the logs. failed!
Logfile:
Sep 26, 2014 3:31:38 PM org.neo4j.server.logging.Logger log
SEVERE: Failed to start Neo Server on port [7474]
Sep 26, 2014 3:33:00 PM org.neo4j.server.logging.Logger log
WARNING: You are using an unsupported Java runtime. Please use Oracle(R) Java(TM) Runtime Environment 7.
Sep 26, 2014 3:33:00 PM org.neo4j.server.logging.Logger log
INFO: Setting startup timeout to: 120000ms based on -1
Sep 26, 2014 3:33:03 PM org.neo4j.server.logging.Logger log
SEVERE:
org.neo4j.server.ServerStartupException: Starting Neo4j Server failed: Error starting org.neo4j.kernel.EmbeddedGraphDatabase, /var/lib/neo4j/data/graph.db
at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:218)
at org.neo4j.server.Bootstrapper.start(Bootstrapper.java:87)
at org.neo4j.server.Bootstrapper.main(Bootstrapper.java:50)
Caused by: java.lang.RuntimeException: Error starting org.neo4j.kernel.EmbeddedGraphDatabase, /var/lib/neo4j/data/graph.db
at org.neo4j.kernel.InternalAbstractGraphDatabase.run(InternalAbstractGraphDatabase.java:330)
at org.neo4j.kernel.EmbeddedGraphDatabase.<init>(EmbeddedGraphDatabase.java:63)
at org.neo4j.graphdb.factory.GraphDatabaseFactory$1.newDatabase(GraphDatabaseFactory.java:92)
at org.neo4j.graphdb.factory.GraphDatabaseBuilder.newGraphDatabase(GraphDatabaseBuilder.java:198)
at org.neo4j.kernel.impl.recovery.StoreRecoverer.recover(StoreRecoverer.java:115)
at org.neo4j.server.preflight.PerformRecoveryIfNecessary.run(PerformRecoveryIfNecessary.java:59)
at org.neo4j.server.preflight.PreFlightTasks.run(PreFlightTasks.java:70)
at org.neo4j.server.AbstractNeoServer.runPreflightTasks(AbstractNeoServer.java:333)
at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:152)
... 2 more
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.impl.transaction.TxManager#61615142' was successfully initialized, but failed to start. Please see attached cause exception.
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:509)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:115)
at org.neo4j.kernel.InternalAbstractGraphDatabase.run(InternalAbstractGraphDatabase.java:307)
... 10 more
Caused by: org.neo4j.graphdb.TransactionFailureException: Unable to start TM
at org.neo4j.kernel.impl.transaction.TxManager.openLog(TxManager.java:824)
at org.neo4j.kernel.impl.transaction.TxManager.start(TxManager.java:198)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:503)
... 12 more
Caused by: java.io.IOException: Branch[����] found for [GlobalId[NENEOK|5930761323375553953|5569093129887285248|13248], BranchId[ ]] but no record list found in map
at org.neo4j.kernel.impl.transaction.TxLog.readBranchAddRecordInto(TxLog.java:520)
at org.neo4j.kernel.impl.transaction.TxLog.getDanglingRecords(TxLog.java:440)
at org.neo4j.kernel.impl.transaction.TxLog.recreateActiveTransactionState(TxLog.java:133)
at org.neo4j.kernel.impl.transaction.TxLog.<init>(TxLog.java:128)
at org.neo4j.kernel.impl.transaction.TxManager.openLog(TxManager.java:796)
... 14 more
Sep 26, 2014 3:33:03 PM org.neo4j.server.logging.Logger log
SEVERE: Failed to start Neo Server on port [7474]
I am using an openjdk 1.7 runtime environment which i think i should upgrade but i don't understand what is the cause of the error since everything was working fine before. Thanks for any help!
You should never ever manually modify anything inside the graph.db directory unless you're 100% sure what you're doing.
To prevent the datastore directory from continuously growing check your setting for keep_logical_logs in neo4j.properties, see http://docs.neo4j.org/chunked/stable/configuration-logical-logs.html.
Unless you require logical logs for online backup or cluster synchronization, you might be save deleting nioneo_logical.log.v* files. Make sure to have a backup first!
Also check your settings for open file limits, http://docs.neo4j.org/chunked/stable/linux-performance-guide.html#_setting_the_number_of_open_files.
There used to be a outdated version of Neo4j having a bug that might corrupt your datastore when running out of disc space. If that has happened to you either need to manually fix it on a binary level (which requires a lot of knowledge on Neo4j internals) or restore a previous backup from a time before you run out of disc.