I have a 3 cassandra nodes (db-stats-01, db-stats-02, db-stats-03), which are all Up and Normal according to nodetool status.
The client is essentially doing this:
Builder builder = Cluster.builder();
builder.addContactPoint(node);
context = getSSLContext();
String[] cipherSuites = SSLOptions.DEFAULT_SSL_CIPHER_SUITES;
builder.withSSL(new SSLOptions(context, cipherSuites)).build();
and failing on the last line with the following exception:
Sep 12 14:31:26 localhost daemon: Defuncting connection to db-stats-01/192.168.105.1
Sep 12 14:31:26 localhost com.datastax.driver.core.ConnectionException: [db-stats-01/192.168.105.1] Unexpected error during transport initialization (com.datastax.driver.core.ConnectionException: [db-stats-01/192.168.105.1] Operation Timeouted)
Sep 12 14:31:26 localhost at com.datastax.driver.core.Connection.initializeTransport(Connection.java:182)
Sep 12 14:31:26 localhost at com.datastax.driver.core.Connection.<init>(Connection.java:132)
Sep 12 14:31:26 localhost at com.datastax.driver.core.Connection.<init>(Connection.java:60)
Sep 12 14:31:26 localhost at com.datastax.driver.core.Connection$Factory.open(Connection.java:419)
Sep 12 14:31:26 localhost at com.datastax.driver.core.ControlConnection.tryConnect(ControlConnection.java:205)
Sep 12 14:31:26 localhost at com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:168)
Sep 12 14:31:26 localhost at com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:81)
Sep 12 14:31:26 localhost at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:794)
Sep 12 14:31:26 localhost at com.datastax.driver.core.Cluster$Manager.access$100(Cluster.java:721)
Sep 12 14:31:26 localhost at com.datastax.driver.core.Cluster.<init>(Cluster.java:82)
Sep 12 14:31:26 localhost at com.datastax.driver.core.Cluster.<init>(Cluster.java:67)
Sometimes, but not everytime, I get the following error on the server side:
DEBUG [New I/O worker #1] 2014-09-12 14:36:14,337 Slf4JLogger.java (line 36) SSLEngine.closeInbound() raised an exception after a handshake failure.
javax.net.ssl.SSLException: Inbound closed before receiving peer's close_notify: possible truncation attack?
at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1619)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1587)
at sun.security.ssl.SSLEngineImpl.closeInbound(SSLEngineImpl.java:1517)
at org.jboss.netty.handler.ssl.SslHandler.setHandshakeFailure(SslHandler.java:1407)
at org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1293)
at org.jboss.netty.handler.ssl.SslHandler.decode(SslHandler.java:913)
at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425)
at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
DEBUG [New I/O worker #1] 2014-09-12 14:36:14,342 Slf4JLogger.java (line 36) Swallowing an exception raised while writing non-app data
java.nio.channels.ClosedChannelException
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:434)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:129)
at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:99)
at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:36)
at org.jboss.netty.channel.Channels.write(Channels.java:725)
at org.jboss.netty.channel.Channels.write(Channels.java:686)
at org.jboss.netty.handler.ssl.SslHandler.wrapNonAppData(SslHandler.java:1153)
at org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1246)
at org.jboss.netty.handler.ssl.SslHandler.channelDisconnected(SslHandler.java:656)
at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:396)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:361)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
However, most of the time I don't even see this error on the server, dispite the server being set to DEBUG.
I don't suspect a connectivity issue, because I have watched tcpdump and I see packets going to and from the client and server. Also, there is briefly an ESTABLISHED connection up on port 9042 when I look at netstat -anp.
Using telnet, I can verify that the server is listening on 9042, and when I send junk through telnet I get a nice error on the server side complaining about a non-SSL record:
ERROR [Native-Transport-Requests:85] 2014-09-12 14:40:23,747 ErrorMessage.java (line 222) Unexpected exception during request
org.jboss.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 640d0a64660d0a
Not sure what else to check or where I may have gone wrong. Before implementing SSL in my client, I didn't see this issue at all, so my thought is it is probably related. Other misc info:
[root#db-stats-01 ~]# rpm -qa | grep cassandra
cassandra20-2.0.9-1.noarch
[root#db-stats-01 ~]# java -version
java version "1.7.0_67"
Java(TM) SE Runtime Environment (build 1.7.0_67-b01)
Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)
The SSL setup was wrong in some way. I regenerated all my key and trust stores according to http://techdocs.acunu.com.s3.amazonaws.com/v5.0/admin/security/ssl.html and everything worked as expected.
Related
I attempted to add the nvm-wrapper plugin (https://plugins.jenkins.io/nvm-wrapper/) to jenkins and reboot and am no longer able to reach the site (This site can’t be reached). It is deployed as an azure VM. The VM is running, I can ssh in and systemctl status jenkins.service is showing:
jenkins.service - LSB: Start Jenkins at boot time
Loaded: loaded (/etc/init.d/jenkins; generated)
Active: failed (Result: exit-code) since Mon 2022-04-04 21:04:43 UTC; 14h ago
Docs: man:systemd-sysv-generator(8)
Process: 2530 ExecStart=/etc/init.d/jenkins start (code=exited, status=7)
Apr 04 21:04:42 pipeline jenkins[2530]: Correct java version found
Apr 04 21:04:42 pipeline jenkins[2530]: * Starting Jenkins Automation Server jenkins
Apr 04 21:04:42 pipeline su[2587]: Successful su for jenkins by root
Apr 04 21:04:42 pipeline su[2587]: + ??? root:jenkins
Apr 04 21:04:42 pipeline su[2587]: pam_unix(su:session): session opened for user jenkins by (uid=0)
Apr 04 21:04:42 pipeline su[2587]: pam_unix(su:session): session closed for user jenkins
Apr 04 21:04:43 pipeline jenkins[2530]: ...fail!
Apr 04 21:04:43 pipeline systemd[1]: jenkins.service: Control process exited, code=exited status=7
Apr 04 21:04:43 pipeline systemd[1]: jenkins.service: Failed with result 'exit-code'.
Apr 04 21:04:43 pipeline systemd[1]: Failed to start LSB: Start Jenkins at boot time.
I had previously been running into Incorrect Java version due to having installed Java 12, so I modified the /etc/init.d/jenkins like so:
# Which Java versions can be used to run Jenkins
JAVA_ALLOWED_VERSIONS=( "18" "110" "120" )
# Work out the JAVA version we are working with:
JAVA_VERSION=$($JAVA -version 2>&1 | sed -n ';s/.* version "\([0-9]*\)\.\([0-9]*\)\..*".*/\1\2/p;')
to allow for Java 12. Any thoughts on how/why either the plugin addition or Java version and/or /etc/init.d/jenkins edit could be impacting things? My sense is that the initial reboot failed due to the Incorrect Java version issue, but not sure how I can resolve things and get it back up and running. It should, by default, be available at 8080 and that is where I am seeing This site can’t be reached.
I also have the networking set up like so:
and port 8080 should allow traffic. I have attempted to restart, hard start and stop/start to no avail as well. "Resource health" says the VM is available which should be obvious since it is running and I can ssh in. Do I need to redeploy perhaps?
First, please delegate the service to a non root user.
Second, azure, means in the net, please use SSL, the times without SSL are gone.
JAVA_ALLOWED_VERSIONS=( "18" "110" "120" )
you are trying java 12, but its configured java 120, i think you should fix that to
JAVA_ALLOWED_VERSIONS=( "18" "11" "12" )
For the past 2 months I run tomcat7 perfectly on my ubuntu server. Today tomcat7 failed to start.
I started tomcat 7
sudo systemctl start tomcat 7
Job for tomcat7.service failed because the control process exited with error code.
See "systemctl status tomcat7.service" and "journalctl -xe" for details.
Then I checked for the status
sudo systemctl status tomcat7
● tomcat7.service - LSB: Start Tomcat.
Loaded: loaded (/etc/init.d/tomcat7; generated)
Active: failed (Result: exit-code) since Sat 2020-05-16 15:47:00 HKT; 8min ago
Docs: man:systemd-sysv-generator(8)
Process: 6043 ExecStart=/etc/init.d/tomcat7 start (code=exited, status=1/FAILURE
May 16 15:46:55 www.example.com systemd[1]: Starting LSB: Start Tomcat....
May 16 15:46:55 www.example.com tomcat7[6043]: * Starting Tomcat servlet engine
May 16 15:47:00 www.example.com tomcat7[6043]: ...fail!
May 16 15:47:00 www.example.com systemd[1]: tomcat7.service: Control process exit
May 16 15:47:00 www.example.com systemd[1]: tomcat7.service: Failed with result '
May 16 15:47:00 www.example.com systemd[1]: Failed to start LSB: Start Tomcat..
Then, I checked for the log error and I have no idea how to fix this
sudo tail /etc/var/log/tomcat7/catalina.out
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
-Djava.endorsed.dirs=/usr/share/tomcat7/endorsed is not supported. Endorsed standards and standalone APIs
in modular form will be supported via the concept of upgradeable modules.
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
-Djava.endorsed.dirs=/usr/share/tomcat7/endorsed is not supported. Endorsed standards and standalone APIs
in modular form will be supported via the concept of upgradeable modules.
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
Somebody please help ....
(Updated as of 8 May 2018)
This is about an issue experienced during the connection phase when deploying EAR to WebSphere Application Server in Jenkins using the "WebSphere Deployer Plugin".
System information:
[Jenkins]
Jenkins version: 2.107.2
java.runtime.version (as seen from Jenkins Admin UI): 1.8.0_144-b01
[Jenkins - WebSphere Deployer Plugin]
Version: 1.6.1
Copied the relevant JAR files (from C:\IBM\WebSphere\AppServer\runtimes and C:\IBM\WebSphere\AppServer\plugins) to C:\Program Files (x86)\Jenkins\plugins\websphere-deployer\WEB-INF\lib.
Cell/Node/Server info is specified in the plugin configuration (same as what is shown in "Show Available Targets").
[IBM WebSphere Application Server]
WAS version: 9.0.0.7
Java version: 1.8.0_151
Administrative security: Enabled
Application security: Disabled
SOAP connection port: 8880 (checked to be not being blocked by firewall)
Telnet to port 8880 is successful
Command line login to wsadmin via port 8880 is successful
The "Test Connection" is successful. Unfortunately, in the actual job run the deploy phase still fails at connection.
The relevant log information is as follows
[Jenkins console log / job log (displayed in the Jenkins Admin UI "Console Output")]
12:48:31 Connecting to IBM WebSphere Application Server... 12:48:31
The following artifacts will be deployed in this order... 12:48:31
------------------------------------------- 12:48:36 [EAR file path]
Last modified on May 08, 2018 12:47:56 12:48:36
------------------------------------------- 12:48:59 Artifact is being deployed to virtual host: default_host 12:49:08 Error deploying to IBM
WebSphere Application Server: Could not determine if artifact
'[App name]' is installed: AdminException: 12:49:08 Performing
rollback of '[App name]' 12:49:08 WARNING: Artifact doesn't exist
rollback repository 12:49:08 Build step 'Deploy To IBM WebSphere
Application Server' changed build result to FAILURE 12:49:12 Finished:
FAILURE
[Jenkins error log (C:\Program Files (x86)\Jenkins\jenkins.err.log)]
May 08, 2018 12:43:47 PM hudson.WebAppMain$3 run INFO: Jenkins is
fully up and running May 08, 2018 12:43:53 PM
javax.jmdns.impl.DNSStatefulObject$DefaultImplementation
waitForAnnounced WARNING: Wait for announced timed out: DNS:
[Host name]. state: announcing 2 task: Announcer([Host name])
state: announcing 2 May 08, 2018 12:45:52 PM
com.ibm.websphere.management.AdminClientFactory WARNING: ADMC0046W:
The system failed to load properties from com.ibm.SOAP.configURL:
file:null/properties/soap.client.props May 08, 2018 12:45:54 PM
com.ibm.ws.ssl.config.SSLConfigManager INFO: CWPKI0051I: The process
has the java security property jdk.certpath.disabledAlgorithms set to
[MD2, MD5, SHA1 jdkCA & usage TLSServer, RSA keySize < 1024, DSA
keySize < 1024, EC keySize < 224]. The WebSphere Application server
is setting the java security property jdk.certpath.disabledAlgorithms
to [MD2, RSA keySize < 1024, MD5]. May 08, 2018 12:45:54 PM
com.ibm.ws.ssl.config.SSLConfigManager INFO: CWPKI0051I: The process
has the java security property jdk.tls.disabledAlgorithms set to
[SSLv3, RC4, MD5withRSA, DH keySize < 768, EC keySize < 224]. The
WebSphere Application server is setting the java security property
jdk.tls.disabledAlgorithms to [SSLv3, RC4, DH keySize < 768,
MD5withRSA]. May 08, 2018 12:45:54 PM
com.ibm.ws.ssl.config.SSLConfigManager INFO: CWPKI0027I: Disabling
default hostname verification for HTTPS URL connections. May 08, 2018
12:45:54 PM com.ibm.ws.security.config.SecurityObjectLocator INFO:
CWSCF0002I: The client code is attempting to load the security
configuration the server and this operation is not allowed. May 08,
2018 12:45:56 PM com.ibm.ws.security.config.SecurityObjectLocator
INFO: CWSCF0002I: The client code is attempting to load the security
configuration the server and this operation is not allowed. May 08,
2018 12:46:06 PM com.ibm.websphere.management.AdminClientFactory
WARNING: ADMC0046W: The system failed to load properties from
com.ibm.SOAP.configURL: file:null/properties/soap.client.props May 08,
2018 12:46:32 PM org.jenkinsci.plugins.github.webhook.WebhookManager$1
run INFO: GitHub webhooks activated for job [Jenkins Job name] with
[GitHubRepositoryName[host=github.com,username=[GitHub user name],repository=[GitHub repository]]]
(events: [PUSH]) May 08, 2018 12:46:37 PM
com.squareup.okhttp.internal.Platform$JdkWithJettyBootPlatform
getSelectedProtocol INFO: ALPN callback dropped: SPDY and HTTP/2 are
disabled. Is alpn-boot on the boot class path? May 08, 2018 12:48:31
PM hudson.model.Run execute INFO: [Jenkins Job name] #38 main build action completed: SUCCESS May
08, 2018 12:48:31 PM com.ibm.websphere.management.AdminClientFactory
WARNING: ADMC0046W: The system failed to load properties from
com.ibm.SOAP.configURL: file:null/properties/soap.client.props May 08,
2018 12:48:50 PM
com.squareup.okhttp.internal.Platform$JdkWithJettyBootPlatform
getSelectedProtocol INFO: ALPN callback dropped: SPDY and HTTP/2 are
disabled. Is alpn-boot on the boot class path?
com.ibm.websphere.management.exception.AdminException: at
com.ibm.websphere.management.application.AppManagementProxy.proxyInvoke(AppManagementProxy.java:192)
at
com.ibm.websphere.management.application.AppManagementProxy.checkIfAppExists(AppManagementProxy.java:269)
at
org.jenkinsci.plugins.websphere.services.deployment.WebSphereDeploymentService.isArtifactInstalled(WebSphereDeploymentService.java:468)
at
org.jenkinsci.plugins.websphere_deployer.WebSphereDeployerPlugin.stopArtifact(WebSphereDeployerPlugin.java:409)
at
org.jenkinsci.plugins.websphere_deployer.WebSphereDeployerPlugin.perform(WebSphereDeployerPlugin.java:245)
at hudson.tasks.BuildStepMonitor$3.perform(BuildStepMonitor.java:45)
at
hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
at
hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
at
hudson.maven.MavenModuleSetBuild$MavenModuleSetBuildExecution.post2(MavenModuleSetBuild.java:1073)
at
hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
at hudson.model.Run.execute(Run.java:1752) at
hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:543) at
hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429) Caused by:
java.lang.reflect.UndeclaredThrowableException at
com.sun.proxy.$Proxy67.invoke(Unknown Source) at
com.ibm.ws.management.AdminClientImpl.invoke(AdminClientImpl.java:224)
at
com.ibm.websphere.management.application.AppManagementProxy.proxyInvoke(AppManagementProxy.java:186)
... 13 more Caused by: [SOAPException:
faultCode=SOAP-ENV:ServerException; msg=The Soap RPC call can't
be unmarshalled.] at
com.ibm.ws.management.connector.soap.SOAPConnectorClient.handleAdminFault(SOAPConnectorClient.java:959)
at
com.ibm.ws.management.connector.soap.SOAPConnectorClient.invokeTemplateOnce(SOAPConnectorClient.java:924)
at
com.ibm.ws.management.connector.soap.SOAPConnectorClient.invokeTemplate(SOAPConnectorClient.java:689)
at
com.ibm.ws.management.connector.soap.SOAPConnectorClient.invokeTemplate(SOAPConnectorClient.java:679)
at
com.ibm.ws.management.connector.soap.SOAPConnectorClient.invoke(SOAPConnectorClient.java:665)
at
com.ibm.ws.management.connector.soap.SOAPConnectorClient.invoke(SOAPConnectorClient.java:487)
... 16 more
[WAS console log (SystemOut.log) (C:\IBM\WebSphere\AppServer\profiles\AppSrv01\logs\server1\SystemOut.log)]
[5/8/18 12:49:00:183 CST] 000000f7 LTPAServerObj E SECJ0369E:
Authentication failed when using LTPA. The exception is
com.ibm.websphere.wim.exception.PasswordCheckFailedException:
CWWIM4513E The password match failed for the 'WASadmin' principal
name.. [5/8/18 12:49:00:332 CST] 000000f6 LTPAServerObj E SECJ0369E:
Authentication failed when using LTPA. The exception is
com.ibm.websphere.wim.exception.PasswordCheckFailedException:
CWWIM4513E The password match failed for the 'WASadmin' principal
name.. [5/8/18 12:49:04:291 CST] 000000f7 LTPAServerObj E SECJ0369E:
Authentication failed when using LTPA. The exception is
com.ibm.websphere.wim.exception.PasswordCheckFailedException:
CWWIM4513E The password match failed for the 'WASadmin' principal
name.. [5/8/18 12:49:07:221 CST] 000000f7 FFDCJanitor I
com.ibm.ws.ffdc.impl.FFDCJanitor doCleanupIfNeeded FFDC0004I: FFDC log
file management removed 20 of 20 files that have reached their
configured maximum age [5/8/18 12:49:07:486 CST] 000000f7 FfdcProvider
W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC
Incident emitted on
C:\IBM\WebSphere\AppServer\profiles\AppSrv01\logs\ffdc\server1_f7987cd2_18.05.08_12.49.07.2212966399582449208005.txt
Unable to authenticate incoming request 534 [5/8/18 12:49:07:489 CST]
000000f7 FfdcProvider W com.ibm.ws.ffdc.impl.FfdcProvider logIncident
FFDC1003I: FFDC Incident emitted on
C:\IBM\WebSphere\AppServer\profiles\AppSrv01\logs\ffdc\server1_f7987cd2_18.05.08_12.49.07.4884954440341702141044.txt
com.ibm.ws.management.connector.soap.SOAPUtils.extractSoapCall 157
[5/8/18 12:49:07:716 CST] 000000f7 FfdcProvider W
com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident
emitted on
C:\IBM\WebSphere\AppServer\profiles\AppSrv01\logs\ffdc\server1_f7987cd2_18.05.08_12.49.07.4909017290418303421157.txt
com.ibm.ws.management.connector.soap.SOAPConnector.service 212
Just released v1.5.6, please try upgrading to this version. You won't have to deal with certificates anymore if you select the following option.
I started a jstatd on the remote server (Ubuntu Server 14.04):
jstatd -J-Djava.security.policy=.jstatd.all.policy -J-Djava.rmi.server.logCalltrue -p 9099
and try to connect to it with jvisualvm on windows. I checked netstat, the connection is established, and on the remote it logs the call:
Sep 11, 2015 12:48:51 PM sun.rmi.server.UnicastServerRef logCall
FINER: RMI TCP Connection(4)-10.82.199.0: [10.82.199.0: sun.rmi.registry.RegistryImpl[0:0:0, 0]: java.rmi.Remote lookup(java.lang.String)]
Sep 11, 2015 12:48:55 PM sun.rmi.server.UnicastServerRef logCall
FINER: RMI TCP Connection(4)-10.82.199.0: [10.82.199.0: sun.rmi.registry.RegistryImpl[0:0:0, 0]: java.rmi.Remote lookup(java.lang.String)]
Sep 11, 2015 12:48:59 PM sun.rmi.server.UnicastServerRef logCall
FINER: RMI TCP Connection(4)-10.82.199.0: [10.82.199.0: sun.rmi.registry.RegistryImpl[0:0:0, 0]: java.rmi.Remote lookup(java.lang.String)]
All signs are saying that it's working. but however no applications is showing in jvisualvm:
Apparently VisualVM expects a consistent DNS name for the server you're trying to connect to remotely (the Ubuntu Server 14.04 in your case). Hence, if you're specifying an IP address instead of a DNS name to VisualVM you should add the following to your jstatd startup line:
-J-Djava.rmi.server.hostname=<the IP address to your Ubuntu server here>
Additionally, I found out that specifying the port option (-p 9099 in your case) is not supported in some VisualVM releases:
Known limitation: In this VisualVM release the jstatd's default port and rminame must be used when starting the jstatd utility, i.e. the use of the -p and -n options is not supported.
VisualVM Troubleshooting Guide
All in all, you should try running the following jstatd line on your Ubuntu Server:
jstatd -J-Djava.security.policy=.jstatd.all.policy -J-Djava.rmi.server.hostname=10.82.83.117 -J-Djava.rmi.server.logCalltrue
Sources:
http://www.catify.com/2012/09/26/remote-monitoring-with-visualvm/
It worked for me :)
jstatd -p 1099 -J-Djava.rmi.sver.hostname=10.250.105.112 -J-Djava.security.policy=<(echo 'grant codebase "file:${java.home}/../lib/tools.jar" {permission java.security.AllPermission;};')
Works for Me Perfectly
In case this helps someone else...
I was running into problems where neither jstatd nor adding a plain JMX connection in VisualVM worked. The former would not give any error messages, it just wouldn't list any apps. The latter would give me an error saying "Cannot connect to some-server:30648 using service:jmx:rmi:///jndi/rmi://some-server:30648/jmxrmi.
Trying to use the excellent sjk-plus tool to manually connect to the JMX service gave the following error:
$ java --add-opens java.base/jdk.internal.perf=ALL-UNNAMED \
--add-opens jdk.attach/sun.tools.attach=ALL-UNNAMED \
-Dsjk.breakCage=false \
-jar scripts/sjk-plus-0.14.jar
mx --get --allMatched -b com.acme.some.package:name=* -f Count \
-s some-server:30648
JMX Connection failed: java.rmi.ConnectException: Connection refused to host: 127.0.1.1; nested exception is:
java.net.ConnectException: Connection refused (Connection refused)
Do you see it? 127.0.1.1, what is that weird IP address doing there?
This was caused by a particular entry in the /etc/hosts file on the server:
user#some-server:~$ cat /etc/hosts
127.0.0.1 localhost
127.0.1.1 some-server
# The following lines are desirable for IPv6 capable hosts
::1 localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
Changing the some-server entry in the hosts file and restarting the process made it work with sjk-plus, and made it discoverable with jstatd as well.
in java 11+
jstatd -J-Djava.rmi.server.logCalls=true \
-J-Djava.security.policy=.jstatd.all.policy \
-J-Djava.net.preferIPv4Stack=true \
-J-Djava.security.policy=<(echo 'grant codebase "jrt:/jdk.jstatd" {permission java.security.AllPermission;}; grant codebase "jrt:/jdk.internal.jvmstat" {permission java.security.AllPermission;};')
i had the /var disk full on my debian server and neo4j stopped as expected. I freed up the space on the disk but the neo4j server does not start throwing the error given below in the logs. I don't have any java or neo4j process running on the server that i may kill.
Things were stable with my neo4j setup for the past 6 months with around 1000 nodes. I am little novice on the java side so please let me know if i have missed out on anything basic.
Output on command line: service neo4j-service restart
Restarting Neo4j Graph Database: neo4jWARNING: Max 1024 open files
allowed, minimum of 40 000 recommended. See the Neo4j manual. WARNING!
You are using an unsupported Java runtime.
* Please use Oracle(R) Java(TM) 7 to run Neo4j Server. Download "Java Platform (JDK) 7" from:
http://www.oracle.com/technetwork/java/javase/downloads/index.html
* Please see http://docs.neo4j.org/ for Neo4j Server installation instructions. Using additional JVM arguments: -server
-XX:+DisableExplicitGC -Dorg.neo4j.server.properties=conf/neo4j-server.properties -Djava.util.logging.config.file=conf/logging.properties -Dlog4j.configuration=file:conf/log4j.properties -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled Starting Neo4j Server...WARNING: not changing user process [4733]... waiting for
server to be ready..... Failed to start within 120 seconds. Neo4j
Server may have failed to start, please check the logs. failed!
Logfile:
Sep 26, 2014 3:31:38 PM org.neo4j.server.logging.Logger log
SEVERE: Failed to start Neo Server on port [7474]
Sep 26, 2014 3:33:00 PM org.neo4j.server.logging.Logger log
WARNING: You are using an unsupported Java runtime. Please use Oracle(R) Java(TM) Runtime Environment 7.
Sep 26, 2014 3:33:00 PM org.neo4j.server.logging.Logger log
INFO: Setting startup timeout to: 120000ms based on -1
Sep 26, 2014 3:33:03 PM org.neo4j.server.logging.Logger log
SEVERE:
org.neo4j.server.ServerStartupException: Starting Neo4j Server failed: Error starting org.neo4j.kernel.EmbeddedGraphDatabase, /var/lib/neo4j/data/graph.db
at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:218)
at org.neo4j.server.Bootstrapper.start(Bootstrapper.java:87)
at org.neo4j.server.Bootstrapper.main(Bootstrapper.java:50)
Caused by: java.lang.RuntimeException: Error starting org.neo4j.kernel.EmbeddedGraphDatabase, /var/lib/neo4j/data/graph.db
at org.neo4j.kernel.InternalAbstractGraphDatabase.run(InternalAbstractGraphDatabase.java:330)
at org.neo4j.kernel.EmbeddedGraphDatabase.<init>(EmbeddedGraphDatabase.java:63)
at org.neo4j.graphdb.factory.GraphDatabaseFactory$1.newDatabase(GraphDatabaseFactory.java:92)
at org.neo4j.graphdb.factory.GraphDatabaseBuilder.newGraphDatabase(GraphDatabaseBuilder.java:198)
at org.neo4j.kernel.impl.recovery.StoreRecoverer.recover(StoreRecoverer.java:115)
at org.neo4j.server.preflight.PerformRecoveryIfNecessary.run(PerformRecoveryIfNecessary.java:59)
at org.neo4j.server.preflight.PreFlightTasks.run(PreFlightTasks.java:70)
at org.neo4j.server.AbstractNeoServer.runPreflightTasks(AbstractNeoServer.java:333)
at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:152)
... 2 more
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.impl.transaction.TxManager#61615142' was successfully initialized, but failed to start. Please see attached cause exception.
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:509)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:115)
at org.neo4j.kernel.InternalAbstractGraphDatabase.run(InternalAbstractGraphDatabase.java:307)
... 10 more
Caused by: org.neo4j.graphdb.TransactionFailureException: Unable to start TM
at org.neo4j.kernel.impl.transaction.TxManager.openLog(TxManager.java:824)
at org.neo4j.kernel.impl.transaction.TxManager.start(TxManager.java:198)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:503)
... 12 more
Caused by: java.io.IOException: Branch[����] found for [GlobalId[NENEOK|5930761323375553953|5569093129887285248|13248], BranchId[ ]] but no record list found in map
at org.neo4j.kernel.impl.transaction.TxLog.readBranchAddRecordInto(TxLog.java:520)
at org.neo4j.kernel.impl.transaction.TxLog.getDanglingRecords(TxLog.java:440)
at org.neo4j.kernel.impl.transaction.TxLog.recreateActiveTransactionState(TxLog.java:133)
at org.neo4j.kernel.impl.transaction.TxLog.<init>(TxLog.java:128)
at org.neo4j.kernel.impl.transaction.TxManager.openLog(TxManager.java:796)
... 14 more
Sep 26, 2014 3:33:03 PM org.neo4j.server.logging.Logger log
SEVERE: Failed to start Neo Server on port [7474]
I am using an openjdk 1.7 runtime environment which i think i should upgrade but i don't understand what is the cause of the error since everything was working fine before. Thanks for any help!
You should never ever manually modify anything inside the graph.db directory unless you're 100% sure what you're doing.
To prevent the datastore directory from continuously growing check your setting for keep_logical_logs in neo4j.properties, see http://docs.neo4j.org/chunked/stable/configuration-logical-logs.html.
Unless you require logical logs for online backup or cluster synchronization, you might be save deleting nioneo_logical.log.v* files. Make sure to have a backup first!
Also check your settings for open file limits, http://docs.neo4j.org/chunked/stable/linux-performance-guide.html#_setting_the_number_of_open_files.
There used to be a outdated version of Neo4j having a bug that might corrupt your datastore when running out of disc space. If that has happened to you either need to manually fix it on a binary level (which requires a lot of knowledge on Neo4j internals) or restore a previous backup from a time before you run out of disc.