Develop Reference

Deploying war to wildfly 19

I am trying to deploy a war file to Wildfly 19 but getting a communications failure error. I have been trying to figure this out for a few days without any success. Google search did not yield anything meaningful :-(. I tried some of the suggestions such as changing the address from localhost to 127.0.0.1, etc. but did not work. Below is the error I am getting.
I have wildfly 19 on Synology (Diskstation) with MariaDB 10. I have a DataSource configured called PMProdDS with the connection URL: jdbc:mysql://localhost:3307/pmprod?useUnicode=true&useJDBCCompliantTimezoneShift=true&useLegacyDatetimeCode=false&serverTimezone=America/Chicago
Thank you for your help!!!
Here is my persistence.xml file:
<?xml version="1.0" encoding="UTF-8"?>
<persistence version="2.2"
xmlns="http://xmlns.jcp.org/xml/ns/persistence"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://xmlns.jcp.org/xml/ns/persistence http://xmlns.jcp.org/xml/ns/persistence/persistence_2_2.xsd">
<persistence-unit name="portfoliomanager"
transaction-type="JTA">
<jta-data-source>java:/PMProdDS</jta-data-source>
<!-- <provider>org.hibernate.jpa.HibernatePersistenceProvider</provider> -->
<properties>
<property name="hibernate.show_sql" value="false" />
<property name="hibernate.dialect"
value="org.hibernate.dialect.MySQL8Dialect" />
<property
name="javax.persistence.schema-generation.database.action"
value="update" />
<property name="hibernate.format_sql" value="false" />
</properties>
</persistence-unit>
</persistence>
Below is the error I am getting...
{"WFLYCTL0080: Failed services" => {"jboss.persistenceunit.\"portfoliomanager.war#portfoliomanager\"" => "javax.persistence.PersistenceException: [PersistenceUnit: portfoliomanager] Unable to build Hibernate SessionFactory
Caused by: javax.persistence.PersistenceException: [PersistenceUnit: portfoliomanager] Unable to build Hibernate SessionFactory
Caused by: org.hibernate.exception.GenericJDBCException: Unable to open JDBC Connection for DDL execution
Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/PMProdDS
Caused by: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/PMProdDS
Caused by: javax.resource.ResourceException: IJ031084: Unable to create connection
Caused by: com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
Caused by: com.mysql.cj.exceptions.CJCommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
Caused by: java.net.ConnectException: Connection refused (Connection refused)"}}

IBM MQ v8 and Glassfish local setup port configuration issue

Given the following local setup:
IBM WebSphere MQ Advanced for Developers V8.0
Payara 4.1.2.172
I'd like to connect to the local queue manager via JMS on other than the default port (1414).
In spite I added several properties to the connection factory to configure port 1415, it seems that the server is still trying to connect via port 1414, as Payara constantly throws java.net.ConnectException.
The relevant part of my domain.xml:
<connector-connection-pool resource-adapter-name="wmq.jmsra" name="jms/testCP" connection-definition-name="javax.jms.ConnectionFactory" transaction-support="XATransaction"></connector-connection-pool>
<connector-resource pool-name="jms/testCP" jndi-name="jms/testCF">
<property name="transportType" value="CLIENT"></property>
<property name="port" value="1415"></property>
<property name="channel" value="CHANNEL1"></property>
<property name="hostName" value="localhost"></property>
<property name="localAddress" value="localhost(1415)"></property>
<property name="connectionNameList" value="localhost(1415)"></property>
<property name="queuemanager" value="testQM"></property>
<property name="username" value="mqm"></property>
</connector-resource>
However the exception in server.log suggests that the resource adapter still wants to connect via port 1414:
[2017-08-20T12:41:47.366+0200] [Payara 4.1] [SEVERE] [] [javax.enterprise.system.core] [tid: _ThreadID=63 _ThreadName=AutoDeployer] [timeMillis: 1503225707366] [levelValue: 1000] [[
Exception while loading the app : EJB Container initialization error
java.lang.Exception
at com.sun.enterprise.connectors.inbound.ConnectorMessageBeanClient.setup(ConnectorMessageBeanClient.java:215)
at org.glassfish.ejb.mdb.MessageBeanContainer.(MessageBeanContainer.java:244)
at org.glassfish.ejb.mdb.MessageBeanContainerFactory.createContainer(MessageBeanContainerFactory.java:63)
at org.glassfish.ejb.startup.EjbApplication.loadContainers(EjbApplication.java:224)
at org.glassfish.ejb.startup.EjbDeployer.load(EjbDeployer.java:290)
at org.glassfish.ejb.startup.EjbDeployer.load(EjbDeployer.java:100)
at org.glassfish.internal.data.ModuleInfo.load(ModuleInfo.java:206)
at org.glassfish.internal.data.ApplicationInfo.load(ApplicationInfo.java:314)
at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:497)
at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:220)
at org.glassfish.deployment.admin.DeployCommand.execute(DeployCommand.java:487)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$2$1.run(CommandRunnerImpl.java:539)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$2$1.run(CommandRunnerImpl.java:535)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$2.execute(CommandRunnerImpl.java:534)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$3.run(CommandRunnerImpl.java:565)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$3.run(CommandRunnerImpl.java:557)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:556)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:1464)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.access$1300(CommandRunnerImpl.java:109)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1846)
at org.glassfish.deployment.autodeploy.AutoOperation.run(AutoOperation.java:164)
at org.glassfish.deployment.autodeploy.AutoDeployer.deploy(AutoDeployer.java:597)
at org.glassfish.deployment.autodeploy.AutoDeployer.deployAll(AutoDeployer.java:484)
at org.glassfish.deployment.autodeploy.AutoDeployer.run(AutoDeployer.java:412)
at org.glassfish.deployment.autodeploy.AutoDeployer.run(AutoDeployer.java:403)
at org.glassfish.deployment.autodeploy.AutoDeployService$1.run(AutoDeployService.java:233)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
Caused by: com.ibm.mq.connector.DetailedResourceAdapterInternalException: MQJCA1011: Failed to allocate a JMS connection., error code: MQJCA1011 An internal error caused an attempt to allocate a connection to fail. See the linked exception for details of the failure.
at com.ibm.mq.connector.services.JCAExceptionBuilder.buildException(JCAExceptionBuilder.java:174)
at com.ibm.mq.connector.services.JCAExceptionBuilder.buildException(JCAExceptionBuilder.java:135)
at com.ibm.mq.connector.inbound.ConnectionHandler.allocateConnection(ConnectionHandler.java:393)
at com.ibm.mq.connector.inbound.MessageEndpointDeployment.acquireConnection(MessageEndpointDeployment.java:288)
at com.ibm.mq.connector.inbound.MessageEndpointDeployment.(MessageEndpointDeployment.java:228)
at com.ibm.mq.connector.ResourceAdapterImpl.endpointActivation(ResourceAdapterImpl.java:531)
at com.sun.enterprise.connectors.inbound.ConnectorMessageBeanClient.setup(ConnectorMessageBeanClient.java:207)
... 31 more
Caused by: com.ibm.msg.client.jms.DetailedIllegalStateException: JMSWMQ0018: Failed to connect to queue manager '' with connection mode 'Client' and host name 'localhost(1414)'.
Check the queue manager is started and if running in client mode, check there is a listener running. Please see the linked exception for more information.
at com.ibm.msg.client.wmq.common.internal.Reason.reasonToException(Reason.java:489)
at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:215)
at com.ibm.msg.client.wmq.internal.WMQConnection.(WMQConnection.java:413)
at com.ibm.msg.client.wmq.internal.WMQXAConnection.(WMQXAConnection.java:67)
at com.ibm.msg.client.wmq.factories.WMQXAConnectionFactory.createV7ProviderConnection(WMQXAConnectionFactory.java:188)
at com.ibm.msg.client.wmq.factories.WMQConnectionFactory.createProviderConnection(WMQConnectionFactory.java:7814)
at com.ibm.msg.client.wmq.factories.WMQXAConnectionFactory.createProviderXAConnection(WMQXAConnectionFactory.java:98)
at com.ibm.msg.client.jms.admin.JmsConnectionFactoryImpl.createXAConnectionInternal(JmsConnectionFactoryImpl.java:347)
at com.ibm.mq.jms.MQXAConnectionFactory.createXAConnection(MQXAConnectionFactory.java:131)
at com.ibm.mq.connector.inbound.ConnectionHandler.allocateConnection(ConnectionHandler.java:268)
... 35 more
Caused by: com.ibm.mq.MQException: JMSCMQ0001: WebSphere MQ call failed with compcode '2' ('MQCC_FAILED') reason '2538' ('MQRC_HOST_NOT_AVAILABLE').
at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:203)
... 43 more
Caused by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2538;AMQ9204: Connection to host 'localhost(1414)' rejected. [1=com.ibm.mq.jmqi.JmqiException[CC=2;RC=2538;AMQ9213: A communications error for 'TCP' occurred. [1=java.net.ConnectException[Connection refused (Connection refused)],3=connnectUsingLocalAddress,4=TCP,5=Socket.connect]],3=localhost(1414),5=RemoteTCPConnection.connnectUsingLocalAddress]
at com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiConnect(RemoteFAP.java:2282)
at com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiConnect(RemoteFAP.java:1294)
at com.ibm.mq.ese.jmqi.InterceptedJmqiImpl.jmqiConnect(InterceptedJmqiImpl.java:376)
at com.ibm.mq.ese.jmqi.ESEJMQI.jmqiConnect(ESEJMQI.java:560)
at com.ibm.msg.client.wmq.internal.WMQConnection.(WMQConnection.java:346)
... 42 more
Caused by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2538;AMQ9213: A communications error for 'TCP' occurred. [1=java.net.ConnectException[Connection refused (Connection refused)],3=connnectUsingLocalAddress,4=TCP,5=Socket.connect]
at com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection.connnectUsingLocalAddress(RemoteTCPConnection.java:838)
at com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection.protocolConnect(RemoteTCPConnection.java:1277)
at com.ibm.mq.jmqi.remote.impl.RemoteConnection.connect(RemoteConnection.java:863)
at com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification.getSessionFromNewConnection(RemoteConnectionSpecification.java:409)
at com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification.getSession(RemoteConnectionSpecification.java:305)
at com.ibm.mq.jmqi.remote.impl.RemoteConnectionPool.getSession(RemoteConnectionPool.java:146)
at com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiConnect(RemoteFAP.java:1730)
... 46 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection$5.run(RemoteTCPConnection.java:823)
at com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection$5.run(RemoteTCPConnection.java:814)
at java.security.AccessController.doPrivileged(Native Method)
at com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection.connnectUsingLocalAddress(RemoteTCPConnection.java:814)
... 52 more
]]
I'm out of ideas why the exception says:
JMSWMQ0018: Failed to connect to queue manager '' with connection mode 'Client' and host name 'localhost(1414)'.
Question 1: Why is queue manager '' (empty)?
Question 2: Why is host name 'localhost(1414)'?
Any help would be very much appreciated!

In reviewing the IBM MQ v8 Knowledge Center page "Installing and testing the resource adapter in GlassFish Server" it appears you have the wrong property name for the host name and queue manager. Try the following config. Checkout the documentation link above to make sure you have installed the RA and the other steps.
<connector-connection-pool resource-adapter-name="wmq.jmsra" name="jms/testCP" connection-definition-name="javax.jms.ConnectionFactory" transaction-support="XATransaction"></connector-connection-pool>
<connector-resource pool-name="jms/testCP" jndi-name="jms/testCF">
<property name="transportType" value="CLIENT"></property>
<property name="port" value="1415"></property>
<property name="channel" value="CHANNEL1"></property>
<property name="host" value="localhost"></property>
<property name="queueManager" value="testQM"></property>
</connector-resource>

I have found that the article referenced by JoshMC is incorrect. Step 6.f should not form part of the "Connector Resource" (connection factory) configuration, but the "Connector Connection Pool" configuration, i.e. between 5.f and 5.g.
If you create a new connection pool using the Admin Console then it displays many properties that can be populated. The following are the important ones:
channel
port
hostName
queueManager
transportType
username
such that the domain.xml should contain the following:
<connector-connection-pool resource-adapter-name="wmq.jmsra" name="jms/testCP" connection-definition-name="javax.jms.ConnectionFactory" transaction-support="XATransaction">
<property name="channel" value="CHANNEL1"></property>
<property name="port" value="1415"></property>
<property name="hostName" value="localhost"></property>
<property name="queueManager" value="testQM"></property>
<property name="transportType" value="CLIENT"></property>
<property name="username" value="mqm"></property>
</connector-connection-pool>

We found that we used the wrong version of the wmq.jmsra adapter.
We were using 7.5.0.4-p750-004-140807. With a newer Version 8.0.0.8-p800-008-171121 everything worked fine.

Remote ejb not working in jboss-eap 6.4

I have an ejb client and able to test remote ejb on tomcat and weblogic.But it is failing in jboss-eap 6.4.
There is no firewall issue as it is working in other servers.I tried placing jboss-client jar shipped with jboss and also tried as bom type. still same error.
Jboss server log:
19:27:45,107 TRACE [org.jboss.remoting.remote] (Remoting "config-based-naming-client-endpoint" read-1) Received message data
19:27:45,108 TRACE [org.jboss.remoting.remote] (Remoting "config-based-naming-client-endpoint" read-1) CAS Channel ID bf292e5d (outbound) of Remoting connection 33fccb04 to /192.168.1.50:4447
old: RS=false WS=false IM=0 OM=0
new: RS=false WS=false IM=1 OM=0
19:27:45,109 TRACE [org.jboss.remoting.remote] (Remoting "config-based-naming-client-endpoint" read-1) Opened inbound message on Channel ID bf292e5d (outbound) of Remoting connection 33fccb04 to /192.168.1.50:4447
19:27:45,116 TRACE [org.jboss.remoting.remote] (Remoting "config-based-naming-client-endpoint" read-1) Received message data
19:27:45,116 TRACE [org.jboss.remoting.remote] (Remoting "config-based-naming-client-endpoint" read-1) CAS Channel ID bf292e5d (outbound) of Remoting connection 33fccb04 to /192.168.1.50:4447
old: RS=false WS=false IM=1 OM=0
new: RS=false WS=false IM=0 OM=0
19:27:45,117 TRACE [org.jboss.remoting.remote] (Remoting "config-based-naming-client-endpoint" read-1) Closed inbound message on Channel ID bf292e5d (outbound) of Remoting connection 33fccb04 to /192.168.1.50:4447
19:27:45,117 TRACE [org.jboss.remoting.remote.connection] (Remoting "config-based-naming-client-endpoint" read-1) Sent message java.nio.HeapByteBuffer[pos=7 lim=7 cap=8192] (direct)
19:27:45,117 TRACE [org.jboss.remoting.remote.connection] (Remoting "config-based-naming-client-endpoint" read-1) Flushed channel (direct)
19:27:45,117 TRACE [org.jboss.remoting.remote] (Remoting "config-based-naming-client-endpoint" read-1) No message ready; returning
Jboss standalone.xml configuration:
<subsystem
xmlns="urn:jboss:domain:remoting:1.2">
<connector name="remoting-connector" socket-binding="remoting" security-realm="ApplicationRealm"/>
<outbound-connections>
<remote-outbound-connection name="remote-ejb-connection" outbound-socket-binding-ref="remote-ejb" username="admin" security-realm="ApplicationRealm">
<properties>
<property name="SASL_POLICY_NOANONYMOUS" value="false"/>
<property name="SSL_ENABLED" value="false"/>
</properties>
</remote-outbound-connection>
</outbound-connections>
</subsystem>
<socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">
<outbound-socket-binding name="remote-ejb">
<remote-destination host="192.168.2.31" port="4447"/>
</outbound-socket-binding>
</socket-binding-group>
Stacktrace is:
callExternalService - Exchange-exception :java.lang.IllegalStateException: EJBCLIENT000025: No EJB receiver available for handling [appName:, moduleName:AS-Test-EJB, distinctName:] combination for invocation context org.jboss.ejb.client.EJBClientInvocationContext#a89afa9
at org.jboss.ejb.client.EJBClientContext.requireEJBReceiver(EJBClientContext.java:747)
at org.jboss.ejb.client.ReceiverInterceptor.handleInvocation(ReceiverInterceptor.java:116)
at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:186)
at org.jboss.ejb.client.EJBInvocationHandler.sendRequestWithPossibleRetries(EJBInvocationHandler.java:255)
at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:200)
at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:183)
at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:146)
at com.sun.proxy.$Proxy406.sayHello(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.camel.component.bean.MethodInfo.invoke(MethodInfo.java:408)
at org.apache.camel.component.bean.MethodInfo$1.doProceed(MethodInfo.java:279)
at org.apache.camel.component.bean.MethodInfo$1.proceed(MethodInfo.java:252)
at org.apache.camel.component.bean.BeanProcessor.process(BeanProcessor.java:177)
at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:109)
at org.apache.camel.component.bean.BeanProcessor.process(BeanProcessor.java:68)
at org.apache.camel.component.bean.BeanProducer.process(BeanProducer.java:38)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197)
at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:109)
at org.apache.camel.processor.UnitOfWorkProducer.process(UnitOfWorkProducer.java:68)
at org.apache.camel.impl.ProducerCache$2.doInProducer(ProducerCache.java:412)
at org.apache.camel.impl.ProducerCache$2.doInProducer(ProducerCache.java:380)
at org.apache.camel.impl.ProducerCache.doInProducer(ProducerCache.java:270)
at org.apache.camel.impl.ProducerCache.sendExchange(ProducerCache.java:380)
at org.apache.camel.impl.ProducerCache.send(ProducerCache.java:221)
at org.apache.camel.impl.DefaultProducerTemplate.send(DefaultProducerTemplate.java:124)
at org.apache.camel.impl.DefaultProducerTemplate$13.call(DefaultProducerTemplate.java:616)
at org.apache.camel.impl.DefaultProducerTemplate$13.call(DefaultProducerTemplate.java:614)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Edited with properties used for lookup.
<endpoint id="Admin__Admin__EJBServiceTest" uri="ejb04-Admin:AS-Test-EJB/ServerTestBean!com.appzillon.test.ejb.ServerTest?method=sayHello"/>
<bean class="org.apache.camel.component.ejb.EjbComponent" id="ejb04-Admin">
<property name="properties" ref="Admin_Admin__EJBServiceTest_jndiProperties"/>
</bean>
<util:properties id="Admin_Admin__EJBServiceTest_jndiProperties">
<prop key="java.naming.provider.url">remote://192.168.1.50:4447</prop>
<prop key="java.naming.factory.initial">org.jboss.naming.remote.client.InitialContextFactory</prop>
<prop key="jboss.naming.client.ejb.context">true</prop>
<prop key="java.naming.security.principal">admin</prop>
<prop key="java.naming.security.credentials">********</prop>
</util:properties>
and we are using apache camel for processing.we read endpoint from xml file and proccess.lookup is taken care by apche camel.

Ehcache - Unable to get distributed caching working

I am trying to enable distributed caching on ehcache for one of my projects between 2 nodes but I am running into issues.
I am using ehcache core 2.6.11 in a Java Spring project
Here are relevant portions of my ehcache config on each of the servers
Node 1
<cacheManagerPeerProviderFactory
class="net.sf.ehcache.distribution.RMICacheManagerPeerProviderFactory"
properties="
peerDiscovery=manual,
rmiUrls=//node2.mydomain.com:40002/myCache1|//node2.mydomain.com:40002/myCache2"/>
<cacheManagerPeerListenerFactory
class="net.sf.ehcache.distribution.RMICacheManagerPeerListenerFactory"
properties="port=40001"/>
Node 2
<cacheManagerPeerProviderFactory
class="net.sf.ehcache.distribution.RMICacheManagerPeerProviderFactory"
properties="
peerDiscovery=manual,
rmiUrls=//node1.mydomain.com:40001/myCache1|//node1.mydomain.com:40001/myCache2"/>
<cacheManagerPeerListenerFactory
class="net.sf.ehcache.distribution.RMICacheManagerPeerListenerFactory"
properties="port=40002"/>
When I update my cache and expect it to be replicated - here are the errors I get on my nodes
Node 1
03.03.2016 06:44:50 DEBUG [net.sf.ehcache.distribution.RMICacheManagerPeerProvider.lookupRemoteCachePeer():126] Lookup URL //node2.mydomain.com:40002/myCache1
03.03.2016 06:44:50 WARN [net.sf.ehcache.distribution.RMIAsynchronousCacheReplicator.writeReplicationQueue():325] Unable to send message to remote peer. Message was: Connection refused to host: <node 2 IP Address>; nested exception is:
java.net.ConnectException: Connection refused
java.rmi.ConnectException: Connection refused to host: <node 2 IP Address>; nested exception is:
java.net.ConnectException: Connection refused
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:130)
at net.sf.ehcache.distribution.RMICachePeer_Stub.send(Unknown Source)
at net.sf.ehcache.distribution.RMIAsynchronousCacheReplicator.writeReplicationQueue(RMIAsynchronousCacheReplicator.java:314)
at net.sf.ehcache.distribution.RMIAsynchronousCacheReplicator.replicationThreadMain(RMIAsynchronousCacheReplicator.java:127)
at net.sf.ehcache.distribution.RMIAsynchronousCacheReplicator.access$000(RMIAsynchronousCacheReplicator.java:58)
at net.sf.ehcache.distribution.RMIAsynchronousCacheReplicator$ReplicationThread.run(RMIAsynchronousCacheReplicator.java:389)
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at java.net.Socket.<init>(Socket.java:434)
at java.net.Socket.<init>(Socket.java:211)
at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40)
at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:148)
at net.sf.ehcache.distribution.ConfigurableRMIClientSocketFactory.createSocket(ConfigurableRMIClientSocketFactory.java:70)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
... 8 more
Node 2
03.03.2016 06:44:50 DEBUG [net.sf.ehcache.distribution.RMICacheManagerPeerProvider.lookupRemoteCachePeer():126] Lookup URL //node1.mydomain.com:40001/myCache1
03.03.2016 06:44:50 WARN [net.sf.ehcache.distribution.RMIAsynchronousCacheReplicator.writeReplicationQueue():325] Unable to send message to remote peer. Message was: Connection refused to host: <node 2 IP Address>; nested exception is:
java.net.ConnectException: Connection refused
java.rmi.ConnectException: Connection refused to host: <node 2 IP Address>; nested exception is:
java.net.ConnectException: Connection refused
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:130)
at net.sf.ehcache.distribution.RMICachePeer_Stub.send(Unknown Source)
at net.sf.ehcache.distribution.RMIAsynchronousCacheReplicator.writeReplicationQueue(RMIAsynchronousCacheReplicator.java:314)
at net.sf.ehcache.distribution.RMIAsynchronousCacheReplicator.replicationThreadMain(RMIAsynchronousCacheReplicator.java:127)
at net.sf.ehcache.distribution.RMIAsynchronousCacheReplicator.access$000(RMIAsynchronousCacheReplicator.java:58)
at net.sf.ehcache.distribution.RMIAsynchronousCacheReplicator$ReplicationThread.run(RMIAsynchronousCacheReplicator.java:389)
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at java.net.Socket.<init>(Socket.java:434)
at java.net.Socket.<init>(Socket.java:211)
at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40)
at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:148)
at net.sf.ehcache.distribution.ConfigurableRMIClientSocketFactory.createSocket(ConfigurableRMIClientSocketFactory.java:70)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
... 8 more
UPDATE
I have changed my configuration now to use Automatic Discovery instead of Manual.
I have 4 servers, 2 in Site A and 2 in Site B.
The replicated is now working as expected on Site A between both nodes but no replication is occuring on Site B or between Site A and Site B.
I used netcat and telnet between each node and the other nodes and all connections were accepted.
Here are the relevant sections of my ehcache config for the automatic discovery
<cacheManagerPeerProviderFactory class="net.sf.ehcache.distribution.RMICacheManagerPeerProviderFactory"
properties="peerDiscovery=automatic,
multicastGroupAddress=230.3.0.0,
multicastGroupPort=4700,
timeToLive=255"/>
<cacheManagerPeerListenerFactory
class="net.sf.ehcache.distribution.RMICacheManagerPeerListenerFactory"
properties="port=43000,
socketTimeoutMillis=20000"/>
Could anyone offer any insight into how the replication is not working on Site B or any ideas I could try?
Thanks

Please check the below steps to fix this issue
Add the host name in the Listener as below.
<cacheManagerPeerListenerFactory
class="net.sf.ehcache.distribution.RMICacheManagerPeerListenerFactory"
properties="hostName=localhost,port=40001"/>
Check the port 40002 is open for access, so that It can be accessed from node1. Means node2.mydomain.com:40002 should be accessible from node1. From console, you can use telnet program to check it. Similarly, check if node1 is accessible from node2. (I believe this is what causing the issue)
Optional: You don't need to have different ports in the two servers. It can have the same port number 40001.

I reverted back to the manual discovery approach as we received advice from a Networking specialist that using Multicast accross data centers opens up a whole world of pain.
Using the manual approach with Thanga's suggestions has worked

Failed to connect to RMI registry

I am running activemq-5.9 on Ubuntu 12.04 and it would be fair to say that I am new to active MQ.
I suspect that I am queueing records and not de-queueing them, so I am looking into using JMX to connect to my active MQ broker and see what's going on. Jconsole fails.
This is hard because most of the docs on how to do this is old, but I finally get down to running activemq list. When I do, it throws a rather lengthy stacktrace:
Connecting to JMX URL: service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi
ERROR: java.lang.RuntimeException: Failed to execute list task. Reason: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:
java.net.ConnectException: Connection refused]
...
Caused by: java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:
java.net.ConnectException: Connection refused
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:342)
at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:118)
... 21 more
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at java.net.Socket.<init>(Socket.java:434)
at java.net.Socket.<init>(Socket.java:211)
at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40)
at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:148)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
... 26 more
Is there still something I have configured wrong? I have no firewall rules. I can't think what else to look for.

There is no 'failure to retrieve RMI server stub' here. There is a failure to connect to the RMI registry. Either it isn't running at all or you're looking in the wrong host, or port.

Make sure your RMI registry is running. You can do this with rmiregistry in another terminal, providing it's in the same working directory.