Develop Reference

Flink job runs to fail on EMR cluster

i am developing flink application as Kafka source. I have completed all required changes to code, but its falling into following error when run on one emr cluster. Could you guys let me know the workaround if anyone already experienced.
Caused by: org.apache.flink.runtime.concurrent.FutureUtils$RetryException: Could not complete the operation. Number of retries has been exhausted.
at org.apache.flink.runtime.concurrent.FutureUtils.lambda$retryOperationWithDelay$8(FutureUtils.java:302)
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
at org.apache.flink.runtime.rest.RestClient.lambda$submitRequest$1(RestClient.java:342)
at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:500)
at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:493)
at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:472)
at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:413)
at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:538)
at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:531)
at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:111)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:323)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:339)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:685)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:632)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:549)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:511)
at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
at org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.util.concurrent.CompletionException: org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: ip-xxx-xxx-xxx.dqa.capitalone.com/xx.xxx.xxx.xxx:38257
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
at java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:957)
at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:940)
... 19 more
Caused by: org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: ip-xxx-xxx-xxx.dqa.capitalone.com/xx.xxx.xxx.xxx:38257
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716)
at org.apache.flink.shaded.netty4.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:336)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:685)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:632)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:549)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:511)
at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
at org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.lang.Thread.run(Thread.java:750)
EMR Cluster-id:

It's failing with a connection failure, trying to talk to ip-xxx-xxx-xxx.dqa.capitalone.com/xx.xxx.xxx.xxx:38257. It seems likely that internal corporate address is not reachable from inside of AWS.

IBM MQ 8 to 9 Migration

We have Sprintboot application on Azure Linux VM which is connecting to IBM MQ 8 and connection goes via KPN.
Recently we have upgraded IBM MQ from 8 to 9. No settings are changed on application, but with IBM MQ 9 - we can see that post 30 mins - connections are getting timed out and new connections are getting created post every 30 mins.
Checked with IBM MQ 9 team - no timeout setting at their end.
Error -
[com.ibm.mq.allclient-9.2.3.0.jar!/:9.2.3.0 - p923-L210708]
Caused by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2009;AMQ9208: Error on receive from host 'Host Details'. [1=javax.net.ssl.SSLException[Connection timed out (Read failed)],3=,4=TCP,5=InputStream.read]
Caused by: javax.net.ssl.SSLException: Connection timed out (Read failed)
Suppressed: java.net.SocketException: Broken pipe (Write failed)
enter code here`Caused by: java.net.SocketException: Connection timed out (Read failed)

Caused by: io.grpc.StatusRuntimeException: INTERNAL: HTTP/2 error code: PROTOCOL_ERROR

I get the following error when trying to connect to mongodb. I am running a spring boot application on java 1.8. Our company built a java client on top of grpc to connect to the mongodb and I see the following error occasionally. It doesn't happen all the time, but only occasionally. Any idea what that is?
Caused by: io.grpc.StatusRuntimeException: INTERNAL: HTTP/2 error code: PROTOCOL_ERROR
Received Goaway
Pseudo-header field ':path' found after regular header.
at io.grpc.Status.asRuntimeException(Status.java:533)
... 13 more

WSO2 API Manager - Caused by: java.net.SocketException: Connection reset

I'm seeing below error in WSO2 API Manager 1.6 after running the wso2server.sh for sometime with average load. I got below error repeatedly on the error log.
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:196)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
... 13 more
[2015-03-03 22:50:48,527] INFO - ReceiverGroup Resending the failed events....
[2015-03-03 22:50:48,527] ERROR - EventPublisher Cannot send events to TCP,localhost:7612,TCP,localhost:7712
org.wso2.carbon.databridge.agent.thrift.exception.EventPublisherException: Cannot send Events
at org.wso2.carbon.databridge.agent.thrift.internal.publisher.client.ThriftEventPublisher.publish(ThriftEventPublisher.java:93)
at org.wso2.carbon.databridge.agent.thrift.internal.publisher.client.EventPublisher.publishEvent(EventPublisher.java:130)
at org.wso2.carbon.databridge.agent.thrift.internal.publisher.client.EventPublisher.run(EventPublisher.java:117)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
at org.wso2.carbon.databridge.commons.thrift.service.general.ThriftEventTransmissionService$Client.recv_publish(ThriftEventTransmissionService.java:146)
at org.wso2.carbon.databridge.commons.thrift.service.general.ThriftEventTransmissionService$Client.publish(ThriftEventTransmissionService.java:133)
at org.wso2.carbon.databridge.agent.thrift.internal.publisher.client.ThriftEventPublisher.publish(ThriftEventPublisher.java:86)
... 5 more
What could be the reason and how can I fix this?

It seems like that you have enabled publishing runtime statistics to BAM. From the log i can see that the server is trying to publish to "TCP,localhost:7612,TCP,localhost:7712" which would be thrift ports. So if you don't have a BAM server running with thrift port 7612 this error would come. You can find more information on publishing runtime statistics to BAM here [1].
[1] https://docs.wso2.com/display/AM160/Publishing+API+Runtime+Statistics

UniversalConnectionPoolManagerMBean Already exists after application "re"start

When I deploy an application.war that contains a servlet to my weblogic server it runs fine. The servlet uses an instance of UniversalConnectionPoolManagerMBean and starts up without any problems.
In the administration console I use the "stop when work completes" command and in my destroy() method it calls:
UniversalConnectionPoolManagerImpl.getUniversalConnectionPoolManager()
.startConnectionPool(mConnectionPoolName);
When I then click on "start servicing requests" command in the administration console I get this exception:
java.sql.SQLException: Unable to start the Universal Connection Pool: java.sql.SQLException: Unable to start the Universal Connection Pool: oracle.ucp.UniversalConnectionPoolException: MBean exception occurred while registering or unregistering the MBean
at com.exzac.dal.jdbc.Database.getConnection(Database.java:134)
at com.exzac.dal.jdbc.Database.executeQuery(Database.java:161)
at com.exzac.profilesystem.ProfileSystemConfiguration.retrieveConfigurationTable(ProfileSystemConfiguration.java:214)
... 49 more
Caused by: java.sql.SQLException: Unable to start the Universal Connection Pool: java.sql.SQLException: Unable to start the Universal Connection Pool: oracle.ucp.UniversalConnectionPoolException: MBean exception occurred while registering or unregistering the MBean
at oracle.ucp.util.UCPErrorHandler.newSQLException(UCPErrorHandler.java:541)
at oracle.ucp.jdbc.PoolDataSourceImpl.throwSQLException(PoolDataSourceImpl.java:587)
at oracle.ucp.jdbc.PoolDataSourceImpl.startPool(PoolDataSourceImpl.java:276)
at oracle.ucp.jdbc.PoolDataSourceImpl.getConnection(PoolDataSourceImpl.java:646)
at oracle.ucp.jdbc.PoolDataSourceImpl.getConnection(PoolDataSourceImpl.java:613)
at oracle.ucp.jdbc.PoolDataSourceImpl.getConnection(PoolDataSourceImpl.java:607)
at com.exzac.dal.jdbc.Database.getConnection(Database.java:132)
... 51 more
Caused by: java.sql.SQLException: Unable to start the Universal Connection Pool: oracle.ucp.UniversalConnectionPoolException: MBean exception occurred while registering or unregistering the MBean
at oracle.ucp.util.UCPErrorHandler.newSQLException(UCPErrorHandler.java:541)
at oracle.ucp.jdbc.PoolDataSourceImpl.throwSQLException(PoolDataSourceImpl.java:587)
at oracle.ucp.jdbc.PoolDataSourceImpl.startPool(PoolDataSourceImpl.java:247)
... 55 more
Caused by: oracle.ucp.UniversalConnectionPoolException: MBean exception occurred while registering or unregistering the MBean
at oracle.ucp.util.UCPErrorHandler.newUniversalConnectionPoolException(UCPErrorHandler.java:421)
at oracle.ucp.util.UCPErrorHandler.newUniversalConnectionPoolException(UCPErrorHandler.java:389)
at oracle.ucp.admin.UniversalConnectionPoolManagerMBeanImpl.getUniversalConnectionPoolManagerMBean(UniversalConnectionPoolManagerMBeanImpl.java:149)
at oracle.ucp.jdbc.PoolDataSourceImpl.startPool(PoolDataSourceImpl.java:242)
... 55 more
Caused by: java.security.PrivilegedActionException: javax.management.InstanceAlreadyExistsException: oracle.ucp.admin:name=UniversalConnectionPoolManagerMBean
at java.security.AccessController.doPrivileged(Native Method)
at oracle.ucp.admin.UniversalConnectionPoolManagerMBeanImpl.getUniversalConnectionPoolManagerMBean(UniversalConnectionPoolManagerMBeanImpl.java:136)
... 56 more
Caused by: javax.management.InstanceAlreadyExistsException: oracle.ucp.admin:name=UniversalConnectionPoolManagerMBean
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:453)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.internal_addObject(DefaultMBeanServerInterceptor.java:1484)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:963)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:917)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:312)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:482)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$27.run(WLSMBeanServerInterceptorBase.java:714)
at java.security.AccessController.doPrivileged(Native Method)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.registerMBean(WLSMBeanServerInterceptorBase.java:709)
at weblogic.management.mbeanservers.internal.JMXContextInterceptor.registerMBean(JMXContextInterceptor.java:448)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$27.run(WLSMBeanServerInterceptorBase.java:712)
at java.security.AccessController.doPrivileged(Native Method)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.registerMBean(WLSMBeanServerInterceptorBase.java:709)
at weblogic.management.jmx.mbeanserver.WLSMBeanServer.registerMBean(WLSMBeanServer.java:462)
at oracle.ucp.admin.UniversalConnectionPoolManagerMBeanImpl$2.run(UniversalConnectionPoolManagerMBeanImpl.java:142)
... 58 more
From then on I hit this exception, even when I stop the server and restart and try to deploy. Only cleaning all published resources seems to clear the exception. I understand that a published resource was not properly cleaned in the destroy method, and I am thinking that the resource is the UniversalConnectionPoolManagerMBean instance, but I thought it would be clean up by the destroyConnectionPool method:
http://docs.oracle.com/cd/B28359_01/java.111/e11990/oracle/ucp/admin/UniversalConnectionPoolManagerMBean.html#destroyConnectionPool_java_lang_String_
How do I avoid hitting this exception?
Just as an interesting (yet unsafe) workaround: For some reason when I insert this into the initialization:
try {
UniversalConnectionPoolManagerImpl.getUniversalConnectionPoolManager().destroyConnectionPool(mConnectionPoolName);
} catch (final Exception e1) {
LOGGER.error(e1);
}
the exception is thrown and caught saying that it could not find a connectionPool with this name to destroy, but the original problem is gone. javax.management.InstanceAlreadyExistsException: oracle.ucp.admin:name=UniversalConnectionPoolManagerMBean is not thrown

This bug seems to have been fixed in versions 11.2.0.4 and 12.1.0.2. I couldn't find any documentation about it, but for me the error disappeared with the above mentioned versions.
JDBC/UCP Download Page:
http://www.oracle.com/technetwork/database/features/jdbc/index-091264.html

This is from Oracle's 11.2.0.4 UCP.jar README.txt.
Fixes in 11.2.0.4
16805157 The name of the UniversalConnectionPoolManagerMBean is made
unique per classloader and hence unique to every UCP instance
even within the same JVM
This fixes the javax.management.InstanceAlreadyExistsException described in the original question. Every connection pool created with have a unique name and will not result in InstanceAlreadyExistsException. However, since you are restarting, you should look into figuring out how to close the connection pool MBean cleanly so you won't run into this problem.