Network unreachable when using Google Maps Geocode API in Google Dataflow - java

I'm using the Google Maps Geocode API (https://github.com/googlemaps/google-maps-services-java) in a Dataflow job. My DoFn prepares the GeoApiContext at Setup. The process element function is done like so:
public void processElement(ProcessContext c) {
String address = c.element().get("Address").toString();
String id = c.element().get("Id").toString();
Gson gson = new GsonBuilder().create();
try {
GeocodingResult[] results = GeocodingApi.newRequest(this.geocodeContext).address(address).language("pt-BR").components(ComponentFilter.country("BR")).await();
if(results.length == 0) {
TableRow outputRow = new TableRow();
outputRow.set("Id", id);
c.output(outputRow);
} else {
for(GeocodingResult r : results) {
TableRow outputRow = convertTableRow(gson.toJson(r).toString());
outputRow.set("Id", id);
c.output(outputRow);
}
}
} catch(ApiException e) {
LOGGER.error("ApiException on address: {}", address, e);
} catch(InterruptedException e) {
LOGGER.error("InterruptedException on address: {}", address, e);
} catch(IOException e) {
LOGGER.error("IOException on address: {}", address, e);
}
}
This code worked fine locally, but when deployed to dataflow it throws a network error:
exception: "java.net.ConnectException: Failed to connect to maps.googleapis.com/2607:f8b0:4001:c05:0:0:0:5f:443
at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:265)
at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:183)
at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.java:224)
at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.java:108)
at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.java:88)
at okhttp3.internal.connection.Transmitter.newExchange(Transmitter.java:169)
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:41)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:94)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:88)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:229)
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:172)
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.net.ConnectException: Network is unreachable (connect failed)
at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
at java.base/java.net.Socket.connect(Socket.java:591)
at okhttp3.internal.platform.Platform.connectSocket(Platform.java:130)
at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:263)
... 22 more
I've ensured that the VM spawned has internet access and I can even ping the maps.googleapis.com endpoint from inside the container:
USER#test-geocode-07020834-qmrj-harness-3k2l ~ $ docker container ls
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b2fd123138aa 3a1cb7aedd54 "/opt/google/dataflo…" 6 minutes ago Up 5 minutes k8s_healthchecker_dataflow-test-geocode-07020834-qmrj-harness-3k2l_default_5648e9815f2ca5beea8b0eb945e12d1f_0
086e36c3dd23 4127911f4769 "/opt/google/dataflo…" 6 minutes ago Up 5 minutes k8s_vmmonitor_dataflow-test-geocode-07020834-qmrj-harness-3k2l_default_5648e9815f2ca5beea8b0eb945e12d1f_0
2890fa415af5 664bd8972b23 "/opt/google/dataflo…" 6 minutes ago Up 6 minutes k8s_shuffle_dataflow-test-geocode-07020834-qmrj-harness-3k2l_default_5648e9815f2ca5beea8b0eb945e12d1f_0
eea757bf6be7 gcr.io/cloud-dataflow/v1beta3/beam-java11-batch "/opt/google/dataflo…" 6 minutes ago Up 6 minutes k8s_java-batch_dataflow-test-geocode-07020834-qmrj-harness-3k2l_default_5648e9815f2ca5beea8b0eb945e12d1f_0
b636784118f5 k8s.gcr.io/pause:3.1 "/pause" 6 minutes ago Up 6 minutes k8s_POD_dataflow-test-geocode-07020834-qmrj-harness-3k2l_default_5648e9815f2ca5beea8b0eb945e12d1f_0
lucas#test-geocode-07020834-qmrj-harness-3k2l ~ $ docker exec -it eea /bin/sh
# ping maps.googleapis.com
PING maps.googleapis.com (172.217.214.95) 56(84) bytes of data.
64 bytes from 172.217.214.95: icmp_seq=1 ttl=115 time=1.08 ms
64 bytes from 172.217.214.95: icmp_seq=2 ttl=115 time=1.28 ms
64 bytes from 172.217.214.95: icmp_seq=3 ttl=115 time=1.15 ms
64 bytes from 172.217.214.95: icmp_seq=4 ttl=115 time=1.41 ms
^C
--- maps.googleapis.com ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3004ms
rtt min/avg/max/mdev = 1.089/1.235/1.414/0.131 ms
#
Regarding versions, I'm using the latest beam version (2.22.0) and the latest google maps version (0.14.0).
No idea what else to look at, and any help is appreciated.
UPDATE
The problem seems to be the fact that the request is done with an ipv6 address. However, GCE machines seems to have no support for ipv6 and the call simply fails without falling back to ipv4.
Considering that, there doesn't seem to be any way out of this problem:
Configuring the JVM to prefer ipv4 address can't be done with Dataflow (JVM flags are ignored)
There's also no way to customize the GCE machine (since a base Dataflow image is used)
The library doesn't seem to open any options to configure ipv4 or ipv6
Thanks

I had this exact same issue come up after upgrading from 2.17 to 2.24 and changing from Java 8 to Java 11. After trying to fix this on 2.24 and Java 11 I gave up and went back to 8 and it's working now.
I couldn't find it documented anywhere but it looks like the userAgent used is based on that -
When I build the self executable with Java 8, Dataflow shows the userAgent as
Apache_Beam_SDK_for_Java/2.24.0(JRE_8_environment)
and with Java 11 it shows Apache_Beam_SDK_for_Java/2.24.0(JDK_11_environment)

Related

mTLS/TLS Redis 6 Issues Java

Currently I am running Redis 6 with ACL and mTLS with a C# client just fine. I am trying to update our Java side to also use ACL and mTLS but have been running into issues. I am primarily focused on mTLS at the moment and have not been getting anywhere with it. This could be user fault in these that I have not used Java for 5-6 years before attempting to do this, so please advise. Not sure what or how to really progress from this error and I have done google searches with not success really. Any help greatly appreciated, again I have not done Java in a long time so that most likely might be the issue.
Trace:
Caused by: io.lettuce.core.RedisConnectionException: Unable to connect to localhost:6379
at io.lettuce.core.RedisConnectionException.create(RedisConnectionException.java:78)
at io.lettuce.core.RedisConnectionException.create(RedisConnectionException.java:56)
at io.lettuce.core.AbstractRedisClient.getConnection(AbstractRedisClient.java:295)
at io.lettuce.core.RedisClient.connect(RedisClient.java:214)
at io.lettuce.core.RedisClient.connect(RedisClient.java:199)
at blah blah blah my code....
... 48 more
Caused by: javax.net.ssl.SSLException: SSLEngine closed already
at io.netty.handler.ssl.SslHandler.wrap(SslHandler.java:834)
at io.netty.handler.ssl.SslHandler.wrapAndFlush(SslHandler.java:797)
at io.netty.handler.ssl.SslHandler.handleUnwrapThrowable(SslHandler.java:1254)
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1230)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1271)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:505)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:444)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514)
at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
... 2 more
Redis Server Debug logs:
1:M 30 Jul 2020 15:23:10.837 - Accepted 10.0.2.2:62023
1:M 30 Jul 2020 15:23:11.024 # Error accepting a client connection: (null)
Java code:
final RedisClient client = RedisClient.create(RedisURI.Builder.redis(hostConfig,portConfig)
.withSsl(true).withVerifyPeer(false).build().toURI().toString());
if (redisTruststorePath != null && !redisTruststorePath.isEmpty()) {
SslOptions sslOptions;
if (redisKeystorePath != null && !redisKeystorePath.isEmpty()) {
sslOptions = SslOptions.builder()
.jdkSslProvider()
.keystore(new File(redisKeystorePath), redisKeystorePass)
.truststore(new File(redisTruststorePath), redisTruststorePass)
.build();
}
else {
sslOptions = SslOptions.builder()
.jdkSslProvider()
.truststore(new File(redisTruststorePath), redisTruststorePass)
.build();
}
client.setOptions(ClientOptions.builder().sslOptions(sslOptions).build());
}
client.connect();
Versions:
Lettuce version(s): 6.0.0.M1 (Running on windows locally)
Redis version: 6.0.5 (Running on linux VM locally)
Notes:
C# client is working fine so doubt its a Redis Server issue.
Redis URI (printed in my real code before set): rediss://localhost:6379
Please check your client-side logs.
16797:M 03 Aug 2020 09:11:11.246 # Error accepting a client connection: (null)
This message above happens when Redis wasn't able to continue with the connection phase. Such a message occurs in SSL arrangements when the SSL handshake wasn't completed successfully, e.g. caused by a failed certificate validation.
Looking at the code above, the client gets created with:
RedisClient.create(RedisURI.Builder.redis(hostConfig,portConfig) .withSsl(true).withVerifyPeer(false).build().toURI().toString());
The RedisURI object gets converted into a string which causes a loss of the verifyPeer flag.
Please change your code to:
RedisClient.create(RedisURI.Builder.redis(hostConfig,portConfig) .withSsl(true).withVerifyPeer(false).build());
by removing .toURI().toString().
As #mp911de mentioned I removed .toURI().toString(); as well as, updated to lettuce-core 6.0.0.RC and started using RESP2 (as suggested here). This resolved my problem. I think the main solution here was switching to RESP2, which again was a suggestion from #mp911de. Thank you for the assistance #mp911de!!

gwhalin memcached java client set object size > 1 MB increase slab size

I am using com.whalin/Memcached-Java-Client library version 3.0.2. Default slab size is 1 MB and I have used -I command to start memcached server - memcached -d -u nobody -m 4096 127.0.0.1 -p 11211 -I 5242880. Memcached server version is 1.4.26. I have a legacy system where I want to increase the memcached slab size to 5 MB. When I try to add/set object size more than 1 MB in memcached then I am getting below exception
Dec 11, 2018 12:13:31 PM com.schooner.MemCached.BinaryClient set
SEVERE: ++++ exception thrown while writing bytes to server on set
Dec 11, 2018 12:13:31 PM com.schooner.MemCached.BinaryClient set
SEVERE: Connection reset by peer
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
at sun.nio.ch.IOUtil.write(IOUtil.java:51)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:466)
at com.schooner.MemCached.SockOutputStream.writeToChannel(Unknown Source)
at com.schooner.MemCached.SockOutputStream.write(Unknown Source)
I have tried to set TransCoder similar to xmemcached client but whalin client has different implementation.
Implement WhalinMemcachedJavaClient example :
private static void implementWhalinMemcachedJavaClient() {
String[] servers = { "127.0.0.1:11211" };
SockIOPool sockIOPool = SockIOPool.getInstance("SanTest");
sockIOPool.setMinConn(2);
sockIOPool.setMaxConn(20);
sockIOPool.setServers(servers);
sockIOPool.setFailover(true);
sockIOPool.setInitConn(30);
sockIOPool.setMaintSleep(90);
sockIOPool.setSocketTO(2000);
sockIOPool.setAliveCheck(true);
sockIOPool.setHashingAlg(SockIOPool.NATIVE_HASH);
sockIOPool.initialize();
MemCachedClient whalinClient = new MemCachedClient("SanTest", true);
//MemCachedClient whalinClient = new MemCachedClient("SanTest");
whalinClient.add("Result", "Success");
System.out.println("get Result=" + whalinClient.get("Result"));
}
I would like to increase the slab size to 5MB. Please advise is there any option available in whalin memcached client to increase the slab size from 1 MB to 5 MB.

Simple Java program 100 times slower after plugging in USB hotspot

I have following Java program:
class Main {
public static void main(String[] args) throws java.io.IOException {
long start = System.nanoTime();
java.io.File.createTempFile("java_test", ".txt").delete();
System.out.println((System.nanoTime() - start ) / 1e9);
}
}
Normally, it takes bout 63 miliseconds to execute:
$ java Main
0.06308555
But, once I connect Android phone as USB hotspot, it takes significantly longer. Depending on the machine anywhere from 3 to 40 seconds:
$ java Main
4.263285528
The strange thing is that nothing here is actually transferred over the network - plugged network adapters shouldn't matter.
I did a backtrace and it looks like majority of time is spent in NetworkInterface.getAll method:
"main" #1 prio=5 os_prio=0 tid=0x00000000023ae000 nid=0x142c runnable [0x000000000268d000]
java.lang.Thread.State: RUNNABLE
at java.net.NetworkInterface.getAll(Native Method)
at java.net.NetworkInterface.getNetworkInterfaces(Unknown Source)
at sun.security.provider.SeedGenerator.addNetworkAdapterInfo(Unknown Source)
at sun.security.provider.SeedGenerator.access$000(Unknown Source)
at sun.security.provider.SeedGenerator$1.run(Unknown Source)
at sun.security.provider.SeedGenerator$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at sun.security.provider.SeedGenerator.getSystemEntropy(Unknown Source)
at sun.security.provider.SecureRandom$SeederHolder.<clinit>(Unknown Source)
at sun.security.provider.SecureRandom.engineNextBytes(Unknown Source)
- locked <0x000000076afa2820> (a sun.security.provider.SecureRandom)
at java.security.SecureRandom.nextBytes(Unknown Source)
- locked <0x000000076af6bdc8> (a java.security.SecureRandom)
at java.security.SecureRandom.next(Unknown Source)
at java.util.Random.nextLong(Unknown Source)
at java.io.File$TempDirectory.generateFile(Unknown Source)
at java.io.File.createTempFile(Unknown Source)
at java.io.File.createTempFile(Unknown Source)
at Main.main(Main.java:4)
which, in turn, seems to spend most of the time in GetIfTable Windows API method:
Child-SP RetAddr Call Site
00000000`0257ed78 000007fe`fd7210ba ntdll!NtDeviceIoControlFile+0xa
00000000`0257ed80 000007fe`fd721252 nsi+0x10ba
00000000`0257ee20 000007fe`fd7211f9 nsi!NsiEnumerateObjectsAllParametersEx+0x2e
00000000`0257ee60 000007fe`fd7217b0 nsi!NsiEnumerateObjectsAllParameters+0xc9
00000000`0257ef00 000007fe`f9c7928d nsi!NsiAllocateAndGetTable+0x184
00000000`0257efd0 00000000`6f8c5a01 IPHLPAPI!GetIfTable+0xa9
00000000`0257f090 00000000`6f8c6980 net!Java_java_net_NetworkInterface_getMTU0+0x1a1
00000000`0257f150 00000000`6f8c6e57 net!Java_java_net_NetworkInterface_isP2P0_XP+0x88
00000000`0257f270 00000000`6f8c6058 net!Java_java_net_NetworkInterface_getAll_XP+0x23
00000000`0257f2a0 00000000`02867f54 net!Java_java_net_NetworkInterface_getAll+0x2c
GetIfTable seems to be the problematic function. I'm observing the same slowdown both in example program from: https://msdn.microsoft.com/en-us/library/windows/desktop/aa365943(v=vs.85).aspx and with following snippet:
#include <iphlpapi.h>
#include <stdlib.h>
int main() {
DWORD dwSize = sizeof(MIB_IFTABLE);
MIB_IFTABLE *pIfTable = malloc(dwSize);
GetIfTable(pIfTable, &dwSize, FALSE);
pIfTable = malloc(dwSize);
GetIfTable(pIfTable, &dwSize, FALSE);
return 0;
}
How do I fix or workaround this problem?
I can create temporary files on my own and avoid calling NetworkInterface.getNetworkInterfaces but SecureRandom is used all over Java standard library.
Is there a way to force SecureRandom not to use GetIfTable?
Java version:
> java -version
java version "1.8.0_101"
Java(TM) SE Runtime Environment (build 1.8.0_101-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode)
Windows version:
OS Name: Microsoft Windows 7 Professional
OS Version: 6.1.7601 Service Pack 1 Build 7601
Problematic network adapter:
Name [00000020] Remote NDIS based Internet Sharing Device
Adapter Type Ethernet 802.3
Product Type Remote NDIS based Internet Sharing Device
Installed Yes
PNP Device ID USB\VID_0FCE&PID_71C4&MI_00\7&6BE3F3B&0&0000
Last Reset 8/14/2016 12:26 PM
Index 20
Service Name usb_rndisx
IP Address 192.168.42.183, fe80::90ab:3786:4396:2870
IP Subnet 255.255.255.0, 64
Default IP Gateway 192.168.42.129
DHCP Enabled Yes
DHCP Server 192.168.42.129
DHCP Lease Expires 8/14/2016 3:27 PM
DHCP Lease Obtained 8/14/2016 2:27 PM
MAC Address 02:18:61:77:7D:72
Driver c:\windows\system32\drivers\usb8023x.sys (6.1.7600.16385, 19.50 KB (19,968 bytes), 7/14/2009 2:09 AM)
Default implementation of SecureRandom scans network interfaces as an additional source of system entropy. In order to avoid this, you need to register a custom java.security.Provider that contains a different implementation of SecureRandomSpi.
Fortunately, JDK for Windows already has a suitable SecureRandomSpi implementation that relies on Microsoft Crypto API: sun.security.mscapi.PRNG. Though this is non-public API, the class exists in all versions of OpenJDK and Oracle JDK from 1.6 to 9, and the fallback is available anyway.
There are two ways to register MS Crypto PRNG as the default SecureRandom algorithm.
1. From within the application by calling WindowsSecureRandom.register() at the very beginning.
import java.security.Provider;
import java.security.Security;
public class WindowsSecureRandom extends Provider {
private static final String MSCAPI = "sun.security.mscapi.PRNG";
private WindowsSecureRandom() {
super("WindowsSecureRandom Provider", 1.0, null);
putService(new Service(this, "SecureRandom", "Windows-PRNG", MSCAPI, null, null));
}
public static void register() {
if (System.getProperty("os.name").contains("Windows")) {
try {
Class.forName(MSCAPI);
Security.insertProviderAt(new WindowsSecureRandom(), 1);
} catch (ClassNotFoundException e) {
// Fallback to default implementation
}
}
}
}
2. By reordering provider list in %JAVA_HOME%\jre\lib\security\java.security file.
security.provider.1=sun.security.mscapi.SunMSCAPI <<<--- make it the first provider
security.provider.2=sun.security.provider.Sun
security.provider.3=sun.security.rsa.SunRsaSign
security.provider.4=sun.security.ec.SunEC
security.provider.5=com.sun.net.ssl.internal.ssl.Provider
...
I've verified that with either solutions SeedGenerator and NetworkInterface classes are no longer loaded.

Random disconnects from master node NoNodeAvailableException using Elastic Cloud/Found

I'm using elastic cloud (former found) with shield and the transport java client. The app communicating with ES runs on heroku. I'm running a stress test on a staging environment with one node
{
"cluster_name": ...,
"status": "yellow",
"timed_out": false,
"number_of_nodes": 1,
"number_of_data_nodes": 1,
"active_primary_shards": 19,
"active_shards": 19,
"relocating_shards": 0,
"initializing_shards": 0,
"unassigned_shards": 7,
"delayed_unassigned_shards": 0,
"number_of_pending_tasks": 0,
"number_of_in_flight_fetch": 0
}
A the beginning everything works perfectly. But after some time (3-4 minutes) I begin to get some errors. I've set the log level to trace and these are the errors I've been getting (I've replaced with ... everything that is irrelevant.
org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes were available: [[...][...][...][inet[...]]{logical_availability_zone=..., availability_zone=..., max_local_storage_nodes=1, region=..., master=true}]
at org.elasticsearch.client.transport.TransportClientNodesService$RetryListener.onFailure(TransportClientNodesService.java:242)
at org.elasticsearch.action.TransportActionNodeProxy$1.handleException(TransportActionNodeProxy.java:78)
at org.elasticsearch.transport.TransportService$3.run(TransportService.java:290)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.transport.SendRequestTransportException: [...][inet[...]][indices:data/read/search]
at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:286)
at org.elasticsearch.shield.transport.ShieldClientTransportService.sendRequest(ShieldClientTransportService.java:41)
at org.elasticsearch.action.TransportActionNodeProxy.execute(TransportActionNodeProxy.java:57)
at org.elasticsearch.client.transport.support.InternalTransportClient$1.doWithNode(InternalTransportClient.java:109)
at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:205)
at org.elasticsearch.client.transport.support.InternalTransportClient.execute(InternalTransportClient.java:106)
at org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:334)
at org.elasticsearch.client.transport.TransportClient.search(TransportClient.java:416)
at org.elasticsearch.action.search.SearchRequestBuilder.doExecute(SearchRequestBuilder.java:1122)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:91)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:65)
...
Caused by: org.elasticsearch.transport.NodeNotConnectedException: [...][inet[...]] Node not connected
at org.elasticsearch.transport.netty.NettyTransport.nodeChannel(NettyTransport.java:936)
at org.elasticsearch.transport.netty.NettyTransport.sendRequest(NettyTransport.java:629)
at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:276)
...
These are my properties
settings = ImmutableSettings.settingsBuilder()
.put("client.transport.nodes_sampler_interval", "5s") //Tried it with 30s, same outcome
.put("client.transport.ping_timeout", "30s")
.put("cluster.name", clusterName)
.put("action.bulk.compress", false)
.put("shield.transport.ssl", true)
.put("request.headers.X-Found-Cluster", clusterName)
.put("shield.user", user + ":" + password)
.put("transport.ping_schedule", "1s") //Tried with 5s, same outcome
.build();
I've also set for every query I make:
max_query_response_size=100000
timeout_seconds=30
I'm using ElasticSearch 1.7.2 and Shield 1.3.2 with corresponding (same version) clients, Java 1.8.0_65 on my machine - Java 1.8.0_40 on the node.
I was getting the same errors without a stress test, but the errors happened very randomly so I wanted to reproduce. That's why I'm running this in a single node.
I spotted another error in my logs
2016-03-07 23:35:52,177 DEBUG [elasticsearch[Vermin][transport_client_worker][T#7]{New I/O worker #16}] ssl.SslHandler (NettyInternalESLogger.java:debug(63)) - Swallowing an exception raised while writing non-app data
java.nio.channels.ClosedChannelException
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:433)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:373)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Hot threads
0.0% (111.6micros out of 500ms) cpu usage by thread 'elasticsearch[...][transport_client_timer][T#1]{Hashed wheel timer #1}'
10/10 snapshots sharing following 5 elements
java.lang.Thread.sleep(Native Method)
org.elasticsearch.common.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:445)
org.elasticsearch.common.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:364)
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
java.lang.Thread.run(Thread.java:745)
After reading this http://blog.trifork.com/2015/04/08/dealing-with-nodenotavailableexceptions-in-elasticsearch/ I came to understand a little better how the whole communication works. I haven't tested this yet, but I believe that the problem lies there. The thing is though, even if I confirm that the problem is closed query connections, how do I handle this? Keep the config as is and just reconnect? Do I disable keepAlive? If yes, should I be worrying over something else?
Citing this link:
https://discuss.elastic.co/t/nonodeavailableexception-with-java-transport-client/37702 by Konrad Beiske
your application could be resolving the ip address at boot time. The
ELB can change ip's at any point in time. For the best reliability
your application should add all ip's of the ELB to the client and
periodically check the DNS service for changes.
The connection timeout of our ELB's are 5 minutes.
Following should help you fix it:
Creating a new TransportClient for every request is not ideal as it
will imply a new connection handshake for every request and this will
hurt your response time. You could have a pool of TransportClients if
you prefer, but it will most likely be an unnecessary overhead as the
client is thread safe.
My suggestion is that you create a small singleton service that
periodically checks for changes to the DNS service and adds any new
ip's to your existing transport client. In theory it could be as naive
as just adding all ip's discovered every time it checks as the
transport client will discard duplicate addresses and also purges old
addresses no longer reachable.

WebRTC java server trouble

I think I am very close to getting my Java server app to talk to a browser page via WebRTC, but I can't quite get it to work. I feel like I am missing something small, so I hope that someone here might have a suggestion.
I took a close look at the WebRTC examples - the Java unit test (org.webrtc.PeerConnectionTest), and the example Android app (trunk/talk/examples/android). Based on what I learned, I put together a java app that uses WebSockets for signalling and attempts to send a video stream to Chrome.
The problem is that there is no video in the browser, even though all of my code (both Javascript and Java) is executed in the order I expect it to, hitting all the correct logging statements. There is some suspicious output in the console log from the native libjingle code, but I am not sure what to make of it. I have highlighted suspicious lines in the log with '>>' below. For example, it seems that the video port allocators are destroyed shortly after being created, so something is obviously wrong. Also, "Changing video state, recv=1 send=0" seems incorrect as well, since the Java side should be sending the video, rather than receiving.... Perhaps I am misusing the OfferToReceiveVideo option?
If you look at the logs below, you'll see that the WebSocket communication with the browser is working perfectly, and that I am able to successfully send the SDP Offer to the browser and receive an SDP Answer from the browser. Setting the Local and Remote Descriptions on the PeerConnections seems to work properly as well. The HTML5 video element gets the source set to a BLOB url, as it should. So, what could I be missing? Do I need to do anything with ICE candidates, even though my client and server are on the same machine right now?
Any advice would be greatly appreciated!
SDP Messages (from Chrome's Javascript Console)
1.134: Java Offer:
v=0
o=- 5893945934600346864 2 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE video
a=msid-semantic: WMS JavaMediaStream
m=video 1 RTP/SAVPF 100 116 117
c=IN IP4 0.0.0.0
a=rtcp:1 IN IP4 0.0.0.0
a=ice-ufrag:dJxTlMlXy7uASrDU
a=ice-pwd:r8BRkXVnc4dqCABUDhuRjpp7
a=ice-options:google-ice
a=mid:video
a=extmap:2 urn:ietf:params:rtp-hdrext:toffset
a=extmap:3 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=sendrecv
a=rtcp-mux
a=crypto:0 AES_CM_128_HMAC_SHA1_80 inline:yq6wOHhk/QfsWuh+1oOEqfB4GjKZzz8XfQnGCDP3
a=rtpmap:100 VP8/90000
a=rtcp-fb:100 ccm fir
a=rtcp-fb:100 nack
a=rtcp-fb:100 goog-remb
a=rtpmap:116 red/90000
a=rtpmap:117 ulpfec/90000
a=ssrc:3720473526 cname:nul6R21KmwAms3Ge
a=ssrc:3720473526 msid:JavaMediaStream JavaMediaStream_v0
a=ssrc:3720473526 mslabel:JavaMediaStream
a=ssrc:3720473526 label:JavaMediaStream_v0
1.149: Received remote stream
1.150: Browsers Answer:
v=0
o=- 4261396844048664099 2 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE video
a=msid-semantic: WMS
m=video 1 RTP/SAVPF 100 116 117
c=IN IP4 0.0.0.0
a=rtcp:1 IN IP4 0.0.0.0
a=ice-ufrag:quzQNsX+ZlUWUQqV
a=ice-pwd:y5A0+7sM8P88AatBLd1fdd5G
a=mid:video
a=extmap:2 urn:ietf:params:rtp-hdrext:toffset
a=extmap:3 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=recvonly
a=rtcp-mux
a=crypto:0 AES_CM_128_HMAC_SHA1_80 inline:WClNA69OfpjdJy3Bv4ujejk/IYnn4DW8kjrB18xP
a=rtpmap:100 VP8/90000
a=rtcp-fb:100 ccm fir
a=rtcp-fb:100 nack
a=rtcp-fb:100 goog-remb
a=rtpmap:116 red/90000
a=rtpmap:117 ulpfec/90000
This seems ok to me. Java's offer includes my video stream.
Native code logging (libjingle)
(Suspicious lines marked with '>>')
Camera '/dev/video0' started with format YUY2 640x480x30, elapsed time 59 ms
Ignored line: c=IN IP4 0.0.0.0
NACK enabled for channel 0
NACK enabled for channel 0
Created channel for video
Jingle:Channel[video|1|__]: NULL DTLS identity supplied. Not doing DTLS
Jingle:Channel[video|2|__]: NULL DTLS identity supplied. Not doing DTLS
Session:5893945934600346864 Old state:STATE_INIT New state:STATE_SENTINITIATE Type:urn:xmpp:jingle:apps:rtp:1 Transport:http://www.google.com/transport/p2p
Setting local video description
AddSendStream {id:JavaMediaStream_v0;ssrcs:[3720473526];ssrc_groups:;cname:nul6R21KmwAms3Ge;sync_label:JavaMediaStream}
Add send ssrc: 3720473526
>> Warning(webrtcvideoengine.cc:2704): SetReceiverBufferingMode(0, 0) failed, err=12606
Changing video state, recv=0 send=0
Transport: video, allocating candidates
Transport: video, allocating candidates
Jingle:Net[eth0:192.168.0.0/24]: Allocation Phase=Udp
Jingle:Port[:1:0::Net[eth0:192.168.0.0/24]]: Port created
Adding allocated port for video
Jingle:Port[video:1:0::Net[eth0:192.168.0.0/24]]: Added port to allocator
Jingle:Net[tun0:192.168.128.6/32]: Allocation Phase=Udp
Jingle:Port[:1:0::Net[tun0:192.168.128.6/32]]: Port created
Adding allocated port for video
Jingle:Port[video:1:0::Net[tun0:192.168.128.6/32]]: Added port to allocator
Ignored line: c=IN IP4 0.0.0.0
Warning(webrtcvideoengine.cc:2309): GetStats: sender information not ready.
Jingle:Channel[video|1|__]: Other side didn't support DTLS.
Jingle:Channel[video|2|__]: Other side didn't support DTLS.
Enabling BUNDLE, bundling onto transport: video
Channel enabled
>> Changing video state, recv=1 send=0
Session:5893945934600346864 Old state:STATE_SENTINITIATE New state:STATE_RECEIVEDACCEPT Type:urn:xmpp:jingle:apps:rtp:1 Transport:http://www.google.com/transport/p2p
Setting remote video description
Hybrid NACK/FEC enabled for channel 0
Hybrid NACK/FEC enabled for channel 0
SetSendCodecs() : selected video codec VP8/1280x720x30fps#2000kbps (min=50kbps, start=300kbps)
Video max quantization: 56
VP8 number of temporal layers: 1
VP8 options : picture loss indication = 0, feedback mode = 0, complexity = normal, resilience = off, denoising = 0, error concealment = 0, automatic resize = 0, frame dropping = 1, key frame interval = 3000
WARNING: no real random source present!
SRTP activated with negotiated parameters: send cipher_suite AES_CM_128_HMAC_SHA1_80 recv cipher_suite AES_CM_128_HMAC_SHA1_80
Changing video state, recv=1 send=0
Session:5893945934600346864 Old state:STATE_RECEIVEDACCEPT New state:STATE_INPROGRESS Type:urn:xmpp:jingle:apps:rtp:1 Transport:http://www.google.com/transport/p2p
Jingle:Net[eth0:192.168.0.0/24]: Allocation Phase=Relay
Jingle:Net[tun0:192.168.128.6/32]: Allocation Phase=Relay
Jingle:Net[eth0:192.168.0.0/24]: Allocation Phase=Tcp
Jingle:Port[:1:0:local:Net[eth0:192.168.0.0/24]]: Port created
Adding allocated port for video
Jingle:Port[video:1:0:local:Net[eth0:192.168.0.0/24]]: Added port to allocator
Jingle:Net[tun0:192.168.128.6/32]: Allocation Phase=Tcp
Jingle:Port[:1:0:local:Net[tun0:192.168.128.6/32]]: Port created
Adding allocated port for video
Jingle:Port[video:1:0:local:Net[tun0:192.168.128.6/32]]: Added port to allocator
Jingle:Net[eth0:192.168.0.0/24]: Allocation Phase=SslTcp
Jingle:Net[tun0:192.168.128.6/32]: Allocation Phase=SslTcp
All candidates gathered for video:1:0
Transport: video, component 1 allocation complete
Transport: video allocation complete
Candidate gathering is complete.
Capture delay changed to 120 ms
Captured frame size 640x480. Expected format YUY2 640x480x30
Capture size changed : selected video codec VP8/640x480x30fps#2000kbps (min=50kbps, start=300kbps)
Video max quantization: 56
VP8 number of temporal layers: 1
VP8 options : picture loss indication = 0, feedback mode = 0, complexity = normal, resilience = off, denoising = 0, error concealment = 0, automatic resize = 1, frame dropping = 1, key frame interval = 3000
VAdapt Frame: 0 / 300 Changes: 0 Input: 640x480 Scale: 1 Output: 640x480 Changed: false
>> Jingle:Port[video:1:0::Net[eth0:192.168.0.0/24]]: Port deleted
>> Jingle:Port[video:1:0::Net[eth0:192.168.0.0/24]]: Removed port from allocator (3 remaining)
Removed port from p2p socket: 3 remaining
Jingle:Port[video:1:0::Net[tun0:192.168.128.6/32]]: Port deleted
Jingle:Port[video:1:0::Net[tun0:192.168.128.6/32]]: Removed port from allocator (2 remaining)
Removed port from p2p socket: 2 remaining
>> Jingle:Port[video:1:0:local:Net[eth0:192.168.0.0/24]]: Port deleted
>> Jingle:Port[video:1:0:local:Net[eth0:192.168.0.0/24]]: Removed port from allocator (1 remaining)
Removed port from p2p socket: 1 remaining
Jingle:Port[video:1:0:local:Net[tun0:192.168.128.6/32]]: Port deleted
Jingle:Port[video:1:0:local:Net[tun0:192.168.128.6/32]]: Removed port from allocator (0 remaining)
Removed port from p2p socket: 0 remaining
HTML
<html lang="en">
<head>
<title>Web Socket Signalling</title>
<link rel="stylesheet" href="css/socket.css">
<script src="js/socket.js"></script>
</head>
<body>
<h2>Repsonse from Server</h2>
<textarea id="responseText"></textarea>
<h2>Video</h2>
<video id="remoteVideo" autoplay></video>
</body>
</html>
Javascript
(function() {
var remotePeerConnection;
var sdpConstraints = {
'mandatory' : {
'OfferToReceiveAudio' : false,
'OfferToReceiveVideo' : true
}
};
var Sock = function() {
var socket;
if (!window.WebSocket) {
window.WebSocket = window.MozWebSocket;
}
if (window.WebSocket) {
socket = new WebSocket("ws://localhost:8080/websocket");
socket.onopen = onopen;
socket.onmessage = onmessage;
socket.onclose = onclose;
} else {
alert("Your browser does not support Web Socket.");
}
function onopen(event) {
getTextAreaElement().value = "Web Socket opened!";
}
function onmessage(event) {
appendTextArea(event.data);
sdpOffer = new RTCSessionDescription(JSON.parse(event.data));
remotePeerConnection = new webkitRTCPeerConnection(null);
remotePeerConnection.onaddstream = gotRemoteStream;
trace("Java Offer: \n" + sdpOffer.sdp);
remotePeerConnection.setRemoteDescription(sdpOffer);
remotePeerConnection.createAnswer(gotRemoteDescription, onCreateSessionDescriptionError, sdpConstraints);
}
function onCreateSessionDescriptionError(error) {
console.log('Failed to create session description: '
+ error.toString());
}
function gotRemoteDescription(answer) {
remotePeerConnection.setLocalDescription(answer);
trace("Browser's Answer: \n" + answer.sdp);
socket.send(JSON.stringify(answer));
}
function gotRemoteStream(event) {
var remoteVideo = document.getElementById("remoteVideo");
remoteVideo.src = URL.createObjectURL(event.stream);
trace("Received remote stream");
}
function onclose(event) {
appendTextArea("Web Socket closed");
}
function appendTextArea(newData) {
var el = getTextAreaElement();
el.value = el.value + '\n' + newData;
}
function getTextAreaElement() {
return document.getElementById('responseText');
}
function trace(text) {
console.log((performance.now() / 1000).toFixed(3) + ": " + text);
}
}
window.addEventListener('load', function() {
new Sock();
}, false);
})();
Java Server
public class PeerConnectionManager {
/**
* Called when the WebSocket handshake is completed
*/
public void createOffer() {
peerConnection = factory.createPeerConnection(
new ArrayList<PeerConnection.IceServer>(),
new MediaConstraints(),
new PeerConnectionObserverImpl());
// Get the video source
videoSource = factory.createVideoSource(VideoCapturer.create(""), new MediaConstraints());
// Create a MediaStream with one video track
MediaStream lMS = factory.createLocalMediaStream("JavaMediaStream");
VideoTrack videoTrack = factory.createVideoTrack("JavaMediaStream_v0", videoSource);
videoTrack.addRenderer(new VideoRenderer(new VideoRendererObserverImpl()));
lMS.addTrack(videoTrack);
peerConnection.addStream(lMS, new MediaConstraints());
// We don't want to receive anything
MediaConstraints sdpConstraints = new MediaConstraints();
sdpConstraints.mandatory.add(new MediaConstraints.KeyValuePair(
"OfferToReceiveAudio", "false"));
sdpConstraints.mandatory.add(new MediaConstraints.KeyValuePair(
"OfferToReceiveVideo", "false"));
// Get the Offer SDP
SdpObserverImpl sdpOfferObserver = new SdpObserverImpl();
peerConnection.createOffer(sdpOfferObserver, sdpConstraints);
SessionDescription offerSdp = sdpOfferObserver.getSdp();
// Set local SDP, don't care for any callbacks
peerConnection.setLocalDescription(new SdpObserverImpl(), offerSdp);
// Serialize Offer and send to the Browser via a WebSocket
JSONObject offerSdpJson = new JSONObject();
offerSdpJson.put("sdp", offerSdp.description);
offerSdpJson.put("type", offerSdp.type.canonicalForm());
webSocketContext.channel().writeAndFlush(
new TextWebSocketFrame(offerSdpJson.toString()));
}
/**
* Called when an SDP Answer arrives via the WebSocket
*/
public void setRemoteDescription(SessionDescription answer) {
peerConnection.setRemoteDescription( new SdpObserverImpl(), answer);
}
}
Ugh. Never mind. Sorry for the stupid question.
The missing part was the exchange of ICE candidates between the browser and Java server. Now that I added the code to do ICE negotiation via my WebSocket, everything works fine!

Categories

Resources