Develop Reference

BigQuery not picking up property set in java

I am trying to connect to BigQuery on a machine that requires all traffic to go through a proxy. I've set the http.proxyHost and http.proxyPort system variable in both Java and command line but I always get the following error:
com.google.cloud.bigquery.BigQueryException: Error getting access token for service account: Read timed out
at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.translate(HttpBigQueryRpc.java:106)
at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.create(HttpBigQueryRpc.java:206)
at com.google.cloud.bigquery.BigQueryImpl$5.call(BigQueryImpl.java:319)
at com.google.cloud.bigquery.BigQueryImpl$5.call(BigQueryImpl.java:316)
at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105)
at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
at com.google.cloud.bigquery.BigQueryImpl.create(BigQueryImpl.java:315)
at com.google.cloud.bigquery.BigQueryImpl.create(BigQueryImpl.java:290)
at com.ad.google.bigquery.GoogleBigQuery.runGBQQuery_CCPA(GoogleBigQuery.java:172)
at com.ad.gogbq.test.Tester_GBQ.runQuery(Tester_GBQ.java:62)
at com.ad.gogbq.test.Tester_GBQ.main(Tester_GBQ.java:274)
Caused by: java.io.IOException: Error getting access token for service account: Read timed out
at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:432)
at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:157)
at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:145)
at com.google.auth.http.HttpCredentialsAdapter.initialize(HttpCredentialsAdapter.java:91)
at com.google.cloud.http.HttpTransportOptions$1.initialize(HttpTransportOptions.java:159)
at com.google.api.client.http.HttpRequestFactory.buildRequest(HttpRequestFactory.java:88)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.buildHttpRequest(AbstractGoogleClientRequest.java:430)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:549)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:482)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:599)
at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.create(HttpBigQueryRpc.java:204)
... 10 more
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
at sun.security.ssl.InputRecord.read(InputRecord.java:503)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397)
at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1334)
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1309)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:259)
at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:108)
at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:79)
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:995)
at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:429)
I have successfully used the above system variables to open and read a web page on the machines in question. It seems like BigQuery isn't using these variables, I was wondering if anyone else has seen this?

In your question, it seems you were setting http.* properties, since all Google API access are through HTTPS, please make sure you set below 2 properties:
System.setProperty("https.proxyHost", "localhost");
System.setProperty("https.proxyPort", "3128");
See more on this doc.

Getting error while pushing data remotely into Azure Cosmos db via gremlin java client

I am trying to parse a CSV file and create nodes and edges into Cosmos db via the help of gremlin-java client ( https://github.com/Azure-Samples/azure-cosmos-db-graph-java-getting-started ).
Some queries I have changed in this program:
For adding data :
Submitting this Gremlin 87246 query:
g.V().has('id','1120-L CONSOLIDATED SCHEDULE L').fold().coalesce(unfold(), addV('sub_schedule').property('id', '1120-L CONSOLIDATED SCHEDULE L').property('name', '1120-L CONSOLIDATED SCHEDULE L'))
For Creating edge:
g.V().hasLabel('parent').has('id', '1120').addE('has').to(g.V().hasLabel('state').has('id', 'FD'))
After running it almost for 6 hours successfully it gives weird error:
Exception in thread "main" java.util.concurrent.ExecutionException: java.io.IOException: An existing connection was forcibly closed by the remote host
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
at GetStarted.WriteWithOriginalCSVData.main(WriteWithOriginalCSVData.java:158)
Caused by: java.io.IOException: An existing connection was forcibly closed by the remote host
at sun.nio.ch.SocketDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:221)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:899)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:275)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:652)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:575)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:489)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:451)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140)
at java.lang.Thread.run(Thread.java:745)
Can somebody please tell me the possible causes for this?
To mention I am invoking this java client remotely from my laptop via open connection and hitting azure services ( in different location ).
Firstly I thought it might be because of my internet connectivity . But it exactly throws error when it has completed some 87-88k gremlin queries.
Can someone please let me know where I am going wrong here?

Java error - Software caused connection abort: recv failed

The full error is:
java.net.SocketException: Software caused connection abort: recv failed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:168)
at java.net.SocketInputStream.read(SocketInputStream.java:121)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at java.io.BufferedReader.fill(BufferedReader.java:154)
at java.io.BufferedReader.readLine(BufferedReader.java:317)
at java.io.BufferedReader.readLine(BufferedReader.java:382)
at chat.run(chat.java:76)
at java.lang.Thread.run(Thread.java:722)
"at chat.run(chat.java:76)" is this line:
System.out.println("[_in_"+(line=ins.readLine())+"_]");
line being a string and ins being a BufferedReader
ins = new BufferedReader(new InputStreamReader(_sock.getInputStream()));
_sock being
Socket _sock = null;
_sock = serv_sock.accept();
This happends on my Server java and not on my client java...
Also, this happends when I dont send anything for a little while
-----EDIT------
I tried it again;
First it went 40 minutes of doing nothing and it still worked,
then I waited 40 minutes again and this time it didn't work.
It seems to be kind of random..
I can't figure out what's causing it.
The files I have are
* [client.java], [chat.java], [vlc.java]
* [server.java], [chat.java], [vlc.java]
the vlc files are the same in both although chat is different
(since I havn't done Threads in both client and server yet)
Here's the code;
SERVER
server.java: pastebin.com/GH8ShcGp
chat.java : pastebin.com/iaL23kSb
vlc.java : pastebin.com/9kyrbh5q
CLIENT
client.java: pastebin.com/HDK450Jg
chat.java : pastebin.com/CfHrEUkE
vlc.java : pastebin.com/SfZgYy58
I think it's a window/network thing.
Someone who please can help me fix this??

I couldn't get your exact problem to replicate on my own computer using your code and localhost-connections, but after reading about "BufferedReader.readLine() throwing java.net.SocketException: Software caused connection abort: recv failed" in several different pages, I believe this is related to your network. The connection seems to be terminated due to TCP timeout or data corrupted in the transmission.
If this happens over localhost-connections (server and client in same computer), it could be a faulty memory, otherwise you might have a broken nic, router or cable somewhere along the line. If you're using wireless, they can be pretty unreliable.

This error occurs exactly due to problem in network. It comes when connection with your database failed and and application is unable to retrieve the required data. The problem may be in your connection wire or with the modem. Try changing them. Otherwise check your database access in sql developer etc if its working fine, there could be a problem in transaction timeout.

There are two possibilities:
The database is down, or
The database server's IP address duplicates an existing server's IP address on the network

java.net.SocketException: Software caused connection abort: recv failed, with java.net.SocketException: Connection reset [duplicate]

This question already has answers here:
Official reasons for "Software caused connection abort: socket write error"
(14 answers)
Closed 5 years ago.
I'm getting an exception in my Java, Hibernate, MySQL project within Netbeans IDE.
I've done some digging to try and fix this strange Socket exception, to no avail.
Some forum solutions suggested implementing a c3p0 connection pool for Hibernate. No fix.
Others suggested disabling AV and firewall (!!) as these could interfere with the socket connection. No fix.
I've tagged this question with Hibernate, as I'm not certain that hibernate is the cause, but the exceptions occur on a Hibernate call (running a query). Other Hibernate code in the app works fine. The Hibernate version is 3.2 .
The exception:
NotifyUtil::java.net.SocketException: Software caused connection abort: recv failed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
at java.io.FilterInputStream.read(FilterInputStream.java:116)
at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:2676)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)
at java.io.InputStreamReader.read(InputStreamReader.java:167)
at java.io.BufferedReader.fill(BufferedReader.java:136)
at java.io.BufferedReader.readLine(BufferedReader.java:299)
at java.io.BufferedReader.readLine(BufferedReader.java:362)
at org.netbeans.modules.web.monitor.server.NotifyUtil$RecordSender.run(NotifyUtil.java:299)
This exception may occur once, or a few times, and be followed by none or a few of this exception:
NotifyUtil::java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:168)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
at java.io.FilterInputStream.read(FilterInputStream.java:116)
at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:2676)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)
at java.io.InputStreamReader.read(InputStreamReader.java:167)
at java.io.BufferedReader.fill(BufferedReader.java:136)
at java.io.BufferedReader.readLine(BufferedReader.java:299)
at java.io.BufferedReader.readLine(BufferedReader.java:362)
at org.netbeans.modules.web.monitor.server.NotifyUtil$RecordSender.run(NotifyUtil.java:299)
Help would be much appreciated. This is a bit of a strange one.
Thanks.

In my case, I was using Tomcat. Right click on Tomcat in Server list and then click properties, then uncheck "Enable HTTP Monitor". This solve my issues

There's a Microsoft Knowledge Base article about this, see if you can find it. Basically this is a result of prior errors writing to the network by the peer that gets the exception. Indicates a network problem rather than software.

It is a network error!
1.If you run your sparkstreaming on windows.just run:
ncat.exe -lk 7777
and download ncat.exe at https://nmap.org/ncat/
2.If you run your sparkstreaming on linux.just run:
nc -lk 7777
and make sure you have install on your linux

java.net.SocketException: Broken pipe

I am getting this for all the database connections from my app server..
This exception occured for couple of hours, then got fixed by itself.
Something to do with network connection from the appserver?
java.net.SocketException: Broken pipe
com.inet.tds.SQLException: java.net.SocketException: Broken pipe
java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at com.inet.tds.a.a(Unknown Source)
at com.inet.tds.a.a(Unknown Source)
at com.inet.tds.a.commit(Unknown Source)
at com.inet.pool.a.commit(Unknown Source)

For MySQL, "By default, the server closes the connection after eight hours if nothing has happened." And, MySQL has a reconnect feature that supports auto-reconnect after the closed connection is detected on the client side.
Eight hours? With the use of connection pooling or a long-running background job, that is possible.
http://dev.mysql.com/doc/refman/5.0/en/gone-away.html

It means you client has disconnected from the server; check if it is running. See here

During a write the connection was severed, this can be due the the destination closing the connection or the destination process has terminated. Its not an error with your implementation.