I am writing following program to download the URL using Apache Common-IO and I am getting ReadTimeOut exception,
Exception
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at sun.security.ssl.InputRecord.readFully(Unknown Source)
at sun.security.ssl.InputRecord.read(Unknown Source)
at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
at sun.security.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
at sun.security.ssl.AppInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read1(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
at sun.net.www.http.HttpClient.parseHTTPHeader(Unknown Source)
at sun.net.www.http.HttpClient.parseHTTP(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(Unknown Source)
at java.net.URL.openStream(Unknown Source)
at org.apache.commons.io.FileUtils.copyURLToFile(FileUtils.java:1456)
at com.touseef.stock.FileDownload.main(FileDownload.java:23)
Program
String urlStr = "https://www.nseindia.com/";
File file = new File("C:\\User\\WorkSpace\\Output.txt");
URL url;
try {
url = new URL(urlStr);
FileUtils.copyURLToFile(url, file);
System.out.println("Successfully Completed.");
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
Other site are able to download. Please suggest.
Using commons-io-2.6 jar.
It seems like this site is protected by some web gateway (DOS protection service like Akamai?). Clients seem to be fingerprinted by TLS connection and the HTTP request (headers) and only valid web browsers can connect to the site.
The following code uses Apache commons http client 4.5 and works at least at the moment:
String urlStr = "https://www.nseindia.com/";
File file = new File("C:\\User\\WorkSpace\\Output.txt");
String userAgent = "-";
CloseableHttpClient httpclient = HttpClients.custom().setUserAgent(userAgent).build();
HttpGet httpget = new HttpGet(urlStr);
httpget.addHeader("Accept-Language", "en-US");
httpget.addHeader("Cookie", "");
System.out.println("Executing request " + httpget.getRequestLine());
try (CloseableHttpResponse response = httpclient.execute(httpget)) {
System.out.println("----------------------------------------");
System.out.println(response.getStatusLine());
String body = EntityUtils.toString(response.getEntity());
System.out.println(body);
Files.writeString(file.toPath(), body);
}
A request that e.g works from within Firefox does not work from Java (because the TLS connection with protocols and ciphers is different). I tried a few combinations using Apache commons http client. but is also fails (even though the same request works from Fiddler).
Hence using this web site from within Java is extremely difficult and even the code above works at the moment, the protection system can be adapted at any time so that it won't work again.
I would assume that such a site provides an API dedicated for program usage. Contact them and ask, that is the only advice I can give to you.
Related
I have run into an issue developing a HTTP client with the use of BouncyCastle libraries.
Target versions (but the error is also reproducible in Java 1.8.0_91 with the same version of BouncyCastle.)
JRE 1.6.0_45-b06
BouncyCastle jdk15to18 167 (bcprov-jdk15to18-167.jar, bcpkix-jdk15to18-167.jar, bctls-jdk15to18-167.jar)
HTTPClient code:
String strURL = "https://www.<WEBSITE>.com";
// CODE to set default TrustStore, KeyStore to be used
// setup BC as SecurityProvider and SSLSocketFactoryProvider
/*
Security.insertProviderAt(new BouncyCastleProvider(), 1);
Security.insertProviderAt(new BouncyCastleJsseProvider(), 2);
Security.setProperty("ssl.KeyManagerFactory.algorithm", "PKIX");
Security.setProperty("ssl.TrustManagerFactory.algorithm", "PKIX");
Security.setProperty("ssl.SocketFactory.provider", "org.bouncycastle.jsse.provider.SSLSocketFactoryImpl");
System.setProperty("jdk.tls.trustNameService", "true");
*/
URL url = new URL( strURL );
HttpsURLConnection conn = (HttpsURLConnection) url.openConnection();
conn.setDoOutput(true);
conn.setRequestMethod("GET");
InputStream is = null;
if (conn.getResponseCode() == HttpURLConnection.HTTP_OK) {
System.out.println("OK");
is = conn.getInputStream();
} else if (conn.getResponseCode() == HttpURLConnection.HTTP_INTERNAL_ERROR) {
System.out.println("ERROR");
is = conn.getErrorStream();
} else {
System.out.println( conn.getResponseCode() );
System.out.println( conn.getResponseMessage() );
}
if (is != null) {
System.out.println( readFullyAsString(is, "UTF-8") );
}
conn.disconnect();
Accessing data on a website with TLS 1.2 without client_auth works fine (there are some issues with specific sites that return handshake_failure(40), but luckily not in our case). But when client_auth is required the code fails with the following (a little cryptic) error:
nov. 27, 2020 5:23:10 PM org.bouncycastle.jsse.provider.ProvTlsClient notifyAlertRaised
WARNING: Client raised fatal(2) internal_error(80) alert: Failed to read record
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at org.bouncycastle.tls.RecordStream$Record.fillTo(RecordStream.java:429)
at org.bouncycastle.tls.RecordStream$Record.readHeader(RecordStream.java:468)
at org.bouncycastle.tls.RecordStream.readRecord(RecordStream.java:201)
at org.bouncycastle.tls.TlsProtocol.safeReadRecord(TlsProtocol.java:768)
at org.bouncycastle.tls.TlsProtocol.readApplicationData(TlsProtocol.java:731)
at org.bouncycastle.jsse.provider.ProvSSLSocketDirect$AppDataInput.read(ProvSSLSocketDirect.java:603)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read1(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
at sun.net.www.http.HttpClient.parseHTTPHeader(Unknown Source)
at sun.net.www.http.HttpClient.parseHTTP(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
at java.net.HttpURLConnection.getResponseCode(Unknown Source)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(Unknown Source)
at HttpsClient.main(HttpsClient.java:109)
nov. 27, 2020 5:23:10 PM org.bouncycastle.jsse.provider.ProvTlsClient notifyAlertRaised
WARNING: Client raised fatal(2) internal_error(80) alert: Failed to read record
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at org.bouncycastle.tls.RecordStream$Record.fillTo(RecordStream.java:429)
at org.bouncycastle.tls.RecordStream$Record.readHeader(RecordStream.java:468)
at org.bouncycastle.tls.RecordStream.readRecord(RecordStream.java:201)
at org.bouncycastle.tls.TlsProtocol.safeReadRecord(TlsProtocol.java:768)
at org.bouncycastle.tls.TlsProtocol.readApplicationData(TlsProtocol.java:731)
at org.bouncycastle.jsse.provider.ProvSSLSocketDirect$AppDataInput.read(ProvSSLSocketDirect.java:603)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read1(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
at sun.net.www.http.HttpClient.parseHTTPHeader(Unknown Source)
at sun.net.www.http.HttpClient.parseHTTP(Unknown Source)
at sun.net.www.http.HttpClient.parseHTTP(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
at java.net.HttpURLConnection.getResponseCode(Unknown Source)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(Unknown Source)
at HttpsClient.main(HttpsClient.java:109)
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at org.bouncycastle.tls.RecordStream$Record.fillTo(RecordStream.java:429)
at org.bouncycastle.tls.RecordStream$Record.readHeader(RecordStream.java:468)
at org.bouncycastle.tls.RecordStream.readRecord(RecordStream.java:201)
at org.bouncycastle.tls.TlsProtocol.safeReadRecord(TlsProtocol.java:768)
at org.bouncycastle.tls.TlsProtocol.readApplicationData(TlsProtocol.java:731)
at org.bouncycastle.jsse.provider.ProvSSLSocketDirect$AppDataInput.read(ProvSSLSocketDirect.java:603)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read1(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
at sun.net.www.http.HttpClient.parseHTTPHeader(Unknown Source)
at sun.net.www.http.HttpClient.parseHTTP(Unknown Source)
at sun.net.www.http.HttpClient.parseHTTP(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
at java.net.HttpURLConnection.getResponseCode(Unknown Source)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(Unknown Source)
at HttpsClient.main(HttpsClient.java:109)
As expected the code runs just fine in Java 1.8 without BC as Security Provider ... but in Java 1.6 BC is needed for TLS 1.2 support.
I looked at the packets with Wireshark, but I don't have enough knowledge to analyze what's happening (failing) with the SSL handshake operation with the server.
What am I missing here? Or it's just a known limitation of BC?
Thanks for any suggestions.
After extensive research and testing (we set up our own dev environment that replicated the endpoint, we were trying to connect to with client cert) we have found a solution (posted below) that works for us in our dev environment, but couldn't be configured on the endpoint. That's why we went to plan B - TLS termination proxy and it worked like a charm in our case, when we didn't have control of the server environment.
Solution for IIS
In our case the web service requiring client auth was running on Microsoft IIS. As stated "Unfortunately our TLS libraries do not support renegotiation (and we do not plan to add it, although there are corresponding TLS 1.3 features that we will)." in this open issue (https://github.com/bcgit/bc-java/issues/593). After reading the IIS documentation and some testing we successfully configured the environment in a way that TLS sessions could be established with BC TLS lib:
Step 1:
Create an IIS site with default settings
Run IISCrypto with the "Best practices" template:
Step 2:
Enable SSLAlwaysNegoClientCert
Save the following text to a file called "Enable_SSL_Renegotiate.js"
var vdirObj=GetObject("IIS://localhost/W3svc/1");
// replace 1 on this line with the number of the web site you wish to configure
WScript.Echo("Value of SSLAlwaysNegoClientCert Before: " + vdirObj.SSLAlwaysNegoClientCert);
vdirObj.Put("SSLAlwaysNegoClientCert", true);
vdirObj.SetInfo();
WScript.Echo("Value of SSLAlwaysNegoClientCert After: " + vdirObj.SSLAlwaysNegoClientCert);
Run the following command from an elevated / administrator command prompt:
cscript.exe enable_ssl_renegotiate.js
I posted the same answer on BouncyCastle Github - #847
I'm running into a java.net.SocketException (Permission denied: connect) when sending a lot of requests to a server. I've tried the -Djava.net.preferIPv4Stack=true option as mentioned in other threads.
This issue only occurs after a lot of connections have already been made. The following code can be used to reproduce the issue:
public class Example {
public static void main(String[] args) {
if(args.length == 1) {
System.out.println(args[0]);
for(int i = 0; i < Integer.MAX_VALUE; i++) {
requestURL(args[0]);
}
}
}
public static void requestURL(String targetUrl) {
URL url = new URL(targetUrl);
HttpURLConnection httpCon = (HttpURLConnection)url.openConnection();
httpCon.setDoInput(true);
BufferedReader rd = new BufferedReader(new InputStreamReader(httpCon.getInputStream()));
//handle response here
rd.close();
httpCon.disconnect();
}
}
The code will successfully send the requests until the Exception occurs. This has been verified in the server's log files.
Please run the sample code only against a server that you are permitted to, as it generates quite a bit of traffic / load on the server.
So the question now is: How to avoid that Exception? Or any workarounds? And what is the actually issue here?
EDIT
added httpCon.disconnect(). (doesn't solve the problem)
EDIT #2 (StackTrace)
java.net.SocketException: Permission denied: connect
at java.net.DualStackPlainSocketImpl.connect0(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(Unknown Source)
at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
at java.net.AbstractPlainSocketImpl.connect(Unknown Source)
at java.net.PlainSocketImpl.connect(Unknown Source)
at java.net.SocksSocketImpl.connect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at sun.net.NetworkClient.doConnect(Unknown Source)
at sun.net.www.http.HttpClient.openServer(Unknown Source)
at sun.net.www.http.HttpClient.openServer(Unknown Source)
at sun.net.www.http.HttpClient.<init>(Unknown Source)
at sun.net.www.http.HttpClient.New(Unknown Source)
at sun.net.www.http.HttpClient.New(Unknown Source)
at sun.net.protocol.http.HttpURLConnection.getNewHttpClient(Unknown Source)
at sun.net.protocol.http.HttpURLConnection.plainConnect0(Unknown Source)
at sun.net.protocol.http.HttpURLConnection.plainConnect(Unknown Source)
at sun.net.protocol.http.HttpURLConnection.connect(Unknown Source)
at Example.requestUrl(Example.java)
at Example.main(Example.java)
When the issue occurs no further requests are sent to the server (no log entries on the server and no SYN packets (sniffed with tcpdump)).
EDIT #3
I've forgotten to run the application with the -Djava.net.preferIPv4Stack=true argument.
It hasn't thrown an exception yet when using both, the aforementioned argument and the httpCon.disconnect() method.
Try to use httpCon.disconnect() to be sure to realease the socket after you finish to read the response.
I'm trying to send some JSON notifications to the GCM servers and I'm getting this list of errors
Response Code: 400
java.io.IOException: Server returned HTTP response code: 400 for URL: https://gcm-http.googleapis.com/gcm/send
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection$10.run(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection$10.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at sun.net.www.protocol.http.HttpURLConnection.getChainedException(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(Unknown Source)
at controller.Notification.post(Notification.java:181)
at controller.App.main(App.java:17)
Caused by: java.io.IOException: Server returned HTTP response code: 400 for URL: https://gcm-http.googleapis.com/gcm/send
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
at java.net.HttpURLConnection.getResponseCode(Unknown Source)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(Unknown Source)
at controller.Notification.post(Notification.java:179)
... 1 more
Weird thing is, this only happens if I try to send to a single device. it works if I try to send to multiple devices, using the registration_id field. The reference says error 400 means the JSON couldn't be parsed, but I tried printing it out and it seems fine:
Multicast sending: (response code: 200, invalid registration)
{"registration_ids":["asdlfiwuhfs"]}
Sending to single device: (responce code: 400, errors above)
{"to":"asdlfiwuhfs"}
The functions to send to a single or multiple devices have only one different line. One of them adds the to field, the other a registration_id field.
obj.put("registration_ids", regIds); // multiple devices
obj.put("to", regId); // single device
What could be the problem?
Edit:
The third line below is line 181 mentioned in the errors:
int responseCode = conn.getResponseCode();
System.out.println("Response Code: " + responseCode);
BufferedReader in = new BufferedReader(new InputStreamReader(conn.getInputStream())); // 181
I found the solution looking at this code on github.
It seems getInputStream() is only used when the response code is 200. Otherwise, getErrorStream() should be used.
So changing my code to this fixed it.
int responseCode = conn.getResponseCode();
System.out.println("Response Code: " + responseCode);
BufferedReader in = null;
if (responseCode == 200)
in = new BufferedReader(new InputStreamReader(conn.getInputStream()));
else
in = new BufferedReader(new InputStreamReader(conn.getErrorStream()));
I am using HttpClient to connect to some third party url.Its working fine.but some times its giving java.net.UnknownHostException and also connection Reset Excetion. Please help to avoid the exception.
Below is my code snippet.
PostMethod post = null;
try
{
for(int i =0 ; i < 72 ; i++)
{
String url ="http://www.accuweather.com/en/in/amritsar/205593//hourly-weather-forecast/205593?hour=0";
post = new PostMethod(url);
post.setRequestHeader("Content-type", "application/x-www-form-urlencoded;charset=" + "UTF-8");
post.setRequestHeader("Accept-Charset", "UTF-8");
HttpClient httpclient = new HttpClient();
httpclient.getParams().setParameter(CoreConnectionPNames.CONNECTION_TIMEOUT, 120000);
httpclient.getParams().setParameter(CoreConnectionPNames.SO_TIMEOUT, 120000);
result = httpclient.executeMethod(post);
}
}
catch(Exception e){}
finaly
{
post.releaseConnection();
}
Belowe is the exception trace:
java.net.UnknownHostException: www.accuweather.com
at java.net.AbstractPlainSocketImpl.connect(Unknown Source)
at java.net.PlainSocketImpl.connect(Unknown Source)
at java.net.SocksSocketImpl.connect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at java.net.Socket.<init>(Unknown Source)
at java.net.Socket.<init>(Unknown Source)
at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:80)
at org.apache.commons.httpclient.protocol.ControllerThreadSocketFactory$1.doit(ControllerThreadSocketFactory.java:91)
at org.apache.commons.httpclient.protocol.ControllerThreadSocketFactory$SocketTask.run(ControllerThreadSocketFactory.java:158)
at java.lang.Thread.run(Unknown Source)
More than likely, you just have a flaky Internet connection. I'm guessing this is on a mobile phone, in which case flaky Internet connections are a common reality, and you'll just have to deal with it in some way.
Exactly what is the best way to deal with it depends on your program and context, of course. Since you state you run this regularly, one way to deal with it could be to simply ignore an error (catch the exception and just silently abort the attempt if it happens) and then wait for the next regular run. Another option could be to repeat the attempt a set number of times. I'm sure there are other conceivable ways as well.
I need sent some requests to server side and get reponse, sometimes when I call specific method to run the flollowing common code, I get one error in line(addToCookieJar(connection);), any idea how this get happened?
URL url = new URL(providerURL);
HttpURLConnection connection = (HttpURLConnection)url.openConnection();
connection.setRequestMethod("POST");
connection.setDoInput(true);
connection.setDoOutput(true);
connection.setUseCaches(false);
connection.setRequestProperty("Content-Type", "application/octet-stream");
// We understand gzip encoding
connection.addRequestProperty("Accept-Encoding", "gzip");
if (cookie != null && cookieHandler != null) {
connection.setRequestProperty("Cookie", cookie);
}
if (cookieHandler == null) {
addFromCookieJar(connection);
}
// Send the request
ObjectOutputStream oos = new ObjectOutputStream(connection.getOutputStream());
oos.writeObject(remote.getName());
oos.writeObject(m.getName()); // method name
oos.writeObject(m.getParameterTypes()); // formal parameters
oos.writeObject(args); // actual parameters
oos.flush();
oos.close();
if (cookieHandler == null) {
cookieJar.put(new URI(providerURL), connection.getHeaderFields());
}
Exception:
java.lang.reflect.UndeclaredThrowableException
at $Proxy0.updateDocument(Unknown Source)
at com.agst.ui.gantt.GanttPanel.doUpdateDocument(GanttPanel.java:1931)
at com.agst.ui.gantt.GanttPanel.save(GanttPanel.java:1419)
at com.agst.ui.gantt.GanttPanel$4.run(GanttPanel.java:1673)
at java.lang.Thread.run(Unknown Source)
Caused by: java.net.SocketException: Connection reset
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection$6.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at sun.net.www.protocol.http.HttpURLConnection.getChainedException(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
at com.agst.rmi.RemoteCallHandler.call(RemoteCallHandler.java:196)
at com.agst.rmi.RemoteCallHandler.invoke(RemoteCallHandler.java:142)
... 5 more
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read1(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
at sun.net.www.http.HttpClient.parseHTTPHeader(Unknown Source)
at sun.net.www.http.HttpClient.parseHTTP(Unknown Source)
at sun.net.www.http.HttpClient.parseHTTP(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.getHeaderFields(Unknown Source)
at com.agst.rmi.RemoteCallHandler.addToCookieJar(RemoteCallHandler.java:529)
at com.agst.rmi.RemoteCallHandler.call(RemoteCallHandler.java:192)
... 6 more
This error indicates that the remote side has closed the connection while your side was still trying to read from it. You should check if
there is a problem on the server (check it's logs) or
you are trying to read more data than the server supplies
What does the addToCookieJar method do? Passing in the connection object might be a problem to this method since you close the OutputStream obtained from the connection. Are you by any chance trying to retrieve and operate on the output stream object in the addToCookieJar method?