In my code, I need to connect one url say http://example.org . If the above url doesn't connect (connectivity issue or server down), I need to connect another url say http://example-1.org .
URL url = new URL("http://example.org");
HttpURLConnection connection = (HttpURLConnection)url.openConnection();
connection.setRequestMethod("GET");
connection.connect();
int code = connection.getResponseCode();
if (code != 200){
//try alternate url connection here.
}
Please suggest improved way to do my usecase.
Also in case the first url has not been connected for n times (or n minutes), then I need to connect the alternate url afterwards. I'll have property(will have value in DB) to set the connectivity of first url has failed. once the first url is up, then I manually change the value in DB to connect the first url.
Please help me.
Related
I'm trying to implement a simple URL availability checker which basically checks if the link is available (No HTTP 403, 404 etc. returned).
I have more than 20,000 links(to different servers/websites) in my database for testing purposes but it doesn't seems to work when I try to create more than 10 Threads.
Here is the code I'm using for opening the connection and reading response code within each WorkerThread.
URL url = new URL(dto.getUrl());
httpUrlConnection = (HttpURLConnection) url.openConnection();
httpUrlConnection.setUseCaches(false);
// httpUrlConnection.setConnectTimeout(6000);
httpUrlConnection.setDoInput(true);
httpUrlConnection.setDoOutput(false);
httpUrlConnection.setRequestMethod("GET");
httpUrlConnection.setRequestProperty("Host", dto.getUrl().replace("http://", ""));
// httpUrlConnection.setRequestProperty("Connection",
// "Keep-Alive");
httpUrlConnection.setRequestProperty("User-Agent", USER_AGENT);
httpUrlConnection.setRequestProperty("Cache-Control", "no-cache");
httpUrlConnection.connect();
int code = httpUrlConnection.getResponseCode();
Few issues I have noticed when having multiple threads opening the connections:
1) Only first 100-200 connections seems to open with no problem, after that, I start getting "Read timeout", "Connection timeout", "Connection reset" etc. Although, if you try to run the code again the links which have thrown above exceptions will return proper response code (if they get processed in first 100).
2) The response code sometimes is not valid (especially if the link was processed after first 100 links). I have noticed that sometimes 404 is returned when in fact it should return 200 (I checked it by putting link in first 100).
I did try using Apache's Http client but it also fails to process links correctly with many threads.
So does anyone know a solution to this problem ? What is the maximum amount of connections you could open using HttpURLConnection using multiple threads ? Is there any other way to open many HTTP connections and check response codes ?
Thank you all in advance !
I am trying an example on httpsURLConnection,I get the following exception
Status code: 302
ejava.lang.IllegalStateException: connection not yet open
Following is the code snippet :
private static String url_s = "https://java.sun.com:443" ;
URL url = new URL(url_s) ;
HttpsURLConnection conn = (HttpsURLConnection) url.openConnection() ;
conn.setDoInput(true);
conn.setRequestMethod("GET") ;
conn.connect() ;
You need to read up a bit about HTTP protocol first if you really wanna try your hands at direct client calling.
302 is temporary redirect header which is sent back from the server. The response has another header named 'Location' whose value is a URL to which the server wants you to hit next. Browsers handle this automatically so you dont see a wait thing.
But if you want to do it yourself be sure ot handle all 301, 302 situations like these.
Best Of Luck.
I use URL connection to download stream in the Internet. But after i reset the modem, i can't continue download this stream caz it error: Connection reset. How i solve it?
Here is my code:
URL url = new URL(_URL);
HttpURLConnection hUC = (HttpURLConnection) url.openConnection();
hUC.connect();
while (true) {
if ((_data.num = is.read(_data.b)) == -1) {
break;
}
//write to file
fos.write(_data.b, 0, _data.num);
}
You can't - at least, not how you may be expecting.
Instead, you need to handle your exception, and determine how much data you've already read. Once your Internet connection is re-established - assuming that the HTTP server you're downloading from supports requestable byte ranges - you can then set custom HTTP Headers on the request and re-download the remaining portions. (This will require a new HttpURLConnection.)
http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35 shows the related HTTP specifications involved to make this work.
This is a bit more complicated if you're looking for a "resume" type feature.
You would need to reissue the request once the internet comes back after a disconnect, and add a header to the request in order to resume the download at the byte number where you left off.
You need to set the Range property in the request header in order to specify how far in you're resuming. Then you would just continue to write to the "fos" object from there.
Check out this url: Java: resume Download in URLConnection
I made an app. for Android which uses the C2DM service from Google. I
made a server simulator from some tutorials and it works fine. My
problem is, I tried to build a Java Servlet. From the Android device
it receives fine the message and saves the Registration ID, but when I
try to send a https POST request to the Google C2DM Server it always
gets a SocketTimeoutException : Timeout while fetching:
https://android.clients.google.com/c2dm/send.
I don't get why this is happening when the same works on the Android
device. Here is the code:
//The AuthToken from Google Client Login
String auth_key = TOKEN;
StringBuilder postDataBuilder = new StringBuilder();
//some parameters to pass, I've checked and it's correct, it's working
//with Fiddler
postDataBuilder.append(PARAM_REGISTRATION_ID).append("=").append(REGISTRATION_ID);
postDataBuilder.append("&").append(PARAM_COLLAPSE_KEY).append("=").append("0");
postDataBuilder.append("&").append("data.payload").append("=").append(URLEncoder.encode(message, UTF8));
byte[] postData = postDataBuilder.toString().getBytes(UTF8);
URL url = new URL("https://android.clients.google.com/c2dm/send");
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setDoOutput(true);
conn.setUseCaches(false);
conn.setRequestMethod("POST");
conn.setRequestProperty("Content-Type","application/x-www-form-urlencoded;charset=UTF-8");
conn.setRequestProperty("Content-Length",Integer.toString(postData.length));
conn.setRequestProperty("Authorization", "GoogleLogin auth="+auth_key);
OutputStream out = conn.getOutputStream();
out.write(postData);
out.close();
int responseCode = conn.getResponseCode();
//here comes the error processing, but I can't reach it, because of
//the exception.
if (responseCode == 401 || responseCode == 403) {
//....
}
Thanks for your help :).
The first obvious thing to check is - if you have thought of this I apologise - are you behind a proxy server e.g. a company firewall? If so a timeout is exactly the symptom I'd expect with the above code. (This catches me out all the time!)
With the latter half of your code (from the HttpURLConnection declaration on), unmodified, I see a timeout; on my system (behind a company firewall), with two changes I get a 200 OK back:
addition of a proxy object passed to the HttpUrlConnection factory as follows:
Proxy proxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress("...", 8080));
HttpURLConnection conn = (HttpURLConnection) url.openConnection(proxy);
accepting the C2DM server's certificate that wasn't trusted by my JVM. For test purposes I overrode the default hostname verifier and TrustManager as described in Trusting all certificates using HttpClient over HTTPS . For production you should look at a more secure solution.
Another thing I spotted; it doesn't seem to matter but http://code.google.com/android/c2dm/index.html#push says to post to https://android.apis.google.com/c2dm/send, not android.clients.google.com - just something to be aware of that might break in future.
I faced same problem and
I had tried :
URL url = new URL("http://android.apis.google.com/c2dm/send");
instead of :
URL url = new URL("https://android.apis.google.com/c2dm/send");
it worked for me.
I have an authentication protected url : www.domain.com/alias
that when requested will return another url: www.another.com/resource.mp4 (not protected)
I would like to know if exists a method in Java that will return the real url from a given one.
Something like: second = resolve(first)
I'm thinking of loading the first and try to read into the response maybe the location attribute, but since I'm not a java guru I would like to know if Java already faces this.
This is a problem i used to have concerning URL redirects. Try the following code:
URL url = new URL(url);
HttpURLConnection ucon = (HttpURLConnection) url.openConnection();
ucon.setInstanceFollowRedirects(false);
URL secondURL = new URL(ucon.getHeaderField("Location"));
URLConnection conn = secondURL.openConnection();
The "magic" here happens in these 2 steps:
ucon.setInstanceFollowRedirects(false);
URL secondURL = new URL(ucon.getHeaderField("Location"));
By default InstanceFollowRedirects are set to true, but you want to set it to false to capture the second URL. To be able to get that second URL from the first URL, you need to get the header field called "Location".
I have eliminated this issue on sites where we have a MikroTik router by using a Layer 7 protocol filter as shown below. This doesn't help the devices off the WiFi network (obviously) but at least gives them some reprieve when they are connected to home and/or work WiFi networks.
Firstly, create the protocol definition:
/ip firewall layer7-protocol
add comment="Frigging javascript redirects on chrome browsers" \
name=Javascript_Redirect \
regexp="^.+(spaces.slimspot.com|mostawesomeoffers.com).*\$"
Now, to actually filter this traffic out
/ip firewall filter
add action=drop chain=forward comment=\
"Block and log Javascript_Redirect L7 Protocol" layer7-protocol=\
Javascript_Redirect log=yes log-prefix=JSredirect_
Other firewalls that have Layer 7 filtering capacity could also block these redirects in a similar way.
If you are using Ktor:
import io.ktor.client.statement.*
val resp = HttpClient.get<HttpResponse>(urlString = yourUrl)
val redirectedUrl = resp.request.url