I have a java program that is trying to read any arbitrary file from URL. However, it return an 404 error when the file clearly exist, try it for yourself with the URL. What is wrong?
URL url = new URL("http://images.all-free-download.com/images/graphiclarge/blue_abstract_background_310971.jpg");
HttpURLConnection myHTTPConTest = null;
myHTTPConTest = (HttpURLConnection) url.openConnection();
int responseCode = myHTTPConTest.getResponseCode(); // Returns 404
Added user-agent, no change:
myHTTPConTest.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.3) Gecko/20100401");
I think you had a bad luck and you just hit the server restart time or something like that.
Because it returns 200 now.
Related
I would like to take some data from www.groupon.pl website. To do this I used code below:
String currentUrl = "https://www.groupon.pl/browse/radom";
URL urlObj = new URL(currentUrl);
HttpURLConnection urlConnection = (HttpURLConnection) urlObj.openConnection();
urlConnection.addRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0");
urlConnection.connect();
final HtmlCleaner cleaner = new HtmlCleaner();
final TagNode tagNodeRoot = cleaner.clean(urlConnection.getInputStream());
The problem is that after the last line:
final TagNode tagNodeRoot = cleaner.clean(urlConnection.getInputStream());
The program doesn't want go further. I think that maybe the problem is with:
urlConnection.addRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0");
Or maybe I use some and old API which is not supported any more? I use debugger and it doesn't stepping out. When I run this code normally without any debugger it also stops. No error at the console. I don't know maybe it is waiting for the response. Could you tell me what is wrong? Thank you.
Is there a way to fetch just Last Modified header by using a HttpConnection?
When I use this code:
HttpURLConnection httpCon = (HttpURLConnection) url.openConnection();
long lastModified = httpCon.getLastModified();
In the log files of the webserver I see:
a.b.c.d - - [26/Dec/2015:10:25:50 +0100] "GET /file.txt HTTP/1.1" 200 484 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0"
If I understand it correct, the GET request fetches the whole content as well. Is there a possibility to perform HEAD request to retrieve the headers only?
You can set the HTTP method, see HttpURLConnection:
Set the method for the URL request, one of:
GET
POST
HEAD
OPTIONS
PUT
DELETE
TRACE
are legal, subject to protocol restrictions. The default method is GET.
Example:
HttpURLConnection httpUrlConnection = (HttpURLConnection) url.openConnection();
httpUrlConnection.setRequestMethod("HEAD");
long lastModified = httpUrlConnection.getLastModified();
This is my code:
URL url = new URL("http://superchillin.com/login2.php");
HttpURLConnection urlConnection = (HttpURLConnection)url.openConnection();
urlConnection.setUseCaches(false);
urlConnection.setRequestMethod("POST");
String data = "email="+URLEncoder.encode(name, "UTF-8")+"&password="+URLEncoder.encode(pass, "UTF-8");
urlConnection.setRequestProperty("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
urlConnection.setRequestProperty("Accept-Encoding", "gzip,deflate");
urlConnection.setRequestProperty("Accept-Language", "en-US,en;q=0.8,lt;q=0.6");
urlConnection.setRequestProperty("Cache-Control", "max-age=0");
urlConnection.setRequestProperty("Connection", "keep-alive");
urlConnection.setRequestProperty("Content-Length", Integer.toString(data.getBytes().length));
urlConnection.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
urlConnection.addRequestProperty("Cookie", "place=1");
urlConnection.addRequestProperty("Cookie", "lvca_unique_user=1");
urlConnection.setRequestProperty("Host", "superchillin.com");
urlConnection.setRequestProperty("Origin", "http://superchillin.com");
urlConnection.setRequestProperty("Referer", "http://superchillin.com/login.php");
urlConnection.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.124 Safari/537.36");
urlConnection.setDoOutput(true);
urlConnection.setDoInput(true);
urlConnection.setInstanceFollowRedirects(true);
DataOutputStream wr = new DataOutputStream(urlConnection.getOutputStream());
wr.writeBytes(data);
wr.flush();
wr.close();
After that code I only read the response. It redirects me to "login.php" and is trying to set cookie "place=1"...
Connecting via browser works great. The reason for so many headers is I thought they may be the problem so I copied all headers from which I see when using a browser.
The response code is 200.
I also noticed that if password or email is incorrect, there's a message saying that in HTML which i retrieve.
When I use a browser I get redirected to index.php and cookie "auth" is set. So that's what I'm expecting from my program aswell. Curently I get redirected back to "login.php".
There is no universal answer to this question, I'm afraid. What you're asking is "why does the remote server not return an auth cookie when I send this exact request?" And that depends entirely on what the server's documentation says about those requests, whether it has any bugs in its implementation, etc.
If you don't have access to the server's own source and logs, then you'll likely have to get by with experimentation. Use something like Firebug or Chrome's Developer Tools to capture the exact requests sent by the browser with the login works successfully. Since these text strings are the only thing the remote server sees, if you replicate them exactly with your Java program you will(/should) get exactly the same responses.
If you think you're sending the same requests from Java and find that you're still not getting the expected responses, there must be some difference. Try recording the network traffic with something like Wireshark in order to see exactly what your app is sending - and then address the differences.
And if you get to the point where e.g. a redirect isn't being followed, and you're not sure how to do that with a URLConnection - then that's a good concrete question to ask.
For this link http://bits.blogs.nytimes.com/2014/09/02/uber-banned-across-germany-by-frankfurt-court/?partner=rss&emc=rss this code doesn`t work but if I put another for exemple: https://www.google.com everything is ok:
URL url = new URL("http://bits.blogs.nytimes.com/2014/09/02/uber-banned-across-germany-by-frankfurt-court/?partner=rss&emc=rss");
URLConnection uc;
uc = url.openConnection();
uc.setRequestProperty("User-Agent", "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_7; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.648.205 Safari/534.16");
uc.addRequestProperty("referer", "http://www.facebook.com");
uc.connect();
this.input = uc.getInputStream();
I get this exception:
java.lang.IllegalArgumentException: protocol = http host = null
at sun.net.spi.DefaultProxySelector.select(DefaultProxySelector.java:170)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:926)
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:850)
at sun.net.www.protocol.http.HttpURLConnection.followRedirect(HttpURLConnection.java:2398)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1557)
at UrlParser.<init>(UrlParser.java:48)
at TikaParser.test_url_parser(TikaParser.java:186)
at TikaParser.run(TikaParser.java:256)
at java.lang.Thread.run(Thread.java:745)
what is wrong with my code?
I encountered this same exception when the URL started with http:/ instead of http://. e.g. http:/www.example.com
This was in the org.springframework.web.client.RestTemplate.exchange() method, so it's not quite the same context, but maybe a similar issue.
Mostly this error occur due to incorrect url. Make sure the Url that you are hitting is correct.As wsams mentioned try to check the // in your url if it is correct or not.
Correct pattern is http://localhost:8080
Looks like your proxy host is not set. Try setting it, then it should work.
See this page for more information on proxy properties.
Easy solution: In gradle-wrapper.properties just hav a look at distribution url.
Make it correct like this;
distributionUrl=https\://services.gradle.org/distributions/gradle-7.4-all.zip
Its working properly
I'm trying to download images from url but got java.io.IOException: exception.
My code is:
public static void main(String[] args) throws MalformedURLException, IOException {
File picutreFile = new File("test.jpg");
FileUtils.copyURLToFile(new java.net.URL("http://paceoil.ca/files/includes/images/images-stories-presentation-october-icon-graphic.jpg"), picutreFile);
}
When run threw :
Exception in thread "main" java.io.IOException: Server returned HTTP response code: 403 for URL: http://paceoil.ca/files/includes/images/images-stories-presentation-october-icon-graphic.jpg
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
at java.net.URL.openStream(Unknown Source)
at org.apache.commons.io.FileUtils.copyURLToFile(FileUtils.java:1460)
at com.hrant.Test.main(Test.java:14)
This code works for some urls but for this I couldnt understand why not.
Advance many Thanks.
try setting user agent property on your url, for example, see if it helps:
File picutreFile = new File("src/test.jpg");
URL url=new URL("http://paceoil.ca/files/includes/images/images-stories-presentation-october-icon-graphic.jpg");
URLConnection conn = url.openConnection();
conn.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Firefox/31.0");
conn.connect();
FileUtils.copyInputStreamToFile(conn.getInputStream(), picutreFile);
Just a suggestion:
When this only happen to this URL (or Domain), it may be possible, that the http-server implementation on the other end is checking some HTTP-Headers before processing the request., And you do not send such informations.
The result may be, that the server will response with 403.
Maybe: Forbidden to unknown Client (e.g User-Agent)
Maybe this Link will help to add headers to your request:
https://stackoverflow.com/a/15555952/3887073
this server maybe dont promise the some request that has invalid HTTP request .
you can send your HTTP header file with you request and server will promise you to download this image if its need the some HTTP header information .
something like this :
URL myURL = new URL(serviceURL);
HttpURLConnection myURLConnection = (HttpURLConnection)myURL.openConnection();
String userCredent = "username:password";
myURLConnection.setRequestProperty ("property", value);
myURLConnection.setRequestMethod("POST");