Develop Reference

HttpURLConnection respond 404 when file clearly exist

I have a java program that is trying to read any arbitrary file from URL. However, it return an 404 error when the file clearly exist, try it for yourself with the URL. What is wrong?
URL url = new URL("http://images.all-free-download.com/images/graphiclarge/blue_abstract_background_310971.jpg");
HttpURLConnection myHTTPConTest = null;
myHTTPConTest = (HttpURLConnection) url.openConnection();
int responseCode = myHTTPConTest.getResponseCode(); // Returns 404
Added user-agent, no change:
myHTTPConTest.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.3) Gecko/20100401");

I think you had a bad luck and you just hit the server restart time or something like that.
Because it returns 200 now.

Program can not login into website

This is my code:
URL url = new URL("http://superchillin.com/login2.php");
HttpURLConnection urlConnection = (HttpURLConnection)url.openConnection();
urlConnection.setUseCaches(false);
urlConnection.setRequestMethod("POST");
String data = "email="+URLEncoder.encode(name, "UTF-8")+"&password="+URLEncoder.encode(pass, "UTF-8");
urlConnection.setRequestProperty("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
urlConnection.setRequestProperty("Accept-Encoding", "gzip,deflate");
urlConnection.setRequestProperty("Accept-Language", "en-US,en;q=0.8,lt;q=0.6");
urlConnection.setRequestProperty("Cache-Control", "max-age=0");
urlConnection.setRequestProperty("Connection", "keep-alive");
urlConnection.setRequestProperty("Content-Length", Integer.toString(data.getBytes().length));
urlConnection.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
urlConnection.addRequestProperty("Cookie", "place=1");
urlConnection.addRequestProperty("Cookie", "lvca_unique_user=1");
urlConnection.setRequestProperty("Host", "superchillin.com");
urlConnection.setRequestProperty("Origin", "http://superchillin.com");
urlConnection.setRequestProperty("Referer", "http://superchillin.com/login.php");
urlConnection.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.124 Safari/537.36");
urlConnection.setDoOutput(true);
urlConnection.setDoInput(true);
urlConnection.setInstanceFollowRedirects(true);
DataOutputStream wr = new DataOutputStream(urlConnection.getOutputStream());
wr.writeBytes(data);
wr.flush();
wr.close();
After that code I only read the response. It redirects me to "login.php" and is trying to set cookie "place=1"...
Connecting via browser works great. The reason for so many headers is I thought they may be the problem so I copied all headers from which I see when using a browser.
The response code is 200.
I also noticed that if password or email is incorrect, there's a message saying that in HTML which i retrieve.
When I use a browser I get redirected to index.php and cookie "auth" is set. So that's what I'm expecting from my program aswell. Curently I get redirected back to "login.php".

There is no universal answer to this question, I'm afraid. What you're asking is "why does the remote server not return an auth cookie when I send this exact request?" And that depends entirely on what the server's documentation says about those requests, whether it has any bugs in its implementation, etc.
If you don't have access to the server's own source and logs, then you'll likely have to get by with experimentation. Use something like Firebug or Chrome's Developer Tools to capture the exact requests sent by the browser with the login works successfully. Since these text strings are the only thing the remote server sees, if you replicate them exactly with your Java program you will(/should) get exactly the same responses.
If you think you're sending the same requests from Java and find that you're still not getting the expected responses, there must be some difference. Try recording the network traffic with something like Wireshark in order to see exactly what your app is sending - and then address the differences.
And if you get to the point where e.g. a redirect isn't being followed, and you're not sure how to do that with a URLConnection - then that's a good concrete question to ask.

Java HttpUrlConnection throws Connection Refused

I know there are several question regarding this topic But I did't find an answer in any of them.
I'm trying to open a connection to my local server but I keep getting connection refused.
I have the server running and I tested the connection with the Browser and with a Google App called Postman and it works.
It's failing when opening the connection as if there where nothing to connect to. or maybe something is blocking the connection? I tested with firewall and antivirus down, no luck.
testing in Postman the URL returns a User as it should...
If I replace the url with "http://www.google.com" It Works fine.
here is my code:
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.MalformedURLException;
import java.net.URL;
/**
*
* #author Gabriel
*/
public class HttpConnection {
public HttpConnection() {
}
public void makeRequest() throws MalformedURLException, IOException {
String url = "http://localhost:8000/users/1";
URL obj = new URL(url);
HttpURLConnection con = (HttpURLConnection) obj.openConnection();
// optional default is GET
con.setRequestMethod("GET");
//add request header
con.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.120 Safari/537.36");
con.setRequestProperty("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
con.setRequestProperty("Accept-Encoding", "gzip,deflate,sdch");
con.setRequestProperty("Accept-Language", "en-US,en;q=0.8,es;q=0.6");
con.setRequestProperty("Connection", "keep-alive");
con.setRequestProperty("Host", "localhost:8000");
int responseCode = con.getResponseCode();
System.out.println("\nSending 'GET' request to URL : " + url);
System.out.println("Response Code : " + responseCode);
BufferedReader in = new BufferedReader(new InputStreamReader(con.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
//print result
System.out.println(response.toString());
}
}

I faced exactly the same problem. Use this instead of localhost:
http://[::1]:8000/index.php

I have similar code that is working, but my request header is a lot simpler. Basically just:
con.setRequestProperty("User-Agent", "Mozilla/5.0");
If simplifying the header does not help, I would capture the traffic when using your browser with something like fiddler and then making the request look exactly like that.

I will make a wild guess what can be the problem. It is possible a IPv4/IPv6 problem.
If so, here is two possible solutions
If the server is only listening on an ipv6 address, change it to listening to ipv4.
If the server is listening to ipv4, then force Java to use ipv4 with
java.net.preferIPv4Stack=true

You can try implementing CORS at the API you are trying to connect by setting access-control-allow-origin:* property in response header.

The code is good and works great. Now the problem must be on the transportation or network part. What I want to mean is you don't request the right server. If you use 127.0.0.1 instead of localhost I think you won't get a problem. So, my guest will be that you have a problem in /etc/hosts or C:\Windows\System32\drivers\etc\hosts.
I advice you to try a simple test: ping the hostname and check in the output if the ip address is good.

Well, put http://localhost:8000/users/1 in your web browser and what do you get? A simple Connection Refused error. It's not you, it's the website. Also, Url returns websites using Protocol Identifiers(http://, https://), Ending Domains(.com, .edu, .gov) that's also another reason why you get an error.

You mentioned that you were opening a connection to your "local server"
I am assuming that you are doing this on the same computer that you're hosting the server on?
Try to open the connection to your local server using a different computer.

java, android, resolve an url, get redirected uri

I have an authentication protected url : www.domain.com/alias
that when requested will return another url: www.another.com/resource.mp4 (not protected)
I would like to know if exists a method in Java that will return the real url from a given one.
Something like: second = resolve(first)
I'm thinking of loading the first and try to read into the response maybe the location attribute, but since I'm not a java guru I would like to know if Java already faces this.

This is a problem i used to have concerning URL redirects. Try the following code:
URL url = new URL(url);
HttpURLConnection ucon = (HttpURLConnection) url.openConnection();
ucon.setInstanceFollowRedirects(false);
URL secondURL = new URL(ucon.getHeaderField("Location"));
URLConnection conn = secondURL.openConnection();
The "magic" here happens in these 2 steps:
ucon.setInstanceFollowRedirects(false);
URL secondURL = new URL(ucon.getHeaderField("Location"));
By default InstanceFollowRedirects are set to true, but you want to set it to false to capture the second URL. To be able to get that second URL from the first URL, you need to get the header field called "Location".

I have eliminated this issue on sites where we have a MikroTik router by using a Layer 7 protocol filter as shown below. This doesn't help the devices off the WiFi network (obviously) but at least gives them some reprieve when they are connected to home and/or work WiFi networks.
Firstly, create the protocol definition:
/ip firewall layer7-protocol
add comment="Frigging javascript redirects on chrome browsers" \
name=Javascript_Redirect \
regexp="^.+(spaces.slimspot.com|mostawesomeoffers.com).*\$"
Now, to actually filter this traffic out
/ip firewall filter
add action=drop chain=forward comment=\
"Block and log Javascript_Redirect L7 Protocol" layer7-protocol=\
Javascript_Redirect log=yes log-prefix=JSredirect_
Other firewalls that have Layer 7 filtering capacity could also block these redirects in a similar way.

If you are using Ktor:
import io.ktor.client.statement.*
val resp = HttpClient.get<HttpResponse>(urlString = yourUrl)
val redirectedUrl = resp.request.url

java.io.IOException: Server returned HTTP response code: 403 for URL

I want to download the mp3 file from url : "http://upload13.music.qzone.soso.com/30671794.mp3", i always got java.io.IOException: Server returned HTTP response code: 403 for URL. But it's ok when open the url using browser. Below is part of my code:
BufferedInputStream bis = null;
BufferedOutputStream bos = null;
try {
URL url = new URL(link);
URLConnection urlConn = url.openConnection();
urlConn.addRequestProperty("User-Agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)");
String contentType = urlConn.getContentType();
System.out.println("contentType:" + contentType);
InputStream is = urlConn.getInputStream();
bis = new BufferedInputStream(is, 4 * 1024);
bos = new BufferedOutputStream(new FileOutputStream(
fileName.toString()));​
Anyone could help me? Thanks in advance!

You can also use
System.setProperty("http.agent", "Chrome");
it worked for me.
//Update
Explanation
Because HttpURLConnection reads the property "http.agent" if set.
You can read it here: https://www.innovation.ch/java/HTTPClient/advanced_info.html
Or you can look it up in the source code of the HttpURLConnection Class:
String agent = java.security.AccessController.doPrivileged(new sun.security.action.GetPropertyAction("http.agent"));

Instead of using URLConnection in java, if you use HttpURLConnection you should beable to access the requested web page from java. Try the following code:
HttpURLConnection httpcon = (HttpURLConnection) url.openConnection();
httpcon.addRequestProperty("User-Agent", "Mozilla/4.76");
Normal java using urlConnection wont be accepted to access the internet. To access the browser it will need to perform a search without theexception HTTP response code : 403 for URL
EDIT (#Mordechai): No need to do the casting, just add the user agent.

When I access the URL with my browser I also get 403. Perhaps you're logged in to the site with your browser?
If that's the case you need to duplicate the cookie from your browser and send it along, perhaps even do more to replicate your browser's signature if the site does any extra checks.
You can set the cookie by adding:
urlConn.setRequestProperty("Cookie", "foo=bar");
Where foo=bar is the key-value pair you'll find when you locate the site's cookie in your browser.

The problem is given by the Status code. 403 means actually "Forbidden" and implies The request was denied for a reason the server does not want to (or has no means to) indicate to the client.
the problem lies at the server-side.

I would also check if the server were the resource is located has an ACL or similar in place, we just resolved a "java.io.IOException: 403" issue this way.
It happens that 403 errors are very generic and you cannot really be sure of the source as it can be just anything.