I use a HttpURLConnection to connect to a website and receive an ResponseCode=404 (HTTP_NOT_FOUND). However I have no problem opening the website in my browser (IE).
Why the difference, and what can I do about it?
This is my Program:
String responseMsg = "";
String cgsUrl = "http://localhost:9081/ntes/";
URL url = new URL(cgsUrl);
System.out.println("ouuuuuuu-->"+url.getContent());
InputStream in = url.openConnection().getInputStream();
StringBuffer respDataBuf = new StringBuffer();
respDataBuf.setLength(0);
int b = -1;
while((b = in.read()) != -1) {
respDataBuf.append((char)b);
}
responseMsg = respDataBuf.toString();
If this is a 404 error, this is certainly a particular server configuration.
Maybe your user-agent is banned, or you're not carrying special headers and so on. I recommend you copying the headers from your browser (all of them) and use them to make the request in your Java program.
Then you throw them away one by one to find the one which is mandatory
Related
I'm trying to write a simple test where I submit a request to http://localhost:12345/%, knowing that this is an illegal URI, because I want to assert that my HTTP Server's error-handling code behaves correctly. However, I am having a hard time forcing Java to do this.
If I try to create a Java 11 HttpRequest with URI.create("localhost:12345/%"), I get a URISyntaxException, which is correct and not helpful.
Similarly, using a ws-rs WebTarget:
ClientBuilder.newBuilder().build().target("http://localhost:12345").path("/%")
builds me a WebTarget pointing to /%25, which would normally be very helpful, but is not what I want in this particular situation.
Is there a way to test my error-handling behavior without resorting to low-level bytestream manipulation?
Another possibility is just to use plain Socket - it's easy enough if you know the protocol (especially if using the new text-block feature). This will allow you to misformat the request in any way you like. Reading the response and analysing the result is - of course - a bit more involved:
String request = """
GET %s HTTP/1.1\r
Host: localhost:%s\r
Connection: close\r
\r
""".formatted("/%", port);
try (Socket client = new Socket("localhost", port);
OutputStream os = client.getOutputStream();
InputStream in = client.getInputStream()) {
os.write(request.getBytes(StandardCharsets.US_ASCII));
os.flush();
// This is optimistic: the server should close the
// connection since we asked for it, and we're hoping
// that the response will be in ASCII for the headers
// and UTF-8 for the body - and that it won't use
// chunk encoding.
byte[] bytes = in.readAllBytes();
String response = new String(bytes, StandardCharsets.UTF_8);
System.out.println("response: " + response);
}
Noah's comment lead me down the right path; I was able to do this with the URL class:
#Test
public void testUriMalformed() throws Exception {
final URL url = new URL(TEST_URI + "/%");
final HttpURLConnection connection = (HttpURLConnection)url.openConnection();
final int code = connection.getResponseCode();
final String contentType = connection.getHeaderField("Content-Type");
final String entity = IOUtils.toString(connection.getErrorStream(), Charsets.UTF_8);
assertEquals(500, code);
assertEquals(MediaType.APPLICATION_JSON, contentType);
assertTrue(entity.contains("error_id"));
}
I'm trying to get HTML by URL in Java. But 301 Moved Permanently is all that I've got. Another URLs work. What's wrong? This is my code:
hh= new URL("http://hh.ru");
in = new BufferedReader(
new InputStreamReader(hh.openStream()));
while ((inputLine = in.readLine()) != null) {
sb.append(inputLine).append("\n");
str=sb.toString();//returns 301
}
You're facing a redirect to other URL. It's quite normal and web site may have many reasons to redirect you. Just follow the redirect based on "Location" HTTP header like that:
URL hh= new URL("http://hh.ru");
URLConnection connection = hh.openConnection();
String redirect = connection.getHeaderField("Location");
if (redirect != null){
connection = new URL(redirect).openConnection();
}
BufferedReader in = new BufferedReader(new InputStreamReader(connection.getInputStream()));
String inputLine;
System.out.println();
while ((inputLine = in.readLine()) != null) {
System.out.println(inputLine);
}
Your browser is following redirects automaticaly, but using URLConnection you should do it by your own. If it bothers you take a look at other Java HTTP client implementations, like Apache HTTP Client. Most of them are able to follow redirect automatically.
found this answer useful and improved a little due to the possibility of multiple redirections (e.g. 307 then 301).
URLConnection urlConnection = url.openConnection();
String redirect = urlConnection.getHeaderField("Location");
for (int i = 0; i < MAX_REDIRECTS ; i++) {
if (redirect != null) {
urlConnection = new URL(redirect).openConnection();
redirect = urlConnection.getHeaderField("Location");
} else {
break;
}
}
There's nothing wrong with your code. The message means that hh.ru is permanently moved to another domain.
I tested your code and it is ok, but when I use "hh.ru", the same problem as yours, and when I use lynx(command line browser) to connect to "hh.ru", it will show me that it is redirecting to another url and then show me that it is moved permanently and after that this alert:
"Alert!: This client does not contain support for HTTPS URLs"
Check if the URL provided is HTTP or HTTPS, consider adding protocol is you are using only domain name like http(s)://domainname.com/resource-name
Read: https://en.wikipedia.org/wiki/HTTP_301
I resolved mine when I put the specific file running on the server.
Instead of http://hh.ru,
I used http://hh.ru/index.php.
It worked for me
I am trying to get a JSON response using this code.
it works fine for short responses (20 to 30 lines) but cuts off in middle of longer ones.
final HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setRequestMethod("GET");
if (connection.getResponseCode() != 200) { throw new Exception(); }
BufferedReader in = new BufferedReader(new InputStreamReader(connection.getInputStream()));
String inline; StringBuilder response = new StringBuilder();
while ((inline = in.readLine()) != null) {
response.append(inline);
}
in.close();
Log.d(TAG, "getJson: {"+url.toString()+"}\n"+response.toString());
return response.toString();
Sample of the logcat output showing cutoff json here.
How can I force it to always download the full response?
Edit:
It seems from the comments that this question generated, as well as further testing, that there is nothing inherent or lacking in the code that is causing the server to truncate, but that this is an android specific problem.
Can anyone, A) identify why long responses (use example link above) truncate specifically on android (whether from above code, or in android browser), and B) - (the real question) is there a way to programatically overcome the issue?
I have an android app that downloads and uses a file at runtime. The file is valid as I can download it via the browser and open it up, etc. However my app kept reporting that the file is corrupted.
After investigation I discovered the server (which I have no control over) is returning an incorrect "Content-Length:" (~180 vs ~120000). The header is the culprit as I confirmed the issue by downloading the file with curl - which also resulted in a truncated file.
After some research I concluded that my use of BufferedInputStream to append to a ByteArrayBuffer is autosizing the byte array to the url connections content length. To circumvent this, I tried to use ByteArrayOutputStream instead, however this solved nothing.
Anybody know of a way to download a file if the Content-Length is incorrectly set? A browser can.
Here's my latest attempt:
public static void downloadFileFromRemoteUrl(String urlString, String destination){
try {
URL url = new URL(urlString);
File file = new File(destination);
URLConnection urlConnection = url.openConnection();
InputStream inputStream = urlConnection.getInputStream();
byte[] buffer = new byte[1024];
int curLength = 0;
int newLength = 0;
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
while((newLength = inputStream.read(buffer))>0)
{
curLength += newLength;
byteArrayOutputStream.write(buffer, 0, newLength);
}
FileOutputStream fos = new FileOutputStream(file);
fos.write(byteArrayOutputStream.toByteArray());
fos.close();
android.util.Log.d("DB UPDATE", "Done downloading database. Size: " + byteArrayOutputStream.toByteArray().length);
}
catch (MalformedURLException e) {
e.printStackTrace();
}
catch (IOException e) {
e.printStackTrace();
}
}
After some research I concluded that my use of BufferedInputStream to append to a ByteArrayBuffer is autosizing the byte array to the url connections content length.
Nonsense. You are crediting those classes with paranormal powers. How could an output stream possibly become aware of the Content-length header? The URLConnection's input stream is being terminated at the content-length. Correctly.
To circumvent this, I tried to use ByteArrayOutputStream instead, however this solved nothing.
Of course not.
Anybody know of a way to download a file if the Content-Length is incorrectly set?
You could use a Socket and engage in HTTP yourself, which is less trivial than it sounds. But the problem is at the server and that's where it should be fixed. Complain. Or else #Zong Yu is correct and the page is HTML containing JavaScript, say.
NB You don't need to read the entire file into memory:
while((newLength = inputStream.read(buffer))>0)
{
curLength += newLength;
fos.write(buffer, 0, newLength);
}
My final "solution" was to realize I was dealing with a 301 redirect response and not the actual resource! I updated the section that handles my url, checking for a 301 and if exists, update the url. The new url contained the Content-Length that corresponded with the file I was downloading.
// start by creating an http url connection object
HttpURLConnection httpURLConnection = (HttpURLConnection) url.openConnection();
// determine if this is a redirect
boolean redirect = false;
int status = httpURLConnection.getResponseCode();
if (status != HttpURLConnection.HTTP_OK) {
if (status == HttpURLConnection.HTTP_MOVED_TEMP
|| status == HttpURLConnection.HTTP_MOVED_PERM
|| status == HttpURLConnection.HTTP_SEE_OTHER)
redirect = true;
}
// if it is, we need a new url
if (redirect) {
String newUrl = httpURLConnection.getHeaderField("Location");
httpURLConnection = (HttpURLConnection) new URL(newUrl).openConnection();
}
Try Fetch. Fetch is an in app download manager for Android. It's very easy to use. Find the GitHub page here. The project comes with several demos that you can try out. Disclaimer: I'm the creator of Fetch, and it is open source.
I am currently working with android and i am using a http connection with some headers (i havent included them or the real url for security purposes) to get a JSON response from an API, and feeding that response back into the application. The problem that i am having is that when using the getContentLength method of the http request, the wrong length is being returned (wrong length returned is 1225, the correct length in characters of the JSON array is 3365).
I have a feeling that the JSON is not fully loaded when my reader starts to read it, and as such is only reading the loaded JSON at that point. Is there any way around this, possibly using a delay on the HTTP connection or waiting until it is fully loaded to read the data?
URL url = new URL("https://www.exampleofurl.com");
HttpURLConnection request = (HttpURLConnection) url.openConnection();
request.connect();
int responseCode = request.getResponseCode();
if(responseCode == HttpURLConnection.HTTP_OK) {
InputStream inputStream = request.getInputStream();
InputStreamReader reader = new InputStreamReader(inputStream);
long contentLength2 = Long.parseLong(request.getHeaderField("Content-Length"));
Log.i("contentLength: ", "Content: " + contentLength2);
I generally don't recommend always relying on "Content-Length" as it may not be available (you get -1), or perhaps affected by intermediate proxy.
Why don't you just read your stream until it is exhausted into memory buffer (say, StringBuilder) and then get the actual size, for example :
BufferedReader br = new BufferedReader(inputStream); // inputStream in your code
String line;
StringBuilder sb = new StringBuilder();
while ((line = br.readLine()) != null) {
sb.append(line);
}
// finished reading
System.out.println("data size = " + sb.length());
JSONObject data = new JSONObject(sb.toString());
// and don't forget finally clauses with closing streams/connections