I'm sending zip file over FTP connection so to fetch file size , I have used :
URLConnection conn = imageURL.openConnection();
long l = conn.getContentLengthLong();
But it returns -1
Similarly for files sent over Http request , I get correct file size.
How to get correct file size in ftp connection in this case ?
for files sent over Http request , I get correct file size.
MAYBE. URLConnection.getContentLength[Long] returns specifically the content-length header. HTTP (and HTTPS) supports several different ways of delimiting bodies, and depending on the HTTP options and versions the server implements, it might use a content-length header or it might not.
Somewhat similarly, an FTP server may provide the size of a 'retrieved' file at the beginning of the operation, or it may not. But it never uses a content-length header to do so, so getContentLength[Long] doesn't get it. However, the implementation code does store it internally if the server provides it, and it can be extracted by the following quite ugly hack:
URL url = new URL ("ftp://anonymous:dummy#192.168.56.2/pub/test");
URLConnection conn = url.openConnection();
try( InputStream is = conn.getInputStream() ){
if( ! conn.getClass().getName().equals("sun.net.www.protocol.ftp.FtpURLConnection") ) throw new Exception("conn wrong");
Field fld1 = conn.getClass().getDeclaredField("ftp");
fld1.setAccessible(true); Object ftp = fld1.get(conn);
if( ! ftp.getClass().getName().equals("sun.net.ftp.impl.FtpClient") ) throw new Exception ("ftp wrong");
Field fld2 = ftp.getClass().getDeclaredField("lastTransSize");
fld2.setAccessible(true); long size = fld2.getLong(ftp);
System.out.println (size);
}
Hacking undocumented internals may fail at any time, and versions of Java above 8 progressively discourage it: 9 through 15 give a warning message about illegal access unless you use --add-opens to permit it and 16 makes it an error (i.e. fatal). Unless you really need the size before reading the data (which implicitly gives you the size) I would not recommend this.
Related
i'm wrote simple download manager and i'm trying to set RESUME for all downloads. after googleing for how to do that. i know must be setRequestProperty for connection, but my code does not work and i get this error:
FATAL EXCEPTION: Thread-882
java.lang.IllegalStateException: Cannot set request property after connection is made
at libcore.net.http.HttpURLConnectionImpl.setRequestProperty(HttpURLConnectionImpl.java:510)
My code is:
URL url = new URL(downloadPath);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
final int fileSize = connection.getContentLength();
File file = new File(filepath);
if (file.exists() && fileSize == file.length()) {
return;
} else if (file.exists()) {
connection.setRequestProperty("Range", "bytes="+(file.length())+"-");
}else
connection.setRequestProperty("Range", "bytes=" + downloadedSize + "-");
connection.setRequestMethod("GET");
connection.setDoInput(true);
connection.setDoOutput(true);
connection.connect();
how to resolve this problem and correctly set setRequestProperty to connection?
The problem is that you're calling connection.getContentLength() before you're calling setRequestProperty(). The content length is only available after you've made a request, at which point you can't set the request property...
It's not entirely clear what you're trying to do, but one option is to use a HEAD request just to get the content length, and then make a separate request if you need to get just a portion of the data. Be aware that it's possible that the content length will change between requests, of course.
However, I would actually suggest keeping more metadata somewhere in your download manager - so that when you first start downloading the data, you keep a record of the total size, so that you don't need to make the HEAD request when resuming - you can tell just from the local information whether or not you've already downloaded a file. (This has the same problem in terms of content changing, but that's a different matter.)
I had the same error than OP.
WHY
The problem is that when you try to set the params to the request to resume the download, you have to be disconnected from the Http.
The moment you invoke the method connection.getContentLenght(); what happens is that connection.connect(); so if you then try to set the properties to the connection you will get the error mentioned.
FIX
What I did was that I closed the connection to Http after I invoked the method long totalFileSize = connection.getContentLength();
connection.disconnect()//Disconnect from http
And after that you can set the the parameters you want to the connection and invoke connection.connect() whenever needed.
TIP
In my particular case I was trying to download a file and needed to support resumable downloads, so to do it what I did was:
Check if file exists.
If file exists then get the lenght of the file:
long bytesDownloaded = file.getLenght();
Use this lenght to set it to the connection so it can resume from exactly the bytes it was paused.
Write the bytes to the end of the file.
You should set properties before getContentLength()
If you set range equal to exist file length you will receive remain bytes when call getContentLength() so if content length was equal to "0" that means that file downloaded completely.
But if you want to build a download manager, #Jon Skeet's method is reasonable.
Edit:
public abstract long getContentLength ()
Added in API level 1 Tells the length of the content, if known.
Returns the number of bytes of the content, or a negative number if
unknown. If the content length is known but exceeds Long.MAX_VALUE, a
negative number is returned.
I want to read the second line of the text at this URL: "http://vuln2014.picoctf.com:51818/" (this is a capture-the-flag competition but only asking for flags or direction to flags breaks the competition rules). I am attempting to open an input stream from the URL but I get an Invalid HTTP Response exception. Any help is appreciated, and I recognize that my error is likely quite foolish.
Code:
URL url = new URL("http://vuln2014.picoctf.com:51818");
URLConnection con = url.openConnection();
InputStream is = con.getInputStream()
The error occurs at the third line.
java.io.IOException: Invalid Http response at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1342) at name.main(name.java:41)
curl happily gets the text from the page, and it is perfectly accessible from a web browser.
When you do this:
URL url = new URL("http://vuln2014.picoctf.com:51818");
URLConnection con = url.openConnection();
You are entering into a contract that says that this URL uses the http protocol. When you call openConnection it expects to get http responses because you used http:// in the URL as the protocol. The Java Documentation says:
If for the URL's protocol (such as HTTP or JAR), there exists a public, specialized URLConnection subclass belonging to one of the following packages or one of their subpackages: java.lang, java.io, java.util, java.net, the connection returned will be of that subclass. For example, for HTTP an HttpURLConnection will be returned, and for JAR a JarURLConnection will be returned.
The server you are connecting to just returns a couple lines of data. I retrieved them with the command nc vuln2014.picoctf.com 51818. There is no http response code like HTTP/1.1 200 OK:
Welcome to the Daedalus Corp Spies RSA Key Generation Service. The public modulus you should use to send your updates is below. Remember to use exponent 65537.
b4ab920c4772c5247e7d89ec7570af7295f92e3b584fc1a1a5624d19ca07cd72ab4ab9c8ec58a63c09f382aa319fa5a714a46ffafcb6529026bbc058fc49fb1c29ae9f414db4aa609a5cab6ff5c7b4c4cfc7c18844f048e3899934999510b2fe25fcf8c572514dd2e14c6e19c4668d9ad82fe647cf9e700dcf6dc23496be30bb
In this case I would use java.net.Socket to establish a connection and then read the lines. This is a simplistic approach that assumes there are 2 lines of data:
Socket theSocket;
try {
theSocket = new Socket("vuln2014.picoctf.com", 51818);
BufferedReader inFile = new BufferedReader(new InputStreamReader(theSocket.getInputStream()));
String strGreet = inFile.readLine();
String strData = inFile.readLine();
} catch (IOException e) {
e.printStackTrace();
}
As for why curl and browsers may render it properly? They are likely more lenient about the data they read and will just dump what is read from the port even if it doesn't conform to the specified protocol (like http)
I'm trying to make multiple calls to a REST API using HttpURLConnection with GET method and retrieving the response through InputStream.
It worked fine previously for 3 months but now it's throwing below exception:
SAXException Occurred during getArtifactsUrl method:: org.xml.sax.SAXParseException; Premature end of file.
at org.apache.xerces.parsers.DOMParser.parse(Unknown Source) [xercesImpl.jar:6.1.0.Final]
at org.apache.xerces.jaxp.DocumentBuilderImpl.parse(Unknown Source) [xercesImpl.jar:6.1.0.Final]
at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:121) [:1.7.0_03]
Below is the line of code where I'm making the second call to parse the response:
request = (HttpURLConnection) endpointUrl.openConnection();
inputstream = request.getInputStream();
doc = dBuilder.parse(inputstream);
First call is working fine using request and inputstream objects but second call is failing. I tried all possible answers I found in google but no luck:
after every call:
inputstream.close();
request.disconnect();
Remember that request is an HttpURLConnection object.
I greatly appreciate if you can be able to solve this as I this is a high prioirity production issue now!
First you should check for error cases and not assume it's always working.
Try this:
request = (HttpURLConnection) endpointUrl.openConnection();
request.connect(); // not really necessary (done automatically)
int statusCode = request.getResponseCode();
if (statusCode == 200) { // or maybe other 2xx codes?
// Success - should work if server gives good response
inputstream = request.getInputStream();
// if you get status code 200 and still have the same error, you should
// consider logging the stream to see what document you get from server.
// (see below *)
doc = dBuilder.parse(inputstream);
} else {
// Something happened
// handle error, try again if it makes sense, ...
if (statusCode == 404) ... // resource not found
if (statusCode == 500) ... // internal server error
// maybe there is something interesting in the body
inputstream = request.getErrorStream();
// read and parse errorStream (but probably this is not the body
// you expected)
}
Have a look at the List of HTTP status codes.
And in some nasty cases, there are other problems which are not easy to detect if you just sit behind HttpURLConnection. Then you could enable logging or snoop the TCP/IP traffic with an apropriate tool (depends on your infrastructure, rights, OS, ...). This SO post might help you.
*) In your case I suppose that you're getting a non-error status code from the server but unparseable XML. If logging the traffic is not your thing, you could read the InputStream, write it to a file and then process the stream like before. Of course you can write the stream to a ByteArrayOutputStream, get the byte[] and write that Bytes to a file and then convert them to a ByteArrayInputStream and give this to your XML-parser. Or you could use Commons IO TeeInputStream to handle that for you.
There are cases where connection.getResponseCode() throws an exception. Then it was not possible to parse the HTTP header. This should only happen if there are strange errors in server software, hardware or perhaps a firewall, proxy or load balancer not behaving well.
One more thing: You might consider choosing an HTTP Client library and not directly use HttpURLConnection.
I use URL connection to download stream in the Internet. But after i reset the modem, i can't continue download this stream caz it error: Connection reset. How i solve it?
Here is my code:
URL url = new URL(_URL);
HttpURLConnection hUC = (HttpURLConnection) url.openConnection();
hUC.connect();
while (true) {
if ((_data.num = is.read(_data.b)) == -1) {
break;
}
//write to file
fos.write(_data.b, 0, _data.num);
}
You can't - at least, not how you may be expecting.
Instead, you need to handle your exception, and determine how much data you've already read. Once your Internet connection is re-established - assuming that the HTTP server you're downloading from supports requestable byte ranges - you can then set custom HTTP Headers on the request and re-download the remaining portions. (This will require a new HttpURLConnection.)
http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35 shows the related HTTP specifications involved to make this work.
This is a bit more complicated if you're looking for a "resume" type feature.
You would need to reissue the request once the internet comes back after a disconnect, and add a header to the request in order to resume the download at the byte number where you left off.
You need to set the Range property in the request header in order to specify how far in you're resuming. Then you would just continue to write to the "fos" object from there.
Check out this url: Java: resume Download in URLConnection
I have Java webserver (no standard software ... self written). Everything seems to work fine, but when I try to call a page that contains pictures, those pictures are not displayed. Do I have to send images with the output stream to the client? Am I missing an extra step?
As there is too much code to post it here, here is a little outline what happens or is supposed to happen:
1. client logs in
2. client gets a session id and so on
3. the client is connected with an output stream
4. we built the response with the HTML-Code for a certain 'GET'-request
5. look what the GET-request is all about
6. send html response || file || image (not working yet)
So much for the basic outline ...
It sends css-files and stuff, but I still have a problem with images!
Does anybody have an idea? How can I send images from a server to a browser?
Thanks.
I check requests from the client and responses from the server with charles. It sends the files (like css or js) fine, but doesn't with images: though the status is "200 OK" the transfer-encoding is chunked ... I have no idea what that means!? Does anybody know?
EDIT:
Here is the file-reading code:
try{
File requestedFile = new File( file );
PrintStream out = new PrintStream( this.getHttpExchange().getResponseBody() );
// File wird geschickt:
InputStream in = new FileInputStream( requestedFile );
byte content[] = new byte[(int)requestedFile.length()];
in.read( content );
try{
// some header stuff
out.write( content );
}
catch( Exception e ){
e.printStackTrace();
}
in.close();
if(out!=null){
out.close();
System.out.println( "FILE " + uri + " SEND!" );
}
}
catch ( /*all exceptions*/ ) {
// catch it ...
}
Your browser will send separate GET image.png HTTP 1.1 requests to your server, you should handle these file-gets too. There is no good way to embed and image browser-independent in HTML, only the <img src="data:base64codedimage"> protocol handler is available in some browsers.
As you create your HTML response, you can include the contents of the external js/css files directly between <script></script> and <style></style> tags.
Edit: I advise to use Firebug for further diagnostics.
Are you certain that you send out the correct MIME type for the files?
If you need a tiny OpenSource webserver to be inspired by, then have a look at http://www.acme.com/java/software/Acme.Serve.Serve.html which serves us well for ad-hoc server needs.
Do I have to send those external files
or images with the output stream to
the client?
The client will make separate requests for those files, which your server will have to serve. However, those requests can arrive over the same persisten connection (a.k.a. keepalive). The two most likely reasons for your problem:
The client tries to send multiple requests over a persistent connection (which is the default with HTTP 1.1) and your server is not handling this correctly. The easiest way to avoid this is to send a Connection: close header with the response.
The client tries to open a separate connection and your server isn't handling it correctly.
Edit:
There's a problem with this line:
in.read( content );
This method is not guaranteed to fill the array; it will read an arbitrary number of bytes and return that number. You have to use it in a loop to make sure everything is read. Since you have to do a loop anyway, it's a good idea to use a smaller array as a buffer to avoid keeping the whole file in memory and running into an OutOfMemoryError with large files.
Proabably step #4 is where you are going wrong:
// 4. we built the response with the HTML-Code for a certain 'GET'-request
Some of the requests will be a 'GET /css/styles.css' or 'GET /js/main.js' or 'GET /images/header.jpg'. Make sure you stream those files in those circumstances - try loading those URLs directly.
Images (and css/js files) are requested by the browser as completely separate GET requests to the page, so there's definitely no need to "send those ... with the output stream". So if you're getting pages served up ok, but images aren't being loaded, my first guess would be that you're not setting your response headers appropriately (for example, setting the Content-Type of the response to text/html), so the browser isn't interpreting it as a proper page & therefore not loading the images.
Some other things to try if that doesn't work:
Check if you can access an image directly
Use something like firebug or fiddler to check whether the browser is actually requesting the image/css/js files & that all your request/response headers look ok
Use an existing web server!