Develop Reference

Java FileInputStream limited to 65535 bytes

I have a program that is opening a fileInputStream, reading it and writing it to an outputstream.
I'm using AIX.
When I run from terminal, I have no issues. However, when my third party application runs it, it encounters an issue where the FileInputStream only reads the first 65535 bytes from the file and then the next call to the .read() function returns -1, even though the file is slightly bigger than 65535 bytes (it's 68372 bytes). This results in a truncated output file.
My question is, what can cause this limitation? It doesn't appear to be an intrinsic limit of FileInputStream. I suspect there is a java option being set somewhere but I can't for the life of me determine where. Could this be an OS limitation somehow?
Here's my basic code:
OutputStream lOut = new FileOutputStream("/home/fileOut.txt");
FileInputStream fIn = new FileInputStream(new File("/home/fileIn.txt"));
int ch;
byte[] buf = new byte[65536];
while((ch = fIn.read(buf)) > 0)
{
lOut.write(buf);
}
fIn.close();
lOut.close();

Leaving aside that your test code is broken; see #EJP's comments. (Yes. He >>is<< correct. Trust me / him.)
My question is, what can cause this limitation?
AFAIK, there is no such limitation.
It doesn't appear to be an intrinsic limit of FileInputStream.
There isn't one.
I suspect there is a java option being set somewhere but I can't for the life of me determine where.
There is no JVM option that would cause this behavior1.
Could this be an OS limitation somehow?
In theory yes. Or it could be a bug / limitation in the file system drivers, a "resource limit" (see "man ulimit") or a storage media error. In practice, all of these explanations are unlikely. But you could confirm this the problem by trying to read ... or copy ... the same file using an OS utility that is known to work.
Actually, I suspect that the real problem is a bug in the third-party application2 you are trying to using. It could be a similar bug to the bug in your test code, or something else.
1 - A slight overstatement. If the 3rd-party app did something really stupid (i.e. catching and squashing Throwable), then setting the heap size too small could in theory "cause" this behavior. If you ever encounter a problem like that, change vendors as soon as you can!
2 - Just to be clear, a "third party application" is an application written & supplied by a third party. You / your organization are the first party, your JVM vendor is the second party ...

Copying file data over network using Java very slow

My code has to read a portion of data from each file it loads, typically about 100,000 bytes. This works fine for local files but can be really slow over my wifi network, even though my network seems adequate (but not blistering fast)
So I created this simple test:
public void testDataCopySpeed() throws Exception
{
File file = new File("Z:\\TestFile.mp3");
System.out.println("start:"+new Date());
FileChannel fc = new FileInputStream(file).getChannel();
ByteBuffer bb = ByteBuffer.allocate(500000); //1/2 MB
fc.read(bb);
System.out.println("end:"+new Date());
}
Would take less than a second on a local file, but over a minute on a networked file.
So I then tried to test my network speed, I cannot see how to just test the wifi but I tested the internet upload/download speed using http://myspeedtestonline.com/ assuming this would be slower than my actual wifi network. It gave me:
Download Speed:512KB/second
Upload Speed :40KB/second
and I ran the same test on another computer and it gave a similar speed
So how is it I can download 1/2 MB of data in one second but it can take a minute to copy 1/2MB of data from one file in Java, the file is hosted on a NAS. ?
EDIT:So I have a couple of good answers below, what I really want to know is what is the best way to get access to the first 100,000 bytes from a set of files for read only access that will work for local and networked files, or should I have different code depending on whether or not it is not networked. Fixing the network is not really the solution, I may be able to fix my network but this software has to work on any computer, many of my customer may not have optiminal networks and would not have the skill to fix their network issues.

Can you try a memory mapped file?
File file = new File("Z:/TestFile.mp3");
System.out.println("start:"+new Date());
FileChannel fc = new FileInputStream(file).getChannel();
MappedByteBuffer bb = fc.map(FileChannel.MapMode.READ_ONLY, 0, fc.size());
System.out.println("end:"+new Date());
This might only appear faster, or it may help hide how long it takes.

You are comparing apples with oranges here. When you access http://myspeedtestonline.com/, the flash plugin is probably using the HTTP protocol, but certainly not CIFS.
When you address a file on a NAS, it's most probably using the CIFS protocol. This protocol is known to have performance problems, especially when implemented on consumer appliances (Buffalo drives, etc.).
Sometimes the MTU size is too big, causing the packets to be fragmented and resent.
So my guess is that Java is not the right address to blame in this case. In any case however, you cannot analyze the problem with a simple Java program. You should use a network sniffer for this.

How can I make my android app receive/send data faster?

I have an app that needs to transfer data back and forth between a server, but the speed is not satisfactory right now. The main part is that I'm receiving and parsing JSON data (about 200 characters long) over 3g from a server, and the fastest it will ever do the task is about 5 seconds, but sometimes it will take long enough to timeout (upwards of 30 seconds). My server is a rackspace cloud server.
I thought I was following best practices, but it can't be so with these kinds of speeds. I am using AsyncTask and the same global HttpClient variable for everything.
Can you help me find a better way?
I've thought about these options:
using TCP instead of HTTP
encoding the data to try to reduce the size (not sure how this would work)
I don't know a lot about TCP, but it seems like it would be less overhead. What would be the pros and cons of using TCP instead of HTTP? is it practical for a cell phone to do?
Thanks
fyi - once I solve the problem I'll accept an answer that's the most helpful. So far I've received some really great answers
EDIT: I made it so that I can see the progress as it downloads and I've noticed that it is staying at 0% for a long time then it is quickly going to 100% -- does anyone have any ideas in light of this new info? It may be relevant that I'm using a Samsung Epic with Froyo.

Try using GZIP to compress the data being sent. Note a code complete example, but it should get you on the right path.
Rejinderi is right; GSON rocks.
HttpGet getRequest = new HttpGet(url);
getRequest.addHeader("Accept-Encoding", "gzip");
InputStream instream = response.getEntity().getContent();
Header contentEncoding = response.getFirstHeader("Content-Encoding");
if (contentEncoding != null && contentEncoding.getValue().equalsIgnoreCase("gzip")) {
instream = new GZIPInputStream(instream);
}

TCP is just HTTP at a lower level and if you really need performance then TCP is the one you should use. HTTP is easier to develop as there are more support and easier to implement as a developer it wraps a lot of things up so you don't have to implement them yourself. The overhead for your case shouldnt be that much.
As for the JSON data. check if its taking a long time, the normal JSON library java has is damn slow take a look here
http://www.cowtowncoder.com/blog/archives/2009/09/entry_326.html
Debug and see if that is the case. if its the json parse speed i suggest you use the gson library. Its cleaner and easy to implement and much MUCH faster.

Sounds like you need to profile the application to find out where your bottleneck is. You said you are sending data of about 200 chars. That is miniscule and I don't see how compression or anything strictly data related is going to make much of an impact on such a small data set.
I think it is more likely that you have some communication issues, perhaps attempting to establish a new connection for every transfer or something along those lines that is giving you all the overhead.
Profiling is the key to resolving your issues, anything else is a shot in the dark.

URL.openStream() is very slow when ran on school's unix server

I am using URL.openStream() to download many html pages for a crawler that I am writing. The method runs great locally on my mac however on my schools unix server the method is extremely slow. But only when downloading the first page.
Here is the method that downloads the page:
public static String download(URL url) throws IOException {
Long start = System.currentTimeMillis();
InputStream is = url.openStream();
System.out.println("\t\tCreated 'is' in "+((System.currentTimeMillis()-start)/(1000.0*60))+"minutes");
...
}
And the main method that invokes it:
LinkedList<URL> ll = new LinkedList<URL>();
ll.add(new URL("http://sheldonbrown.org/bicycle.html"));
ll.add(new URL("http://www.trentobike.org/nongeo/index.html"));
ll.add(new URL("http://www.trentobike.org/byauthor/index.html"));
ll.add(new URL("http://www.myra-simon.com/bike/travel/index.html"));
for (URL tmp : ll) {
System.out.println();
System.out.println(tmp);
CrawlerTools.download(tmp);
}
Output locally (Note: all are fast):
http://sheldonbrown.org/bicycle.html
Created 'is' in 0.00475minutes
http://www.trentobike.org/nongeo/index.html
Created 'is' in 0.005083333333333333minutes
http://www.trentobike.org/byauthor/index.html
Created 'is' in 0.0023833333333333332minutes
http://www.myra-simon.com/bike/travel/index.html
Created 'is' in 0.00405minutes
Output on School Machine Server (Note: All are fast except the first one. The first one is slow regardless of what the first site is):
http://sheldonbrown.org/bicycle.html
Created 'is' in 3.2330666666666668minutes
http://www.trentobike.org/nongeo/index.html
Created 'is' in 0.016416666666666666minutes
http://www.trentobike.org/byauthor/index.html
Created 'is' in 0.0022166666666666667minutes
http://www.myra-simon.com/bike/travel/index.html
Created 'is' in 0.009533333333333333minutes
I am not sure if this is a Java issue (*A problem in my Java code) or a server issue. What are my options?
When run on the server this is the output of the time command:
real 3m11.385s
user 0m0.277s
sys 0m0.113s
I am not sure if this is relevant... What should I do to try and isolate my problem..?

You've answered your own question. It's not a Java issue, it has to do with your school's network or server.
I'd recommend that you report your timings in milliseconds and see if they're repeatable. Run that test in a loop - 1,000 or 10,000 times - and keep track of all the values you get. Import them into a spreadsheet and calculate some statistics. Look at the distribution of values. You don't know if the one data point that you have is an outlier or the mean value. I'd recommend that you do this for both networks in exactly the same way.
I'd also recommend using Fiddler or some other tool to watch network traffic as you download. You can get better insight into what's going on and perhaps ferret out the root cause.
But it's not Java. It's your code, your network. If this was a bug in the JDK it would have been fixed a long time ago. Suspect yourself first, last, and always.
UPDATE:
My network admin assured me that this
was a bad java implementation Not a
network problem. What do you think?
"Assured" you? What evidence did s/he produce to support this conclusion? What data? What measurements were taken? Sounds like laziness and ignorance to me.
It certainly doesn't explain why all the other requests behave just fine. What changed in Java between the first and subsequent calls? Did the JVM suddenly rewrite itself?
You can accept it if you want, but I'd say shame on your network admin for not being more curious. It would have been more honorable to be honest and say they didn't know, didn't have time, and weren't interested.

By Default Java prefers to use IPv6. My school's firewall
drops all IPv6 traffic (with no warning). After 3 minutes, 15 seconds Java falls back to IPv4. Seems strange to me that it takes so long to fall back to IPv4.
duffymo's answer, essentially: "Go talk to your network admin", helped me to solve the problem however I think that this is a problem caused by a strange Java implementation and a strange network configuration.
My network admin assured me that this was a bad java implementation Not a network problem. What do you think?

How to transfer a file in a client/server Java application

I am currently working on a homework assignment and I am thoroughly stuck. I am on the last question and I just can not figure out how to go about accomplishing the last task. Below is the tasks I had to complete:
The client should
save the file in the "client"
subdirectory of the home directory.
Test your program. Be sure it works
with binary files, not just text
files. Be sure it works when both
programs are on the same machine as
well as when they are separated over
the network.
Thus far when I start the server it asks what port I want to use. Then I start the client and it asks what IP and port to use. The the server immediately sends a list of the files in the home directory "server" folder. I then respond with the client with the file number I wish to download. This is where I get stuck. I can't seem to find any information about how to do this. So as you can see in my code posted below, I am trying to use a FileInputReader to convert the file to an array of bytes. Then I am sending that to the client. I am then trying to FileOutputReader the recieved array of bytes to a file. But I can't seem to find the correct methods to do that, or even if I am doing that correctly.
CLIENT
int i = 0;
while(i < 1000){
String modifiedSentence = inFromServer.readLine();
System.out.println("From Server: " + modifiedSentence);
i++;
}
while(j < 1000) {
int byteString = inFromServer.read();
ArrayList<byte[]> bytes = new ArrayList<byte[]>();
bytes.add(byteString);
}
Integer byteInt = new Integer(byteString);
FileOutputStream fo = new FileOutputStream(System.getProperty("user.home")+ "/client/text.txt");
fo.write(byteInt.byteValue());
}
}
SERVER
byte[] bytes = new byte[1024];
FileInputStream fi = new FileInputStream(file.toString() + fileArray[userChoiceInt]);
fi.read(bytes, 0, 1024);
outToClient.write(bytes, 0, 1024);
}
}
}
If anyone could offer any advice or the correct classes or methods to use I would appreciate it.
Thank you in advance.

Without spoiling the whole thing here's some hints.
This can be easily accomplish by using Socket (Server & Client). Using byte[] for transfering the file(s) will ensure that your program will work with both ascii and binary file(s).
Another approach would be to use the build in Remote Method Invocation (RMI). I haven't transfered file using this approach but I'm sure it's feasible.
And in case you didn't know, getting the user home directory is accomplished with the following call: System.getProperty( "user.home" );

You can take a look at this tutorial by Sun (Oracle). That should give you a basic understanding of sockets.
What I do seem to notice however, in the client side, you iterate a specific amount of times (1000) which is not a good idea since generally, you do not know the size of the file to be sent, the tutorial should show this and how to make the appropriate changes. Also, you keep creating the structures within the loop, so you loose any information that you have received, besides creating new datastructures each and every time, which is inefficient.
What you have to do is to move the initialization of the structures from outside the loop. Also, in the case of the modifiedSentence variable, you might want to change it from string to StringBuilder.

If you can use an IO library for this, I would suggest Netty or Mina. There are some netty examples here: http://jboss.org/netty/documentation.html
Even if you cannot use a library, these may be helpful for learning how things are done.

You probably should not ask how to do homework in the class on Web sites like this. It is not appropriate. Your server does look mostly good. Your program will only read files up to 1024 bytes though. You should look at java.io.File in more detail. There is a length method you can use to find the length of the file, so you know how much to send.