Develop Reference

Efficient way of reading, loading and unloading to memory parts of a very big binary file in java.

I need to read the contents of a binary file containing floats that represent the positions of a big number of particles, and pass this information to my program so they can be rendered to the screen. The file contains the position of every particle at every frame, and thus can be as big as 10 or more GB.
My plan is to fill all available memory with the next frames the program will need. Once a frame has been shown, I need to free that space and load a new frame from the file. If a certain frame is requested (say, frame 12), I need to go to this frame in the file and read it, as well as all other successive frames that fit in the memory.
The question is, how can I read, and then store this information in a way that is efficient?
I have done some research, and read similar questions.
Arrays and Vectors are not an option since they can fit about 2gb of data, when I could be loading more than that to ram. Besides, initializing an array of such a size seems to lead to an "out of memory exception", and expanding an array takes too much time.
Buffer a large file; BufferedInputStream limited to 2gb; Arrays limited to 2^31 bytes
MappedByteBuffer could be an option, but it is also limited to 2 gb, and I do not understand exactly how it works.
I could also use external libraries that save data outside of the buffer (but which one?), or use a ByteBuffer, I guess, but what is the best option in this case?
In short, I need to read the file fast (probably in parts, unless there is another way), and save as much in memory as I can, and pass this information to another thread, frame by frame. How do I read this file, and where do I store the data in memory?
Read large files in Java
This source seems to address something similar, but it is about reading text, and it seems he could use a MappedByteBuffer or a BufferedInputStream, since his file is of only 1.5 GB.

most efficient way to temporarily store discontinuous data composing a larger file

We are emulating a p2p network in java. So we divide the file into chunks (with checksums) so that the individual chunks can be recompiled into the original file once we have all the parts. What is the best way to store the individual parts while they are being downloaded?
I was thinking of just storing each chunk as a separate file...but if there are 20000 chunks, it would create as many files. is this the best way?
Thanks

Either keep chunks in memory or in files. There is no much to discuss here about. Findd the perfect ratio between chunks count and the actual size of it, to suit your needs.
Files sounds more reasonable as data would not be totally lost in case of application crash and continue of download would be possible.

I would write to memory until you reach some threshold, at which point you dump your memory to disk, and keep reading into memory. When the file transfer completes, you can take what is currently stored in memory, and concatenate it with what may have been stored on disk.

Write files into randomly non-contiguous hard disk positions

I need to do some performance tests for my program and I need to evaluate the worst IO access time to some data stored in files, so I think that the best way to evaluate this is to randomly store these data into different HD sectors in order to avoid contiguous data access and caching improvements. I think that the only way to do this is using some low-level OS commands, like dd in UNIX where you can specify the sector where you write the data, but if I'm not mistaken, this is an insecure method. Someone know a good alternative to do this?
PS: Any solution for any OS will work, the only requirement is that I have to do the tests over different data size, accessing the data through a JAVA program.

I think that the only way to do this is using some low-level OS commands
No... RandomAccessFile has .seek():
final RandomAccessFile f = new RandomAccessFile(someFile, "rw");
f.seek(someRandomLong);
f.write(...);
Now, it is of course up to you to ensure that writes don't collide with one another.
Another solution is to map the file in memory and set the buffer's position to some random position before writing.

Java handling billions bytes

I'm creating a compression algorithm in Java;
to use my algorithm I require a lot of information about the structure of the target file.
After collecting the data, I need to reread the file. <- But I don't want to.
While rereading the file, I make it a good target for compression by 'converting' the data of the file to a rather peculiar format. Then I compress it.
The problems now are:
I don't want to open a new FileInputStream for rereading the file.
I don't want to save the converted file which is usually 150% the size of the target file to the disk.
Are there any ways to 'reset' a FileInputStream for moving to the start of the file, and how would I store the huge amount 'converted' data efficiently without writing to the disk?

You can use one or more RandomAccessFiles. You can memory map them to ByteBuffer() which doesn't consume heap (actually they use about 128 bytes) or direct memory but can be accessed randomly.
Your temporary data can be storing in a direct ByteBuffer(s) or more memory mapped files. Since you have random access to the original data, you may not need to duplicate as much data in memory as you think.
This way you can access the whole data with just a few KB of heap.

There's the reset method, but you need to wrap the FileInputStream in a BufferedInputStream.

You could use RandomAccessFile, or java.nio ByteBuffer is what you are looking for. (I do not know.)
Resources might be saved by pipes/streams: immediately writing to a compressed stream.
To answer your questions on reset: not possible; the base class InputStream has provisions for mark and reset-to-mark, but FileInputStream was made optimal for several operating systems and does purely sequential input. Closing and opening is best.

How would you change a single byte in a file?

What is the best way to change a single byte in a file using Java? I've implemented this in several ways. One uses all byte array manipulation, but this is highly sensitive to the amount of memory available and doesn't scale past 50 MB or so (i.e. I can't allocate 100MB worth of byte[] without getting OutOfMemory errors). I also implemented it another way which works, and scales, but it feels quite hacky.
If you're a java io guru, and you had to contend with very large files (200-500MB), how might you approach this?
Thanks!

I'd use RandomAccessFile, seek to the position I wanted to change and write the change.

If all I wanted to do was change a single byte, I wouldn't bother reading the entire file into memory. I'd use a RandomAccessFile, seek to the byte in question, write it, and close the file.