Is there any way of reliably "allocating" (reserving) hard disk space via "standard" Java (J2SE 5 or later)?
Take for example the case of a multithreaded application, executing in a thread pool, where every thread downloads files. How can the application make sure that its download won't be interrupted as a result of disk space exhaustion?
At least, if it knows beforehand the size of the file it is downloading, can it do some sort of "reservation", which would guarantee file download, irrespective of what the other threads are doing?
(There is a similar question in StackOverflow, but it does not discuss multithreading and also uses NIO.)
EDIT: After some further testing, the solution proposed on the similar question does not seem to work, as one can set ANY allowed length via the suggested RandomAccessFile approach, irrespective of the underlying hard disk space. For example, on a partition with only a few gigabytes available, I was able to create TB (terrabyte!) files at will.
The solution would have been sufficient if the getFreeSpace() method of the File class reported a decreased amount of available space every time one created a new file, but it actually doesn't, thus confirming the zero length which, in practice, these files seem to have.
These are at least the results I am seeing on a CentOS 5.6 virtual machine, running in VMWare Player 4.0.0.
Write zeros to the file. That will ensure you have allocated disk space (unless drive compression or some other variable-size encoding of the file is in use).
You might get away with writing a single zero for every block, but determining the blocksize may not be trivial.
This is to avoid the creation of a sparse file which does not have all your space allocated.
Related
Memory mapping a large file on Android in Java works good. But when mapping more than ~1.5GB in total even with multiple mapping calls it fails with:
mmap failed: ENOMEM (Out of memory)
See the full discussion here. Note: It does not fail on a server Linux. The android:largeHeap="true" is enabled for the application.
The following Java code is called a few hundred times requesting ~1MB per call:
ByteBuffer buf = raFile.getChannel().map(allowWrites ? FileChannel.MapMode.READ_WRITE : FileChannel.MapMode.READ_ONLY, offset, byteCount);
to avoid requesting one large contiguous memory chunk which is often harder to be found. See the full code here. Keep in mind that doubling the 'segment size' (i.e. the size of a single map call) has no effect, which means it stops at the similar memory position. Also it is important to note that 2 apps with both slightly under the limit are executing fine (hinting to a per process limit).
Related questions are here, here, here, here, here, here, here and here.
Will using multiple files instead of one file with multiple mappings help?
I've read that this could be a per process limit for the virtual address space. Where can I find more about this? Can I change this setting with NDK e.g. how to call ulimit? Could madvise help me a bit here?
Update
See my answer here for a mmap tool usable in Java
Your problem is surely caused by virtual address space exhausting. Probably your problem reproduces on 32-bit Android devices, where available to user address space is physically limited to 2GB and cannot be bumped. (Although it may be 3GB (unlikely so) and it is configured during OS build process). Probably ~500 MB is used for system libraries, JVM and its heap. And ~1.5 GB is available for you.
The only way in this situation IMO - keep being mapped only file sections that are really used now and unmap unused ones as soon as possible. You can utilize some kind of sliding window where only small part of file will be mapped to memory, and when you finish - unmap that part, advance your window position and map that updated window, so on.
Also when you map whole large file - your process becomes an attractive victim for system's Out-Of-Memory killer. Because when you read such mapped file - consumption of physical memory raises and at some moment process will be killed.
As we cannot increase the virtual address limit on Android via an API (that does not need root access), also I've not yet seen this in the Android source code. The only possible solution I see is to implement kind of a cache, which mmaps segments on access and releases older segments if a certain number of segments is already mmapped. This means we are doing the work the OS normally does for us automatically, which is a bit ugly.
To make it working under Android one can use this answer / util-mmap. Hopefully someone can implement such a mmap cache for Android at some point, maybe even us :)
try to add largeHeap in your manifest. May be it works
I am trying to test a Java program that I wrote, in order to prove its efficiency. I have many different tests to run, and I am trying to be precise in the runtime analysis. The problem is that different tests may access information on the disk that is common. So, in order to be "fair" in my experimental results I would like to somehow programmatically clear the internal memory being used by my Java program, in between experiments. In other words, I want each experiment to have the same "empty memory/cache".
I tried reading in a large file in between experiments. I also tried restarting my machine. Interestingly, the times are much much worse when I read in a large file, then when I simply restart the machine (say 40 sec to 5 sec). What is the correct way to clear internal memory (i.e., avoid the artificial speedup for experiments from common disk accesses) beyond restarting my machine between each experiment, which is not feasible?
What is the correct way to clear internal memory
Restart the computer. As you say this might not be the slowest option.
Computers are design to be efficient as possible, making them deliberately inefficient means making some assumption about how inefficient you want to make it, so there is no standard way of doing this.
One way of clearing the cache for files you have written is to delete them. If you want to clear the cache for file in disk you want to keep, you can write a large set of files, larger than your main memory and delete them.
If you are using Windows it doesn't use all the memory to cache files in any case. What you can do is to write large files of a few hundred MB at a time and time how long this takes. When the cache is exhausted you see a sudden jump in the time it takes from Java and you know at this point you have probably clearer the cache. After this happens, delete them and your cache is likely to be empty as a result.
This can help you get reproducible results but this may not be exactly the same as rebooting. Note: unless you expect to reboot your computer every time you run your application, the time it takes after a reboot may not be meaningful anyway.
BTW: The disk cache you should be worrying about the in the OS. Java doesn't cache files except in buffers you create and in the code you write so these should be under your control.
I want to write a big file to the local disk.
I split the big file into many small files and I tried to write it to the disk. But I observed that when I split the files and tried to write, there was a big increase in disk write time.
Also, I copy the files from a disk and write it another computer's disk(reducer). I observed that there was a big increase in read time as well. Can anybody explain me the reason? I am working with hadoop.
Thanks!
That's due to the underlying file system and hardware.
There's overhead for each file in addition to its contents, for example MFT for NTFS(on Windows). So for a single large file the file system could do less bookkeeping.Thus it's faster.
As arranged by your OS, single big file tends to be written on consecutive sectors of the hard drive where possible, but multiple small files may or may not be written as such. So the resulting increased seek time may account for the increased reading time for many small files.
The efficiency of your OS may also play a big part. For example whether it prefetches file contents, how it makes use of buffer, etc. For many small files it's more difficult for the OS to use the buffer(and deal with other issues) efficiently.(Under different scenarios it can behave differently.)
EDIT: As for the copy process you mentioned, generally your OS do it in the following steps:
read data from disk->writing data to buffer->read from buffer->write to (possibly another) disk
This is usually done in multiple threads. When dealing with many small files, the OS may fail to coordinate these threads efficiently(Some threads are very busy, while others must wait). For a single large file the OS doesn't have to deal with these issues.
Every file system has a smallest unit(non sharable) defined to store the data named page. Say for example, in the file system, you have a page size of 4KB. Now if you save a big file of 8 KB, it will consume 2 pages on the disk. But if you break the file in 4 files, each of size 2KB, then it will consume 4 half filled pages on the disk consuming size 16KB disk space.
Similarly, if you break the file in 8 small files, each of size 1KB, then it will consume 8 pages in the disk though partially filled and your 32KB of the disk space is consumed.
Same is true in the reading overhead. If your file as several pages, then might be scattered. This will lead into high overhead in seektime/access time.
This looks like a long question because of all the context. There are 2 questions inside the novel below. Thank you for taking the time to read this and provide assistance.
Situation
I am working on a scalable datastore implementation that can support working with data files from a few KB to a TB or more in size on a 32-bit or 64-bit system.
The datastore utilizes a Copy-on-Write design; always appending new or modified data to the end of the data file and never doing in-place edits to existing data.
The system can host 1 or more database; each represented by a file on-disk.
The details of the implementation are not important; the only important detail being that I need to constantly append to the file and grow it from KB, to MB, to GB to TB while at the same time randomly skipping around the file for read operations to answer client requests.
First-Thoughts
At first glance I knew I wanted to use memory-mapped files so I could push the burden of efficiently managing the in-memory state of the data onto the host OS and out of my code.
Then all my code needs to worry about is serializing the append-to-file operations on-write, and allowing any number of simultaneous readers to seek in the file to answer requests.
Design
Because the individual data-files can grow beyond the 2GB limit of a MappedByteBuffer, I expect that my design will have to include an abstraction layer that takes a write offset and converts it into an offset inside of a specific 2GB segment.
So far so good...
Problems
This is where I started to get hung up and think that going with a different design (proposed below) might be the better way to do this.
From reading through 20 or so "memory mapped" related questions here on SO, it seems mmap calls are sensitive to wanting contiguous runs of memory when allocated. So, for example, on a 32-bit host OS if I tried to mmap a 2GB file, due to memory fragmentation, my chances are slim that mapping will succeed and instead I should use something like a series of 128MB mappings to pull an entire file in.
When I think of that design, even say using 1024MB mmap sizes, for a DBMS hosting up a few huge databases all represented by say 1TB files, I now have thousands of memory-mapped regions in memory and in my own testing on Windows 7 trying to create a few hundred mmaps across a multi-GB file, I didn't just run into exceptions, I actually got the JVM to segfault every time I tried to allocate too much and in one case got the video in my Windows 7 machine to cut out and re-initialize with a OS-error-popup I've never seen before.
Regardless of the argument of "you'll never likely handle files that large" or "this is a contrived example", the fact that I could code something up like that with those type of side effects put my internal alarm on high-alert and made consider an alternative impl (below).
BESIDES that issue, my understanding of memory-mapped files is that I have to re-create the mapping every time the file is grown, so in the case of this file that is append-only in design, it literally constantly growing.
I can combat this to some extent by growing the file in chunks (say 8MB at a time) and only re-create the mapping every 8MB, but the need to constantly be re-creating these mappings has me nervous especially with no explicit unmap feature supported in Java.
Question #1 of 2
Given all of my findings up to this point, I would dismiss memory-mapped files as a good solution for primarily read-heavy solutions or read-only solutions, but not write-heavy solutions given the need to re-create the mapping constantly.
But then I look around at the landscape around me with solutions like MongoDB embracing memory-mapped files all over the place and I feel like I a missing some core component here (I do know it allocs in something like 2GB extents at a time, so I imagine they are working around the re-map cost with this logic AND helping to maintain sequential runs on-disk).
At this point I don't know if the problem is Java's lack of an unmap operation that makes this so much more dangerous and unsuitable for my uses or if my understanding is incorrect and someone can point me North.
Alternative Design
An alternative design to the memory-mapped one proposed above that I will go with if my understanding of mmap is correct is as follows:
Define a direct ByteBuffer of a reasonable configurable size (2, 4, 8, 16, 32, 64, 128KB roughly) making it easily compatible with any host platform (don't need to worry about the DBMS itself causing thrashing scenarios) and using the original FileChannel, perform specific-offset reads of the file 1 buffer-capacity-chunk at a time, completely forgoing memory-mapped files at all.
The downside being that now my code has to worry about things like "did I read enough from the file to load the complete record?"
Another down-side is that I don't get to make use of the OS's virtual memory logic, letting it keep more "hot" data in-memory for me automatically; instead I just have to hope the file cache logic employed by the OS is big enough to do something helpful for me here.
Question #2 of 2
I was hoping to get a confirmation of my understanding of all of this.
For example, maybe the file cache is fantastic, that in both cases (memory mapped or direct reads), the host OS will keep as much of my hot data available as possible and the performance difference for large files is negligible.
Or maybe my understanding of the sensitive requirements for memory-mapped files (contiguous memory) are incorrect and I can ignore all that.
You might be interested in https://github.com/peter-lawrey/Java-Chronicle
In this I create multiple memory mappings to the same file (the size is a power of 2 up to 1 GB) The file can be any size (up to the size of your hard drive)
It also creates an index so you can find any record at random and each record can be any size.
It can be shared between processes and used for low latency events between processes.
I make the assumption you are using a 64-bit OS if you want to use large amounts of data. In this case a List of MappedByteBuffer will be all you ever need. It makes sense to use the right tools for the job. ;)
I have found it performance well even with data sizes around 10x your main memory size (I was using a fast SSD drive so YMMV)
I think you shouldn't worry about mmap'ping files up to 2GB in size.
Looking at the sources of MongoDB as an example of DB making use of memory mapped files you'll find it always maps full data file in MemoryMappedFile::mapWithOptions() (which calls MemoryMappedFile::map()). DB data spans across multiple files each up to 2GB in size. Also it preallocates data files so there's no need to remap as the data grows and this prevents file fragmentation. Generally you can inspire yourself with the source code of this DB.
Is there any way to access the number of blocks allocated to a file with the standard Java File API? Or even do it with some unsupported & undocumented API underneat. Anything to avoid native code plugins.
I'm talking about the st_blocks field of struct stat that the fstat/stat syscalls work on in Unix.
What I want to do is to create a sparse copy of a file that now has lots of redundant data, i.e. make a new copy of it, only containing the active data but sparsely written to it. Then swap the two files with an atomic rename/link operation. But I need a way to find out how many blocks are allocated to the file beforehand, it might already have been sparsely copied. The old file is then removed.
This will be used to free up disk space in a database application that is 100% Java. The benefit on relying on sparse file support in the filesystem is that I would not have to change the index that point out the location where the data is, that increases the complexity of the task at hand.
I think I can do somewhat well by relying on the file timestamp to see if files have already been cleaned up. But this intrigued me. I can not even find anything in the java 7 NIO.2 API for file attribute access at this level.
The only way I can think of is to use ls -s filename to get the actual size of the file on disk. http://www.lrdev.com/lr/unix/sparsefile.html