Develop Reference

Does java.nio.file.Files.copy call sync() on the file system?

i'm developing an application that has to reboot the system after a file has been uploaded and verified. The file system is on an sd card, so it must be synced to be sure the uploaded file has actually been saved on the device.
I was wondering if java.io.file.Files.copy does the sync or not.
My code runs like this:
public int save(MultipartFile multipart) throws IOException {
Files.copy(multipart.getInputStream(), file, standardCopyOption.REPLACE_EXISTING);
if (validate(file)) {
sync(file); <-- is it useless?
reboot();
return 0;
} else {
Files.delete(file);
return -1;
}
}
I tried to find a way to call sync on the fs in the nio package, but the only solution that i've found is:
public void sync(Path file) {
final FileOutputStream fos = new FileOutputStream(file.toFile());
final FileDescriptor fd = fos.getFD();
fd.sync();
}
which relies on old java.io.File .

If you look at the source code for Files.copy(...), you will see that it doesn't perform a sync(). In the end, it will perform a copy of an input stream into an output stream corresponding to the first 2 arguments passed to Files.copy(...).
Furthermore, the FileDescriptor is tied to the stream from which it is obtained. If you don't perform any I/O operation with this stream, other than creating a file with new FileOutputStream(...), there will be nothing to sync() with the fie system, as is the case with the code you shared.
Thus, the only way I see to accomplish your goal is to "revert" to the old-fashioned java.io API and implement a stream-to-stream copy yourself. This will allow you to sync() on the file descriptor obtained from the same FileOutputStream that is used for the copy operation.

I'll say the copy operation is depending on your OS JRE code, so if you want to be sure of the file Copy at OS level, continue to explicitly call the sync() method.

This was because SYNC and DSYNC were annoyingly omitted from StandardCopyOption enum, yet were provided in StandardOpenOption enum for file targets, so you need to use FileChannel and SeekableByteChannel if supported by FileSystemProvider, like :
Set<? extends OpenOption> TARGET_OPEN_OPTIONS = EnumSet.of(StandardOpenOption.CREATE_NEW, StandardOpenOption.WRITE);
FileChannel.out = target.getFileSystem().provider().newFileChannel(target, TARGET_OPEN_OPTIONS);
SeekableByteChannel = Files.newByteChannel(source, StandardOpenOption.READ);
out.transferFrom(source, 0, source.size());
out.force(boolean metadata); // false == DSYNC, true == SYNC
Using java.io.FileOutputStream.getFD().sync() is an obsolete "solution", because you lose all support for NIO2 FileSystems, like the often bundled ZipFileSystem, and it can still fail if not supported by the native class implementations or OS!
Using DSYNC or SYNC when opening an OutputStream via a FileSystemProvider is another option, but may cause premature flushing of a FileSystem cache.

Java 7 filechannel not closing properly after calling a map method

I'm working on a sc2replay parsing tool. I build it on top of MPQLIB http://code.google.com/p/mpqlib/
Unfortunately the tool uses filechannels to read through the bzip files,
and uses map(MapMode.READ_ONLY, hashtablePosition, hashTableSize);
After calling that function closing the file channel does not release the file in the process.
To be specific I cannot rename/move the file.
The problem occurs in Java 7 and it works fine on Java 6.
Here is a simple code snippet to replicate it:
FileInputStream f = new FileInputStream("test.SC2Replay");
FileChannel fc = f.getChannel();
fc.map(MapMode.READ_ONLY, 0,1);
fc.close();
new File("test.SC2Replay").renameTo(new File("test1.SC2Replay"));
commenting out the fc.map will allow you to rename the file.
P.S. from here Should I close the FileChannel?
It states that you do not need to close both filechannel and filestream because closing one will close another. I also tried closing either or both and still did not worked.
Is there a workaround on renaming the file after reading the data using FileChannel.map on Java 7, because every one seems to have Java 7 nowadays?

Good day,
it seems that FileChannel.map causes the problem on java 7. if you use FileChannel.map, you can no longer close the the file.
a quick work around is instead of using FileChannel.map(MapMode.READ_ONLY, position, length)
you can use
ByteBuffer b = ByteBuffer.allocate(length);
fc.read(b,position);
b.rewind();

It's a documented bug. The bug report referes to Java 1.4, and they consider it a documentation bug. Closing the filechannel does not close the underlying stream.

If you're using Sun JRE, you can cheat by casting to their implementation and telling it to release itself. I'd only recommend doing this if you're not reliant on the file being closed or never plan to use another JRE.
At some point, I hope that something like this will make it into the proper public API.
try (FileInputStream stream = new FileInputStream("test.SC2Replay");
FileChannel channel = stream.getChannel()) {
MappedByteBuffer mappedBuffer = channel.map(FileChannel.MapMode.READ_ONLY, 0, 1);
try {
// do stuff with it
} finally {
if (mappedBuffer instanceof DirectBuffer) {
((DirectBuffer) mappedBuffer).cleaner().clean();
}
}
}

How to estimate zip file size in java before creating it

I am having a requirement wherein i have to create a zip file from a list of available files. The files are of different types like txt,pdf,xml etc.I am using java util classes to do it.
The requirement here is to maintain a maximum file size of 5 mb. I should select the files from list based on timestamp, add the files to zip until the zip file size reaches 5 mb. I should skip the remaining files.
Please let me know if there is a way in java where in i can estimate the zip file size in advance without creating actual file?
Or is there any other approach to handle this

Wrap your ZipOutputStream into a personalized OutputStream, named here YourOutputStream.
The constructor of YourOutputStream will create another ZipOutputStream (zos2) which wraps a new ByteArrayOutputStream (baos)
public YourOutputStream(ZipOutputStream zos, int maxSizeInBytes)
When you want to write a file with YourOutputStream, it will first write it on zos2
public void writeFile(File file) throws ZipFileFullException
public void writeFile(String path) throws ZipFileFullException
etc...
if baos.size() is under maxSizeInBytes
Write the file in zos1
else
close zos1, baos, zos2 an throw an exception. For the exception, I can't think of an already existant one, if there is, use it, else create your own IOException ZipFileFullException.
You need two ZipOutputStream, one to be written on your drive, one to check if your contents is over 5MB.
EDIT : In fact I checked, you can't remove a ZipEntry easily.
http://download.oracle.com/javase/6/docs/api/java/io/ByteArrayOutputStream.html#size()

+1 for Colin Herbert: Add files one by one, either back up the previous step or removing the last file if the archive is to big. I just want to add some details:
Prediction is way too unreliable. E.g. a PDF can contain uncompressed text, and compress down to 30% of the original, or it contains already-compressed text and images, compressing to 80%. You would need to inspect the entire PDF for compressibility, basically having to compress them.
You could try a statistical prediction, but that could reduce the number of failed attempts, but you would still have to implement above recommendation. Go with the simpler implementation first, and see if it's enough.
Alternatively, compress files individually, then pick the files that won't exceedd 5 MB if bound together. If unpacking is automated, too, you could bind the zip files into a single uncompressed zip file.

There is a better option. Create a dummy LengthOutputStream that just counts the written bytes:
public class LengthOutputStream extends OutputStream {
private long length = 0L;
#Override
public void write(int b) throws IOException {
length++;
}
public long getLength() {
return length;
}
}
You can just simply connect the LengthOutputStream to a ZipOutputStream:
public static long sizeOfZippedDirectory(File dir) throws FileNotFoundException, IOException {
try (LengthOutputStream sos = new LengthOutputStream();
ZipOutputStream zos = new ZipOutputStream(sos);) {
... // Add ZIP entries to the stream
return sos.getLength();
}
}
The LengthOutputStream object counts the bytes of the zipped stream but stores nothing, so there is no file size limit. This method gives an accurate size estimation but almost as slow as creating a ZIP file.

I dont think there is any way to estimate the size of zip that will be created because the zips are processed as streams. Also it would not be technically possible to predict the size of the created compressed format unless you actually compress it.

I did this once on a project with known input types. We knew that general speaking our data compressed around 5:1 (it was all text.) So, I'd check the file size and divide by 5...
In this case, the purpose for doing so was to check that files would likely be below a certain size. We only needed a rough estimate.
All that said, I have noticed zip applications like 7zip will create a zip file of a certain size (like a CD) and then split the zip off to a new file once it reaches the limit. You could look at that source code. I have actually used the command line version of that app in code before. They have a library you can use as well. Not sure how well that will integrate with Java though.
For what it is worth, I've also used a library called SharpZipLib. It was very good. I wonder if there is a Java port to it.

Maybe you could add a file each time, until you reach the 5MB limit, and then discard the last file. Like #Gopi, I don't think there is any way to estimate it without actually compressing the file.
Of course, file size will not increase (or maybe a little, because of the zip header?), so at least you have a "worst case" estimation.

just wanted to share how we implemented manual way
int maxSizeForAllFiles = 70000; // Read from property
int sizePerFile = 22000; // Red from property
/**
* Iterate all attachment list to verify if ZIP is required
*/
for (String attachFile : inputAttachmentList) {
File file = new File(attachFile);
totalFileSize += file.length();
/**
* if ZIP required ??? based on the size
*/
if (file.length() >= sizePerFile) {
toBeZipped = true;
logger.info("File: "
+ attachFile
+ " Size: "
+ file.length()
+ " File required to be zipped, MAX allowed per file: "
+ sizePerFile);
break;
}
}
/**
* Check if all attachments put together cross MAX_SIZE_FOR_ALL_FILES
*/
if (totalFileSize >= maxSizeForAllFiles) {
toBeZipped = true;
}
if (toBeZipped) {
// Zip Here iterating all attachments
}

Java downloading files sometimes result in CRC

I've written code to automatically download a batch of files using an InputStream and a FileOutputStream.
The code is very straightforward:
is = urlConn.getInputStream();
fos = new FileOutputStream(outputFile);
eventBus.fireEvent(this, new DownloadStartedEvent(item));
int read;
byte[] buffer = new byte[2048];
while ((read = is.read(buffer)) != -1) {
fos.write(buffer, 0, read);
}
eventBus.fireEvent(this, new DownloadCompletedEvent(item));
At first sight this works very well, files get downloaded without any problems, however,
occasionally while trying to extract a batch of downloaded rar files, extraction fails with one of the rar parts having a CRC error.
As this happened a few times already, although not consistently, I started to suspect that something in this code is not correct/optimal.
It will be helpful to know that there are 4 downloads executing concurrently using the JDK FixedThreadPool mechanism:
execService.execute(new Runnable() {
#Override
public void run() {
if (item.getState().equals(DownloadCandidateState.WAITING)) {
Downloader downloader = new Downloader(eventBus);
downloader.download(item, item.getName());
}
}
});
But because every download thread uses a new instance of the Downloader class, I believe this problem is not a side effect of concurrency?
Any ideas if this occasional CRC error has to do with the code or if it has to do with something else?
UPDATE
I can verify that the file size of a problematic file is correct.
I also did a diff (on linux) on the automatically downloaded file and the manually downloaded file.
The filesize is the exact same for both files, however, diff says that the binary content differs between the 2 files:
Binary files file.rar and file(2).rar differ
UPDATE 2
I used a visual binary diff tool and could see that a sequence of 128 bytes was different, somewhere in the middle of the file. I don't understand how that could happen, as the file being downloaded doesn't change and it is being read byte per byte using an input stream. Any ideas??

You can also use Apache's HttpClient if you don't want to handle that entity streaming yourself. It's a well written and documented library. There are several usable entity / entity wrapper classes available.
Here you can have a look at entity retrieval: http://hc.apache.org/httpcomponents-client-4.0.1/tutorial/html/fundamentals.html#d4e152

You should run a diff (unix tool) comparing the original with the result to find out what has actually changed. You May see a pattern right away.

I would start by flushing (or closing) the FileOutputStream

Your code is correct provided everything is closed and no exceptions are thrown. The problem lies elsewhere, probably in the original files.

Problem seemed to have been the Linux atheros driver for my NIC.

Native JDK code to copy files

Is there a native JDK code to copy files(buffers, streams, or whatever)?

This is the preferred way to copy a file since JDK 1.4 and later
public static void copyFile(final File sourceFile, final File destFile) throws IOException
{
if (!destFile.exists())
{
destFile.createNewFile();
}
FileChannel source = null;
FileChannel destination = null;
try
{
source = new FileInputStream(sourceFile).getChannel();
destination = new FileOutputStream(destFile).getChannel();
destination.transferFrom(source, 0, source.size());
}
finally
{
source.close();
destination.close();
}
}
public abstract long transferFrom(ReadableByteChannel src,
long position,
long count)
throws IOException
... This method is potentially much
more efficient than a simple loop that
reads from this channel and writes to
the target channel. Many operating
systems can transfer bytes directly
from the filesystem cache to the
target channel without actually
copying them. ...

If by "native" you mean "part of the Java standard API" (rather than platform-dependant code, which is usually called "native" in the Java world) and by "copy files" you mean "single method that takes a file and a target path and produces a copy of the file's contents" then no, there is no such method in the standard API. You have to open an InputStream and an OutputStream (optionally get their more efficient FileChannels) and use a buffer to transfer bytes. Convenient single methods to call are found in Apache Commons IO.
Update: Since Java 7, file copy functionality has become part of the Standard API in java.nio.file.Files

Your best option is to use Java NIO:
http://java.sun.com/javase/6/docs/api/java/nio/package-summary.html
For buffers see:
http://java.sun.com/javase/6/docs/api/java/nio/package-summary.html#buffers
For stream however, see the following article:
http://java.sun.com/docs/books/tutorial/essential/io/file.html#readStream
There are frameworks built on top of this, namely Mina and Netty:
Mina - http://mina.apache.org/
Netty - http://www.jboss.org/netty

Just to add that JDK7 defines several copy methods in java.nio.file.Files, including copying files and copy files to/from streams.