Is there a native JDK code to copy files(buffers, streams, or whatever)?
This is the preferred way to copy a file since JDK 1.4 and later
public static void copyFile(final File sourceFile, final File destFile) throws IOException
{
if (!destFile.exists())
{
destFile.createNewFile();
}
FileChannel source = null;
FileChannel destination = null;
try
{
source = new FileInputStream(sourceFile).getChannel();
destination = new FileOutputStream(destFile).getChannel();
destination.transferFrom(source, 0, source.size());
}
finally
{
source.close();
destination.close();
}
}
public abstract long transferFrom(ReadableByteChannel src,
long position,
long count)
throws IOException
... This method is potentially much
more efficient than a simple loop that
reads from this channel and writes to
the target channel. Many operating
systems can transfer bytes directly
from the filesystem cache to the
target channel without actually
copying them. ...
If by "native" you mean "part of the Java standard API" (rather than platform-dependant code, which is usually called "native" in the Java world) and by "copy files" you mean "single method that takes a file and a target path and produces a copy of the file's contents" then no, there is no such method in the standard API. You have to open an InputStream and an OutputStream (optionally get their more efficient FileChannels) and use a buffer to transfer bytes. Convenient single methods to call are found in Apache Commons IO.
Update: Since Java 7, file copy functionality has become part of the Standard API in java.nio.file.Files
Your best option is to use Java NIO:
http://java.sun.com/javase/6/docs/api/java/nio/package-summary.html
For buffers see:
http://java.sun.com/javase/6/docs/api/java/nio/package-summary.html#buffers
For stream however, see the following article:
http://java.sun.com/docs/books/tutorial/essential/io/file.html#readStream
There are frameworks built on top of this, namely Mina and Netty:
Mina - http://mina.apache.org/
Netty - http://www.jboss.org/netty
Just to add that JDK7 defines several copy methods in java.nio.file.Files, including copying files and copy files to/from streams.
Related
i'm developing an application that has to reboot the system after a file has been uploaded and verified. The file system is on an sd card, so it must be synced to be sure the uploaded file has actually been saved on the device.
I was wondering if java.io.file.Files.copy does the sync or not.
My code runs like this:
public int save(MultipartFile multipart) throws IOException {
Files.copy(multipart.getInputStream(), file, standardCopyOption.REPLACE_EXISTING);
if (validate(file)) {
sync(file); <-- is it useless?
reboot();
return 0;
} else {
Files.delete(file);
return -1;
}
}
I tried to find a way to call sync on the fs in the nio package, but the only solution that i've found is:
public void sync(Path file) {
final FileOutputStream fos = new FileOutputStream(file.toFile());
final FileDescriptor fd = fos.getFD();
fd.sync();
}
which relies on old java.io.File .
If you look at the source code for Files.copy(...), you will see that it doesn't perform a sync(). In the end, it will perform a copy of an input stream into an output stream corresponding to the first 2 arguments passed to Files.copy(...).
Furthermore, the FileDescriptor is tied to the stream from which it is obtained. If you don't perform any I/O operation with this stream, other than creating a file with new FileOutputStream(...), there will be nothing to sync() with the fie system, as is the case with the code you shared.
Thus, the only way I see to accomplish your goal is to "revert" to the old-fashioned java.io API and implement a stream-to-stream copy yourself. This will allow you to sync() on the file descriptor obtained from the same FileOutputStream that is used for the copy operation.
I'll say the copy operation is depending on your OS JRE code, so if you want to be sure of the file Copy at OS level, continue to explicitly call the sync() method.
This was because SYNC and DSYNC were annoyingly omitted from StandardCopyOption enum, yet were provided in StandardOpenOption enum for file targets, so you need to use FileChannel and SeekableByteChannel if supported by FileSystemProvider, like :
Set<? extends OpenOption> TARGET_OPEN_OPTIONS = EnumSet.of(StandardOpenOption.CREATE_NEW, StandardOpenOption.WRITE);
FileChannel.out = target.getFileSystem().provider().newFileChannel(target, TARGET_OPEN_OPTIONS);
SeekableByteChannel = Files.newByteChannel(source, StandardOpenOption.READ);
out.transferFrom(source, 0, source.size());
out.force(boolean metadata); // false == DSYNC, true == SYNC
Using java.io.FileOutputStream.getFD().sync() is an obsolete "solution", because you lose all support for NIO2 FileSystems, like the often bundled ZipFileSystem, and it can still fail if not supported by the native class implementations or OS!
Using DSYNC or SYNC when opening an OutputStream via a FileSystemProvider is another option, but may cause premature flushing of a FileSystem cache.
I've googled around for quite a while for this, but all the results point to pre-Java 7 NIO solutions. I've used the NIO stuff to read in files from the a specific place on the file system, and it was so much easier than before (Files.readAllBytes(path)). Now, I'm wanting to read in a file that is packaged in my WAR and on the classpath. We currently do that with code similar to the following:
Input inputStream = this.getClass().getClassLoader().getResourceAsStream(fileName);
ByteArrayOutputStream byteStream = new ByteArrayOutputStream();
/* iterate through the input stream to get all the bytes (no way to reliably find the size of the
* file behind the inputStream (see http://docs.oracle.com/javase/6/docs/api/java/io/InputStream.html#available()))
*/
int byteInt = -1;
try
{
byteInt = inputStream.read();
while (byteInt != -1)
{
byteStream.write(byteInt);
byteInt = inputStream.read();
}
byteArray = byteStream.toByteArray();
inputStream.close();
return byteArray;
}
catch (IOException e)
{
//...
}
While this works, I was hoping there was an easier/better way to do this with the NIO stuff in Java 7. I'm guessing I'll need to get a Path object that represents this path on the classpath, but I'm not sure how to do that.
I apologize if this is some super easy thing to do. I just cannot figure it out. Thanks for the help.
This works for me.
import java.nio.file.Files;
import java.nio.file.Paths;
// fileName: foo.txt which lives under src/main/resources
public String readFileFromClasspath(final String fileName) throws IOException, URISyntaxException {
return new String(Files.readAllBytes(
Paths.get(getClass().getClassLoader()
.getResource(fileName)
.toURI())));
}
A Path represents a file on the file system. It doesn't help to read a resource from the classpath. What you're looking after is a helper method that reads everything fro a stream (more efficiently than how you're doing) and writes it to a byte array. Apache commons-io or Guava can help you with that. For example with Guava:
byte[] array =
ByteStreams.toByteArray(this.getClass().getClassLoader().getResourceAsStream(resourceName));
If you don't want to add Guava or commons-io to your dependencies just for that, you can always read their source code and duplicate it to your own helper method.
As far as I understand, what you want is to open a ReadableByteChannel to your resource, so you can use NIO for reading it.
This should be a good start,
// Opens a resource from the current class' defining class loader
InputStream istream = getClass().getResourceAsStream("/filename.txt");
// Create a NIO ReadableByteChannel from the stream
ReadableByteChannel channel = java.nio.channels.Channels.newChannel(istream);
You should look at ClassLoader.getResource(). This returns a URL which represents the resource. If it's local to the file system, it will be a file:// URL. At that point you can strip off the scheme etc., and then you have the file name with which you can do whatever you want.
However, if it's not a file:// path, then you can fall back to the normal InputStream.
I'm working on a sc2replay parsing tool. I build it on top of MPQLIB http://code.google.com/p/mpqlib/
Unfortunately the tool uses filechannels to read through the bzip files,
and uses map(MapMode.READ_ONLY, hashtablePosition, hashTableSize);
After calling that function closing the file channel does not release the file in the process.
To be specific I cannot rename/move the file.
The problem occurs in Java 7 and it works fine on Java 6.
Here is a simple code snippet to replicate it:
FileInputStream f = new FileInputStream("test.SC2Replay");
FileChannel fc = f.getChannel();
fc.map(MapMode.READ_ONLY, 0,1);
fc.close();
new File("test.SC2Replay").renameTo(new File("test1.SC2Replay"));
commenting out the fc.map will allow you to rename the file.
P.S. from here Should I close the FileChannel?
It states that you do not need to close both filechannel and filestream because closing one will close another. I also tried closing either or both and still did not worked.
Is there a workaround on renaming the file after reading the data using FileChannel.map on Java 7, because every one seems to have Java 7 nowadays?
Good day,
it seems that FileChannel.map causes the problem on java 7. if you use FileChannel.map, you can no longer close the the file.
a quick work around is instead of using FileChannel.map(MapMode.READ_ONLY, position, length)
you can use
ByteBuffer b = ByteBuffer.allocate(length);
fc.read(b,position);
b.rewind();
It's a documented bug. The bug report referes to Java 1.4, and they consider it a documentation bug. Closing the filechannel does not close the underlying stream.
If you're using Sun JRE, you can cheat by casting to their implementation and telling it to release itself. I'd only recommend doing this if you're not reliant on the file being closed or never plan to use another JRE.
At some point, I hope that something like this will make it into the proper public API.
try (FileInputStream stream = new FileInputStream("test.SC2Replay");
FileChannel channel = stream.getChannel()) {
MappedByteBuffer mappedBuffer = channel.map(FileChannel.MapMode.READ_ONLY, 0, 1);
try {
// do stuff with it
} finally {
if (mappedBuffer instanceof DirectBuffer) {
((DirectBuffer) mappedBuffer).cleaner().clean();
}
}
}
Given
URL of an archive (e.g. a zip file)
Full name (including path) of a file inside that archive
I'm looking for a way (preferably in Java) to create a local copy of that file, without downloading the entire archive first.
From my (limited) understanding it should be possible, though I have no idea how to do that. I've been using TrueZip, since it seems to support a large variety of archive types, but I have doubts about its ability to work in such a way. Does anyone have any experience with that sort of thing?
EDIT: being able to also do that with tarballs and zipped tarballs is also important for me.
Well, at a minimum, you have to download the portion of the archive up to and including the compressed data of the file you want to extract. That suggests the following solution: open a URLConnection to the archive, get its input stream, wrap it in a ZipInputStream, and repeatedly call getNextEntry() and closeEntry() to iterate through all the entries in the file until you reach the one you want. Then you can read its data using ZipInputStream.read(...).
The Java code would look something like this:
URL url = new URL("http://example.com/path/to/archive");
ZipInputStream zin = new ZipInputStream(url.getInputStream());
ZipEntry ze = zin.getNextEntry();
while (!ze.getName().equals(pathToFile)) {
zin.closeEntry(); // not sure whether this is necessary
ze = zin.getNextEntry();
}
byte[] bytes = new byte[ze.getSize()];
zin.read(bytes);
This is, of course, untested.
Contrary to the other answers here, I'd like to point out that ZIP entries are compressed individually, so (in theory) you don't need to download anything more than the directory and the entry itself. The server would need to support the Range HTTP header for this to work.
The standard Java API only supports reading ZIP files from local files and input streams. As far as I know there's no provision for reading from random access remote files.
Since you're using TrueZip, I recommend implementing de.schlichtherle.io.rof.ReadOnlyFile using Apache HTTP Client and creating a de.schlichtherle.util.zip.ZipFile with that.
This won't provide any advantage for compressed TAR archives since the entire archive is compressed together (beyond just using an InputStream and killing it when you have your entry).
Since TrueZIP 7.2, there is a new client API in the module TrueZIP Path. This is an implementation of an NIO.2 FileSystemProvider for JSE 7. Using this API, you can access HTTP URI as follows:
Path path = new TPath(new URI("http://acme.com/download/everything.tar.gz/README.TXT"));
try (InputStream in = Files.newInputStream(path)) {
// Read archive entry contents here.
...
}
I'm not sure if there's a way to pull out a single file from a ZIP without downloading the whole thing first. But, if you're the one hosting the ZIP file, you could create a Java servlet which reads the ZIP file and returns the requested file in the response:
public class GetFileFromZIPServlet extends HttpServlet{
#Override
public void doGet(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException{
String pathToFile = request.getParameter("pathToFile");
byte fileBytes[];
//get the bytes of the file from the ZIP
//set the appropriate content type, maybe based on the file extension
response.setContentType("...");
//write file to the response
response.getOutputStream().write(fileBytes);
}
}
Is there a standard Java library that handles common file operations such as moving/copying files/folders?
Here's how to do this with java.nio operations:
public static void copyFile(File sourceFile, File destFile) throws IOException {
if(!destFile.exists()) {
destFile.createNewFile();
}
FileChannel source = null;
FileChannel destination = null;
try {
source = new FileInputStream(sourceFile).getChannel();
destination = new FileOutputStream(destFile).getChannel();
// previous code: destination.transferFrom(source, 0, source.size());
// to avoid infinite loops, should be:
long count = 0;
long size = source.size();
while((count += destination.transferFrom(source, count, size-count))<size);
}
finally {
if(source != null) {
source.close();
}
if(destination != null) {
destination.close();
}
}
}
Not yet, but the New NIO (JSR 203) will have support for these common operations.
In the meantime, there are a few things to keep in mind.
File.renameTo generally works only on the same file system volume. I think of this as the equivalent to a "mv" command. Use it if you can, but for general copy and move support, you'll need to have a fallback.
When a rename doesn't work you will need to actually copy the file (deleting the original with File.delete if it's a "move" operation). To do this with the greatest efficiency, use the FileChannel.transferTo or FileChannel.transferFrom methods. The implementation is platform specific, but in general, when copying from one file to another, implementations avoid transporting data back and forth between kernel and user space, yielding a big boost in efficiency.
Check out:
http://commons.apache.org/io/
It has copy, and as stated the JDK already has move.
Don't implement your own copy method. There are so many floating out there...
Previous answers seem to be outdated.
Java's File.renameTo() is probably the easiest solution for API 7, and seems to work fine.
Be carefull IT DOES NOT THROW EXCEPTIONS, but returns true/false!!!
Note that there seem to be problems with it in previous versions (same as NIO).
If you need to use a previous version, check here.
Here's an example for API7:
File f1= new File("C:\\Users\\.....\\foo");
File f2= new File("C:\\Users\\......\\foo.old");
System.err.println("Result of move:"+f1.renameTo(f2));
Alternatively:
System.err.println("Move:" +f1.toURI() +"--->>>>"+f2.toURI());
Path b1=Files.move(f1.toPath(), f2.toPath(), StandardCopyOption.ATOMIC_MOVE ,StandardCopyOption.REPLACE_EXISTING ););
System.err.println("Move: RETURNS:"+b1);
Google's Guava library also has these:
http://guava-libraries.googlecode.com/svn/trunk/javadoc/com/google/common/io/Files.html
Try to use org.apache.commons.io.FileUtils (General file manipulation utilities). Facilities are provided in the following methods:
(1) FileUtils.moveDirectory(File srcDir, File destDir) => Moves a
directory.
(2) FileUtils.moveDirectoryToDirectory(File src, File destDir, boolean
createDestDir) => Moves a directory to another directory.
(3) FileUtils.moveFile(File srcFile, File destFile) => Moves a file.
(4) FileUtils.moveFileToDirectory(File srcFile, File destDir, boolean
createDestDir) => Moves a file to a directory.
(5) FileUtils.moveToDirectory(File src, File destDir, boolean
createDestDir) => Moves a file or directory to the destination
directory.
It's simple, easy and fast.
Interesting observation:
Tried to copy the same file via various java classes and printed time in nano seconds.
Duration using FileOutputStream byte stream: 4 965 078
Duration using BufferedOutputStream: 1 237 206
Duration using (character text Reader: 2 858 875
Duration using BufferedReader(Buffered character text stream: 1 998 005
Duration using (Files NIO copy): 18 351 115
when using Files Nio copy option it took almost 18 times longer!!!
Nio is the slowest option to copy files and BufferedOutputStream looks like the fastest. I used the same simple text file for each class.