I had the following code:
#Nonnull
#SneakyThrows
private Pair<InputStream, Long> probeSize(#Nonnull final InputStream image) {
final String tmpId = UUID.randomUUID().toString();
final File probeFile = new File(tmpDir, tmpId + ".jpg");
try (final FileChannel outChannel = FileChannel.open(probeFile.toPath(), CREATE);
final ReadableByteChannel innChannel = Channels.newChannel(image)) {
outChannel.transferFrom(innChannel, 0, Long.MAX_VALUE);
}
final Long fileSize = probeFile.length();
return Pair.of(new FileInputStream(probeFile), fileSize);
}
This code consistently threw the following exception:
Caused by: java.nio.file.NoSuchFileException: /tmp/4bbc9008-e91c-4f18-b0f2-c61eed35066e.jpg
at sun.nio.fs.UnixException.translateToIOException(Unknown Source)
at sun.nio.fs.UnixException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.UnixException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.UnixFileSystemProvider.newFileChannel(Unknown Source)
looking at the javadoc of FileChannel.open(path, option) and the associated StandardOpenOption, there is no documentation that alludes to the fact that, to create a file, you must also open it for write.
The only options that work:
FileChannel.open(probeFile.toPath(), CREATE, WRITE)
FileChannel.open(probeFile.toPath(), CREATE_NEW, WRITE)
I only determined this by going through the UnixChannelFactory.newFileChannel and noticed the following:
UnixChannelFactory:
protected static FileDescriptor open(int dfd,
UnixPath path,
String pathForPermissionCheck,
Flags flags,
int mode)
throws UnixException
{
// map to oflags
int oflags;
if (flags.read && flags.write) {
oflags = O_RDWR;
} else {
oflags = (flags.write) ? O_WRONLY : O_RDONLY;
}
if (flags.write) {
if (flags.truncateExisting)
oflags |= O_TRUNC;
if (flags.append)
oflags |= O_APPEND;
// create flags
if (flags.createNew) {
byte[] pathForSysCall = path.asByteArray();
// throw exception if file name is "." to avoid confusing error
if ((pathForSysCall[pathForSysCall.length-1] == '.') &&
(pathForSysCall.length == 1 ||
(pathForSysCall[pathForSysCall.length-2] == '/')))
{
throw new UnixException(EEXIST);
}
oflags |= (O_CREAT | O_EXCL);
} else {
if (flags.create)
oflags |= O_CREAT;
}
}
Which shows that, unless you specify WRITE option, the file will never be created.
Is this a bug or an intended functionality, that FileChannel.open cannot create a file unless it is opened for write?
I'm looking at the JDK 7 Javadoc for FileChannel.open(...).
The doc for the method says:
The READ and WRITE options determine if the file should be opened for reading and/or writing. If neither option (or the APPEND option) is contained in the array then the file is opened for reading.
The doc for CREATE_NEW says:
This option is ignored when the file is opened only for reading.
The doc for CREATE says:
This option is ignored if the CREATE_NEW option is also present or the file is opened only for reading.
Putting these three snippets together, yes, this is expected behavior.
Related
I am working on a utility that zips up a number of files (for diagnostics purposes). At it's core, it uses the following function:
private void write(ZipOutputStream zipStream, String entryPath, ByteSource content) throws IOException {
try (InputStream contentStream = content.openStream()) {
zipStream.putNextEntry(new ZipEntry(entryPath));
ByteStreams.copy(contentStream, zipStream);
zipStream.closeEntry();
}
}
But one of the files I want to read is a log file that another application runs and locks. Because that file is locked, I get an IO exception.
<ERROR>java.io.IOException: The process cannot access the file because another process has locked a portion of the file
at java.base/java.io.FileInputStream.readBytes(Native Method)
at java.base/java.io.FileInputStream.read(FileInputStream.java:257)
at com.google.common.io.ByteStreams.copy(ByteStreams.java:112)
If I am willing to accept that I might get some garbage because of conflicts between my reads and the other application's writes, what is the best/easiest way to work around this? Is there a file reader that ignores locks or perhaps only reads all the unlocked sections only?
Update -- To clarify, I am looking to read a log file, or as much of it as possible. So, I could just start reading the file, wait until I get a block I can't read, catch the error, append a file end and go. Notepad++ and other programs can read files that are partially locked. I'm just looking for a way to do that without re-inventing the ByteStreams.copy function to create a "Copy as much as I can" function.
I should have perhaps asked "How to read all the unlocked parts of a log file" and I will update the title.
One possible answer (which I don't like) is to create a method almost identical to ByteStreams.copy(), which I call "copyUntilLock" which catches any IOException, then it checks to see if the exception is a because another process has locked a portion of the file.
If that is the case, then simply stop writing and return the number of bytes so far. If its some other exception go ahead and throw it. (You could also write a note to the stream like "READING FAILED DUE TO LOCK").
Still looking for a better answer. Code included below.
private static long copyUntilLock (InputStream from, OutputStream to) throws IOException {
checkNotNull(from);
checkNotNull(to);
byte[] buf = createBuffer();
long total = 0;
try {
while (true) {
int r = from.read(buf);
if (r == -1) {
break;
}
to.write(buf, 0, r);
total += r;
}
return total;
} catch (IOException iox) {
if (iox.getMessage() != null && iox.getMessage().contains("another process has locked a portion of the file")) {
return total;
} else {
throw iox;
}
}
}
Is there a simple way in Java to translate file extensions to specific file types? That is, I'd like to translate ".doc" to "Microsoft Word Document." But I also do NOT want to look inside each file to determine mime type etc because of the performance hit that entails.
Is there a library or database file listing all the currently accepted extensions and their meanings? Something I can load programmatically and then search when I need to?
Microsoft has a support article on common file name extensions.
The wikipedia page on 'list of file formats' seems pretty exhaustive.
It should be easy to copy / paste one (or both) of those lists, tweak the formatting with a text editor, and push it into your code via a hard-coded array or external resource file.
Just keep in mind that if you go this route then you are trusting that the files contain the type of data that the extension claims. This is going to be a somewhat brittle solution because it will only take one upstream error to break things.
Much better IMO is to read the magic numbers (ie. file signature) from the first handful of bytes in the file itself. Just about any production or commercial software will do this (at an absolute minimum) instead of trusting the file extension. It would still be possible for a malicious user to fake the signature, but that requires intentional action and not just a coding error.
The cost of doing this versus checking the extension is really not substantially different; unless you are processing an enormous number of files or have some super tight deadlines (and in that case, Java might not be the best choice), both approaches require not much beyond asking the OS to read a handful of bytes from disk. Checking the magic numbers will just require a few more bytes to be read, and the overhead of opening/closing a stream on each file.
As far as I know, those names are only available in the registry. I’ve seen attempts to use Swing’s FileView to get them, but I wouldn’t rely on that, especially for headless code.
public static String getFullFileTypeName(String extension)
throws IOException,
InterruptedException {
if (!extension.startsWith(".")) {
extension = "." + extension;
}
String progID = readDefaultValue(extension);
String name = readDefaultValue(progID);
return name;
}
private static String readDefaultValue(String node)
throws IOException,
InterruptedException {
String registryPath = "HKEY_CLASSES_ROOT\\" + node;
ProcessBuilder builder = new ProcessBuilder(
"reg.exe", "query", registryPath, "/ve");
builder.redirectError(ProcessBuilder.Redirect.INHERIT);
String value = null;
Process process = builder.start();
try (BufferedReader reader = new BufferedReader(
new InputStreamReader(process.getInputStream()))) {
String line;
while ((line = reader.readLine()) != null) {
int regSZIndex = line.indexOf("REG_SZ");
if (regSZIndex >= 0) {
value = line.substring(regSZIndex + 6).trim();
break;
}
}
// Consume remaining output.
if (line != null) {
int c;
do {
c = reader.read();
} while (c >= 0);
// As of Java 11, the above loop can be replaced with:
//reader.transferTo(Writer.nullWriter());
}
}
int returnCode = process.waitFor();
if (returnCode != 0) {
throw new IOException("Got return code " + returnCode
+ " from " + builder.command());
}
if (value == null) {
throw new IOException(
"Could not find value for \"" + registryPath + "\"");
}
return value;
}
I have a watch service running on a folder, when I am trying to modify and existing file using evenKind == Modify (basically pasting a same file without removing the current file) I am getting FileNotFoundException (The process cannot access the file because it is being used by another process.)
if (eventKind == StandardWatchEventKinds.ENTRY_MODIFY) {
String newFileChecksum = null;
if (eventPath.toFile().exists()) {
newFileChecksum = getFileChecksum(eventPath.toFile());
}
if (fileMapper.containsKey(eventPath)) {
String existingFileChecksum = fileMapper.get(eventPath);
if (!existingFileChecksum.equals(newFileChecksum)) {
fileMapper.replace(eventPath, existingFileChecksum, newFileChecksum);
log.info("listener.filemodified IN");
for (DirectoryListener listener : this.listeners) {
listener.fileModified(this, eventPath);
}
log.info("listener.filemodified OUT");
} else {
log.info("existing checksum");
log.debug(String.format(
"Checksum for file [%s] has not changed. Skipping plugin processing.",
eventPath.getFileName()));
}
}
}
In the code when...getFileChecksum() is called
if (eventPath.toFile().exists()) {
newFileChecksum = getFileChecksum(eventPath.toFile());
}
So ideally, eventPath.toFile().exists() is TRUE, hence code is going inside if but when getFileChecksum() is called, it goes to method...
private synchronized String getFileChecksum(File file) throws IOException, NoSuchAlgorithmException {
MessageDigest md5Digest = MessageDigest.getInstance("MD5");
FileInputStream fis = null;
if(file.exists()) {
try {
fis = new FileInputStream(file);
} catch(Exception e) {
e.printStackTrace();
}
} else {
log.warn("File not detected.");
}
byte[] byteArray = new byte[1024];
int bytesCount = 0;
while ((bytesCount = fis.read(byteArray)) != -1) {
md5Digest.update(byteArray, 0, bytesCount);
};
fis.close();
byte[] bytes = md5Digest.digest();
StringBuilder stringBuilder = new StringBuilder();
for (int i=0; i< bytes.length ;i++) {
stringBuilder.append(Integer.toString((bytes[i] & 0xff) + 0x100, 16).substring(1));
}
return stringBuilder.toString();
}
}
An exception is coming fis = new FileInputStream(file); even if the file is present in the folder.
FileNotFoundException (The process cannot access the file because it is being used by another process.)
I created a RandomAccessFile and a channel to release any LOCK placed on file, but it is not working. Please suggest what could be happening here.
//UPDATE --> This is the infinite while loop that I have,
WHAT IS HAPPENING? WHEN I PUT A FILE 1 create and 2 update are getting called, suppose, when I am deleting the file, 1 delete 1 modify is being called, and IF I PUT THE SAME FILE BACK TO FOLDER, I GET CREATE but before CREATE is finishing, MODIFY IS BEING called. and create is not running instead modify is running.
I fixed this issue by putting Thread.sleep(500) between
WatchKey wk = watchService.take();
Thread.sleep(500)
for (WatchEvent<?> event : wk.pollEvents()) {
But I dont think I can justify use of sleep here. Please help
WatchService watchService = null;
WatchKey watchKey = null;
while (!this.canceled && (watchKey == null)) {
watchService = watchService == null
? FileSystems.getDefault().newWatchService() : watchService;
watchKey = this.directory.register(watchService,
StandardWatchEventKinds.ENTRY_MODIFY, StandardWatchEventKinds.ENTRY_DELETE,
StandardWatchEventKinds.ENTRY_CREATE);
}
while (!this.canceled) {
try {
WatchKey wk = watchService.take();
for (WatchEvent<?> event : wk.pollEvents()) {
Kind<?> eventKind = event.kind();
System.out.println("Event kind : " + eventKind);
Path dir = (Path)wk.watchable();
Path eventPath = (Path) event.context();
Path fullPath = dir.resolve(eventPath);
fireEvent(eventKind, fullPath);
}
wk.reset();
}
I have a better approach, use and a while loop on a var isFileReady like this...
var isFileReady = false;
while(!isFile...) {
}
inside while create a try and catch.
try {
FileInputStream fis = new FileInputStream();
isFileReady = true;
} catch () {
catch exception or print file not ready.
}
This will solve your problem.
The WatchService is verbose and may report multiple ENTRY_MODIFY events for save operation - even when another application is part way through or doing writes repeatedly. Your code is probably acting on a modify event while the other app is still writing and there may be a second ENTRY_MODIFY on its way.
A safer strategy for using the WatchService is to collate the events you receive and only act on the changes when there is a pause. Something like this will ensure that you block on first event but then poll the watch service with small timeout to see if more changes are present before you act on the previous set:
WatchService ws = ...
HashSet<Path> modified = new HashSet<>();
while(appIsRunning) {
int countNow = modified.size();
WatchKey k = countNow == 0 ? ws.take() : ws.poll(1, TimeUnit.MILLISECONDS);
if (k != null) {
// Loop through k.pollEvents() and put modify file path into modified set:
// DO NOT CALL fireEvent HERE, save the path instead:
...
if (eventKind == ENTRY_MODIFY)
modified.add(filePath);
}
// Don't act on changes unless no new events:
if (countNow == modified.size()) {
// ACT ON modified list here - the watch service did not report new changes
for (Path filePath : modified) {
// call fireEvent HERE:
fireEvent(filePath);
}
// reset the list so next watch call is take() not poll(1)
modified.clear();
}
}
If you are also looking out for CREATE and DELETE operations with MODIFY you will have to collate and ignore some of the earlier events because the last recorded event type can take precedence over a previously recorded type. For example, if calling take() then poll(1) until nothing new is reported:
Any DELETE then CREATE => you might want to consider as MODIFY
Any CREATE then MODIFY => you might want to consider as CREATE
Any CREATE or MODIFY then a DELETE => treat as DELETE
Your logic would also want to only act when value of modified.size() + created.size() + deleted.size() gets changed between runs.
let me guess...
modify event gets called when you modify a file. to modify the file you most likely use a seperate tool like notepad that opens and LOCKS the file.
your watcher gets an event that the file gets modified (right now) but you can not modify it again (which fileinputstream wants to do) since it is locked already.
I've build a simple test which creates and deletes a file (name does not change) in an infinite loop. The test does run for a couple of seconds (sometimes over 77,000 iterations!) and then fails with this exception:
Exception in thread "main" java.io.IOException: Access is denied
at java.io.WinNTFileSystem.createFileExclusively(Native Method)
at java.io.File.createNewFile(Unknown Source)
at DeleteTest.main(DeleteTest.java:11)
Here's the test logic:
final File f = new File(pathname);
while (true) {
final boolean create = f.createNewFile();
if (!create) {
System.out.println("crate failed");
} else {
final boolean delete = f.delete();
if (!delete) {
System.out.println("delete failed");
}
}
}
How is this possible? The delete call does not fail. It would tell. So delete always succeeds but createNewFile fails. This is what MSDN says about win32 api function DeleteFile:
The DeleteFile function marks a file for deletion on close. Therefore,
the file deletion does not occur until the last handle to the file is
closed. Subsequent calls to CreateFile to open the file fail with
ERROR_ACCESS_DENIED.
So createNewFile does not close the file? The openjdk source tells us that the file is closed:
JNIEXPORT jboolean JNICALL
Java_java_io_Win32FileSystem_createFileExclusively(JNIEnv *env, jclass cls,
jstring pathname)
{
jboolean rv = JNI_FALSE;
DWORD a;
WITH_PLATFORM_STRING(env, pathname, path) {
int orv;
int error;
JVM_NativePath((char *)path);
orv = JVM_Open(path, JVM_O_RDWR | JVM_O_CREAT | JVM_O_EXCL, 0666);
if (orv < 0) {
if (orv != JVM_EEXIST) {
error = GetLastError();
// If a directory by the named path already exists,
// return false (behavior of solaris and linux) instead of
// throwing an exception
a = GetFileAttributes(path);
if ((a == INVALID_FILE_ATTRIBUTES) ||
!(a & FILE_ATTRIBUTE_DIRECTORY)) {
SetLastError(error);
JNU_ThrowIOExceptionWithLastError(env, path);
}
}
} else {
JVM_Close(orv);
rv = JNI_TRUE;
}
} END_PLATFORM_STRING(env, path);
return rv;
}
Can anyone explain this behaviour?
I've found an explanation while writing the question. I still posted the question because I wanted to share what I learned.
My application is not the only process on the system accessing files. The Windows Search Index Service for example could open this file because it wants to add it to it's index. Or the windows Explorer if it is updating the view.
This issue reminds me a problem I experienced recently with the File.renameTo() method. It is (was?) due to this bug in the jvm :
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6213298
A weird workaround is to call System.gc() and to retry renaming the file again (and it works...).
Not sure it has a link with your issue, but it may be worth exploring...
Try this:
final File f = new File("file");
while (true) {
final boolean create = f.createNewFile();
if (!create) {
System.out.println("crate failed");
} else {
final boolean delete = f.delete();
try {
Thread.sleep(10);
} catch (InterruptedException e) {
System.out.println("...");
}
if (!delete) {
System.out.println("delete failed");
}
}
}
In this way we ensure that the file is released by the delete before invoking createNewFile.
I'm getting an NPE while trying to read in an image file, and I can't for the life of me figure out why. Here is my line:
BufferedImage source = ImageIO.read(new File(imgPath));
imgPath is basically guaranteed to be valid and right before it gets here it copies the file from the server. When it hits that line, I get this stack trace:
Exception in thread "Thread-26" java.lang.NullPointerException
at com.ctreber.aclib.image.ico.ICOReader.getICOEntry(ICOReader.java:120)
at com.ctreber.aclib.image.ico.ICOReader.read(ICOReader.java:89)
at javax.imageio.ImageIO.read(ImageIO.java:1400)
at javax.imageio.ImageIO.read(ImageIO.java:1286)
at PrintServer.resizeImage(PrintServer.java:981) <---My function
<Stack of rest of my application here>
Also, this is thrown into my output window:
Can't create ICOFile: Can't read bytes: 2
I have no idea what is going on, especially since the File constructor is succeeding. I can't seem to find anybody who has had a similar problem. Anybody have any ideas? (Java 5 if that makes any difference)
I poked around some more and found that you can specify which ImageReader ImageIO will use and read it in that way. I poked around our codebase and found that we already had a function in place for doing EXACTLY what I was trying to accomplish here. Just for anybody else who runs into a similar issue, here is the crux of the code (some of the crap is defined above, but this should help anybody who tries to do it):
File imageFile = new File(filename);
Iterator<ImageReader> imageReaders = ImageIO.getImageReadersByFormatName("jpeg");
if ( imageReaders.hasNext() ) {
imageReader = (ImageReader)imageReaders.next();
stream = ImageIO.createImageInputStream(imageFile);
imageReader.setInput(stream, true);
ImageReadParam param = imageReader.getDefaultReadParam();
curImage = imageReader.read(0, param);
}
Thanks for the suggestions and help all.
The File constructor will almost certainly succeed, regardless of whether it points to a valid/existing file. At the very least, I'd check whether your underlying file exists via the exists() method.
Also note that ImageIO.read is not thread-safe (it reuses cached ImageReaders which are not thread-safe).
This means you can't easily read multiple files in parallel. To do that, you'll have to deal with ImageReaders yourself.
Have you considered that the file may simply be corrupted, or that ImageIO is trying to read it as the wrong type of file?
Googling for the ICOReader class results in one hit: IconsFactory from jide-common.
Apparently they had the same problem:
// Using ImageIO approach results in exception like this.
// Exception in thread "main" java.lang.NullPointerException
// at com.ctreber.aclib.image.ico.ICOReader.getICOEntry(ICOReader.java:120)
// at com.ctreber.aclib.image.ico.ICOReader.read(ICOReader.java:89)
// at javax.imageio.ImageIO.read(ImageIO.java:1400)
// at javax.imageio.ImageIO.read(ImageIO.java:1322)
// at com.jidesoft.icons.IconsFactory.b(Unknown Source)
// at com.jidesoft.icons.IconsFactory.a(Unknown Source)
// at com.jidesoft.icons.IconsFactory.getImageIcon(Unknown Source)
// at com.jidesoft.plaf.vsnet.VsnetMetalUtils.initComponentDefaults(Unknown Source)
// private static ImageIcon createImageIconWithException(final Class<?> baseClass, final String file) throws IOException {
// try {
// InputStream resource =
// baseClass.getResourceAsStream(file);
// if (resource == null) {
// throw new IOException("File " + file + " not found");
// }
// BufferedInputStream in =
// new BufferedInputStream(resource);
// return new ImageIcon(ImageIO.read(in));
// }
// catch (IOException ioe) {
// throw ioe;
// }
// }
What did they do instead?
private static ImageIcon createImageIconWithException(
final Class<?> baseClass, final String file)
throws IOException {
InputStream resource = baseClass.getResourceAsStream(file);
final byte[][] buffer = new byte[1][];
try {
if (resource == null) {
throw new IOException("File " + file + " not found");
}
BufferedInputStream in = new BufferedInputStream(resource);
ByteArrayOutputStream out = new ByteArrayOutputStream(1024);
buffer[0] = new byte[1024];
int n;
while ((n = in.read(buffer[0])) > 0) {
out.write(buffer[0], 0, n);
}
in.close();
out.flush();
buffer[0] = out.toByteArray();
} catch (IOException ioe) {
throw ioe;
}
if (buffer[0] == null) {
throw new IOException(baseClass.getName() + "/" + file
+ " not found.");
}
if (buffer[0].length == 0) {
throw new IOException("Warning: " + file
+ " is zero-length");
}
return new ImageIcon(Toolkit.getDefaultToolkit().createImage(
buffer[0]));
}
So you might want to try the same approach: read the raw bytes and use Toolkit to create an image from them.
"it's a jpeg but doesn't have a jpeg
extension."
That might be it.
It appears that the library AC.lib-ICO is throwing the NPE. Since this library is intended to read the Microsoft ICO file format, a JPEG might be a problem for it.
Consider explicitly providing the format using an alternative method.