Develop Reference

How to read/copy a (partially locked) log file, or at least the unlocked parts?

I am working on a utility that zips up a number of files (for diagnostics purposes). At it's core, it uses the following function:
private void write(ZipOutputStream zipStream, String entryPath, ByteSource content) throws IOException {
try (InputStream contentStream = content.openStream()) {
zipStream.putNextEntry(new ZipEntry(entryPath));
ByteStreams.copy(contentStream, zipStream);
zipStream.closeEntry();
}
}
But one of the files I want to read is a log file that another application runs and locks. Because that file is locked, I get an IO exception.
<ERROR>java.io.IOException: The process cannot access the file because another process has locked a portion of the file
at java.base/java.io.FileInputStream.readBytes(Native Method)
at java.base/java.io.FileInputStream.read(FileInputStream.java:257)
at com.google.common.io.ByteStreams.copy(ByteStreams.java:112)
If I am willing to accept that I might get some garbage because of conflicts between my reads and the other application's writes, what is the best/easiest way to work around this? Is there a file reader that ignores locks or perhaps only reads all the unlocked sections only?
Update -- To clarify, I am looking to read a log file, or as much of it as possible. So, I could just start reading the file, wait until I get a block I can't read, catch the error, append a file end and go. Notepad++ and other programs can read files that are partially locked. I'm just looking for a way to do that without re-inventing the ByteStreams.copy function to create a "Copy as much as I can" function.
I should have perhaps asked "How to read all the unlocked parts of a log file" and I will update the title.

One possible answer (which I don't like) is to create a method almost identical to ByteStreams.copy(), which I call "copyUntilLock" which catches any IOException, then it checks to see if the exception is a because another process has locked a portion of the file.
If that is the case, then simply stop writing and return the number of bytes so far. If its some other exception go ahead and throw it. (You could also write a note to the stream like "READING FAILED DUE TO LOCK").
Still looking for a better answer. Code included below.
private static long copyUntilLock (InputStream from, OutputStream to) throws IOException {
checkNotNull(from);
checkNotNull(to);
byte[] buf = createBuffer();
long total = 0;
try {
while (true) {
int r = from.read(buf);
if (r == -1) {
break;
}
to.write(buf, 0, r);
total += r;
}
return total;
} catch (IOException iox) {
if (iox.getMessage() != null && iox.getMessage().contains("another process has locked a portion of the file")) {
return total;
} else {
throw iox;
}
}
}

Determining File Type from Extension

Is there a simple way in Java to translate file extensions to specific file types? That is, I'd like to translate ".doc" to "Microsoft Word Document." But I also do NOT want to look inside each file to determine mime type etc because of the performance hit that entails.
Is there a library or database file listing all the currently accepted extensions and their meanings? Something I can load programmatically and then search when I need to?

Microsoft has a support article on common file name extensions.
The wikipedia page on 'list of file formats' seems pretty exhaustive.
It should be easy to copy / paste one (or both) of those lists, tweak the formatting with a text editor, and push it into your code via a hard-coded array or external resource file.

Just keep in mind that if you go this route then you are trusting that the files contain the type of data that the extension claims. This is going to be a somewhat brittle solution because it will only take one upstream error to break things.
Much better IMO is to read the magic numbers (ie. file signature) from the first handful of bytes in the file itself. Just about any production or commercial software will do this (at an absolute minimum) instead of trusting the file extension. It would still be possible for a malicious user to fake the signature, but that requires intentional action and not just a coding error.
The cost of doing this versus checking the extension is really not substantially different; unless you are processing an enormous number of files or have some super tight deadlines (and in that case, Java might not be the best choice), both approaches require not much beyond asking the OS to read a handful of bytes from disk. Checking the magic numbers will just require a few more bytes to be read, and the overhead of opening/closing a stream on each file.

As far as I know, those names are only available in the registry. I’ve seen attempts to use Swing’s FileView to get them, but I wouldn’t rely on that, especially for headless code.
public static String getFullFileTypeName(String extension)
throws IOException,
InterruptedException {
if (!extension.startsWith(".")) {
extension = "." + extension;
}
String progID = readDefaultValue(extension);
String name = readDefaultValue(progID);
return name;
}
private static String readDefaultValue(String node)
throws IOException,
InterruptedException {
String registryPath = "HKEY_CLASSES_ROOT\\" + node;
ProcessBuilder builder = new ProcessBuilder(
"reg.exe", "query", registryPath, "/ve");
builder.redirectError(ProcessBuilder.Redirect.INHERIT);
String value = null;
Process process = builder.start();
try (BufferedReader reader = new BufferedReader(
new InputStreamReader(process.getInputStream()))) {
String line;
while ((line = reader.readLine()) != null) {
int regSZIndex = line.indexOf("REG_SZ");
if (regSZIndex >= 0) {
value = line.substring(regSZIndex + 6).trim();
break;
}
}
// Consume remaining output.
if (line != null) {
int c;
do {
c = reader.read();
} while (c >= 0);
// As of Java 11, the above loop can be replaced with:
//reader.transferTo(Writer.nullWriter());
}
}
int returnCode = process.waitFor();
if (returnCode != 0) {
throw new IOException("Got return code " + returnCode
+ " from " + builder.command());
}
if (value == null) {
throw new IOException(
"Could not find value for \"" + registryPath + "\"");
}
return value;
}

PlaceHolderAPI not working

As part of my plugin i have a clear chat command and at the end of the blank messages there is an option to display text. My problem is that the PlaceHolderAPI isn't working as it should.
Command Code:
if (label.equalsIgnoreCase("clearchat") || label.equalsIgnoreCase("mcc")) {
if (p.hasPermission("mystic.chat.admin.clearchat")) {
for (int i = 0; i < getConfig().getInt("clearChat.blankLines"); i++) {
Bukkit.broadcastMessage(" ");
}
for (String s : getConfig().getStringList("clearChat.endMessage")) {
s = PlaceholderAPI.setPlaceholders(p, s);
// This is here to check if the PlaceHolderAPI even knows there is place holders in it
p.sendMessage(String.valueOf(PlaceholderAPI.containsPlaceholders(s)));
Bukkit.broadcastMessage(ChatColor.translateAlternateColorCodes('&', s));
}
return true;
} else {
p.sendMessage(ChatColor.RED + "You are lacking the required permission node!");
return true;
}
}
Config File section:
clearChat:
blankLines: 256
endMessage:
- '&bChat was cleared by %player_name%'
When i run the command "/mcc" or "/clearchat" it always says false (for not recognizing any placeholders) and none of the place holders are replaced.
I do have the API correctly in the build path, and the command words perfectly, other than the place holders not converting.
I feel as if im making a stupid mistake, or that im doing this the complete wrong way...

You shouldn't need to use another api using p.getName() should suffice then using String.replace to replace the %name%

You are using the API wrong
You did this
s = PlaceholderAPI.setPlaceholders(p, s);
If the API would throw an error if there are no placeholders just surround that line with a try{} catch (Exception e) then send it to the player
p.sendMessage(s);
There is no need for the String.valueOf(s) as the API [is expected to] return a String, anyways setting the String s = PlaceholderAPI.setPlaceholders(p, s); will cast whatever object is there to a string.

Java File.exists and other File operations returning wrong results for an existing File (network, macosx)

The filesystem AirportHDD is mounted (AFP) from the beginning and the file exists when I start this little program.
I tried to figure out the whole day why the following is not working, but couldnt find any solution:
public static void main(String[] arguments)
{
while(1==1)
{
File f=new File(
"/Volumes/AirportHDD/test/lock.csv");
System.out.println(f.exists());
AmySystem.sleep(100);
}
}
the output is:
true, true, ...
as soon as I remove the file from a different computer (AirportHDD is a mounted harddisk over network) then the output keeps saying:
true, true, ...
when I open the finder and goto this directory the output changes to: false, false, ...
when the file is added again (via another pc) the output is still:
false, false, ...
but if you open the finder again and click on the directory and finder shows the existing file, the output changes suddenly to: false, true, true, true, ...
NOTE:
also all other file operations like opening for read are failing as long as java 'thinks' the file is not there
if the program itself is creating and deleting the files then problem is not occurring
just found out while testing that with samba sharing everything is ok, but with AFP it just wont work
is there a way to tell java to do the same thing as finder, like a refresh, or do not try to cache, whatever?

I think you might be looking for the WatchService. Oracle was also kind enough to provide a tutorial.
Because the longevity of these links aren't guaranteed, I'll edit in an example code in a couple of minutes. I just wanted to let you know I think I found something in case you want to start looking at it for yourself.
UPDATE
Following the linked tutorial, I came up with code like this. I'm not sure it'll work (don't have time to test it), but it might be enough to get you started. The WatchService also has a take() method that will wait for events, which means you could potentially assume the file's existence (or lack thereof) based on the last output you gave. That will really depend on what this program will be interacting with.
If this works, good. If not, maybe we can figure out how to fix it based on whatever errors you're getting. Or maybe someone else will come along and give a better version of this code (or better option altogether) if they're more acquainted with this than I am.
public static void main(String[] arguments) {
Path path = Paths.get("/Volumes/AirportHDD/test/lock.csv");
WatchService watcher = FileSystems.getDefault().newWatchService();
WatchKey key = null;
try {
key = path.register(watcher,
ENTRY_CREATE,
ENTRY_DELETE);
} catch (IOException x) {
System.err.println(x);
}
while(true) {//I tend to favor this infinite loop, but that's just preference.
key = watcher.poll();
if(key != null) {
for (WatchEvent<?> event: key.pollEvents()) {
WatchEvent.Kind<?> kind = event.kind();
if (kind == OVERFLOW || kind == ENTRY_DELETE) {
System.out.println(false);
}
else if (kind == ENTRY_CREATE) {
System.out.println(true);
}
}//for(all events)
}//if(file event occured)
else {
File f=new File(path);
System.out.println(f.exists());
}//else(no file event occured)
AmySystem.sleep(100);
}//while(true)
}//main() method

Here is a JUnit test that shows the problem
The problem still happens using Samba on OSX Mavericks. A possible reason
is explaned by the statement in:
http://appleinsider.com/articles/13/06/11/apple-shifts-from-afp-file-sharing-to-smb2-in-os-x-109-mavericks
It aggressively caches file and folder properties and uses opportunistic locking to enable better caching of data.
Please find below a checkFile that will actually attempt to read a few bytes and forcing a true file access to avoid the caching misbehaviour ...
JUnit test:
/**
* test file exists function on Network drive
* #throws Exception
*/
#Test
public void testFileExistsOnNetworkDrive() throws Exception {
String testFileName="/Volumes/bitplan/tmp/testFileExists.txt";
File testFile=new File(testFileName);
testFile.delete();
for (int i=0;i<10;i++) {
Thread.sleep(50);
System.out.println(""+i+":"+OCRJob.checkExists(testFile));
switch (i) {
case 3:
// FileUtils.writeStringToFile(testFile, "here we go");
Runtime.getRuntime().exec("/usr/bin/ssh phobos /usr/bin/touch "+testFileName);
break;
}
}
}
checkExists source code:
/**
* check if the given file exists
* #param f
* #return true if file exists
*/
public static boolean checkExists(File f) {
try {
byte[] buffer = new byte[4];
InputStream is = new FileInputStream(f);
if (is.read(buffer) != buffer.length) {
// do something
}
is.close();
return true;
} catch (java.io.IOException fnfe) {
}
return false;
}

The problem is the network file system AFP. With the use of SAMBA everything works like expected.
Maybe the OS returns the wrong file info in OSX with the use of AFP in these scenarios.

What to do if logfile has rotated?

If I am continually parsing a log file and I detect that it has rotated, what is the best practice for handling this?
Update my internal hashmap to reflect a new filePointer (end of
rotated file)
Or should I update the hashmap to 0 so it can read the
rotated file from the start
My concern is that in the case of an anomaly in situation no. 2 mentioned above, I may indirectly parse a large file from the start and put significant load on the host.
However if I use no. 1 I may miss something critical that I was parsing the log file for.
This is the code that I've put together.
currentFilePointer = util.FileManagement.getLastFilePointerFromFile(file.getName());
lastFilePointer = Long.parseLong(lastReadFiles_.get(file.getName()).toString());
if (currentFilePointer < lastFilePointer) // file has grown
{
processLineByLine(file.getName(), currentFilePointer, lastFilePointer);
} else if (currentFilePointer > lastFilePointer) // file has been rotated
{
lastReadFiles_.put(file.getName(), currentFilePointer); // Option 1
}

From the code you posted, I can't see any Hashmap... Hard to tell which solution is better :)
Here's a snippet of code I used to "tail" a file, which also supports rotates (to be run in a separate Thread):
#Override
public void start() {
File file = ...
long filePointer = 0;
try {
while (running) {
Thread.sleep(updateInterval);
long len = file.length();
if (len < filePointer) {
// File was rotated
filePointer = 0;
} else if (len > filePointer) {
// File must have had something added to it!
RandomAccessFile raf = new RandomAccessFile(file, "r");
raf.seek(filePointer);
// Here you can read the next lines using for example a BufferedReader
filePointer = raf.getFilePointer();
raf.close();
}
}
} catch (Exception e) {
// ...
}
}

I think that your issues is how to manage log files in general. Such operations can be done using script language like bash or some general script language like python. An extensive tutorial for using java is managing Logs for the Java Subsystems.