If I am continually parsing a log file and I detect that it has rotated, what is the best practice for handling this?
Update my internal hashmap to reflect a new filePointer (end of
rotated file)
Or should I update the hashmap to 0 so it can read the
rotated file from the start
My concern is that in the case of an anomaly in situation no. 2 mentioned above, I may indirectly parse a large file from the start and put significant load on the host.
However if I use no. 1 I may miss something critical that I was parsing the log file for.
This is the code that I've put together.
currentFilePointer = util.FileManagement.getLastFilePointerFromFile(file.getName());
lastFilePointer = Long.parseLong(lastReadFiles_.get(file.getName()).toString());
if (currentFilePointer < lastFilePointer) // file has grown
{
processLineByLine(file.getName(), currentFilePointer, lastFilePointer);
} else if (currentFilePointer > lastFilePointer) // file has been rotated
{
lastReadFiles_.put(file.getName(), currentFilePointer); // Option 1
}
From the code you posted, I can't see any Hashmap... Hard to tell which solution is better :)
Here's a snippet of code I used to "tail" a file, which also supports rotates (to be run in a separate Thread):
#Override
public void start() {
File file = ...
long filePointer = 0;
try {
while (running) {
Thread.sleep(updateInterval);
long len = file.length();
if (len < filePointer) {
// File was rotated
filePointer = 0;
} else if (len > filePointer) {
// File must have had something added to it!
RandomAccessFile raf = new RandomAccessFile(file, "r");
raf.seek(filePointer);
// Here you can read the next lines using for example a BufferedReader
filePointer = raf.getFilePointer();
raf.close();
}
}
} catch (Exception e) {
// ...
}
}
I think that your issues is how to manage log files in general. Such operations can be done using script language like bash or some general script language like python. An extensive tutorial for using java is managing Logs for the Java Subsystems.
Related
I am working on a utility that zips up a number of files (for diagnostics purposes). At it's core, it uses the following function:
private void write(ZipOutputStream zipStream, String entryPath, ByteSource content) throws IOException {
try (InputStream contentStream = content.openStream()) {
zipStream.putNextEntry(new ZipEntry(entryPath));
ByteStreams.copy(contentStream, zipStream);
zipStream.closeEntry();
}
}
But one of the files I want to read is a log file that another application runs and locks. Because that file is locked, I get an IO exception.
<ERROR>java.io.IOException: The process cannot access the file because another process has locked a portion of the file
at java.base/java.io.FileInputStream.readBytes(Native Method)
at java.base/java.io.FileInputStream.read(FileInputStream.java:257)
at com.google.common.io.ByteStreams.copy(ByteStreams.java:112)
If I am willing to accept that I might get some garbage because of conflicts between my reads and the other application's writes, what is the best/easiest way to work around this? Is there a file reader that ignores locks or perhaps only reads all the unlocked sections only?
Update -- To clarify, I am looking to read a log file, or as much of it as possible. So, I could just start reading the file, wait until I get a block I can't read, catch the error, append a file end and go. Notepad++ and other programs can read files that are partially locked. I'm just looking for a way to do that without re-inventing the ByteStreams.copy function to create a "Copy as much as I can" function.
I should have perhaps asked "How to read all the unlocked parts of a log file" and I will update the title.
One possible answer (which I don't like) is to create a method almost identical to ByteStreams.copy(), which I call "copyUntilLock" which catches any IOException, then it checks to see if the exception is a because another process has locked a portion of the file.
If that is the case, then simply stop writing and return the number of bytes so far. If its some other exception go ahead and throw it. (You could also write a note to the stream like "READING FAILED DUE TO LOCK").
Still looking for a better answer. Code included below.
private static long copyUntilLock (InputStream from, OutputStream to) throws IOException {
checkNotNull(from);
checkNotNull(to);
byte[] buf = createBuffer();
long total = 0;
try {
while (true) {
int r = from.read(buf);
if (r == -1) {
break;
}
to.write(buf, 0, r);
total += r;
}
return total;
} catch (IOException iox) {
if (iox.getMessage() != null && iox.getMessage().contains("another process has locked a portion of the file")) {
return total;
} else {
throw iox;
}
}
}
I'm using platform windows and Java(for writing to the filing) and C# Unity3D for reading the memory mapped file.
I'm using that for Java
File f = new File("c:\\tmp\\mapped.txt");
f.delete();
FileChannel fc = new RandomAccessFile(f, "rw").getChannel();
long bufferSize=8*1000;
MappedByteBuffer mem =fc.map(FileChannel.MapMode.READ_WRITE, 0, bufferSize);
int start = 0;
long counter=1;
long startT = System.currentTimeMillis();
long noOfMessage = 1000;
for(;;)
{
if(!mem.hasRemaining())
{
start+=mem.position();
mem =fc.map(FileChannel.MapMode.READ_WRITE, start, bufferSize);
}
mem.putLong(counter);
counter++;
if(counter > noOfMessage )
break;
Thread.sleep(400);
}
For C# Unity3D I'm reading the file in memory
// Update is called once per frame
void Update()
{
using (MemoryMappedFile mappedFile = MemoryMappedFile.OpenExisting("C:\\tmp\\mapped.txt"))
{
using (var accessor = mappedFile.CreateViewAccessor())
{
accessor.Read(1, out int omegay);
Debug.Log("counter " + omegay.ToString());
}
}
}
Current Problems.
Java file writing to the file, has nulls inside the file and not integers as it supposed to do.
I get can't open file exception in C# Unity3D
Try specifying the access mode as read-only:
MemoryMappedFile.OpenExisting("C:\\tmp\\mapped.txt", MemoryMappedFileRights.Read))
Apply the same also to the view accessor.
I have not tried that, but if Java is not taking a fully exclusive lock then opening the file as read-only should work. Give it a try ;)
~Pino
I built a classic Hoffman code, with encoder and decoder. I noticed that I had a problem, I use code in "bitset", to compress the input file. But the "bitset" - does not decode all the files I send to, for example when I send a txt file, it works great, but when I send other files like BMP. It doesn't work.
Before I used bitset - the code worked - but without any compression - so I'm afraid the problem is with bitset.
The decoder I built is:
public void Decompress(String[] input_names, String[] output_names) {
HuffmanVerticle tree = new HuffmanVerticle();
tree = readTreeFile(output_names);
restoreInput(tree, output_names, input_names);
}
public static void restoreInput(HuffmanVerticle tree, String[] binary_names, String[] original_names) {
BitSet huffmanCodeBit;
try {
FileOutputStream to_original = new FileOutputStream(original_names[0]);
FileInputStream binary = new FileInputStream(binary_names[0]);
ObjectInputStream s = new ObjectInputStream(binary);
huffmanCodeBit = (BitSet) s.readObject();
System.out.println(huffmanCodeBit.toString());
int index = 0;
while(huffmanCodeBit.length() > index)
{
HuffmanVerticle tmp = tree;
while (!tmp.isNullTree())
{
boolean bit = huffmanCodeBit.get(index);
index++;
System.out.println(bit);
if (!bit)
tmp = tmp.left;
else
tmp = tmp.right;
}
to_original.write(tmp.character);
}
binary.close();
to_original.close();
} catch (Exception e) {
e.printStackTrace();
}
}
What am I missing here? Why doesn't the code work for certain files? I'm trying to run the code on some files but it doesn't work, the files that come back don't work.
The code does not work for bmp files at all, even after half an hour, for example txt files, it runs very fast.
Thank for your help.
Stackoverflowers,
I am doing a simple project using Android smartphones to create 3D forms. I am using Android Processing to make a simple App.
My code makes a 3D shape and saves it as an .STL file. It works on my laptop and saves the .STL file, but in the App. version, I need it to save to the External storage/SD Card of my phone (HTC Sensation). It does not, because of the way the “save” function (writeSTL) in the Processing library I am using has been written.
I have posted for help here (my code more complete code is here too):
http://forum.processing.org/two/discussion/4809/exporting-geometry-stl-obj-dfx-modelbuilder-and-android
...and Marius Watz who wrote the library says that the writeSTL() code is pretty much standalone and the only thing missing is (or should be) replacing the code creating the output stream, which needs to be modified to work with Android. Basically, this line:
FileOutputStream out=(FileOutputStream)UIO.getOutputStream(p.sketchPath(filename));
I am not a great programmer in that I can usually get Processing to do what I need to do but no more; this problem has me beaten. I am looking for ideas for the correct code to replace the line:...
FileOutputStream out=(FileOutputStream)UIO.getOutputStream(p.sketchPath(filename));
...with something “Android-friendly”. Calling getExternalStorageDirectory() should work but I am at a loss to find the correct structure.
The code for the writeSTL function is below.
import java.io.*;
import java.nio.ByteBuffer;
import java.nio.ByteOrder;
/**
* Output binary STL file of mesh geometry.
* #param p Reference to PApplet instance
* #param filename Name of file to save to
*/
public void customWriteSTL(UGeometry geo, PApplet p, String filename) {
byte [] header;
ByteBuffer buf;
UFace f;
try {
if (!filename.toLowerCase().endsWith("stl")) filename+=".stl";
FileOutputStream out=(FileOutputStream)UIO.getOutputStream(p.sketchPath(filename));
buf = ByteBuffer.allocate(200);
header=new byte[80];
buf.get(header, 0, 80);
out.write(header);
buf.rewind();
buf.order(ByteOrder.LITTLE_ENDIAN);
buf.putInt(geo.faceNum);
buf.rewind();
buf.get(header, 0, 4);
out.write(header, 0, 4);
buf.rewind();
UUtil.logDivider("Writing STL '"+filename+"' "+geo.faceNum);
buf.clear();
header=new byte[50];
if (geo.bb!=null) UUtil.log(geo.bb.toString());
for (int i=0; i<geo.faceNum; i++) {
f=geo.face[i];
if (f.n==null) f.calcNormal();
buf.rewind();
buf.putFloat(f.n.x);
buf.putFloat(f.n.y);
buf.putFloat(f.n.z);
for (int j=0; j<3; j++) {
buf.putFloat(f.v[j].x);
buf.putFloat(f.v[j].y);
buf.putFloat(f.v[j].z);
}
buf.rewind();
buf.get(header);
out.write(header);
}
out.flush();
out.close();
UUtil.log("Closing '"+filename+"'. "+geo.faceNum+" triangles written.\n");
}
catch (Exception e) {
e.printStackTrace();
}
}
Any suggestions are gratefully received.
Thank you in advance.
There are a few ways of doing this - some that will just work and some that are proper ... as with all things Processing/Java. It's really not that different from regular Java though - the only quirk is the root SD path, and checking if it exists or not (note that some phones have "internal" rather than "external" storage (i.e. not removable/swappable), but Android should interpret these the same AFAIK.
In classic Java fashion, you should really be checking IF the SD Card is present beforehand... I use the following structure, taken from this answer by #kaolick
String state = Environment.getExternalStorageState();
if (state.equals(Environment.MEDIA_MOUNTED)) {
// Storage is available and writeable - ALL GOOD
} else if (state.equals(Environment.MEDIA_MOUNTED_READ_ONLY)) {
// Storage is only readable - RUH ROH
} else {
// Storage is neither readable nor writeable - ABORT
}
Note that he provides a full class for you to use, which is great, and has a few convenience functions.
The second thing you might want to look at is creating a custom directory on the SD Card of the device, probably in setup() - something like this:
try{
String dirName = "//sdcard//MyAppName";
File newFile = new File(dirName);
if(newFile.exists() && newFile.isDirectory()) {
println("Directory Exists... All Good");
}
else {
println("Directory Doesn't Exist... We're Making It");
newFile.mkdirs();
}
}
catch(Exception e) {
e.printStacktrace();
}
Of course, instead of HardCoding the Path name, you should do something like
String dirName = Environment.getExternalStorageDirectory().getAbsolutePath() + "/MyAppName";
instead...
Also, note that the above try/catch should go INSIDE the case statement of "if (state.equals(Environment.MEDIA_MOUNTED))" ... or should be wrapped in a separate function anc called from there.
Then, finally, saving it. If you wanted to use a BufferedWriter, it would look like this:
BufferedWriter writer = new BufferedWriter(new FileWriter(dirName, true));
writer.write(STL_STUFF);
writer.flush();
writer.close();
I've only use a FileOutputStream within a BufferedOutput Stream, and it looked like this:
try {
String fileName = "SOME_UNIQUE_NAME_PER_FILE";
String localFile = dirName + "/" +filename;
OutputStream output = new BufferedOutputStream(newFileOutputStream(localFile));
}
catch(Exception e) {
e.printStackTrace();
}
Finally, give my regards to Marius if you talk to him! ;-)
I'm currently using jpathwatch to watch for new files created in a folder. All fine, but I need to find out when a program finished writing to a file.
The library's author describes on his website (http://jpathwatch.wordpress.com/faq/) how that's done but somehow I don't have a clue how to do that. Maybe it's described a bit unclear or I just don't get it.
I would like to ask whether you could give me a snippet which demonstrates how to do that.
This is the basic construct:
public void run() {
while (true) {
WatchKey signalledKey;
try {
signalledKey = watchService.take();
} catch (InterruptedException ix) {
continue;
} catch (ClosedWatchServiceException cwse) {
break;
}
List<WatchEvent<?>> list = signalledKey.pollEvents();
signalledKey.reset();
for (WatchEvent<?> e : list) {
if (e.kind() == StandardWatchEventKind.ENTRY_CREATE) {
Path context = (Path) e.context();
String filename = context.toString();
// do something
} else if (e.kind() == StandardWatchEventKind.ENTRY_DELETE) {
Path context = (Path) e.context();
String filename = context.toString();
// do something
} else if (e.kind() == StandardWatchEventKind.OVERFLOW) {
}
}
}
}
From the FAQ for jpathwatch, the author says that you will get an ENTRY_MODIFY event regularly when a file is being written and that event will stop being generated when the file writing is complete. He is suggesting that you keep a list of files and the time stamp for the last generated event for each file.
At some interval (which he refers to as a timeout), you scan through the list of files and their timestamps. If any file has a time stamp that is older than your timeout interval, then that should mean that it isn't being updated anymore and is probably complete.
He even suggests you try to determine the rate at a file is growing and calculate out when it should complete so that you can set your poll time to the expected completion duration.
Does that clear it up at all? Sorry I'm not up to expressing that in code :)