I am generating a log file and what i want is that i want to read the data periodically without having to read from the beginning each time. can anyone help.
Open the file and have a loop which,
get the size and compare with the size you have read already.
if the size has grown, read that many bytes and no more. Doing this means you can read more later.
if the size has shrink, close the file and start again.
You can use FileInputStream or RandomAccessFile.
use unix command 'tail', the option '-f' and '-F' is for the same command is very handy as well.
See here http://www.thegeekstuff.com/2009/08/10-awesome-examples-for-viewing-huge-log-files-in-unix/ for examples or just google around for examples.
If you want to Run a program to read your log file periodically then you can use schedulers like, Quartz Scheduler to run it periodically.
RandomAccessFile is a good option. If you leave the application you will have to persist the place of your last read before leaving, in order to avoid rereading information.
Log files, on the other hand, tend to become quite large for heavy event flow. Rotating log files will allow you to shift your problem a little towards file naming. Your can configure your system to produce one log file per day like here:
app_access.2011-11-28.log,
app_access.2011-11-29.log,
app_access.2011-11-30.log,
...
If the files you get are still very large, you may rotate them by date and time and you will have also the hour as part of the file name. Your files could then rotate, let's say, every three hours or even every hour. This will give you more log files to read, but they will be smaller, thus easier to process. The date and time range you want to seek for will be part of the file name.
You could also additionally rotate by file size. If you select a maximum file size you can deal with you could avoid accessing randomly a huge file completely.
Related
I've got a huge csv file that keeps increasing forever [although sometimes it gets reset], I know that's not good but unfortunately I can't change the design since it's another application that keeps adding stuff there.
I have split this file into new smaller files considering that new stuff is going to appear in that csv file every time.For example, one csv file for each 1000 values or something like that.
I'm thinking about writing a small program to do it and run it periodically via Windows Scheduled Tasks, is it the best way of fixing this problem? If so, Can you help me with the code [Java, VB, C#...]. If it's not the best solution, which path should I follow?
What would be a good way to read new entries in a text file that is continuously getting updated by another thread without having too much system strain?
Only new entries are being appended that I want to read line by line, process them, and then output them in a new file.
The log file could grow a few GBs big and have around 5 million lines before being deleted.
Both the log file and this code will run on 64-bit Windows 7 with Java 7 (if that matters).
I am very new to programming, so would someone be willing to provide some sample code? Thanks.
Apache Commons has a Tailer class.
Other than that, I guess you could open a BufferedReader, and sleep if readLine comes back null (in other words, the end of the file has been reached).
But the Apache Tailer class is much better, handles rolling log files and such.
I'm currently trying to write a simple journal-like program in Java that allows me to add "entries" and be able to browse all the "entries" I have added since the very beginning. The problem is, if I run the program, add two entries, exit the program, and then run the program again, I want to be able to have access to the two entries I previously added. I guess my questions is then, how am I able to "save" (if that's the right word) the entries that I add so that they won't be wiped out every time the program terminates?
I did some looking around, and it appears there's a tool I can use called the Java Cache System, but I'm not entirely sure if that's what I need for my situation. I'd appreciate if somebody could point me in the right direction.
When you run the program and create the entries your storing them in primary storage aka RAM. As you have discovered these entries will not persist across different executions of your program.
You need to store the entries in secondary storage aka the hard drive. This can be done by writing the entries to a file saved on disk and then reading those entries upon startup of the program. Java provides several mechanisms to read and write files to the file system on a machine.
Some applications use a database to store information in a relational manner so that it is available via a SQL request, however I would recommend using a simple file to store your entries.
The simplest way would be to store this data somehow in a file, and then read it from the file when the application starts, a few simple examples on how to write/read from file:
http://www2.cs.uic.edu/~sloan/CLASSES/java/MyFileReader.java
http://www2.cs.uic.edu/~sloan/CLASSES/java/MyFileReader.txt
http://www2.cs.uic.edu/~sloan/CLASSES/java/MyFileWriter.java
http://www2.cs.uic.edu/~sloan/CLASSES/java/MyFileWriter.txt
Now, you store your objects in memory instead of this you can try to serialize them to some format like xml. And then in next run load them from xml. Or you can try to use dataBase for storing objects.
I faced same problem in past but little bit different.I clearly understood your problem , My solution is whatever the journal you are entering and getting saved should be saved in a particular location in your Location such as "C:\Your_Directory\Journal_folder\"
so it will be easier when you initially enter the journal it stores in above location ,again if u exit and reopen the application just try to retrieve the data from the above Mentioned target Location.
therefore every time when ever you enter the application it retrieves the data from that location if not it displays empty
I want to keep track of multiple files in a location in a unix box, and delete them if no one is using them for a long time.
I am trying to search for reference counting but did not get much help on google.
I also saw FileTime in java 7 which can give me the last accessed time , but i have to the above work using java 6.
If anyone has any ideas or good reference to reference counting and how i can use it, it will be great.
You can use apache common IO. Set observer on your dir which checks the dir and notifying listeners of create, change or delete eventslisten. By this it is possible to track on which dir has been working by user and others are not.
Okay, so we want last access time.
You can have a read of Get the Last Access Time for a File for some further info.
You could try and have a look at http://jdevel.wordpress.com/2011/04/08/file-last-access-time-in-java-on-linux/ for a possible soultion, but this is Linux, not Unix.
Finally, you could borrow the same idea and use JNA or JNI
There is no module for File watching till JDK 1.6. What you can do is, you can write your own file watcher by seeing the
lastModified() method of the java.io.File class.
It returns you the last modified time as a long value and you keep a watch on this file during a regular interval. If the total time difference
(presentTime - lastModifiedTime)
exceeds you time criteria, you can delete the file.
I'm trying to edit configuration file in Java. What I really need to do is to change single line, so reading the whole file and writing it back would be waste of time, since configuration file can be big.
Is there a more efficient way to do this? Except reading in/editing/writing out file. I thouhgt of converting entire file to string, replacing the line I want and writting it back.
I don't know how efficient would that be, can someone give me some other suggestions or the one I mentioned are ok, execution time is important.
I would recommend to use the Preferences API instead. Then on the Windows platform your preferences is stored in the registry. On other platforms the corresponding way to save application preferences is used. See also Preferences API Overview.
How big of a configuration file are we talking here? 1k lines? 10k? 1m lines? If the line you want to edit is the last line, just seek to the start of the line, truncate the file there and write the new one. If it's not... you will need to read it whole and write it again.
Oh, and the 2 options you mention are actually the same (read/edit/write).
On the third hand, I think it's irrelevant (unless you have weird constraints, like a flash storage device which takes too long to write, and has limited write cycles), given the sizes of most config files.