I try to compare two files by byte blocks. Block by block. But I have a problem with my loop-tree.
public void compare() {
File file1 = arrayOfFiles.get(i);
File file2 = arrayOfFiles.get(y);
if (file1.length() != file2.length()) {
break;
}
else
{
for (int z = 0; ; z++) {
byte[] b1 = getParts(file1, z);
byte[] b2 = getParts(file2, z);
if (b1.length != b2.length) {
break;
}
else
{
for (int x = 0; ; x++) {
if (b1[x] != b2[x]) {
break;
}
else
{
//how can I find the end of file? and compare last [x] of b1 and b2?
}
}
}
}
}
}
private static byte[] getParts(File file, int z) throws IOException {
byte [] bytes = new byte[1024];
int point = z * 1024;
RandomAccessFile raf = new RandomAccessFile(file, "r");
FileChannel fc = raf.getChannel();
MappedByteBuffer buffer = fc.map(FileChannel.MapMode.READ_ONLY, point, 1024);
buffer.get(bytes);
buffer.clear();
return bytes;
}
Is there another way to compare two files by bytes and do it with block of different size?
To compare last byte block of files your program could benefit from minor modification. Start iterating blocks from the last block. Changed the for clauses as following to iterate backwards.
import java.lang.Math;
import java.nio.channels.FileChannel;
import java.io.File;
import java.io.IOException;
import java.io.RandomAccessFile;
import java.nio.MappedByteBuffer;
public class Compare
{
public static final int BLOCK_SIZE = 1024;
public boolean compare(File file1, File file2)
{
//File file1 = arrayOfFiles.get(i);
//File file2 = arrayOfFiles.get(y);
boolean equal = file1.length() != file2.length();
for (int z = getSizeInBlocks(file1) - 1; equal && 0 <= z ; z--)
{
MappedByteBuffer b1 = getParts(file1, z);
MappedByteBuffer b2 = getParts(file2, z);
if (b1.remaining() != b2.remaining())
{
equal = false;
break;
}
else
{
for (int x = getBlockSize() - 1; equal && 0 <= x; x--)
{
if (b1[x] != b2[x])
{
equal = false;
break;
}
else
{
//how can I find the end of file? and compare last [x] of b1 and b2?
}
}
}
}
return equal;
}
private static int getSizeInBlocks(File file)
{
return (int) Math.ceil((double)getBlockSize()/file.length());
}
private static int getBlockSize()
{
return BLOCK_SIZE;
}
private static ByteBuffer getParts(File file, int z)
throws IOException
{
int point = z * getBlockSize();
RandomAccessFile raf = new RandomAccessFile(file, "r");
FileChannel fc = raf.getChannel();
MappedByteBuffer buffer = fc.map(
FileChannel.MapMode.READ_ONLY,
point,
getBlockSize());
return buffer;
}
}
Related
I have tried to get the size of the image by .length in java.
However the original size of the image is several bytes higher than that.
What is the reason for this? Is there any code to get the original size?
import java.awt.image.BufferedImage;
import java.io.ByteArrayOutputStream;
import java.io.File;
import javax.imageio.ImageIO;
public class Array {
public static void main(String argv[]) throws IOException {
String imageFile1 = "C:/Users/Desktop/4.jpg";
File file = new File(imageFile1);
BufferedImage originalImage = ImageIO.read(file);
ByteArrayOutputStream baos = new ByteArrayOutputStream();
ImageIO.write(originalImage, "jpg", baos);
byte[] imageInByte = baos.toByteArray();
System.out.println("The length in bytes " + imageInByte.length);
}
}
I think it's about jpeg file header size.
If you want to get an original size when copying image file.
You can use just file copy rather then image file copy.
Or, you can make your own jpeg library, if you really want.
For just one example,
this is one of the old-style code by using java's NIO.
private static void fileCopy(String from, String to) {
FileChannel fromCh = null;
FileChannel toCh = null;
FileInputStream fin = null;
FileOutputStream fout = null;
try {
fin = new FileInputStream(new File(from));
fromCh = fin.getChannel();
fout = new FileOutputStream(new File(to));
toCh = fout.getChannel();
fromCh.transferTo(0, fin.available(), toCh);
} catch (IOException e) {
e.printStackTrace();
} finally {
if (fin != null)
try {
fin.close();
} catch (IOException e) {
}
if (fout != null)
try {
fout.close();
} catch (IOException e) {
}
}
}
You can get a file from the original file with the same size.
Check the reference site below:
https://en.wikipedia.org/wiki/JPEG_File_Interchange_Format
I tested it the difference between two file, one is the original another is the copy one. I got an jpeg image from googling.
I modified some code form here in order to analyze the jpeg header file .
Here is the method:
final public static ImageProperties getJpegProperties(File file) throws FileNotFoundException, IOException {
BufferedInputStream in = null;
try {
in = new BufferedInputStream(new FileInputStream(file));
// check for "magic" header
byte[] buf = new byte[2];
int count = in.read(buf, 0, 2);
if (count < 2) {
throw new RuntimeException("Not a valid Jpeg file!");
}
if ((buf[0]) != (byte) 0xFF || (buf[1]) != (byte) 0xD8) {
throw new RuntimeException("Not a valid Jpeg file!");
}
int width = 0;
int height = 0;
char[] comment = null;
boolean hasDims = false;
boolean hasComment = false;
int ch = 0;
int totalHeaderLen = 0;
while (ch != 0xDA && !(hasDims && hasComment)) {
/* Find next marker (JPEG markers begin with 0xFF) */
while (ch != 0xFF) {
ch = in.read();
}
/* JPEG markers can be padded with unlimited 0xFF's */
while (ch == 0xFF) {
ch = in.read();
}
/* Now, ch contains the value of the marker. */
int length = 256 * in.read();
length += in.read();
totalHeaderLen += length;
if (length < 2) {
throw new RuntimeException("Not a valid Jpeg file!");
}
/* Now, length contains the length of the marker. */
if (ch >= 0xC0 && ch <= 0xC3) {
in.read();
height = 256 * in.read();
height += in.read();
width = 256 * in.read();
width += in.read();
for (int foo = 0; foo < length - 2 - 5; foo++) {
in.read();
}
hasDims = true;
} else if (ch == 0xFE) {
// that's the comment marker
comment = new char[length - 2];
for (int foo = 0; foo < length - 2; foo++)
comment[foo] = (char) in.read();
hasComment = true;
} else {
// just skip marker
for (int foo = 0; foo < length - 2; foo++) {
in.read();
}
}
}
if(comment == null) comment = "no comment".toCharArray();
return (new ImageProperties(width, height, new String(comment), totalHeaderLen, "jpeg"));
} finally {
if (in != null) {
try {
in.close();
} catch (IOException e) {
}
}
}
}
The main method is :
public static void main(String argv[]) throws IOException {
String imageFile1 = "resource/4.jpg";
String imageFile2 = "resource/4_jpg.jpg";
//copyImage(imageFile1);
ImageProperties origin = getJpegProperties(new File(imageFile1));
ImageProperties copyed = getJpegProperties(new File(imageFile2));
System.out.println("============ Original one ===========");
System.out.println("comments(origin) : " + origin.getComments());
System.out.println("Height(origin) : " + origin.getHeight());
System.out.println("Width(origin) : " + origin.getWidth());
System.out.println("Header Length(origin) : " + origin.getHeaderLen());
//System.out.println("suffix(origin) : " + origin.getSuffix());
System.out.println();
System.out.println("============ Copy one ===========");
System.out.println("comments(copy) : " + copyed.getComments());
System.out.println("Height(copy) : " + copyed.getHeight());
System.out.println("Width(copy) : " + copyed.getWidth());
System.out.println("Header Length(copy) : " + copyed.getHeaderLen());
//System.out.println("suffix(copy) : " + copyed.getSuffix());
}
I copy the original image first using copyImage method here.
static BufferedImage copyImage(BufferedImage source) {
BufferedImage b = new BufferedImage(source.getWidth(), source.getHeight(), source.getType());
Graphics g = b.getGraphics();
g.drawImage(source, 0, 0, null);
g.dispose();
return b;
}
I can see the difference of the two images in the explorer.
I got a different header size when running the program.
The output:
============ Original one ===========
comments(origin) : no comment
Height(origin) : 534
Width(origin) : 800
Header Length(origin) : 21269
============ Copy one ===========
comments(copy) : no comment
Height(copy) : 534
Width(copy) : 800
Header Length(copy) : 603
Header Length is different as you can see the result.
Here is full test code.
package stackoverflow;
import java.awt.Graphics;
import java.awt.image.BufferedImage;
import java.io.BufferedInputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.nio.channels.FileChannel;
public class Misc {
public static void main(String argv[]) throws IOException {
String imageFile1 = "resource/4.jpg";
String imageFile2 = "resource/4_jpg.jpg";
String imageFile3 = "resource/4_org.jpg";
fileCopy(imageFile1, imageFile3);
//copyImage(imageFile1);
ImageProperties origin = getJpegProperties(new File(imageFile1));
ImageProperties copyed = getJpegProperties(new File(imageFile2));
System.out.println("============ Original one ===========");
System.out.println("comments(origin) : " + origin.getComments());
System.out.println("Height(origin) : " + origin.getHeight());
System.out.println("Width(origin) : " + origin.getWidth());
System.out.println("Header Length(origin) : " + origin.getHeaderLen());
//System.out.println("suffix(origin) : " + origin.getSuffix());
System.out.println();
System.out.println("============ Copy one ===========");
System.out.println("comments(copy) : " + copyed.getComments());
System.out.println("Height(copy) : " + copyed.getHeight());
System.out.println("Width(copy) : " + copyed.getWidth());
System.out.println("Header Length(copy) : " + copyed.getHeaderLen());
//System.out.println("suffix(copy) : " + copyed.getSuffix());
}
static class ImageProperties {
private final int width;
private final int height;
private final String comments;
private final int headerLen;
private final String suffix;
public ImageProperties(
final int width, final int height, final String comments, final int headerLen,
final String suffix)
{
this.width = width;
this.height = height;
this.comments = comments;
this.suffix = suffix;
this.headerLen = headerLen;
}
public int getWidth() {
return width;
}
public int getHeight() {
return height;
}
public String getComments() {
return comments;
}
public String getSuffix() {
return suffix;
}
public int getHeaderLen() {
return headerLen;
}
}
final public static ImageProperties getJpegProperties(File file) throws FileNotFoundException, IOException {
BufferedInputStream in = null;
try {
in = new BufferedInputStream(new FileInputStream(file));
// check for "magic" header
byte[] buf = new byte[2];
int count = in.read(buf, 0, 2);
if (count < 2) {
throw new RuntimeException("Not a valid Jpeg file!");
}
if ((buf[0]) != (byte) 0xFF || (buf[1]) != (byte) 0xD8) {
throw new RuntimeException("Not a valid Jpeg file!");
}
int width = 0;
int height = 0;
char[] comment = null;
boolean hasDims = false;
boolean hasComment = false;
int ch = 0;
int totalHeaderLen = 0;
while (ch != 0xDA && !(hasDims && hasComment)) {
/* Find next marker (JPEG markers begin with 0xFF) */
while (ch != 0xFF) {
ch = in.read();
}
/* JPEG markers can be padded with unlimited 0xFF's */
while (ch == 0xFF) {
ch = in.read();
}
/* Now, ch contains the value of the marker. */
int length = 256 * in.read();
length += in.read();
totalHeaderLen += length;
if (length < 2) {
throw new RuntimeException("Not a valid Jpeg file!");
}
/* Now, length contains the length of the marker. */
if (ch >= 0xC0 && ch <= 0xC3) {
in.read();
height = 256 * in.read();
height += in.read();
width = 256 * in.read();
width += in.read();
for (int foo = 0; foo < length - 2 - 5; foo++) {
in.read();
}
hasDims = true;
} else if (ch == 0xFE) {
// that's the comment marker
comment = new char[length - 2];
for (int foo = 0; foo < length - 2; foo++)
comment[foo] = (char) in.read();
hasComment = true;
} else {
// just skip marker
for (int foo = 0; foo < length - 2; foo++) {
in.read();
}
}
}
if(comment == null) comment = "no comment".toCharArray();
return (new ImageProperties(width, height, new String(comment), totalHeaderLen, "jpeg"));
} finally {
if (in != null) {
try {
in.close();
} catch (IOException e) {
}
}
}
}
static BufferedImage copyImage(BufferedImage source) {
BufferedImage b = new BufferedImage(source.getWidth(), source.getHeight(), source.getType());
Graphics g = b.getGraphics();
g.drawImage(source, 0, 0, null);
g.dispose();
return b;
}
private static void fileCopy(String from, String to) {
FileChannel fromCh = null;
FileChannel toCh = null;
FileInputStream fin = null;
FileOutputStream fout = null;
try {
fin = new FileInputStream(new File(from));
fromCh = fin.getChannel();
fout = new FileOutputStream(new File(to));
toCh = fout.getChannel();
fromCh.transferTo(0, fin.available(), toCh);
} catch (IOException e) {
e.printStackTrace();
} finally {
if (fin != null)
try {
fin.close();
} catch (IOException e) {
}
if (fout != null)
try {
fout.close();
} catch (IOException e) {
}
}
}
}
So, I think the extra information of the jpeg image skipped by the image io library in java.
So I have a couple of methods and when I test them using an assertion test it fails, and I can't for the life of me figure out why, so if anyone can help me out, I'd greatly appreciate it.
The methods are supposed to write a block into an existing file, then read said block and make sure that the assertion works.
The Write Method:
public void writeBlock(int blockNum, AbstractDBFile f, AbstractBlock b)
throws IOException {
DBFile dbf = (DBFile) f;
Block blk = (Block) b;
if (blockNum >= dbf.totalNumOfBlocks){
dbf.totalNumOfBlocks++;
}
int header = 4096;
RandomAccessFile file = new RandomAccessFile(dbf.fileName, "rw");
file.seek(header + (blockNum) * 4096);
file.write(blk.getData(), 0, 4096);
file.close();
f.curBlockPos = blockNum + 1;
}
The Read Method:
public AbstractBlock readBlock(int blockNum, AbstractDBFile f)
throws IOException {
f.setCurBlockPos(blockNum);
DBFile f2 = new DBFile();
f2.setCurBlockPos(f.getCurBlockPos());
f2.setFileName(f.getFileName());
Block block = new Block();
int currentByte = f2.getCurBlockPos() * 4096;
byte[] data = new byte[4096];
String filename = f2.getFileName();
File ourFile = new File(filename);
RandomAccessFile file = new RandomAccessFile(ourFile, "r");
FileChannel inChannel = file.getChannel();
ByteBuffer bb = ByteBuffer.allocate(currentByte);
inChannel.read(bb);
while(inChannel.read(bb)>0){
bb.flip();
for (int i =0; i<bb.limit(); i++){
data[i]=bb.get();
block.setData(data);
}
bb.clear();
}
inChannel.close();
file.close();
return block;
}
The Block Class:
public class Block extends AbstractBlock {
#Override
public byte[] getData() {
// TODO Auto-generated method stub
/*byte[] data = new byte [4096];
data[0] = 10;
data[5] = 20;*/
return data;
}
#Override
public void setData(byte[] d) throws IOException {
int freeByte = -1;
for(int i = 0; i < data.length && freeByte < 0; i++)
if(data[i] == 0)
freeByte = i;
for(int i = freeByte, j = 0; j < d.length; i++, j++)
data[i] = d[j];
}
}
The Assertion test:
public void testWriteRead() throws IOException {
StorageManager manager = new StorageManager();
DBFile file = (DBFile)manager.createFile("File1");
byte [] write = new byte[4096];
write[0] = 10;
write[5] = 20;
Block block = new Block();
block.setData(write);
manager.writeBlock(0, file, block);
Block b = (Block) manager.readBlock(0, file);
assertTrue(areEqual(write,b.getData()));
}
private boolean areEqual(byte [] array1, byte [] array2) {
if(array1.length != array2.length)
return false;
for(int i = 0 ; i < array1.length ; i++)
if(array1[i] != array2[i])
return false;
return true;
}
I wont be doing this if I was not desperate, any help would be greatly appreciated.
I want to split my audio file (.wav format) in frames of 32 milliseconds each. Sampling frequency - 16khz, number of channels - 1(mono), pcm signal, sample size = 93638.
After getting the data in the byte format, I am converting the byte array storing the wav file data to double array since I require it to pass it to a method which accepts a double array, I am using the following code can someone tell me how to proceed?
import javax.sound.sampled.AudioFileFormat;
import javax.sound.sampled.AudioInputStream;
import javax.sound.sampled.AudioSystem;
import java.io.ByteArrayOutputStream;
import java.io.File;
import java.nio.ByteBuffer;
public class AudioFiles
{
public static void main(String[] args)
{
String file = "D:/p.wav";
AudioFiles afiles = new AudioFiles();
byte[] data1 = afiles.readAudioFileData(file);
byte[] data2 = afiles.readWAVAudioFileData(file);
System.out.format("data len1: %d\n", data1.length);
System.out.format("data len2: %d\n", data2.length);
/* for(int i=0;i<data2.length;i++)
{
System.out.format("\t"+data2[i]);
}*/
System.out.println();
/* for(int j=0;j<data1.length;j++)
{
System.out.format("\t"+data1[j]);
}*/
System.out.format("diff len: %d\n", data2.length - data1.length);
double[] d = new double[data1.length];
d = toDoubleArray(data1);
for (int j = 0; j < data1.length; j++)
{
System.out.format("\t" + d[j]);
}
daub a = new daub();
a.daubTrans(d);
}
public static double[] toDoubleArray(byte[] byteArray)
{
int times = Double.SIZE / Byte.SIZE;
double[] doubles = new double[byteArray.length / times];
for (int i = 0; i < doubles.length; i++)
{
doubles[i] = ByteBuffer.wrap(byteArray, i * times, times).getDouble();
}
return doubles;
}
public byte[] readAudioFileData(final String filePath)
{
byte[] data = null;
try
{
final ByteArrayOutputStream baout = new ByteArrayOutputStream();
final File file = new File(filePath);
final AudioInputStream audioInputStream = AudioSystem
.getAudioInputStream(file);
byte[] buffer = new byte[4096];
int c;
while ((c = audioInputStream.read(buffer, 0, buffer.length)) != -1)
{
baout.write(buffer, 0, c);
}
audioInputStream.close();
baout.close();
data = baout.toByteArray();
}
catch (Exception e)
{
e.printStackTrace();
}
return data;
}
public byte[] readWAVAudioFileData(final String filePath)
{
byte[] data = null;
try
{
final ByteArrayOutputStream baout = new ByteArrayOutputStream();
final AudioInputStream audioInputStream = AudioSystem.getAudioInputStream(new File(filePath));
AudioSystem.write(audioInputStream, AudioFileFormat.Type.WAVE, baout);
audioInputStream.close();
baout.close();
data = baout.toByteArray();
}
catch (Exception e)
{
e.printStackTrace();
}
return data;
}
}
I want to pass the double array d to method performing wavelet transform, in the frames of 32 millisecond since it accepts a double array.
In my previous question I was given a reply that:
At 16kHz sample rate you'll have 16 samples per millisecond. Therefore, each 32ms frame would be 32*16=512 mono samples. Multiply by the number of bytes-per-sample (typically 2 or 4) and that will be the number of bytes per frame.
I want to know whether my frame size changes when I convert my array from byte format to double format or does it remains the same??
My Previous Question.
I have an audio file that I am converting into a byte array, but then you cannot tell when that byte value is actually played in the song. So I am trying to stretch it out over the length of the song.
So while the song is playing, it outputs the byte value. How is this possible?
Here is my code so far:
public class Main {
private static final String FILENAME = "assets/pf.wav";
private static double[] endResult = null;
public static void convert() throws IOException{
ByteArrayOutputStream out = new ByteArrayOutputStream();
BufferedInputStream in = new BufferedInputStream(new FileInputStream(FILENAME));
int read;
byte[] buff = new byte[1024];
while ((read = in.read(buff)) > 0)
{
out.write(buff, 0, read);
}
out.flush();
byte[] audioBytes = out.toByteArray();
endResult = calculateFFT(audioBytes);
}
public static double[] calculateFFT(byte[] signal)
{
final int mNumberOfFFTPoints =1024;
double mMaxFFTSample;
double temp;
Complex[] y;
Complex[] complexSignal = new Complex[mNumberOfFFTPoints];
double[] absSignal = new double[mNumberOfFFTPoints/2];
for(int i = 0; i < mNumberOfFFTPoints; i++){
temp = (double)((signal[2*i] & 0xFF) | (signal[2*i+1] << 8)) / 32768.0F;
complexSignal[i] = new Complex(temp,0.0);
}
y = FFT.fft(complexSignal);
mMaxFFTSample = 0.0;
int mPeakPos = 0;
for(int i = 0; i < (mNumberOfFFTPoints/2); i++)
{
absSignal[i] = Math.sqrt(Math.pow(y[i].re(), 2) + Math.pow(y[i].im(), 2));
if(absSignal[i] > mMaxFFTSample)
{
mMaxFFTSample = absSignal[i];
mPeakPos = i;
}
}
return absSignal;
}
public static void main(String[] args) throws UnsupportedAudioFileException, IOException {
File file = new File(FILENAME);
AudioInputStream audioInputStream = AudioSystem.getAudioInputStream(file);
AudioFormat format = audioInputStream.getFormat();
long frames = audioInputStream.getFrameLength();
final double durationInSeconds = (frames+0.0) / format.getFrameRate();
try {
convert();
for(int i = 0; i < endResult.length; i++) {
System.out.println(endResult[i]);
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
How can I print out the value of the byte array (endResult[i]) over time and not immediately?
Whenever you print out a value, do:
Thread.sleep(100);
To wait 100 milliseconds (0.1 seconds) before printing the next value. This is adjustable of course.
I want to read file in opposite direction from end to the start my file,
[1322110800] LOG ROTATION: DAILY
[1322110800] LOG VERSION: 2.0
[1322110800] CURRENT HOST STATE:arsalan.hussain;DOWN;HARD;1;CRITICAL - Host Unreachable (192.168.1.107)
[1322110800] CURRENT HOST STATE: localhost;UP;HARD;1;PING OK - Packet loss = 0%, RTA = 0.06 ms
[1322110800] CURRENT HOST STATE: musewerx-72c7b0;UP;HARD;1;PING OK - Packet loss = 0%, RTA = 0.27 ms
i use code to read it in this way,
String strpath="/var/nagios.log";
FileReader fr = new FileReader(strpath);
BufferedReader br = new BufferedReader(fr);
String ch;
int time=0;
String Conversion="";
do {
ch = br.readLine();
out.print(ch+"<br/>");
} while (ch != null);
fr.close();
I would prefer to read in reverse order using buffer reader
I had the same problem as described here. I want to look at lines in file in reverse order, from the end back to the start (The unix tac command will do it).
However my input files are fairly large so reading the whole file into memory, as in the other examples was not really a workable option for me.
Below is the class I came up with, it does use RandomAccessFile, but does not need any buffers, since it just retains pointers to the file itself, and works with the standard InputStream methods.
It works for my cases, and empty files and a few other things I've tried. Now I don't have Unicode characters or anything fancy, but as long as the lines are delimited by LF, and even if they have a LF + CR it should work.
Basic Usage is :
in = new BufferedReader (new InputStreamReader (new ReverseLineInputStream(file)));
while(true) {
String line = in.readLine();
if (line == null) {
break;
}
System.out.println("X:" + line);
}
Here is the main source:
package www.kosoft.util;
import java.io.BufferedReader;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.RandomAccessFile;
public class ReverseLineInputStream extends InputStream {
RandomAccessFile in;
long currentLineStart = -1;
long currentLineEnd = -1;
long currentPos = -1;
long lastPosInFile = -1;
public ReverseLineInputStream(File file) throws FileNotFoundException {
in = new RandomAccessFile(file, "r");
currentLineStart = file.length();
currentLineEnd = file.length();
lastPosInFile = file.length() -1;
currentPos = currentLineEnd;
}
public void findPrevLine() throws IOException {
currentLineEnd = currentLineStart;
// There are no more lines, since we are at the beginning of the file and no lines.
if (currentLineEnd == 0) {
currentLineEnd = -1;
currentLineStart = -1;
currentPos = -1;
return;
}
long filePointer = currentLineStart -1;
while ( true) {
filePointer--;
// we are at start of file so this is the first line in the file.
if (filePointer < 0) {
break;
}
in.seek(filePointer);
int readByte = in.readByte();
// We ignore last LF in file. search back to find the previous LF.
if (readByte == 0xA && filePointer != lastPosInFile ) {
break;
}
}
// we want to start at pointer +1 so we are after the LF we found or at 0 the start of the file.
currentLineStart = filePointer + 1;
currentPos = currentLineStart;
}
public int read() throws IOException {
if (currentPos < currentLineEnd ) {
in.seek(currentPos++);
int readByte = in.readByte();
return readByte;
}
else if (currentPos < 0) {
return -1;
}
else {
findPrevLine();
return read();
}
}
}
Apache Commons IO has the ReversedLinesFileReader class for this now (well, since version 2.2).
So your code could be:
String strpath="/var/nagios.log";
ReversedLinesFileReader fr = new ReversedLinesFileReader(new File(strpath));
String ch;
int time=0;
String Conversion="";
do {
ch = fr.readLine();
out.print(ch+"<br/>");
} while (ch != null);
fr.close();
The ReverseLineInputStream posted above is exactly what I was looking for. The files I am reading are large and cannot be buffered.
There are a couple of bugs:
File is not closed
if the last line is not terminated the last 2 lines are returned on the first read.
Here is the corrected code:
package www.kosoft.util;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStream;
import java.io.RandomAccessFile;
public class ReverseLineInputStream extends InputStream {
RandomAccessFile in;
long currentLineStart = -1;
long currentLineEnd = -1;
long currentPos = -1;
long lastPosInFile = -1;
int lastChar = -1;
public ReverseLineInputStream(File file) throws FileNotFoundException {
in = new RandomAccessFile(file, "r");
currentLineStart = file.length();
currentLineEnd = file.length();
lastPosInFile = file.length() -1;
currentPos = currentLineEnd;
}
private void findPrevLine() throws IOException {
if (lastChar == -1) {
in.seek(lastPosInFile);
lastChar = in.readByte();
}
currentLineEnd = currentLineStart;
// There are no more lines, since we are at the beginning of the file and no lines.
if (currentLineEnd == 0) {
currentLineEnd = -1;
currentLineStart = -1;
currentPos = -1;
return;
}
long filePointer = currentLineStart -1;
while ( true) {
filePointer--;
// we are at start of file so this is the first line in the file.
if (filePointer < 0) {
break;
}
in.seek(filePointer);
int readByte = in.readByte();
// We ignore last LF in file. search back to find the previous LF.
if (readByte == 0xA && filePointer != lastPosInFile ) {
break;
}
}
// we want to start at pointer +1 so we are after the LF we found or at 0 the start of the file.
currentLineStart = filePointer + 1;
currentPos = currentLineStart;
}
public int read() throws IOException {
if (currentPos < currentLineEnd ) {
in.seek(currentPos++);
int readByte = in.readByte();
return readByte;
} else if (currentPos > lastPosInFile && currentLineStart < currentLineEnd) {
// last line in file (first returned)
findPrevLine();
if (lastChar != '\n' && lastChar != '\r') {
// last line is not terminated
return '\n';
} else {
return read();
}
} else if (currentPos < 0) {
return -1;
} else {
findPrevLine();
return read();
}
}
#Override
public void close() throws IOException {
if (in != null) {
in.close();
in = null;
}
}
}
The proposed ReverseLineInputStream works really slow when you try to read thousands of lines. At my PC Intel Core i7 on SSD drive it was about 60k lines in 80 seconds. Here is the inspired optimized version with buffered reading (opposed to one-byte-at-a-time reading in ReverseLineInputStream). 60k lines log file is read in 400 milliseconds:
public class FastReverseLineInputStream extends InputStream {
private static final int MAX_LINE_BYTES = 1024 * 1024;
private static final int DEFAULT_BUFFER_SIZE = 1024 * 1024;
private RandomAccessFile in;
private long currentFilePos;
private int bufferSize;
private byte[] buffer;
private int currentBufferPos;
private int maxLineBytes;
private byte[] currentLine;
private int currentLineWritePos = 0;
private int currentLineReadPos = 0;
private boolean lineBuffered = false;
public ReverseLineInputStream(File file) throws IOException {
this(file, DEFAULT_BUFFER_SIZE, MAX_LINE_BYTES);
}
public ReverseLineInputStream(File file, int bufferSize, int maxLineBytes) throws IOException {
this.maxLineBytes = maxLineBytes;
in = new RandomAccessFile(file, "r");
currentFilePos = file.length() - 1;
in.seek(currentFilePos);
if (in.readByte() == 0xA) {
currentFilePos--;
}
currentLine = new byte[maxLineBytes];
currentLine[0] = 0xA;
this.bufferSize = bufferSize;
buffer = new byte[bufferSize];
fillBuffer();
fillLineBuffer();
}
#Override
public int read() throws IOException {
if (currentFilePos <= 0 && currentBufferPos < 0 && currentLineReadPos < 0) {
return -1;
}
if (!lineBuffered) {
fillLineBuffer();
}
if (lineBuffered) {
if (currentLineReadPos == 0) {
lineBuffered = false;
}
return currentLine[currentLineReadPos--];
}
return 0;
}
private void fillBuffer() throws IOException {
if (currentFilePos < 0) {
return;
}
if (currentFilePos < bufferSize) {
in.seek(0);
in.read(buffer);
currentBufferPos = (int) currentFilePos;
currentFilePos = -1;
} else {
in.seek(currentFilePos);
in.read(buffer);
currentBufferPos = bufferSize - 1;
currentFilePos = currentFilePos - bufferSize;
}
}
private void fillLineBuffer() throws IOException {
currentLineWritePos = 1;
while (true) {
// we've read all the buffer - need to fill it again
if (currentBufferPos < 0) {
fillBuffer();
// nothing was buffered - we reached the beginning of a file
if (currentBufferPos < 0) {
currentLineReadPos = currentLineWritePos - 1;
lineBuffered = true;
return;
}
}
byte b = buffer[currentBufferPos--];
// \n is found - line fully buffered
if (b == 0xA) {
currentLineReadPos = currentLineWritePos - 1;
lineBuffered = true;
break;
// just ignore \r for now
} else if (b == 0xD) {
continue;
} else {
if (currentLineWritePos == maxLineBytes) {
throw new IOException("file has a line exceeding " + maxLineBytes
+ " bytes; use constructor to pickup bigger line buffer");
}
// write the current line bytes in reverse order - reading from
// the end will produce the correct line
currentLine[currentLineWritePos++] = b;
}
}
}}
#Test
public void readAndPrintInReverseOrder() throws IOException {
String path = "src/misctests/test.txt";
BufferedReader br = null;
try {
br = new BufferedReader(new FileReader(path));
Stack<String> lines = new Stack<String>();
String line = br.readLine();
while(line != null) {
lines.push(line);
line = br.readLine();
}
while(! lines.empty()) {
System.out.println(lines.pop());
}
} finally {
if(br != null) {
try {
br.close();
} catch(IOException e) {
// can't help it
}
}
}
}
Note that this code reads the hole file into memory and then starts printing it. This is the only way you can do it with a buffered reader or anry other reader that does not support seeking. You have to keep this in mind, in your case you want to read a log file, log files can be very big!
If you want to read line by line and print on the fly then you have no other alternative than using a reader that support seeking such as java.io.RandomAccessFile and this anything but trivial.
As far as I understand, you try to read backwards line by line.
Suppose this is the file you try to read:
line1
line2
line3
And you want to write it to the output stream of the servlet as follows:
line3
line2
line1
Following code might be helpful in this case:
List<String> tmp = new ArrayList<String>();
do {
ch = br.readLine();
tmp.add(ch);
out.print(ch+"<br/>");
} while (ch != null);
for(int i=tmp.size()-1;i>=0;i--) {
out.print(tmp.get(i)+"<br/>");
}
I had a problem with your solution #dpetruha because of this:
Does RandomAccessFile.read() from local file guarantee that exact number of bytes will be read?
Here is my solution: (changed only fillBuffer)
import java.io.File;
import java.io.IOException;
import java.io.InputStream;
import java.io.RandomAccessFile;
public class ReverseLineInputStream extends InputStream {
private static final int MAX_LINE_BYTES = 1024 * 1024;
private static final int DEFAULT_BUFFER_SIZE = 1024 * 1024;
private RandomAccessFile in;
private long currentFilePos;
private int bufferSize;
private byte[] buffer;
private int currentBufferPos;
private int maxLineBytes;
private byte[] currentLine;
private int currentLineWritePos = 0;
private int currentLineReadPos = 0;
private boolean lineBuffered = false;
public ReverseLineInputStream(File file) throws IOException {
this(file, DEFAULT_BUFFER_SIZE, MAX_LINE_BYTES);
}
public ReverseLineInputStream(File file, int bufferSize, int maxLineBytes) throws IOException {
this.maxLineBytes = maxLineBytes;
in = new RandomAccessFile(file, "r");
currentFilePos = file.length() - 1;
in.seek(currentFilePos);
if (in.readByte() == 0xA) {
currentFilePos--;
}
currentLine = new byte[maxLineBytes];
currentLine[0] = 0xA;
this.bufferSize = bufferSize;
buffer = new byte[bufferSize];
fillBuffer();
fillLineBuffer();
}
#Override
public int read() throws IOException {
if (currentFilePos <= 0 && currentBufferPos < 0 && currentLineReadPos < 0) {
return -1;
}
if (!lineBuffered) {
fillLineBuffer();
}
if (lineBuffered) {
if (currentLineReadPos == 0) {
lineBuffered = false;
}
return currentLine[currentLineReadPos--];
}
return 0;
}
private void fillBuffer() throws IOException {
if (currentFilePos < 0) {
return;
}
if (currentFilePos < bufferSize) {
in.seek(0);
buffer = new byte[(int) currentFilePos + 1];
in.readFully(buffer);
currentBufferPos = (int) currentFilePos;
currentFilePos = -1;
} else {
in.seek(currentFilePos - buffer.length);
in.readFully(buffer);
currentBufferPos = bufferSize - 1;
currentFilePos = currentFilePos - bufferSize;
}
}
private void fillLineBuffer() throws IOException {
currentLineWritePos = 1;
while (true) {
// we've read all the buffer - need to fill it again
if (currentBufferPos < 0) {
fillBuffer();
// nothing was buffered - we reached the beginning of a file
if (currentBufferPos < 0) {
currentLineReadPos = currentLineWritePos - 1;
lineBuffered = true;
return;
}
}
byte b = buffer[currentBufferPos--];
// \n is found - line fully buffered
if (b == 0xA) {
currentLineReadPos = currentLineWritePos - 1;
lineBuffered = true;
break;
// just ignore \r for now
} else if (b == 0xD) {
continue;
} else {
if (currentLineWritePos == maxLineBytes) {
throw new IOException("file has a line exceeding " + maxLineBytes
+ " bytes; use constructor to pickup bigger line buffer");
}
// write the current line bytes in reverse order - reading from
// the end will produce the correct line
currentLine[currentLineWritePos++] = b;
}
}
}
}