Develop Reference

How to compare the following file format?

I have to compare the two files having 50K records in each file.
The records are in text file but in following format :
Each line is having records.
If the records are same in both the files (comparing line by line) then we have to find the difference in corresponding records.
If the records are different (check at line 4) then increment the line in text file2 by one line and print this record of file2 as a new record not found in file1 and increment till it finds the record in file1 then compare the records in both lines.
Is this possible that we can compare the two files in this format.
EDIT
private void compareFiles(File sourceFile, File targetFile, XlxsDataUtility resultFile)
throws IOException {
FixedFormatManager manager = new FixedFormatManagerImpl();
FileInputStream fis = new FileInputStream(sourceFile);
DataInputStream dis = new DataInputStream(fis);
BufferedReader sourceReader = new BufferedReader(new InputStreamReader(dis));
String sourceLine;
FileInputStream fis2 = new FileInputStream(targetFile);
DataInputStream dis2 = new DataInputStream(fis2);
BufferedReader targetReader = new BufferedReader(new InputStreamReader(dis2));
String targetLine;
sourceReader.readLine();
targetReader.readLine();
StringBuilder stringBuilder = new StringBuilder();
StringBuilder differetLines = new StringBuilder();
int line = 1;
while ((sourceLine = sourceReader.readLine()) != null && (targetLine = targetReader.readLine()) != null) {
line++;
// here i have used fixedformatManger ancientprogramming api to parse the text.
Record1 record1 = manager.load(Record1.class, sourceLine);
Record2 record2 = manager.load(Record2.class, targetLine);
if (record1.getBlock().trim().equals(record2.getBlock().trim())
&& record1.getId().trim().equals(record2.getId().trim())) {
int minimum = Math.min(sourceLine.length(), targetLine.length());
int maximum = Math.max(sourceLine.length(), targetLine.length());
int index = 0;
String fromIndex = null;
String toIndex = null;
while (index < minimum) {
char sourceChar = sourceLine.charAt(index);
char targetChar = targetLine.charAt(index);
if (sourceChar != targetChar) {
stringBuilder.append(stringBuilder.length() > 0 ? ", " : "").append(index + 1).append(" - ");
while ((index < minimum) && (sourceChar != targetChar))
index++;
if (index == minimum) {
stringBuilder.append(maximum);
index = maximum;
} else {
stringBuilder.append(index);
}
index++;
resultFile.addRowData(record2.getId().trim(), String.valueOf(sourceChar),
String.valueOf(targetChar), stringBuilder.toString(), record1.getBlock(),
String.valueOf(line));
}
index++;
// resultFile.addRowData(stringBuilder.toString());
stringBuilder.delete(0, stringBuilder.length());
}
if (minimum != maximum && index < maximum) {
stringBuilder.append(stringBuilder.length() > 0 ? ", " : "").append(minimum + 1).append(" - ")
.append(maximum);
resultFile.addRowData(record1.getId().trim(), record2.getId().trim(), stringBuilder.toString(),
record1.getBlock(), String.valueOf(line));
stringBuilder.delete(0, stringBuilder.length());
}
// System.out.println(stringBuilder.toString());
} else {
// records in both lines are different
targetReader.readLine(); // I am not sure it works here or not
differetLines.append(record1.getBlock() + record1.getId().trim() + " is not found in "
+ record2.getBlock() + record2.getId().trim() + " at line Number :: " + line + "\n");
}
}
sourceReader.close();
targetReader.close();
writeDifferenceTofile(differetLines.toString(),"Flat_File New_Records");
}
}

Java: reading utf-8 file page by page using FileInputStream

I need some code that will allow me to read one page at a time from a UTF-8 file.
I've used the code;
File fileDir = new File("DIRECTORY OF FILE");
BufferedReader in = new BufferedReader(new InputStreamReader(new FileInputStream(fileDir), "UTF8"));
String str;
while ((str = in.readLine()) != null) {
System.out.println(str);
}
in.close();
}
After surrounding it with a try catch block it runs but outputs the entire file!
Is there a way to amend this code to just display ONE PAGE of text at a time?
The file is in UTF-8 format and after viewing it in notepad++, i can see the file contains FF characters to denote the next page.

You will need to look for the form feed character by comparing to 0x0C.
For example:
char c = in.read();
while ( c != -1 ) {
if ( c == 0x0C ) {
// form feed
} else {
// handle displayable character
}
c = in.read();
}
EDIT added an example of using a Scanner, as suggested by Boris
Scanner s = new Scanner(new File("a.txt")).useDelimiter("\u000C");
while ( s.hasNext() ) {
String str = s.next();
System.out.println( str );
}

If the file is valid UTF-8, that is, the pages are split by U+00FF, aka (char) 0xFF, aka "\u00FF", 'ÿ', then a buffered reader can do. If it is a byte 0xFF there would be a problem, as UTF-8 may use a byte 0xFF.
int soughtPageno = ...; // Counted from 0
int currentPageno = 0;
try (BufferedReader in = new BufferedReader(new InputStreamReader(
new FileInputStream(fileDir), StandardCharsets.UTF_8))) {
String str;
while ((str = in.readLine()) != null && currentPageno <= soughtPageno) {
for (int pos = str.indexOf('\u00FF'; pos >= 0; )) {
if (currentPageno == soughtPageno) {
System.out.println(str.substring(0, pos);
++currentPageno;
break;
}
++currentPageno;
str = str.substring(pos + 1);
}
if (currentPageno == soughtPageno) {
System.out.println(str);
}
}
}
For a byte 0xFF (wrong, hacked UTF-8) use a wrapping InputStream between FileInputStream and the reader:
class PageInputStream implements InputStream {
InputStream in;
int pageno = 0;
boolean eof = false;
PageInputSTream(InputStream in, int pageno) {
this.in = in;
this.pageno = pageno;
}
int read() throws IOException {
if (eof) {
return -1;
}
while (pageno > 0) {
int c = in.read();
if (c == 0xFF) {
--pageno;
} else if (c == -1) {
eof = true;
in.close();
return -1;
}
}
int c = in.read();
if (c == 0xFF) {
c = -1;
eof = true;
in.close();
}
return c;
}
Take this as an example, a bit more work is to be done.

You can use a Regex to detect form-feed (page break) characters. Try something like this:
File fileDir = new File("DIRECTORY OF FILE");
BufferedReader in = new BufferedReader(new InputStreamReader(new FileInputStream(fileDir), "UTF8"));
String str;
Regex pageBreak = new Regex("(^.*)(\f)(.*$)")
while ((str = in.readLine()) != null) {
Match match = pageBreak.Match(str);
bool pageBreakFound = match.Success;
if(pageBreakFound){
String textBeforeLineBreak = match.Groups[1].Value;
//Group[2] will contain the form feed character
//Group[3] will contain the text after the form feed character
//Do whatever logic you want now that you know you hit a page boundary
}
System.out.println(str);
}
in.close();
The parenthesis around portions of the Regex denote capture groups, which get recorded in the Match object. The \f matches on the form feed character.
Edited Apologies, for some reason I read C# instead of Java, but the core concept is the same. Here's the Regex documentation for Java: http://docs.oracle.com/javase/tutorial/essential/regex/

Count the amount of times a string appears in a file

I'm trying to figure out how I would read a file, and then count the amount of times a certain string appears.
This is what my file looks like, it's a .txt:
Test
Test
Test
Test
I want the method to then return how many times it is in the file. Any idea's on how I could go about doing this? I mainly need help with the first part. So if I was searching for the string "Test" I would want it to return 4.
Thanks in advanced! Hope I gave enough info!

Add this method to your class, pass your FileInputStream to it, and it should return the number of words in a file. Keep in mind, this is case sensitive.
public int countWord(String word, FileInputStream fis) {
BufferedReader in = new BufferedReader(new InputStreamReader(fis));
String readLine = "";
int count = 0;
while((readLine = in.readLine()) != null) {
String words = readLine.split(" ");
for(String s : words) {
if(s.equals(word)) count++;
}
return count;
}
Just wrote that now, and it's untested, so let me know if it works. Also, make sure that you understand what I did if this is a homework question.

Here you are:
public int countStringInFile(String stringToLookFor, String fileName){
int count = 0;
try{
FileInputStream fstream = new FileInputStream(fileName);
DataInputStream in = new DataInputStream(fstream);
BufferedReader br = new BufferedReader(new InputStreamReader(in));
String strLine;
while ((strLine = br.readLine()) != null) {
int startIndex = strLine.indexOf(stringToLookFor);
while (startIndex != -1) {
count++;
startIndex = base.indexOf(stringToLookFor,
startIndex +stringToLookFor.length());
}
}
in.close();
}catch (Exception e){//Catch exception if any
System.err.println("Error: " + e.getMessage());
}
return count;
}
Usage: int count = countStringInFile("SomeWordToLookFor", "FileName");

If you have got to the point of reading in each file into a string I would suggest looking at the String method split.
Give it the string code 'Test' and it will return an array of type string - count the number of elements per line. Sum them up to get your total occurrence.
import java.io.*;
public class StringCount {
public static void main(String args[]) throws Exception{
String testString = "Test";
String filePath = "Test.txt";
String strLine;
int numRead=0;
try {
FileInputStream fstream = new FileInputStream(filePath);
DataInputStream in = new DataInputStream(fstream);
BufferedReader br = new BufferedReader(new InputStreamReader(in));
while ((strLine = br.readLine()) != null) {
strLine = strLine + " ";
String [] strArry = strLine.split(testString);
if (strArry.length > 1) {
numRead = numRead + strArry.length - 1;
}
else {
if (strLine == testString) {
numRead++;
}
}
}
in.close();
System.out.println(testString + " was found " + numRead + " times.");
}catch (Exception e){
}
}
}

I would do this:
open and read the file line by line,
check how oft a line contains the given word...
increase a global counter for that..
public static void main(String[] args) throws IOException {
BufferedReader in = new BufferedReader(new FileReader("Test.txt"));
Scanner sc = new Scanner(System.in);
System.out.print("Enter the subtring to look for: ");
String word = sc.next();
String line = in.readLine();
int count = 0;
do {
count += (line.length() - line.replace(word, "").length()) / word.length();
line = in.readLine();
} while (line != null);
System.out.print("There are " + count + " occurrences of " + word + " in ");
}

The first element discarded while sorting text file using arrays in java

I have this code to sort a text file using arrays in java, but it always discard the first line of the text while sorting.
Here is my code:
import java.io.*;
public class Main {
public static int count(String filename) throws IOException {
InputStream is = new BufferedInputStream(new FileInputStream(filename));
try {
byte[] c = new byte[1024];
int count = 0;
int readChars = 0;
while ((readChars = is.read(c)) != -1) {
for (int i = 0; i < readChars; ++i) {
if (c[i] == '\n') {
++count;
}
}
}
return count;
} finally {
is.close();
}
}
public static String[] getContents(File aFile) throws IOException {
String[] words = new String[count(aFile.getName()) + 1];
BufferedReader input = new BufferedReader(new FileReader(aFile));
String line = null; //not declared within while loop
int i = 0;
while ((line = input.readLine()) != null) {
words[i] = line;
i++;
}
java.util.Arrays.sort(words);
for (int k = 0; k < words.length; k++) {
System.out.println(words[k]);
}
return words;
}
public static void main(String[] args) throws IOException {
File testFile = new File("try.txt");
getContents(testFile);
}
}
Here is the text file try.txt:
Daisy
Jane
Amanda
Barbara
Alexandra
Ezabile
the output is:
Alexandra
Amanda
Barbara
Ezabile
Jane
Daisy
To solve this problem I have to insert an empty line in the beginning of the text file, is there a way not to do that? I don't know what goes wrong?

I compiled your code (on a Mac) and it works for me. Try opening the file in a hexeditor and see if there is some special character at the beginning of your file. That might be causing the sorting to be incorrect for the first line.

You probably have a BOM (Byte Order Marker) at the beginning at the file. By definition they will be interpreted as zero-width non-breaking-space.
So if you have
String textA = new String(new byte[] { (byte)0xef, (byte)0xbb, (byte) 0xbf, 65}, "UTF-8");
String textB = new String(new byte[] { 66}, "UTF-8");
System.err.println(textA + " < " + textB + " = " + (textA.compareTo(textB) < 0));
The character should show up in your length of the strings, so try printing the length of each line.
System.out.println(words[k] + " " + words[k].length());
And use a list or some other structure so you don't have to read the file twice.

Try something simpler, like this:
public static String[] getContents(File aFile) throws IOException {
List<String> words = new ArrayList<String>();
BufferedReader input = new BufferedReader(new FileReader(aFile));
String line;
while ((line = input.readLine()) != null)
words.add(line);
Collections.sort(words);
return words.toArray(new String[words.size()]);
}
public static void main(String[] args) throws IOException {
File testFile = new File("try.txt");
String[] contents = getContents(testFile);
for (int k = 0; k < contents.length; k++) {
System.out.println(contents[k]);
}
}
Notice that you don't have to iterate over the file to determine how many lines it has, instead I'm adding the lines to an ArrayList, and at the end, converting it to an array.

Use List and the add() method to read your file contents.
Then use Collections.sort() to sort the List.

Quickly read the last line of a text file?

What's the quickest and most efficient way of reading the last line of text from a [very, very large] file in Java?

Below are two functions, one that returns the last non-blank line of a file without loading or stepping through the entire file, and the other that returns the last N lines of the file without stepping through the entire file:
What tail does is zoom straight to the last character of the file, then steps backward, character by character, recording what it sees until it finds a line break. Once it finds a line break, it breaks out of the loop. Reverses what was recorded and throws it into a string and returns. 0xA is the new line and 0xD is the carriage return.
If your line endings are \r\n or crlf or some other "double newline style newline", then you will have to specify n*2 lines to get the last n lines because it counts 2 lines for every line.
public String tail( File file ) {
RandomAccessFile fileHandler = null;
try {
fileHandler = new RandomAccessFile( file, "r" );
long fileLength = fileHandler.length() - 1;
StringBuilder sb = new StringBuilder();
for(long filePointer = fileLength; filePointer != -1; filePointer--){
fileHandler.seek( filePointer );
int readByte = fileHandler.readByte();
if( readByte == 0xA ) {
if( filePointer == fileLength ) {
continue;
}
break;
} else if( readByte == 0xD ) {
if( filePointer == fileLength - 1 ) {
continue;
}
break;
}
sb.append( ( char ) readByte );
}
String lastLine = sb.reverse().toString();
return lastLine;
} catch( java.io.FileNotFoundException e ) {
e.printStackTrace();
return null;
} catch( java.io.IOException e ) {
e.printStackTrace();
return null;
} finally {
if (fileHandler != null )
try {
fileHandler.close();
} catch (IOException e) {
/* ignore */
}
}
}
But you probably don't want the last line, you want the last N lines, so use this instead:
public String tail2( File file, int lines) {
java.io.RandomAccessFile fileHandler = null;
try {
fileHandler =
new java.io.RandomAccessFile( file, "r" );
long fileLength = fileHandler.length() - 1;
StringBuilder sb = new StringBuilder();
int line = 0;
for(long filePointer = fileLength; filePointer != -1; filePointer--){
fileHandler.seek( filePointer );
int readByte = fileHandler.readByte();
if( readByte == 0xA ) {
if (filePointer < fileLength) {
line = line + 1;
}
} else if( readByte == 0xD ) {
if (filePointer < fileLength-1) {
line = line + 1;
}
}
if (line >= lines) {
break;
}
sb.append( ( char ) readByte );
}
String lastLine = sb.reverse().toString();
return lastLine;
} catch( java.io.FileNotFoundException e ) {
e.printStackTrace();
return null;
} catch( java.io.IOException e ) {
e.printStackTrace();
return null;
}
finally {
if (fileHandler != null )
try {
fileHandler.close();
} catch (IOException e) {
}
}
}
Invoke the above methods like this:
File file = new File("D:\\stuff\\huge.log");
System.out.println(tail(file));
System.out.println(tail2(file, 10));
Warning
In the wild west of unicode this code can cause the output of this function to come out wrong. For example "Mary?s" instead of "Mary's". Characters with hats, accents, Chinese characters etc may cause the output to be wrong because accents are added as modifiers after the character. Reversing compound characters changes the nature of the identity of the character on reversal. You will have to do full battery of tests on all languages you plan to use this with.
For more information about this unicode reversal problem read this:
https://codeblog.jonskeet.uk/2009/11/02/omg-ponies-aka-humanity-epic-fail/

Apache Commons has an implementation using RandomAccessFile.
It's called ReversedLinesFileReader.

Have a look at my answer to a similar question for C#. The code would be quite similar, although the encoding support is somewhat different in Java.
Basically it's not a terribly easy thing to do in general. As MSalter points out, UTF-8 does make it easy to spot \r or \n as the UTF-8 representation of those characters is just the same as ASCII, and those bytes won't occur in multi-byte character.
So basically, take a buffer of (say) 2K, and progressively read backwards (skip to 2K before you were before, read the next 2K) checking for a line termination. Then skip to exactly the right place in the stream, create an InputStreamReader on the top, and a BufferedReader on top of that. Then just call BufferedReader.readLine().

Using FileReader or FileInputStream won't work - you'll have to use either FileChannel or RandomAccessFile to loop through the file backwards from the end. Encodings will be a problem though, as Jon said.

You can easily change the below code to print the last line.
MemoryMappedFile for printing last 5 lines:
private static void printByMemoryMappedFile(File file) throws FileNotFoundException, IOException{
FileInputStream fileInputStream=new FileInputStream(file);
FileChannel channel=fileInputStream.getChannel();
ByteBuffer buffer=channel.map(FileChannel.MapMode.READ_ONLY, 0, channel.size());
buffer.position((int)channel.size());
int count=0;
StringBuilder builder=new StringBuilder();
for(long i=channel.size()-1;i>=0;i--){
char c=(char)buffer.get((int)i);
builder.append(c);
if(c=='\n'){
if(count==5)break;
count++;
builder.reverse();
System.out.println(builder.toString());
builder=null;
builder=new StringBuilder();
}
}
channel.close();
}
RandomAccessFile to print last 5 lines:
private static void printByRandomAcessFile(File file) throws FileNotFoundException, IOException{
RandomAccessFile randomAccessFile = new RandomAccessFile(file, "r");
int lines = 0;
StringBuilder builder = new StringBuilder();
long length = file.length();
length--;
randomAccessFile.seek(length);
for(long seek = length; seek >= 0; --seek){
randomAccessFile.seek(seek);
char c = (char)randomAccessFile.read();
builder.append(c);
if(c == '\n'){
builder = builder.reverse();
System.out.println(builder.toString());
lines++;
builder = null;
builder = new StringBuilder();
if (lines == 5){
break;
}
}
}
}

as far as I know The fastest way to read the last line of a text file is using FileUtils Apache class which is in "org.apache.commons.io". I have a two-million-line file and by using this class, it took me less than one second to find the last line. Here is the my code:
LineIterator lineIterator = FileUtils.lineIterator(newFile(filePath),"UTF-8");
String lastLine="";
while (lineIterator.hasNext()){
lastLine= lineIterator.nextLine();
}

try(BufferedReader reader = new BufferedReader(new FileReader(reqFile))) {
String line = null;
System.out.println("======================================");
line = reader.readLine(); //Read Line ONE
line = reader.readLine(); //Read Line TWO
System.out.println("first line : " + line);
//Length of one line if lines are of even length
int len = line.length();
//skip to the end - 3 lines
reader.skip((reqFile.length() - (len*3)));
//Searched to the last line for the date I was looking for.
while((line = reader.readLine()) != null){
System.out.println("FROM LINE : " + line);
String date = line.substring(0,line.indexOf(","));
System.out.println("DATE : " + date); //BAM!!!!!!!!!!!!!!
}
System.out.println(reqFile.getName() + " Read(" + reqFile.length()/(1000) + "KB)");
System.out.println("======================================");
} catch (IOException x) {
x.printStackTrace();
}

In C#, you should be able to set the stream's position:
From: http://bytes.com/groups/net-c/269090-streamreader-read-last-line-text-file
using(FileStream fs = File.OpenRead("c:\\file.dat"))
{
using(StreamReader sr = new StreamReader(fs))
{
sr.BaseStream.Position = fs.Length - 4;
if(sr.ReadToEnd() == "DONE")
// match
}
}

To avoid the Unicode problems related to reverting the string (or the StringBuilder), as discussed in Eric Leschinski excellent answer, one can read to a byte list, from the end of the file, revert it to a byte array and then create the String from the byte array.
Below are the changes to Eric Leschinski answer's code, to do it with a byte array. The code changes are below the commented lines of code:
static public String tail2(File file, int lines) {
java.io.RandomAccessFile fileHandler = null;
try {
fileHandler = new java.io.RandomAccessFile( file, "r" );
long fileLength = fileHandler.length() - 1;
//StringBuilder sb = new StringBuilder();
List<Byte> sb = new ArrayList<>();
int line = 0;
for(long filePointer = fileLength; filePointer != -1; filePointer--){
fileHandler.seek( filePointer );
int readByte = fileHandler.readByte();
if( readByte == 0xA ) {
if (filePointer < fileLength) {
line = line + 1;
}
} else if( readByte == 0xD ) {
if (filePointer < fileLength-1) {
line = line + 1;
}
}
if (line >= lines) {
break;
}
//sb.add( (char) readByte );
sb.add( (byte) readByte );
}
//String lastLine = sb.reverse().toString();
//Revert byte array and create String
byte[] bytes = new byte[sb.size()];
for (int i=0; i<sb.size(); i++) bytes[sb.size()-1-i] = sb.get(i);
String lastLine = new String(bytes);
return lastLine;
} catch( java.io.FileNotFoundException e ) {
e.printStackTrace();
return null;
} catch( java.io.IOException e ) {
e.printStackTrace();
return null;
}
finally {
if (fileHandler != null )
try {
fileHandler.close();
} catch (IOException e) {
}
}
}

Code is 2 lines only
// Please specify correct Charset
ReversedLinesFileReader rlf = new ReversedLinesFileReader(file, StandardCharsets.UTF_8);
// read last 2 lines
System.out.println(rlf.toString(2));
Gradle:
implementation group: 'commons-io', name: 'commons-io', version: '2.11.0'
Maven:
<dependency>
<groupId>commons-io</groupId><artifactId>commons-io</artifactId><version>2.11.0</version>
</dependency>