Develop Reference

How to compare the following file format?

I have to compare the two files having 50K records in each file.
The records are in text file but in following format :
Each line is having records.
If the records are same in both the files (comparing line by line) then we have to find the difference in corresponding records.
If the records are different (check at line 4) then increment the line in text file2 by one line and print this record of file2 as a new record not found in file1 and increment till it finds the record in file1 then compare the records in both lines.
Is this possible that we can compare the two files in this format.
EDIT
private void compareFiles(File sourceFile, File targetFile, XlxsDataUtility resultFile)
throws IOException {
FixedFormatManager manager = new FixedFormatManagerImpl();
FileInputStream fis = new FileInputStream(sourceFile);
DataInputStream dis = new DataInputStream(fis);
BufferedReader sourceReader = new BufferedReader(new InputStreamReader(dis));
String sourceLine;
FileInputStream fis2 = new FileInputStream(targetFile);
DataInputStream dis2 = new DataInputStream(fis2);
BufferedReader targetReader = new BufferedReader(new InputStreamReader(dis2));
String targetLine;
sourceReader.readLine();
targetReader.readLine();
StringBuilder stringBuilder = new StringBuilder();
StringBuilder differetLines = new StringBuilder();
int line = 1;
while ((sourceLine = sourceReader.readLine()) != null && (targetLine = targetReader.readLine()) != null) {
line++;
// here i have used fixedformatManger ancientprogramming api to parse the text.
Record1 record1 = manager.load(Record1.class, sourceLine);
Record2 record2 = manager.load(Record2.class, targetLine);
if (record1.getBlock().trim().equals(record2.getBlock().trim())
&& record1.getId().trim().equals(record2.getId().trim())) {
int minimum = Math.min(sourceLine.length(), targetLine.length());
int maximum = Math.max(sourceLine.length(), targetLine.length());
int index = 0;
String fromIndex = null;
String toIndex = null;
while (index < minimum) {
char sourceChar = sourceLine.charAt(index);
char targetChar = targetLine.charAt(index);
if (sourceChar != targetChar) {
stringBuilder.append(stringBuilder.length() > 0 ? ", " : "").append(index + 1).append(" - ");
while ((index < minimum) && (sourceChar != targetChar))
index++;
if (index == minimum) {
stringBuilder.append(maximum);
index = maximum;
} else {
stringBuilder.append(index);
}
index++;
resultFile.addRowData(record2.getId().trim(), String.valueOf(sourceChar),
String.valueOf(targetChar), stringBuilder.toString(), record1.getBlock(),
String.valueOf(line));
}
index++;
// resultFile.addRowData(stringBuilder.toString());
stringBuilder.delete(0, stringBuilder.length());
}
if (minimum != maximum && index < maximum) {
stringBuilder.append(stringBuilder.length() > 0 ? ", " : "").append(minimum + 1).append(" - ")
.append(maximum);
resultFile.addRowData(record1.getId().trim(), record2.getId().trim(), stringBuilder.toString(),
record1.getBlock(), String.valueOf(line));
stringBuilder.delete(0, stringBuilder.length());
}
// System.out.println(stringBuilder.toString());
} else {
// records in both lines are different
targetReader.readLine(); // I am not sure it works here or not
differetLines.append(record1.getBlock() + record1.getId().trim() + " is not found in "
+ record2.getBlock() + record2.getId().trim() + " at line Number :: " + line + "\n");
}
}
sourceReader.close();
targetReader.close();
writeDifferenceTofile(differetLines.toString(),"Flat_File New_Records");
}
}

Directory not being recognized

So I have a method that reads all the files in a folder and creates new classes in a List with the variables read from the files. For some reason it doesn't ever get past the if(mainDir.isDirectory()){ part, even though the paths are correct and I double checked the folders were there.
public static void loadIntoClass(String dir, int temp){
try {
File mainDir = new File(dir);
if(mainDir.isDirectory()){ //Checks if the dir is a folder and not a file
String[] fileNames = mainDir.list(); //Grabs a list of all the filenames in the dir
for(int x = 0; x > fileNames.length; x++){ //loops through all the files
File currFile = new File(dir + fileNames[x]); //Creates the object we will be gathering information from
if(currFile.isFile()){ //Checks to make sure the file is a file and not a folder
BufferedReader br = new BufferedReader(new FileReader(currFile));
String line = br.readLine();
int currLoop = 1;
boolean collides = false;
while(line != null){ //Will keep checking files until it reaches a blank line
currLoop ++; //Keeps track of how many times it loops
test = line.split("="); //Splits up the variable from the declaration
String toString = test[0].trim(); //Trims off any extra blank spaces on either side
System.out.println("Reading: " + toString + " on line " + currLoop); //For debugging
String toString2 = test[1].trim(); //Trims the second string
parse[currLoop] = Integer.parseInt(toString2); //Turns the string into an integer then puts it into the array
if(toString.equalsIgnoreCase("Collides")){
if(toString2.equalsIgnoreCase("true")){
collides = true;
}
}
if(toString.equalsIgnoreCase("Image Path")){
//path = toString2;
}
line = br.readLine();
}
if(temp == 1){
types.add(new Type(parse[1], parse[2], parse[3], parse[4], parse[5], parse[6], parse[7]));
}
if(temp == 2){
tiles.add(new Tiles(parse[1], collides, null));
}
if(temp == 3){
abilities.add(new Abilities(parse[1], parse[2], parse[3], parse[4]));
}
br.close();
}
}
}
} catch(Exception e) {
System.err.println("ERROR: " + e);
}
}
After that if I change it some other path like "C:/test" it works only to freeze at the for loop. Here's the declaration:
loadIntoClass("C:/Program Files(x86)/GameNameHere/config/enemies", 1);

The methods isDirectory() and isFile() doe not work if the underlying FS-Objects do not exist.

There are multiple possible issues, which you are not taking into consideration...
Your not checking to see if the dir exists
Your not making sure to close your files in case of an read error (or other associated error)
You making life tough for yourself using File#list, instead use File#listFiles which will return an array of File
Make better use of exceptions...
For example...
public static void loadIntoClass(String dir, int temp) throws IOException {
File mainDir = new File(dir);
if(mainDir.exists) { // Check to see if the abstract path actually exists
if (mainDir.isDirectory()){ //Checks if the dir is a folder and not a file
File[] fileNames = mainDir.listFiles(); //Grabs a list of all the filenames in the dir
//String[] fileNames = mainDir.list(); //Grabs a list of all the filenames in the dir
if (fileNames != null && fileNames.length > 0) {
//for(int x = 0; x > fileNames.length; x++){ //loops through all the files
for(File currFile : fileNames){ //loops through all the files
//File currFile = new File(dir + fileNames[x]); //Creates the object we will be gathering information from
if(currFile.isFile()){ //Checks to make sure the file is a file and not a folder
BufferedReader br = null;
try {
br = new BufferedReader(new FileReader(currFile));
String line = null;
int currLoop = 1;
boolean collides = false;
while((line = br.readLine()) != null){ //Will keep checking files until it reaches a blank line
//...//
}
//...//
// Make sure you make all best attempts to close the reader...
} finally {
try {
br.close();
} catch (Exception exp) {
}
}
}
}
} else {
// You may not care, but...
throw new IOException(dir + " does not contain any files");
}
} else {
throw new IOException(dir + " is not a directory");
}
} else {
throw new IOException(dir + " does not exist");
}
}

Search an array with variable column sizes java

I want to display some of the content of the .txt file on the screen after copying it into a new file. The content of the text file is not exactly the same structure throughout.
When i did this
if (m[11].equals("Channel") && m[12].equals("ID")){ System.out.println("Wavelenght ID = " + m[9]);
I got an error array out of bounds.
TESTSTEP: DEBUG * Fpga Config ECFG *: EED 3 : Channel ID
If this line was found i want it to display Wavelenght ID = 3
String p = path.replace("\\", "\\\\");
File file = new File(p);
File[] files = file.listFiles();
BufferedReader inputStream = null;
PrintWriter outputStreamI = null;
PrintWriter outputStreamO = null;
try {
String l,c;
for (int i=0; i<files.length; i++){
inputStream = new BufferedReader(new FileReader(files[i].getAbsolutePath()));
outputStreamI = new PrintWriter(new FileWriter("S:\\WRED_IBP\\" + files[i].getName().replaceFirst("[.][^.]+$", "")+ "Ingress.txt"));
outputStreamO = new PrintWriter(new FileWriter("S:\\WRED_IBP\\" +files[i].getName().replaceFirst("[.][^.]+$", "")+ "Egress.txt"));
while ((l = inputStream.readLine())!= null) {
String[] m=l.split(" ");
for(int d=0; d<m.length; d++){
c=m[d];
if (c.equalsIgnoreCase("ied_wred") ){outputStreamI.println(l); if (outputStreamI.checkError()){System.err.println("Error in output stream");}} //|| c.equalsIgnoreCase("WARNING") || c.equals("ERROR")
else if (c.equalsIgnoreCase("eed_brc_ibp")){outputStreamO.println(l); if (outputStreamO.checkError()){System.err.println("Error in output stream");}} // || c.equalsIgnoreCase("WARNING") || c.equals("ERROR")
}
}
}
} catch(IOException e){
System.err.println("Caught IOException: " + e.getMessage());
}//wait(10);
finally {
if (inputStream != null) {
try{ inputStream.close();} catch (IOException e) {e.printStackTrace();}}
if (outputStreamI != null) {outputStreamI.close();}
if (outputStreamO != null) {outputStreamO.close();}
}
}

You can use:
if (m.length > 12 && m[11].equals("Channel") && m[12].equals("ID")){
System.out.println("Wavelenght ID = " + m[9]);
}
So index out of bound won't happen here.
Make sure you put these lines before the for loop.

TESTSTEP: DEBUG * Fpga Config ECFG *: EED 3 : Channel ID
I'm assuming this is the text in the file. When you read this, you are trying to split it based on spaces. What you should be doing is split it based on :.

Find file with the most update information

I have a list of log files, and I need to find which one has a latest edition of a specific line, and all or none could have this line.
The lines in the files look like this:
2013/01/06 16:01:00:283 INFO ag.doLog: xxxx xxxx xxxx xxxx
And I need a line lets say
xx/xx/xx xx:xx:xx:xxx INFO ag.doLog: the line i need
I know how to get an array of files, and if I scan backwards I could find the latest latest line in each file (if it exists).
Biggest problem is that the file could be big (2k lines?) and I want to find the line in a relative fast way (a few seconds), so I am open for suggestion.
Personal ideas:
If a file has the line at X time, then any file that has not found the line before X time should not be scan anymore. This will require to search all files at the same time, which i dont know how.
Atm the code breaks, and I suppose if lack of memory.
Code:
if(files.length>0) { //in case no log files exist
System.out.println("files.length: " + files.length);
for(int i = 0; i < files.length; i++) { ///for each log file look for string
System.out.println("Reading file: " + i + " " + files[i].getName());
RandomAccessFile raf = new RandomAccessFile(files[i].getAbsoluteFile(), "r"); //open log file
long lastSegment = raf.length(); //Finds how long is the files
lastSegment = raf.length()-5; //Sets a point to start looking
String leido = "";
byte array[] = new byte[1024];
/*
* Going back until we find line or file is empty.
*/
while(!leido.contains(lineToSearch)||lastSegment>0) {
System.out.println("leido: " + leido);
raf.seek(lastSegment); //move the to that point
raf.read(array); //Reads 1024 bytes and saves in array
leido = new String(array); //Saves what is read as a string
lastSegment = lastSegment-15; //move the point a little further back
}
if(lastSegment<0) {
raf.seek(leido.indexOf(lineToSearch) - 23); //to make sure we get the date (23 characters long) NOTE: it wont be negative.
raf.read(array); //Reads 1024 bytes and saves in array
leido = new String(array); //make the array into a string
Date date = new SimpleDateFormat("MMMM d, yyyy", Locale.ENGLISH).parse(leido.substring(0, leido.indexOf(" INFO "))); //get only the date part
System.out.println(date);
//if date is bigger than the other save file name
}
}
}

I find the code difficult to verify. One could split the task in a backwards reader, which reads lines from file end to start. And use that for parsing dates line wise.
Mind, I am not going for nice code, but something like this:
public class BackwardsReader implements Closeable {
private static final int BUFFER_SIZE = 4096;
private String charset;
private RandomAccessFile raf;
private long position;
private int readIndex;
private byte[] buffer = new byte[BUFFER_SIZE];
/**
* #param file a text file.
* #param charset with bytes '\r' and '\n' (no wide chars).
*/
public BackwardsReader(File file, String charset) throws IOException {
this.charset = charset;
raf = new RandomAccessFile(file, "r");
position = raf.length();
}
public String readLine() throws IOException {
if (position + readIndex == 0) {
raf.close();
raf = null;
return null;
}
String line = "";
for (;;) { // Loop adding blocks without newline '\n'.
// Search line start:
boolean lineStartFound = false;
int lineStartIndex = readIndex;
while (lineStartIndex > 0) {
if (buffer[lineStartIndex - 1] == (byte)'\n') {
lineStartFound = true;
break;
}
--lineStartIndex;
}
String line2;
try {
line2 = new String(buffer, lineStartIndex, readIndex - lineStartIndex,
charset).replaceFirst("\r?\n?", "");
readIndex = lineStartIndex;
} catch (UnsupportedEncodingException ex) {
Logger.getLogger(BackwardsReader.class.getName())
.log(Level.SEVERE, null, ex);
return null;
}
line = line2 + line;
if (lineStartFound) {
--readIndex;
break;
}
// Read a prior block:
int toRead = BUFFER_SIZE;
if (position - toRead < 0) {
toRead = (int) position;
}
if (toRead == 0) {
break;
}
position -= toRead;
raf.seek(position);
raf.readFully(buffer, 0, toRead);
readIndex = toRead;
if (buffer[readIndex - 1] == (byte)'\r') {
--readIndex;
}
}
return line;
}
#Override
public void close() throws IOException {
if (raf != null) {
raf.close();
}
}
}
And a usage example:
public static void main(String[] args) {
try {
File file = new File(args[0]);
BackwardsReader reader = new BackwardsReader(file, "UTF-8");
int lineCount = 0;
for (;;) {
String line = reader.readLine();
if (line == null) {
break;
}
++lineCount;
System.out.println(line);
}
reader.close();
System.out.println("Lines: " + lineCount);
} catch (IOException ex) {
Logger.getLogger(App.class.getName()).log(Level.SEVERE, null, ex);
}
}

File Comparison via Byte Array issues

I am coding a class that compares the files of two directories via comparing the Byte arrays of each file. I am however not getting the expected results; identical files are not being resolved as identical files.
First problem:
Matching the files Byte[] with equals() resolves to false with matching files (Checked with only one file as to circumvent the possible index misalignment issue; the check still resolves to false.).
Second problem:
When using Vector's containsAll() for checking that both Vectors of Byte[] match (One Vector per directory with Byte[] for each file) this check results in false even with identical directories (This check has been removed from the code below.). So is there an issue with the way I am aligning the two vectors? (I have checked this with using two directories with matching files in the same order loaded into matching indeces; this still results in a Vector mismatch).
Third problem:
When there are subdirectories in the directories being checked a file not found exception is thrown stating that access is denied. Why is this happening? How can I circumvent this? I do not want to check the files contained within the subdirectories, but I am designing the code so that the end user need not worry about the subdirectories of the directories being compared. This only happens when there are subdirectories, it work fine when there are no subdirectories in the directories being checked.
Example Exception:
Byte reading error!
Byte reading error!
java.io.FileNotFoundException: C:\Dir1\Dir2\Dir3\Dir4\SubDir (Access is denied)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(Unknown Source)
at tools.filesystem.filecomparison.FileComparator.getBytes(FileComparator.java:166)
at tools.filesystem.filecomparison.FileComparator.main(FileComparator.java:102)
java.io.FileNotFoundException: C:\Dir1\Dir2\Dir3\Dir4\SubDir Files (Access is denied)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(Unknown Source)
at tools.filesystem.filecomparison.FileComparator.getBytes(FileComparator.java:166)
at tools.filesystem.filecomparison.FileComparator.main(FileComparator.java:111)
Here is the code:
package tools.filesystem.filecomparison;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.util.Scanner;
import java.util.Vector;
public class FileComparator
{
public static void main(String[] args)
{
String workingDir1 = "";
String workingDir2 = "";
File[] fileArr1 = null;
File[] fileArr2 = null;
Vector<File> fileVec1 = new Vector<File>();
Vector<File> fileVec2 = new Vector<File>();
Scanner console = new Scanner(System.in);
while (true)
{
System.out.println("Enter working directory one . . . .");
workingDir1 = console.nextLine();
workingDir1.replace("\\", "\\\\");
System.out.println("Enter working directory two . . . .");
workingDir2 = console.nextLine();
workingDir2.replace("\\", "\\\\");
File folder1 = new File(workingDir1);
File[] listOfFiles1 = folder1.listFiles();
File folder2 = new File(workingDir1);
File[] listOfFiles2 = folder2.listFiles();
fileArr1 = listOfFiles1;
fileArr2 = listOfFiles2;
System.out.println("\nWorking Directory 1 Files\n");
for (int i = 0; i < listOfFiles1.length; i++)
{
if (listOfFiles1[i].isFile())
{
System.out.println(" " + listOfFiles1[i].getName());
}
/* else if (listOfFiles1[i].isDirectory())
{
System.out.println("Directory " + listOfFiles1[i].getName());
}*/
}
System.out.println("\nWorking Directory 2 Files\n");
for (int i = 0; i < listOfFiles2.length; i++)
{
if (listOfFiles2[i].isFile())
{
System.out.println(" " + listOfFiles2[i].getName());
}
/* else if (listOfFiles2[i].isDirectory())
{
System.out.println("Directory " + listOfFiles2[i].getName());
}*/
}
for (File fle : fileArr1)
{
fileVec1.add(fle);
}
for (File fle : fileArr2)
{
fileVec2.add(fle);
}
if (fileVec1.containsAll(fileVec2))
break;
else
{
System.out.println("Directories do not contain the same files!\nContinue anyways? y/n?");
if (console.nextLine().equalsIgnoreCase("y"))
break;
else if (console.nextLine().equalsIgnoreCase("n"))
continue;
}
}
Vector<Vector<File>> alignedVectors = align(fileVec1, fileVec2);
fileVec1 = alignedVectors.elementAt(0);
fileVec2 = alignedVectors.elementAt(1);
Vector<byte[]> fileByteVect1 = new Vector<byte[]>();
Vector<byte[]> fileByteVect2 = new Vector<byte[]>();
try
{
fileByteVect1 = getBytes(fileVec1);
}
catch (IOException e)
{
System.out.println("Byte reading error!");
e.printStackTrace();
}
try
{
fileByteVect2 = getBytes(fileVec2);
}
catch (IOException e)
{
System.out.println("Byte reading error!");
e.printStackTrace();
}
boolean[] check = new boolean[fileByteVect1.capacity()];
int i1 = 0;
//debug
for (byte[] e : fileByteVect1)
{
System.out.println("Vector 1 count " + i1);
System.out.println(e.toString());
for (byte b : e)
{
System.out.print(b + " ");
}
i1++;
}
int i2 = 0;
//debug
for (byte[] e : fileByteVect2)
{
System.out.println("Vector 2 count " + i2);
System.out.println(e.toString());
for (byte b : e)
{
System.out.print(b + " ");
}
i2++;
}
if (fileByteVect1.size() == fileByteVect2.size())
{
System.out.println(fileByteVect1.size());
for (int i = 0; i < fileByteVect1.size(); i++ )
{
if (fileByteVect1.elementAt(i).equals(fileByteVect2.elementAt(i)))
{
check[i] = true;
System.out.println("File at index " + i + " are identical");
}
else
{
check[i] = false;
System.out.println("File at index " + i + " are not identical");
}
}
}
else
System.out.println("Files do not match!");
}
public static Vector<Vector<File>> align(Vector<File> fileVect1, Vector<File> fileVect2)
{
Vector<Vector<File>> mainBuffer = new Vector<Vector<File>>();
Vector<File> bufferFileVect = new Vector<File>();
for (File fle1 : fileVect1)
{
for (File fle2 : fileVect2)
{
if (fle1.getName().equals(fle2.getName()))
bufferFileVect.add(fle2);
}
}
mainBuffer.add(fileVect1);
mainBuffer.add(bufferFileVect);
return mainBuffer;
}
public static Vector<byte[]> getBytes(Vector<File> fileVector) throws IOException
{
Vector<byte[]> outVector = new Vector<byte[]>();
for (File file : fileVector)
{
InputStream is = new FileInputStream(file);
// Get the size of the file
long length = file.length();
if (length > Integer.MAX_VALUE)
{
System.out.println("File is too large!");
}
// Create the byte array to hold the data
byte[] bytes = new byte[(int) length];
// Read in the bytes
int offset = 0;
int numRead = 0;
while (offset < bytes.length && (numRead = is.read(bytes, offset, bytes.length - offset)) >= 0)
{
offset += numRead;
}
// Ensure all the bytes have been read in
if (offset < bytes.length)
{
throw new IOException("Could not completely read file " + file.getName());
}
// Close the input stream and return bytes
outVector.add(bytes);
is.close();
}
return outVector;
}
}

The equals function isn't doing a deep comparison, rather for a byte[] you're comparing addresses. Instead you should use
Arrays.equals(fileByteVect1.elementAt(i), fileByteVect2.elementAt(i))
to perform the deep comparison of the byte arrays.
More detail on Arrays.equals.
As for your third question, you're not actually filtering for just files. When you iterate through to print out the filename you should construct the Vector storing the files:
for (File fle : fileArr1) {
if (fle.isFile()) {
fileVec1.add(fle);
System.out.println(" " + fle.getName());
}
}
You will, of course, have to do this for fileArr2 and fileVec2 as well.

Simple. The equals(Object) method on an array is inherited from Object, and hence is equivalent to the == operator; i.e. it is just a reference comparison.
This is specified in JLS 6.4.5.
If you want to compare arrays by value, use the java.util.Arrays.equals(array1, array2) methods. There are overloads for arrays of each primitive type and arrays of Object.
(Note that it is the semantics of each element type's implementation of equals method that determines if Arrays.equals(Object[], Object[]) is a "deep" or "shallow" comparison.)
FOLLOW UP
I suspect that the third problem happens because your application is trying to open the subdirectory as a file. That won't work. Instead, you need to:
Use File.isFile() and File.isDirectory() to determine whether you should be reading the directory entries as files or dirctories (or not at all).
For a directory, you should recursively use File.listFiles() or similar to iterate over the subdirectory contents.