Develop Reference

Java code hangs when try to compare huge files

I am exploring an option to compare two files in Java and show the difference in html.
Below is the code, I am using -
import java.io.File;
import java.io.IOException;
import org.apache.commons.io.FileUtils;
import org.apache.commons.io.LineIterator;
import org.apache.commons.text.diff.CommandVisitor;
import org.apache.commons.text.diff.StringsComparator;
public class FileDiff {
public static void main(String[] args) throws IOException {
// Read both files with line iterator.
LineIterator file1 = FileUtils.lineIterator(new File("file-1.txt"), "utf-8");
LineIterator file2 = FileUtils.lineIterator(new File("file-2.txt"), "utf-8");
// Initialize visitor.
FileCommandsVisitor fileCommandsVisitor = new FileCommandsVisitor();
// Read file line by line so that comparison can be done line by line.
while (file1.hasNext() || file2.hasNext()) {
/*
* In case both files have different number of lines, fill in with empty
* strings. Also append newline char at end so next line comparison moves to
* next line.
*/
String left = (file1.hasNext() ? file1.nextLine() : "") + "\n";
String right = (file2.hasNext() ? file2.nextLine() : "") + "\n";
// Prepare diff comparator with lines from both files.
StringsComparator comparator = new StringsComparator(left, right);
if (comparator.getScript().getLCSLength() > (Integer.max(left.length(), right.length()) * 0.4)) {
/*
* If both lines have atleast 40% commonality then only compare with each other
* so that they are aligned with each other in final diff HTML.
*/
comparator.getScript().visit(fileCommandsVisitor);
} else {
/*
* If both lines do not have 40% commanlity then compare each with empty line so
* that they are not aligned to each other in final diff instead they show up on
* separate lines.
*/
StringsComparator leftComparator = new StringsComparator(left, "\n");
leftComparator.getScript().visit(fileCommandsVisitor);
StringsComparator rightComparator = new StringsComparator("\n", right);
rightComparator.getScript().visit(fileCommandsVisitor);
}
}
fileCommandsVisitor.generateHTML();
}
}
/*
* Custom visitor for file comparison which stores comparison & also generates
* HTML in the end.
*/
class FileCommandsVisitor implements CommandVisitor<Character> {
// Spans with red & green highlights to put highlighted characters in HTML
private static final String DELETION = "<span style=\"background-color: #FB504B\">${text}</span>";
private static final String INSERTION = "<span style=\"background-color: #45EA85\">${text}</span>";
private String left = "";
private String right = "";
#Override
public void visitKeepCommand(Character c) {
// For new line use <br/> so that in HTML also it shows on next line.
String toAppend = "\n".equals("" + c) ? "<br/>" : "" + c;
// KeepCommand means c present in both left & right. So add this to both without
// any
// highlight.
left = left + toAppend;
right = right + toAppend;
}
#Override
public void visitInsertCommand(Character c) {
// For new line use <br/> so that in HTML also it shows on next line.
String toAppend = "\n".equals("" + c) ? "<br/>" : "" + c;
// InsertCommand means character is present in right file but not in left. Show
// with green highlight on right.
right = right + INSERTION.replace("${text}", "" + toAppend);
}
#Override
public void visitDeleteCommand(Character c) {
// For new line use <br/> so that in HTML also it shows on next line.
String toAppend = "\n".equals("" + c) ? "<br/>" : "" + c;
// DeleteCommand means character is present in left file but not in right. Show
// with red highlight on left.
left = left + DELETION.replace("${text}", "" + toAppend);
}
public void generateHTML() throws IOException {
// Get template & replace placeholders with left & right variables with actual
// comparison
String template = FileUtils.readFileToString(new File("difftemplate.html"), "utf-8");
String out1 = template.replace("${left}", left);
String output = out1.replace("${right}", right);
// Write file to disk.
FileUtils.write(new File("finalDiff.html"), output, "utf-8");
System.out.println("HTML diff generated.");
}
}
For smaller files this works good and gives me good results on my laptop. But if file size is more (200MB) with half a million of rows then my IntelliJ seems to hang. RAM for my laptop is 16GB.
How can I improve this to handle large files for comparison?
Thanks

The way you wrote FileCommandsVisitor might cause it to fail to get optimized. What you're doing is adding strings for every character visited, for instance:
left = left + toAppend;
right = right + toAppend;
That might cause a new instance of a String to happen for every addition you do - new instance of a string that by the end is nearly 200 MB long. A new one for every character you visit. And old ones will need to get garbage collected. If your class held StringBuilders instead, and you used append() method it might drastically speed up. For more details read String concatenation: concat() vs "+" operator
For clarity (since according to comments you missed the point twice now):
class FileCommandsVisitor implements CommandVisitor<Character> {
//StringBuilder as properties
private StringBuilder left = new StringBuilder();
private StringBuilder right = new StringBuilder();
#Override
public void visitKeepCommand(Character c) {
String toAppend = "\n".equals("" + c) ? "<br/>" : "" + c;
// append to the StringBuilders where you would concat strings
left.append(toAppend);
right.append(toAppend);
}
//same as above for other methods
..
public void generateHTML() throws IOException {
String template = FileUtils.readFileToString(new File("difftemplate.html"), "utf-8");
//turn StringBuilders into Strings only when you actually need a String.
String out1 = template.replace("${left}", left.toString());
String output = out1.replace("${right}", right.toString());
FileUtils.write(new File("finalDiff.html"), output, "utf-8");
System.out.println("HTML diff generated.");
}
}
If that doesn't help however, and it was optimized at runtime - I don't see anything else fundamentally wrong with the way you're doing it. Comparing huge files is not a cheap operation, it won't be faster than the speed with which you can read two files line by line from your hard drive. You're still making a shortcut (that increases speed, not decreases) in having your FileCommandsVisitor hold both diffs in memory instead of writing it as it goes, which means that at best your code can diff a file of a size equal to half your available RAM. I note however, that you never mentioned how long it actually takes, so it's hard to say if the time you're seeing is expected or an anomaly.

My CSV reader doesn't recognize missing value at the beginning and at the end of a row

Hi I'm working on a simple imitation of Panda's fillna method which requires me to replace a null/missing value in a csv file with an input (in terms of parameter). Almost everything is working fine but I have one issue. My CSV reader can't recognize the null/missing at the beginning and at the end of a row. For example,
Name,Age,Class
John,20,CLass-1
,18,Class-1
,21,Class-3
It will return errors.
Same goes to this example ..
Name,Age,Class
John,20,CLass-1
Mike,18,
Tyson,21,
But for this case (at the end of the row problem), I can solve this by adding another comma at the end. Like this
Name,Age,Class
John,20,CLass-1
Mike,18,,
Tyson,21,,
However, for the beginning of the row problem, I have no idea how to solve it.
Here's my code for the CSV file reader:
public void readCSV(String fileName) {
fileLocation = fileName;
File csvFile = new File(fileName);
Scanner sfile;
// noOfColumns = 0;
// noOfRows = 0;
data = new ArrayList<ArrayList>();
int colCounter = 0;
int rowCounter = 0;
try {
sfile = new Scanner(csvFile);
while (sfile.hasNextLine()) {
String aLine = sfile.nextLine();
Scanner sline = new Scanner(aLine);
sline.useDelimiter(",");
colCounter = 0;
while (sline.hasNext()) {
if (rowCounter == 0)
data.add(new ArrayList<String>());
data.get(colCounter).add(sline.next());
colCounter++;
}
rowCounter++;
sline.close();
}
// noOfColumns = colCounter;
// noOfRows = rowCounter;
sfile.close();
} catch (FileNotFoundException e) {
System.out.println("File to read " + csvFile + " not found!");
}
}

Unless you write a CSV file yourself, the writer mechanism will never arbitrarily add delimiters to suit the needs of your application method so, give up on that train of thought altogether because you shouldn't do it either. If you do indeed have access to the CSV file creation process then the simple solution would be to not allow the possibility of null or empty values to enter the file. In other words, have the defaults (in such a case) placed into empty elements as the CSV file is being written.
The Header line within a CSV file is there for a reason, it tells you the number of data columns and the names of those columns within each line (row) that make up the file. Between the header line and the actual data in the file you can also establish a pretty good idea of what each column Data Type should be.
In my opinion, the first thing your readCSV() method should do is read this Header Line (if it exists) and gather some information about the file that the method is about to iterate through. In your case the Header Line consists of:
Name,Age,Class
Right off the start we know that each line within the file consists of three (3) data columns. The first column contains the name of Name, the second column contains the name of Age, and the third column contains the name of Class. Based on all the information provided within the CSV file we can actually quickly assume the data types:
Name (String)
Age (Integer)
Class (String)
I'm only pointing this out because in my opinion, although not mandatory, I think it would be better to store the CSV data in an ArrayList or List Interface of an Object class, for example:
ArrayList<Student> studentData = new ArrayList<>();
// OR //
List<Student> studentData = new ArrayList<>();
where Student is an object class.
You seem to want everything within a 2D ArrayList so with that in mind, below is a method to read CSV files and place its' contents into this 2D ArrayList. Any file column elements that contain the word null or nothing at all will have a default string applied. There are lots of comments within the code explaining what is going on and I suggest you give them a read. This code can be easily modified to suit your needs. At the very least I hope it gives you an idea of what can be done to apply defaults to empty values within the CSV file:
/**
* Reads a supplied CSV file with any number of columnar rows and returns
* the data within a 2D ArrayList of String ({#code ArrayList<ArrayList<String>>}).
* <br><br>File delimited data that contains 'null' or nothing (a Null String (""))
* will have a supplied common default applied to that column element before it is
* stored within the 2D ArrayList.<br><br>
*
* Modify this code to suit your needs.<br>
*
* #param fileName (String) The CSV file to process.<br>
*
* #param csvDelimiterUsed (String) // The delimiter use in CSV file.<br>
*
* #param commonDefault (String) A default String value that can be common
* to all columnar elements within the CSV file that contains the string
* 'null' or nothing at all (a Null String ("")). Those empty elements will
* end up containing this supplied string value postfixed with the name of
* that column. As an Example, If the CSV file Header line was
* 'Name,Age,Class Room' and if the string "Unknown " is supplied to the
* commonDefault parameter and during file parsing a specific data column
* (let's say Age) contained the word 'null' or nothing at all (ex:
* Bob,null,Class-Math OR Bob,,Class-Math) then this line will be stored
* within the 2D ArrayList as:<pre>
*
* Bob, Unknown Age, Class-Math</pre>
*
* #return (2D ArrayList of String Type - {#code ArrayList<ArrayList<String>>})
*/
public ArrayList<ArrayList<String>> readCSV(final String fileName, final String csvDelimiterUsed,
final String commonDefault) {
String fileLocation = fileName; // The student data file name to process.
File csvFile = new File(fileLocation); // Create a File Object (use in Scanner reader).
/* The 2D ArrayList that will be returned containing all the CSV Row/Column data.
You should really consider creating a Class to hold Student instances of this
data however, this can be accomplish by working the ArrayList later on when it
is received. */
ArrayList<ArrayList<String>> fileData = new ArrayList<>();
// Open the supplied data file using Scanner (as per OP).
try (Scanner reader = new Scanner(csvFile)) {
/* Read the Header Line and gather information... This array
will ultimately be setup to hold default values should
any file columnar data hold null OR null-string (""). */
String[] columnData = reader.nextLine().split("\\s*\\" + csvDelimiterUsed + "\\s*");
/* How many columns of data will be expected per row.
This will be used in the String#split() method later
on as the limiter when we parse each file data line.
This limiter value is rather important in this case
since it ensures that a Null String ("") is in place
of where valid Array element should be should there
be no data available instead of just providing an
array of 'lesser length'. */
int csvValuesPerLineCount = columnData.length;
// Copy column Names Array: To just hold the column Names.
String[] columnName = new String[columnData.length];
System.arraycopy(columnData, 0, columnName, 0, columnData.length);
/* Create default data for columns based on the supplied
commonDefault String. Here the supplied default prefixes
the actual column name (see JavaDoc). */
for (int i = 0; i < columnData.length; i++) {
columnData[i] = commonDefault + columnData[i];
}
// An ArrayList to hold each row of columnar data.
ArrayList<String> rowData;
// Iterate through in each row of file data...
while (reader.hasNextLine()) {
rowData = new ArrayList<>(); // Initialize a new ArrayList.
// Read file line and trim off any leading or trailing white-spaces.
String aLine = reader.nextLine().trim();
// Only Process lines that contain something (blank lines are ignored).
if (!aLine.isEmpty()) {
/* Split the read in line based on the supplied CSV file
delimiter used and the number of columns established
from the Header line. We do this to determine is a
default value will be reguired for a specific column
that contains no value at all (null or Null String("")). */
String[] aLineParts = aLine.split("\\s*\\" + csvDelimiterUsed + "\\s*", csvValuesPerLineCount);
/* Here we determine if default values will be required
and apply them. We then add the columnar row data to
the rowData ArrayList. */
for (int i = 0; i < aLineParts.length; i++) {
rowData.add((aLineParts[i].isEmpty() || aLineParts[i].equalsIgnoreCase("null"))
? columnData[i] : aLineParts[i]);
}
/* Add the rowData ArrayList to the fileData
ArrayList since we are now done with this
file row of data and will now iterate to
the next file line for processing. */
fileData.add(rowData);
}
}
}
// Process the 'File Not Found Exception'.
catch (FileNotFoundException ex) {
System.err.println("The CSV file to read (" + csvFile + ") can not be found!");
}
// Return the fileData ArrayList to the caller.
return fileData;
}
And to use the method above you might do this:
ArrayList<ArrayList<String>> list = readCSV("MyStudentsData.txt", ",", "Unknown ");
if (list == null) { return; }
StringBuilder sb;
for (int i = 0; i < list.size(); i++) {
sb = new StringBuilder("");
for (int j = 0; j < list.get(i).size(); j++) {
if (!sb.toString().isEmpty()) { sb.append(", "); }
sb.append(list.get(i).get(j));
}
System.out.println(sb.toString());
}

How to write data on specified col/index in txt file using java

For example I have data:
000001!1 |TEST 3 |18 01/22 01/23 |789
I will write the data based on specified col/index.
I had been search on internet for the alternative is using string format, but I have been rules that define the specified col on txt and the data.
the example of the rules are:
No.Record on col 1,
Name on col 12.
How I can do this?
Thank you..

I have some pseudo code that might help you for getting the formatted code in the text file ..this is some sort of tabular data
public static void main(String[] args) throws IOException {
File dir = new File("C:\\Test");
dir.mkdirs();
File file = new File(dir, "filename.txt");
FileWriter fileWriter = new FileWriter(file);
int SNIndexlength = "Serial_Number".length();
SNIndexlength = SNIndexlength-3+20;
fileWriter.write(String.format("%s %20s %20s \r\n", "Serial_Number", "Name", "Count"));
List<Data> datas = new ArrayList<Data>();
datas.add(new Data("001", "TEST1", 3));
datas.add(new Data("002", "TEST2", 5));
datas.add(new Data("003", "TEST3", 7));
datas.add(new Data("004", "TEST4", 10));
datas.add(new Data("005", "TEST5", 0));
for (Data data : datas) {
fileWriter.write(String.format("%s %"+SNIndexlength+"s %20s \r\n", data.sNum, data.name, data.count));
}
fileWriter.flush();
fileWriter.close();
}
static class Data{
public String sNum;
public String name;
public int count;
public Data(String no, String name, int count){
this.sNum = no;
this.name = name;
this.count = count;
}
}
I have updated the answer where I have changed the writing to text file as dynamic using a collection, Hope it may help.

It seems you have fixed sized records in text format. One can use a RandomAccessFile. Given the record length and column widths you then can calculate the absolute file position and overwrite bytes padded to the cell width.
This is rather awkward, might fit your use-case or not.

Pseudo-code:
public boolean editFile(AFileObject file, Integer line, Integer col, String value) {
file.openInReadMode();
// We write the results in a temporary file
AFileObject tempFile = AFileObject.getNewTempFile();
tempFile.openInWriteMode();
// We start looping until we'll reach the line to edit
for (int i = 0; ! file.eof() || i < line ; i++) {
tempFile.write(file.readLine());
}
// If we reached the end of file before, do nothing and returns false
if file.eof() {
file.close();
tempFile.delete();
return false;
}
// We edit the line here, then we write it in the temporary file
// in place of the original line
String lineToEdit = file.readLine();
lineToEdit = editLineColumn(lineToEdit, col, value);
tempFile.write(lineToEdit);
// Then we write the remaining lines
while (! file.eof()) {
tempFile.write(file.readLine());
}
// Work done. We just have now to close the files and
// Replace the source file by the temporary one
file.close();
tempFile.close();
tempFile.rename(file.getFileName());
return true;
}
private boolean String editLineColumn(String line, Integer col, String value) {
// Your code here :)
}
As you can see, it's pseudo-code. You'll have to look for the IO package which fits your needs (try with Java.IO one with FileReader and FileWriter objects for example). Start producing your own code, then we'll be able to help.

Generating custom text files in java

public class ScriptCreator {
public static void main(String[] args) throws IOException {
#Choose the CSV file that I am importing the data from
String fName = "C:\\Users\\MyUser\\Downloads\\CurrentApplications (1).csv";
String thisLine;
int count = 0;
FileInputStream fis = new FileInputStream(fName);
DataInputStream myInput = new DataInputStream(fis);
int i = 0;
#Prints the List of names in the CSV file
while((thisLine = myInput.readLine()) != null){
String strar[] = thisLine.split(",");
Printer(strar[0]);
}
}
public static void Printer(String arg) throws IOException{
#Want to pull from the String strar[0] from above
#Says that it cannot be resolved to a variable
String name = arg;
String direc = "C:/Users/MyUser/Documents/";
String path = "C:/Users/MyUser/Documents";
Iterable<String> lines = Arrays.asList("LOGIN -acceptssl ServerName","N " + name + " " + direc ,"cd " + name,"import " + path + "*.ppf" + " true","scan", "publishassessase -aseapplication " + name,"removeassess *","del " + name );
Path file = Paths.get(name + ".txt");
Files.write(file, lines, Charset.forName("UTF-8"));
}
}
Hello everyone and thank you in advance for any help that you may be able to give me. I am trying to create a java program that will pull names from a CSV file and take those names to generate custom outputs for text files. I am having a hard time being able to set a variable that I can use to grab the names that are being printed and using them to generate a text file by setting the name variable.
I am also going to need some help in making sure that it creates the amount of scripts for the amount of names in the CSV file. Ex. 7 names in CSV makes 7 custom .txt files, each with its appropriate name.
Any help is greatly appreciated!
Edit: I have updated my code to match the correction that was needed to make the code work.

It looks like you have some scoping issues. Whenever you declare a variable, it only exists within the boundaries of its closest set of braces. By declaring strar in your main method, the only place you can explicitly use it is within your main method. Your Printer() method doesn't have any previous mention of strar, and the only way it can know about it is by passing it as an argument to the function.
i.e.
Printer(String[] args)
Or, better yet:
Printer(String arg)
and then call it in your while loop with
Printer(strar[0]);
Also, your Printer method begins with a "for each" loop called on strar[0], which is not a valid target for a foreach loop anyway, because if I recall correctly, String isn't an Iterable object. If you implemented the Printer function in the way I recommended, you won't need a for each loop anyway, as there will only be one name passed at a time.

files.equal not returning true

Basically, I have two Strings that are Fully Qualified File Names. I want to compare that the two files are the same thing. So I converted both Strings to File Objects. Using google's Files.equal(File file, File file2) method, I tried to compare them, but the value returned was false. However, wondering what was wrong, I converted both file objects to byte arrays and output those which equaled the same number. So, does anyone know why Files.equal is considering them false.
I'm just curious why the method is returning false because after reading the doc Files.equal compares the two files by bytes.
Thanks.
Code:
public class WhenEncrypting {
private String[] args = new String[4];
/**
* encrypts a plain text file
*
* #throws IOException
* IOException could occur
*/
#Test()
public void normalEncryption() throws IOException {
this.args[0] = "-e";
this.args[1] = "./src/decoderwheel/tests/valid.map";
this.args[2] = "./src/decoderwheel/tests/input.txt";
this.args[3] = "./src/decoderwheel/tests/crypt.txt";
DecoderWheel.main(this.args);
File plainFile = new File("./src/decoderwheel/tests/input.txt");
File crypted = new File("./src/decoderwheel/tests/crypt.txt");
byte[] f1 = Files.toByteArray(plainFile);
byte[] f2 = Files.toByteArray(crypted);
int number = f1.length;
int size = f2.length;
Files.equal(crypted, plainFile);
System.out.println(number);
System.out.println(size);
System.out.println(Files.equal(crypted, plainFile));
assertTrue(Files.equal(crypted, plainFile));
}
}
Output:
360
360
false

Based on what you've shown us, I think that the problem is most likely to be that the two files' contents are NOT equal.
The fact that the two byte arrays (read from the files) have the same lengths does not mean that their contents (and hence the files' contents) are the same.
Add something like this:
for (int i = 0; i < f1.length; i++) {
if (f1[i] != f2[i]) {
System.out.println("File content mismatch at index " + i + ": " +
f1[i] + " != " + f2[i]);
}
}