Develop Reference

Reading Large file A, Search Records matching records from file B and write file C in java

I have two files assume its already sorted.
This is just example data, in real ill have around 30-40 Millions of records each file Size 7-10 GB file as row length is big, and fixed.
It's a simple text file, once searched record is found. ill do some update and write to file.
File A may contain 0 or more records of matching ID from File B
Motive is to complete this processing in least amount of time possible.
I am able to do but its time taking process...
Suggestions are welcome.
File A
1000000001,A
1000000002,B
1000000002,C
1000000002,D
1000000002,D
1000000003,E
1000000004,E
1000000004,E
1000000004,E
1000000004,E
1000000005,E
1000000006,A
1000000007,A
1000000008,B
1000000009,B
1000000010,C
1000000011,C
1000000012,C
File B
1000000002
1000000004
1000000006
1000000008
1000000010
1000000012
1000000014
1000000016
1000000018\
// Not working as of now. due to logic is wrong.
private static void readAndWriteFile() {
System.out.println("Read Write File Started.");
long time = System.currentTimeMillis();
try(
BufferedReader in = new BufferedReader(new FileReader(Commons.ROOT_PATH+"input.txt"));
BufferedReader search = new BufferedReader(new FileReader(Commons.ROOT_PATH+"search.txt"));
FileWriter myWriter = new FileWriter(Commons.ROOT_PATH+"output.txt");
) {
String inLine = in.readLine();
String searchLine = search.readLine();
boolean isLoopEnd = true;
while(isLoopEnd) {
if(searchLine == null || inLine == null) {
isLoopEnd = false;
break;
}
if(searchLine.substring(0, 10).equalsIgnoreCase(inLine.substring(0,10))) {
System.out.println("Record Found - " + inLine.substring(0, 10) + " | " + searchLine.substring(0, 10) );
myWriter.write(inLine + System.lineSeparator());
inLine = in.readLine();
}else {
inLine = in.readLine();
}
}
in.close();
myWriter.close();
search.close();
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
System.out.println("Read and Write to File done in - " + (System.currentTimeMillis() - time));
}

My suggestion would be to use a database. As said in this answer. Using txt files has a big disadvantage over DBs. Mostly because of the lack of indexes and the other points mentioned in the answer.
So what I would do, is create a Database (there are lots of good ones out there such as MySQL, PostgreSQL, etc). Create the tables that are needed, and read the file afterward. Insert each line of the file into the DB and use the db to search and update them.
Maybe this would not be an answer to your concrete question on
Motive is to complete this processing in the least amount of time possible.
But this would be a worthy suggestion. Good luck.

With this approach I am able to process 50M Records in 150 Second on i-3, 4GB Ram and SSD Hardrive.
private static void readAndWriteFile() {
System.out.println("Read Write File Started.");
long time = System.currentTimeMillis();
try(
BufferedReader in = new BufferedReader(new FileReader(Commons.ROOT_PATH+"input.txt"));
BufferedReader search = new BufferedReader(new FileReader(Commons.ROOT_PATH+"search.txt"));
FileWriter myWriter = new FileWriter(Commons.ROOT_PATH+"output.txt");
) {
String inLine = in.readLine();
String searchLine = search.readLine();
boolean isLoopEnd = true;
while(isLoopEnd) {
if(searchLine == null || inLine == null) {
isLoopEnd = false;
break;
}
// Since file is already sorted, i was looking for the //ans i found here..
long seachInt = Long.parseLong(searchLineSubString);
long inInt = Long.parseLong(inputLineSubString);
if(searchLine.substring(0, 10).equalsIgnoreCase(inLine.substring(0,10))) {
System.out.println("Record Found - " + inLine.substring(0, 10) + " | " + searchLine.substring(0, 10) );
myWriter.write(inLine + System.lineSeparator());
}
// Which pointer to move..
if(seachInt < inInt) {
searchLine = search.readLine();
}else {
inLine = in.readLine();
}
}
in.close();
myWriter.close();
search.close();
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
System.out.println("Read and Write to File done in - " + (System.currentTimeMillis() - time));
}

Java - Delete file after reading from it

I am trying to figure out why my inputFile.delete() will not delete the file. After looking at numerous topics it looks like something is still using the file and hence it won't delete. But I can't figure it out. What am I missing??
File inputFile = new File("data/Accounts.txt");
File tempFile = new File("data/tmp.txt");
try {
tempFile.createNewFile();
BufferedReader reader = new BufferedReader(new FileReader(inputFile));
BufferedWriter writer = new BufferedWriter(new FileWriter(tempFile));
String line;
int i = 0;
for (User u : data) {
String toRemove = getIDByUsername(username);
while ((line = reader.readLine()) != null) {
if (line.contains(toRemove + " ")) {
line = (i + " " + username + " " + getStatusByUsername(username) + " " + password);
}
writer.write(line + "\n");
i++;
}
}
reader.close();
writer.close();
} catch (FileNotFoundException e) {
ex.FileNotFound();
} catch (IOException ee) {
ex.IOException();
} finally {
inputFile.delete();
tempFile.renameTo(inputFile);
}

You can have that much shorter and easier by using java.nio:
public static void main(String[] args) {
// provide the path to your file, (might have to be an absolute path!)
Path filePath = Paths.get("data/Accounts.txt");
// lines go here, initialize it as empty list
List<String> lines = new ArrayList<>();
try {
// read all lines (alternatively, you can stream them by Files.lines(...)
lines = Files.readAllLines(filePath);
// do your logic here, this is just a very simple output of the content
System.out.println(String.join(" ", lines));
// delete the file
Files.delete(filePath);
} catch (FileNotFoundException fnfe) {
// handle the situation of a non existing file (wrong path or similar)
System.err.println("The file at " + filePath.toAbsolutePath().toString()
+ " could not be found." + System.lineSeparator()
+ fnfe.toString());
} catch (FileSystemException fse) {
// handle the situation of an inaccessible file
System.err.println("The file at " + filePath.toAbsolutePath().toString()
+ " could not be accessed:" + System.lineSeparator()
+ fse.toString());
} catch (IOException ioe) {
// catch unexpected IOExceptions that might be thrown
System.err.println("An unexpected IOException was thrown:" + System.lineSeparator()
+ ioe.toString());
}
}
This prints the content of the file and deletes it afterwards.
You will want to do something different instead of just printing the content, but that will be possible, too ;-) Try it...

I am trying to figure out why my inputFile.delete() will not delete the file.
That's because the old file API is crappy specifically in this way: It has no ability to tell you why something is not succeeding. All it can do, is return 'false', which it will.
See the other answer, by #deHaar which shows how to do this with the newer API. Aside from being cleaner code and the newer API giving you more options, the newer API also fixes this problem where various methods, such as File.delete(), cannot tell you the reason for why it cannot do what you ask.
There are many, many issues with your code, which is why I strongly suggest you go with deHaar's attempt. To wit:
You aren't properly closing your resources; if an exception happens, your file handlers will remain open.
Both reading and writing here is done with 'platform default encoding', whatever that might be. Basically, never use those FileReader and FileWriter constructors. Fortunately, the new API defaults to UTF_8 if you fail to specify an encoding, which is more sensible.
your exception handling is not great (you're throwing away any useful messages, whatever ex.FileNotFound() might be doing here) - and you still try to delete-and-replace even if exceptions occur, which then fail, as your file handles are still open.
The method should be called getIdByUsername
Your toRemove string is the same every time, or at least, the username variable does not appear to be updated as you loop through. If indeed it never updates, move that line out of your loop.

Java.IO.FileNotFoundException Handling

When doing this, I am trying to make it "dummy proof" for my coworkers, so if they put in a file path instead of a file - the program doesn't stop so that they can just keep going in the application - it'll just ask them again. However currently
FileNotFoundException: C:\Users\usersname\Desktop\userImports\current (Access is denied)
java.io.FileNotFoundException: C:\Users\usersname\Desktop\userImports\current (Access is denied)
How do I work for this? I don't want it to crash like this, I'd rather it just say, "thats not a file - please try again"
How do I better handle file not found exceptions?
File f;
do {
System.out.print("Please give me the " + type + "file: " );
String file = console.nextLine();
f = new File(file);
} while (!f.exists());
Scanner readMe = null;
try {
readMe = new Scanner(f);
} catch (FileNotFoundException e) {
System.err.println("FileNotFoundException: " + e.getMessage());
e.printStackTrace();
}
return readMe;
}

I'm not sure I understood what you exactly want but this is my answer to what I understood :
Just loop while u don't find the file and you can also add a counter like after 5 times u exit the program.
File f;
boolean found = false ;
while (!found) {
do {
System.out.print("Please give me the " + type + "file: " );
String file = console.nextLine();
f = new File(file);
} while (!f.exists());
Scanner readMe = null;
try {
readMe = new Scanner(f);
found = true ;
} catch (FileNotFoundException e) {
found = false ;
System.err.println("FileNotFoundException: " + e.getMessage());
system.out.printl ( "A problem occured while loading the file please try again ");
//e.printStackTrace();
}
}
return readMe;
}

"Access is denied" means that the file does exist but the user who ran the program is not allowed to access - in your case read - it. It could be, i.a., that the user does not have the necessary authority, or that the file is being held by another program.

Try to use canRead() instead of exists()
do {
...
} while (!f.canRead());

Filewriter issues - Java Android

My filewriter does not seem to create a file. This is my code:
public void peopledetails_write(ArrayList<PeopleDetails> peopledetails_file) {
////numbers is the arraylist of numbers.
///names is the arraylist of names of people.
///Written to file like 01235 678 908, Daniel; 01245 645 123, Bob Marley; etc.
///Like a CSV file.
try{
FileWriter writer_file = new FileWriter("PeopleDetailsFile");
String filestring = ""; ///initializes filestring, which is written to the file.
for(PeopleDetails person : peopledetails_file){
String person_detail_string = "";
person_detail_string = person.name + "," + person.number;
filestring = filestring + person_detail_string + ";";
}
writer_file.write(filestring);
writer_file.close();
}
catch (IOException e) {
Log.e("ERROR", e.toString());
}finally{
///Hopefully won't get an error here.
Intent switch_menu = new Intent(this, MenuList.class);
startActivity(switch_menu);
}
}
It acts on the finally, and takes the user back to the main menu of my app. I have managed to debug the section where this code is, and reckon that this is faulty code, as I get a FileNotFound exception, after this section should have written a file.
What is wrong with this code?

here where your going wrong, unless api points to some specific directory, you should always
use absolute file path(complete file path).
FileWriter writer_file = new FileWriter(complete_file_path);

Java writing to a text file not working properly

The Java application that I support is logging some details in a flat file. the problem I face some times is that, the entry is very low compared to the previous day. This entry is most essential because our reports are generated based on the file. I went thro code for writing I couldn't figure out any issues. the method which is writing is sync method.
Any suggestions? I can also provide the code for you is you may need?
public synchronized void log (String connID, String hotline, String callerType,
String cli, String lastMenu, String lastInput,
String status, String reason)
{
//String absoluteFP = LOG_LOC + ls + this.getFilename();
//PrintWriter pw = this.getPrintWriter(absoluteFP, true, true);
try
{
pw.print (this.getDateTime ()+ ","+connID +","+hotline+","+callerType+","+ cli+"," + lastMenu + "," + lastInput + "," + status + "," + reason);
//end 1006
pw.print (ls);
pw.flush ();
//pw.close();
}
catch (Exception e)
{
e.printStackTrace ();
return;
}
}
private synchronized PrintWriter getPrintWriter (String absoluteFileName,
boolean append, boolean autoFlush)
{
try
{
//set absolute filepath
File folder = new File (absoluteFileName).getParentFile ();//2009-01-23
File f = new File (absoluteFileName);
if (!folder.exists ())//2009-01-23
{
//System.out.println ("Call Detailed Record folder NOT FOUND! Creating a new);
folder.mkdirs ();
//System.out.println ("Configure log folder");
this.setHiddenFile (LOG_LOC);//set tmp directory to hidden folder
if (!f.exists ())
{
//System.out.println ("Creating a new Call Detailed Record...");//2009-01-23
f.createNewFile ();//2009-01-23
}
}
else
{
if (!f.exists ())
{
//System.out.println ("Creating a new Call Detailed Record...");//2009-01-23
f.createNewFile ();//2009-01-23
}
}
FileOutputStream tempFOS = new FileOutputStream (absoluteFileName, append);
if (tempFOS != null)
{
return new PrintWriter (tempFOS, autoFlush);
}
else
{
return null;
}
}
catch (Exception ex)
{
ex.printStackTrace ();
return null;
}
}
/**
* Set the given absolute file path as a hidden file.
* #param absoluteFile String
*/
private void setHiddenFile (String absoluteFile)
{
//set hidden file
//2009-01-22, KC
Runtime rt = Runtime.getRuntime ();
absoluteFile = absoluteFile.substring (0, absoluteFile.length () - 1);//2009-01-23
try
{
System.out.println (rt.exec ("attrib +H " + "\"" + absoluteFile + "\"").getInputStream ().toString ());
}
catch (IOException e)
{
e.printStackTrace ();
}
}
private String getDateTime ()
{
//2011-076-09, KC-format up to milliseconds to prevent duplicate PK in CDR table.
//return DateUtils.now ("yyyy/MM/dd HH:mm:ss");
return DateUtils.now ("yyyy/MM/dd HH:mm:ss:SSS");
//end 0609
}
private String getFilename ()
{
///return "CDR_" + port + ".dat";//2010-10-01
return port + ".dat";//2010-10-01
}
public void closePW ()
{
if (pw != null)
{
pw.close ();
}
}

You've created a FileOutputStream, but aren't closing that stream. Close that stream and try again. That might be causing the problem.
Messages are getting logged sometime because the garbage collector kicks in at some intervals and closes the FileOutStream. This then allows messages to be logged again. You're getting the unreachable error since you have a return statement in both the if & else blocks. You'll have to take the PrintWriter and FileOutStreamWriter out of the getPrintWriter put it where you usually call the getPrintWriter(). Then you'll be able to close the streams correctly. getPrintWriter should only ensure file exists, so rename it to ensureFileExistance

If you can use Apache Common IO, try this:
public synchronized void log(String connID, String hotline, String callerType,
String cli, String lastMenu, String lastInput,
String status, String reason) {
String absoluteFP = LOG_LOC + ls + this.getFilename();
File file = new File(absoluteFP);
String message = this.getDateTime() + "," + connID + "," + hotline + "," + callerType + "," + cli + "," + lastMenu + "," + lastInput + "," + status + "," + reason;
try {
// note that you must explicitly add new line character if you want the line to end with newline
FileUtils.write(file, message + "\n", "UTF-8", true);
} catch (IOException ex) {
ex.printStackTrace ();
}
}
In Common IO 2.1, you can append a file that you are writting to. You can now get rid of the closePW and getPrintwriter and since the log method is synchronized, the file can be written one at a time from the same object. However, if you try to write the same file from different object at the same time, you will end up having overwritting problem.
Also, Common IO create the missing parent folder for you automatically. There is no need to explicitly check and create the folder.