Reading multiple records from a flat file in Java - java

I have a text file dump that I need to convert to a delimited file. The file contains a series of "records" (for lack of a better word) formatted like this:
User: abc123
Date: 7/3/12
Subject: the foo is bar
Project: 123456
Problem: foo bar in multiple lines of text
Resolution: foo un-barred in multiple lines of text
User: abc123
Date: 7/3/12
Subject: the foo is bar
Project: 234567
Problem: foo bar in multiple lines of text
Resolution: foo un-barred in multiple lines of text
...
My end result is to get a flat file of delimited values. Using the records above, we would see:
abc123;7/3/12;the foo is bar;123456;foo bar in multiple lines of text;foo un-barred in multiple lines of text
abc123;7/3/12;the foo is bar;234567;foo bar in multiple lines of text;foo un-barred in multiple lines of text
Code appears below, and following that, the problem I'm experiencing.
import java.util.*;
import java.io.*;
import java.nio.file.*;
//
public class ParseOutlookFolderForSE
{
public static void main(String args[])
{
String user = "";
String PDLDate = "";
String name = "";
String PDLNum = "";
String problemDesc = "test";
String resolutionDesc = "test";
String delim = ";";
int recordCounter = 0;
//
try
{
Path file = Paths.get("testfile2.txt");
FileInputStream fstream = new FileInputStream("testfile2.txt");
// Get the object of DataInputStream
/* DataInputStream in = new DataInputStream(fstream); */
BufferedReader br = new BufferedReader(new InputStreamReader(fstream)); //Buffered Reader
String inputLine = null; //String
StringBuffer theText = new StringBuffer(); //StringBuffer
// problem: output contains last record ONLY. program is cycling through the entire file, overwriting records until the end.
// add a for loop based on recordCounter
for(recordCounter=0;recordCounter<10;recordCounter++)
{
while((inputLine=br.readLine())!=null)
{
if(inputLine.toLowerCase().startsWith("from:"))
{
/* recordCounter = recordCounter++; */ // commented out when I added recordCounter++ to the for loop
user = inputLine.trim().substring(5).trim();
}
else
if(inputLine.toLowerCase().startsWith("effective date"))
{
PDLDate = inputLine.trim().substring(15).trim();
}
else
if(inputLine.toLowerCase().startsWith("to:"))
{
name = inputLine.trim().substring(3).trim();
}
else
if(inputLine.toLowerCase().startsWith("sir number"))
{
PDLNum = inputLine.trim().substring(12).trim();
}
} //close for loop
} // close while
System.out.println(recordCounter + "\n" + user + "\n" + name + "\n" + PDLNum + "\n" + PDLDate + "\n" + problemDesc + "\n" + resolutionDesc);
System.out.println(recordCounter + ";" + user + ";" + name + ";" + PDLNum + ";" + PDLDate + ";" + problemDesc + ";" + resolutionDesc);
String lineForFile = (recordCounter + ";" + user + ";" + name + ";" + PDLNum + ";" + PDLDate + ";" + problemDesc + ";" + resolutionDesc + System.getProperty("line.separator"));
System.out.println(lineForFile);
try
{
BufferedWriter out = new BufferedWriter(new FileWriter("testfileoutput.txt"));
out.write(lineForFile);
out.close();
}
catch (IOException e)
{
System.out.println("Exception ");
}
} //close try
catch (Exception e)
{
System.err.println("Error: " + e.getMessage());
}
}
}
My final output is ONLY the last record. I believe that what's happening is that the program is reading every line, but only the LAST one doesn't get overwritten with the next record. Makes sense. So I added a FOR loop, incrementing by 1 if(inputLine.toLowerCase().startsWith("user:")) and outputting the counter variable with my data to validate what's happening.
My FOR loop begins after step 3 in my pseudocode...after BufferedReader but before my IF statements. I terminate it after I write to the file in step 6. I'm using for(recCounter=0;recCounter<10;recCounter++) and while I get ten records in my output file, they are all instances of the LAST record of the input file, numbered 0-9.
Leaving the for loop in the same place, I modified it to read for(recCounter=0;recCounter<10;) and placed recCounter's increment WITHIN the IF statement, incrementing every time the line starts with User:. In this case, I also got ten records in my output file, they were ten instances of the last record in the input file, and all the counters are 0.
EDIT: Given how the file is formatted, the ONLY way to determine w=one record from the next is a subsequent instance of the word "User:" at the start of the line. Each time that occurs, until the NEXT time it occurs is what constitutes a single record.
It appears as though I'm not setting my "recCounter" appropriately, or I'm not interpreting the results of what IS being set as "start a new record".
Anyone have any suggestions for how to read this file as multiple records?

Okay, so your pseudo-code should go something like this:
declare variables
open file
while not eof
read input
if end of set
format output
write output
clear variables
figure out which variable
store in correct variable
end-while
There might be a trick to figuring out when you've finished one set and can start the next. If a set is supposed to be terminated by a blank line as appears from your example, then you could just check for the blank line. Otherwise, how do you know? Does a set always start with "user"?
Also, don't forget to write the last record. You don't want to leave unwritten stuff in your buffer/table.

From your description it sounds like the following is the case: you are actually not writing the output strings as you complete them, but instead doing all of the writing at the end. It does not sound like you are saving the output strings outside of the loop, and so each time you find a record, you are overwriting the output string you previously calculated.
You should test that you are actually writing to the file after each record is found and has its output string created.
Without posting your code, I am not sure I can help you much further.

Related

How to use PrintStream to write to a text file without leaving any blank line between each row?

I'm doing a phone book project for my school homework assignment. What I have to do is to store information of each person into a text file and read it every time the program runs. However, every time it stores the information into the text file, it leaves a blank row between each line. That causes the program to read the blank row between when I need it to read information from the text file. That causes an error in my code.
I tried to use trim() in my reading function to skip the empty lines.
while((currentLine= reader.readLine()) != null){
String emptySpace = currentLine.trim();
if(!emptySpace.equals("")) {
...
...
}
This is my storing function
public static void storePhoneBook (String FileName, Entry[] entryList, int totalEntry) throws Exception{
PrintStream P = new PrintStream(FileName);
for (int i=0; i < totalEntry; i++) {
P.println(entryList[i].name + "\t" +
entryList [i].number + "\t" +
entryList [i].notes + "\n");
}
P.close();
System.out.println("Phone book stored.");
}
And what is written into the text file is this. They have blank space between each row.
Bill 419-536-1234 Bill Gates
JB 510-0114 Charlie's place
Jones 413-257-1234 Karen and Jason
wm 419-257-1234 Walmart
Can I write into a file without leaving any blank row between each line? The trim() function works in this instance but I don't think it's the most efficient way to do it.
use p.print instead of p.println the println adds a '[l]i[n]e'-break after the print in your example though you cna just not put "\n" at the end which is the string notation for a [n]ew line

How to replace line without loosing other lines in text file when equals condition is met?

I have a text file containing client data, ID, Name and Surname, Balance and Date,4 lines for one client in text file, then there is a space and another client data begins and so on.
ID:33
Client: Michael Reedus
Balance: 30000 Eur
Date: 32.03.2019
ID:34
Client: Michael Snow
Balance: 31900 Eur
Date: 32.03.2019
I need to create line replacement for specific client ID block, to avoid replacing same line for other person without using ID.
I tried realizing an idea, where when code finds ID which I need, it stops there, jumps, for example, a line lower and edits that line, but instead, I lose all other lines except line I am replacing.
private static void updateLine(String fails, String ID, String toUpdate, String updated) throws IOException {
BufferedReader file = new BufferedReader(new FileReader(fails));
String line;
String input = "";
while ((line = file.readLine()) != null) {
if (line.equals(ID)) {
line = file.readLine();
input += line + System.lineSeparator();
input = input.replace(toUpdate, updated);
}
}
FileOutputStream os = new FileOutputStream(fails);
os.write(input.getBytes());
file.close();
os.close();
}
I expect getting
ID:33
Client: Michael Jordan
Balance: 30000 Eur
Date: 32.03.2019
not
Client: Michael Jordan
There are a number of reasons why you were having difficulties, here are some:
if (line.equals(ID)) {
line = file.readLine();
input += line + System.lineSeparator();
As you can see in the above wee bit of code you are effectively taking the line you just read in and then applying directly to your String which will be written to file. There is no change in the data here. It should be:
if (line.equals(ID)) {
line = file.readLine();
input += updated + System.lineSeparator();
Which opens another can of worms. What if the Original Name supplied doesn't match the ID number supplied. Maybe an entry mistake was made. Before updating the particular item in file compare it to what is contained within the toUpdate parameter:
if (line.equals(ID)) {
line = file.readLine();
if (line.equals(toUpdate)) {
input += updated + System.lineSeparator();
}
This next line really puzzles me:
input = input.replace(toUpdate, updated);
You do realize that the input String variable will eventually hold ALL the data contained within your file. What if the item you want to update is in several locations under many different ID numbers? The line above will change all of them. Get rid of this scary line of code. If anything it should only be applied to the line variable (the file line currently read in).
Below I have posted a modified version of your updateLine() method. This version allows you to change any client field except the ID field, for example:
updateLine("clients.txt", "ID:33", "Date: 32.03.2019", "Date: 31.03.2019");
updateLine("clients.txt", "ID:34", "Client: Michael Snow", "Client: John Smith");
updateLine("clients.txt", "ID:34", "Balance: 31900", "Balance: 1253672");
Here is the code (most of it is comments):
private static void updateLine(String fails, String ID, String toUpdate, String updated) {
// Try With Resources is used to autoclose the BufferedReader
try (BufferedReader file = new BufferedReader(new FileReader(fails))) {
String line;
String input = "";
while ((line = file.readLine()) != null) {
if (line.equals(ID)) {
// Append the ID to String
input+= ID + System.lineSeparator();
/* Loop through this client's data and make
changes where necessary... */
while ((line = file.readLine()) != null) {
/* If we get to this point where we read an ID again
then we've gone too far. The item to update could
not be found under the supplied ID Number. */
if (line.startsWith("ID:")) {
// Append the original ID to String.
System.out.println("The item to update (" + toUpdate +
") could not be found under the ID of: " + ID);
// Add this line to string anyways.
input+= line + System.lineSeparator();
break; // Break out of this inner lop
}
// Does file line match the supplied toUpdate?
if (line.equals(toUpdate)) {
// Yes - Append the new item to String
input+= updated + System.lineSeparator();
break; // Get out of inner loop. Let main loop take over again.
}
else {
// Append the original item to String.
input+= line + System.lineSeparator();
}
}
}
else {
input+= line + System.lineSeparator();
}
}
// Re-Write File with new data
// Try With Resources is used to autoclose the Stream
try (FileOutputStream os = new FileOutputStream(fails)) {
os.write(input.getBytes());
os.flush();
}
}
catch (FileNotFoundException ex) {
ex.printStackTrace();
}
catch (IOException ex) {
ex.printStackTrace();
}
}

Why does my program read only the first record in my files?

I have written this program to compare 2 files. They are 500mb to 2.8gb in size and are created every 6 hours. I have 2 files from 2 sources (NMD and XMP). They are broken up into lines of text that have fields separated by the pipe(|) character. Each line is a single record and may be up to 65,000 characters long. The data is about TV shows and movies, showing times and descriptive content. I have determined that any particular show or movie has a minimum of 3 pieces of data that will uniquely identify that show or movie. IE: CallSign, ProgramId and StartLong. The two sources for this data are systems called NMD and XMP hence that acronym added to various variables. So my goal is to compare a file created by NMD and one created by XMP and confirm that everything that NMD produces is also produced by XMP and that the data in each matched record is the same.
What I am trying to accomplish here is this:
1. Read the NMD file record by record for the 3 unique data fields.
2. Read the XMP file record by record and look for a match for the current record in the NMD file.
3.The NMD file should iterate one record at a time. Each NMD record should then be searched for in the entire XMD file, record by record for that same record.
4. Write a log entry in one of 2 files indicating success or failure and what that data was.
What is happening is the first record in each file is read, but then no records after that get read. As a result, the end of neither file is ever reached and no matches are ever found. My success.log and failure.log file never show any data in them. In the outer do/while loop, System.out displays a single line of text. IE: The first record in the file. In the inner do/while loop System.out prints the same data over and over and over which is also from the first record in the file. Isn't this proof that the program isn't iterating record by record through the two source files?
So onto the actual code...
import java.io.*;
public class FileParse {
public static void main(String[] args) throws java.io.IOException {
String epgsRecordNMD = null;
String epgsRecordXMP = null;
BufferedWriter logSuccessWriter = null;
BufferedWriter logFailureWriter = null;
BufferedReader readXMP = null;
BufferedReader readNMD = null;
readNMD = new BufferedReader(new FileReader("d:testdataNMD.txt"));
do {
epgsRecordNMD = readNMD.readLine();
String[] epgsSplitNMD = epgsRecordNMD.split("\\|");
String epgsCallSignNMD = epgsSplitNMD[0];
String epgsProgramIdNMD = epgsSplitNMD[2];
String epgsStartLongNMD = epgsSplitNMD[9];
System.out.println("epgsCallsignNMD: " + epgsCallSignNMD + " epgsProgramIdNMD: " + epgsProgramIdNMD + " epgsStartLongNMD: " + epgsStartLongNMD );
do {
readXMP = new BufferedReader(new FileReader("d:testdataXMP.txt"));
epgsRecordXMP = readXMP.readLine();
String[] epgsSplitXMP = epgsRecordXMP.split("\\|");
String epgsCallSignXMP = epgsSplitXMP[0];
String epgsProgramIdXMP = epgsSplitXMP[2];
String epgsStartLongXMP = epgsSplitXMP[9];
System.out.println("epgsCallsignXMP: " + epgsCallSignXMP + " epgsProgramIdXMP: " + epgsProgramIdXMP + " epgsStartLongXMP: " + epgsStartLongXMP);
if (epgsCallSignXMP.equals(epgsCallSignNMD) && epgsProgramIdXMP.equals(epgsProgramIdNMD) && epgsStartLongXMP.equals(epgsStartLongNMD)) {
logSuccessWriter = new BufferedWriter (new FileWriter("d:success.log", true));
logSuccessWriter.write("NMD match found in XMP" + "epgsCallsignNMD: " + epgsCallSignNMD + " epgsProgramIdNMD: " + epgsProgramIdNMD + " epgsStartLongNMD: " + epgsStartLongNMD);
logSuccessWriter.write("\n");
logSuccessWriter.close();
System.out.println ("Match found");
}
} while (epgsRecordXMP != null);
logFailureWriter = new BufferedWriter (new FileWriter("d:failure.log", true));
logFailureWriter.write("NMD match not found in XMP" + "epgsCallsignNMD: " + epgsCallSignNMD + " epgsProgramIdNMD: " + epgsProgramIdNMD + " epgsStartLongNMD: " + epgsStartLongNMD);
logFailureWriter.write("\n");
logFailureWriter.close();
System.out.println ("Match NOT found");
} while (epgsRecordNMD != null);
readNMD.close();
readXMP.close();
}
}
do {
readXMP = new BufferedReader(new FileReader("d:testdataXMP.txt"));
You sure you want this in a loop? You are opening the same file over and over again.
Why not do something like this?
BufferedReader br = new BufferedReader(new FileReader("whateverfile.dat"));
BufferedReader br2 = new BufferedReader(new FileReader("whateverfile2.dat"));
String data;
String data2;
while(data = br.readLine() != null)
{
// do whatever you want, for example found 3 unique records
while(data2 = br2.readLine() != null)
{
// do whatever you want
}
}
Just like what OldProgrammer said, you're opening the file multiple times. You just need to ensure that the reader / streamer is open and iterate whatever you want then close it.

BufferedWriter not writing to my text file

File outFile = new File("D:\\output.txt");
BufferedWriter wb = new BufferedWriter(new FileWriter(outFile));
while (resultSet.next()) {
int attr_id = resultSet.getInt("int_id");
String stringValue = resultSet.getString("StringValue");
String name = resultSet.getString("Name");
int index = stringValue.indexOf(".");
int valueLength = stringValue.length();
if(isNumeric(stringValue)) {
//if(index != -2 ) {
if(index != (valueLength - 2)) {
String string1 = Double.valueOf(stringValue).toString();
System.out.println("converted values : " +string1);
System.out.println("stringValue : " +stringValue);
System.out.println("intValue : " +int_id);
wb.write( stringValue + "," + int_id + "," + string1 );
wb.newLine();
}
}
}
Above is my part of the code, from resultset i'm writing the data into a file. However the code is not printing values in output.txt file but i could see the result in console.
if i remove the commented line and comment if(index != (valueLength - 2)) { this line, the java code is creating output.txt with values.
What's wrong?
probably need to just call flush and/or close on your BufferedWriter when you are done.
The reason why your code is not writing to the file is because you never push the data from the Writer to the actual file.
Think of BufferedWriters as an email you are sending to a coworker. When you use:
wb.write( stringValue + "," + int_id + "," + string1 );
it is like you are typing the email to your coworker, and defining the message to them. BUT, since you did not press "send email", your coworker will never see the message. The same concept can be applied here.
You are filling the writer with a bunch of data, but you are never sending the data to the file that you are trying to write to. There are two ways that you can do this, the first being to "flush" the writer (see documentation HERE). You can do this by calling
wb.flush();
This is the equivalent of pressing "send email", and will write the data to the file that you are editing. The second method would be to "close" the writer (see documenation HERE). The reason that this works is because the BufferedWriter's close method flushes the stream first, before closing the stream. This can be done by calling
wb.close();
Although, it would not be wise to do so until you are fully done editing the file, as it can no longer be accessed.
The following is your code edited to flush the stream after every record you are writing, and then close the stream after all records have been processed. Note the locations of wb.flush() and wb.close().
File outFile = new File("D:\\output.txt");
BufferedWriter wb = new BufferedWriter(new FileWriter(outFile));
while (resultSet.next()) {
int attr_id = resultSet.getInt("int_id");
String stringValue = resultSet.getString("StringValue");
String name = resultSet.getString("Name");
int index = stringValue.indexOf(".");
int valueLength = stringValue.length();
if(isNumeric(stringValue)) {
//if(index != -2 ) {
if(index != (valueLength - 2)) {
String string1 = Double.valueOf(stringValue).toString();
System.out.println("converted values : " +string1);
System.out.println("stringValue : " +stringValue);
System.out.println("intValue : " +int_id);
wb.write( stringValue + "," + int_id + "," + string1 );
wb.newLine();
wb.flush();
}
}
}
wb.close();

java - Can't get "append mode" on Filewriter to work for me

I've been all over the search and it seems like the given answers just don't work for me.
My code is relatively simple, it generates an array of Objects, populates it with some random strings and then tries to output to a file. The idea is basically to generate a CSV file with some names, login names, passwords, etc. and the names are strings of random letters (long story, it's for mass-populating an environment with users...)
I have a "Writer" class like this:
public class Writer {
public static void log(String message) throws IOException {
PrintWriter out = new PrintWriter(new FileWriter("testlog.txt"), true);
out.println(message);
out.close();
}
}
And a loop like this:
for (int y=0; y < num_names; y++) {
try {
Writer.log(arrayTest[y].first + "," + arrayTest[y].last + "," + arrayTest[y].loginName + "," + arrayTest[y].password +
"," + arrayTest[y].email);
} catch (IOException ex) {
Logger.getLogger(Csvgenerator.class.getName()).log(Level.SEVERE, null, ex);
}
System.out.println(arrayTest[y].first + "," + arrayTest[y].last + "," + arrayTest[y].loginName + "," + arrayTest[y].password +
"," + arrayTest[y].email);
}
My expectation is that I will loop through and for each object in arrayTest[] I output a single line of data to the file.
I included the System.out.println just for debugging.
When I run my code, the System.out.println proves that it works properly -- I get a list of 10 rows. (num_names = 10 here) so this proves that each time I get to this line of code, I have a unique "row" of data which gets printed out.
However, at the end of the run, the file "testlog.txt" only contains a single line -- the very last row in my output.
I've tried "out.append" instead of "out.println" but no difference. It appears as though every time I call the logger it's creating the file anew for some reason.
So in other words if my console output (from the system.out.println) looks like this:
nxoayISPaX,aNQWbAjvWE,nanqwbajvwe,P#ssw0rd!,nanqwbajvwe#mylab.com
RpZDZAovgv,QOfyNRtIAN,rqofynrtian,P#ssw0rd!,rqofynrtian#mylab.com
SajEwHhfZz,VziPeyXmAc,svzipeyxmac,P#ssw0rd!,svzipeyxmac#mylab.com
sifahXTtBx,MRmewORtGZ,smrmewortgz,P#ssw0rd!,smrmewortgz#mylab.com
PlepqHzAxE,MQUJsHgEgy,pmqujshgegy,P#ssw0rd!,pmqujshgegy#mylab.com
VKYjYGLCfV,nuRKBJUuxW,vnurkbjuuxw,P#ssw0rd!,vnurkbjuuxw#mylab.com
YgvgeWmomA,ysKLVSZvaI,yysklvszvai,P#ssw0rd!,yysklvszvai#mylab.com
feglvfOBUX,UTIPxdEriq,futipxderiq,P#ssw0rd!,futipxderiq#mylab.com
RAQPPNajxR,vzdIwzFHJY,rvzdiwzfhjy,P#ssw0rd!,rvzdiwzfhjy#mylab.com
DeXgVFClyg,IEuUuvdWph,dieuuuvdwph,P#ssw0rd!,dieuuuvdwph#mylab.com
Then testlog.txt only contains a single line:
DeXgVFClyg,IEuUuvdWph,dieuuuvdwph,P#ssw0rd!,dieuuuvdwph#mylab.com
How do I force this to keep using the same file and just append new lines?
On the constructor PrintWriter(Writer out, boolean autoFlush), the second boolean argument is actually for autoflush, not append mode.
I think you intended to use FileWriter(File file, boolean append) constructor instead, ie:
PrintWriter out = new PrintWriter(new FileWriter("testlog.txt", true));
instead of
PrintWriter out = new PrintWriter(new FileWriter("testlog.txt"), true);

Categories

Resources