Develop Reference

algorithm arraylist remove String duplicates and save to new text file

I am currently writing an algorithm that creates an ArrayList from a .txt file, checks it with a loop for duplicates (where the loop should look like this:
Line one is written to new .txt & boolean found is set to true because the string was already found.
Line 2 is written to new .txt etc.
But if two strings are identical, the duplicate, i.e. the second string should just be ignored and continue with the next one).
public class test {
public static void main(String[] args) throws IOException {
String suche = "88 BETRAG-MINUS VALUE 'M'.";
String suche2 = "88 BETRAG-PLUS VALUE 'P'";
boolean gefunden = false;
File neueDatei = new File("C:\\Dev\\xx.txt");
if (neueDatei.createNewFile()) {
System.out.println("Datei wurde erstellt");
}
if (gefunden == false) {
dateiEinlesen(null, gefunden);
ArrayList<String> arr = null;
inNeueDateischreiben(neueDatei, gefunden, arr, suche, suche2);
}
}
public static void dateiEinlesen(File neueDatei, boolean gefunden) {
BufferedReader reader;
String zeile = null;
try {
reader = new BufferedReader(new FileReader("C:\\Dev\\Test.txt"));
zeile = reader.readLine();
ArrayList<String[]> arr = new ArrayList<String[]>();
while (zeile != null) {
arr.add(zeile.split(" "));
zeile = reader.readLine();
}
System.out.println(arr);
} catch (IOException e) {
System.err.println("Error2 :" + e);
}
}
public static void inNeueDateischreiben(File neueDatei, boolean gefunden, ArrayList<String> arr, String suche2,
String suche22) throws IOException {
FileWriter writer = new FileWriter(suche22);
String lastValue = null;
for (Iterator<String> i = arr.iterator(); i.hasNext();) {
String currentValue = i.next();
if (lastValue != null && currentValue.equals(lastValue)) {
i.remove();
{
writer.write(suche2.toString());
gefunden = true;
}
}
writer.close();
}
}
}

Your variable namings (suche2, suche22) makes reading the code difficult.
Other than that, your writing algorithm looks funny. You only compare adjacent lines while duplicate lines could be anywhere. In addition, writer.write only hits when you find a duplicate. Also how you call it and other things don't look right.
Here are some general steps to write this correctly:
Open the file so you can read it line by line.
Create a file writer
Create a set or dictionary like data structure that enables you to look up items in constant time.
For each line that you read do the following:
Look if the line exists in the dictionary.
If not, write it to the new file
If it already exists in the dictionary, skip to step 4.
Add that line to the dictionary for later comparisons and go to step 4.
When the lines are exhausted close both files.
I suggest, you rewrite your code completely as the current version is very difficult to amend.

How to count duplicate entries in a .csv file?

I have a .csv file that is formated like this:
ID,date,itemName
456,1-4-2020,Lemon
345,1-3-2020,Bacon
345,1-4-2020,Sausage
123,1-1-2020,Apple
123,1-2-2020,Pineapple
234,1-2-2020,Beer
345,1-4-2020,Cheese
I have already implemented the algorithm to go through the file, scan for the first number and sort it in a descending order and make a new output:
123,1-1-2020,Apple
123,1-2-2020,Pineapple
234,1-2-2020,Beer
345,1-3-2020,Bacon
345,1-4-2020,Cheese
345,1-4-2020,Sausage
456,1-4-2020,Lemon
My question is, how do I implement my algorithm to make an output that counts the duplicate first number entries and reformat it to make it look like this...
123,1-1-2020,1,Apple
123,1-2-2020,1,Pineapple
234,1-2-2020,1,Beer
345,1-3-2020,1,Bacon
345,1-4-2020,2,Cheese,Sausage
456,1-4-2020,1,Lemon
...so that it counts the number of occurrence for each ID, denote it with the number of times, and if the date of that ID is also the same, combine the item names to the same line. Below is my source code (each line in the .csv is made into an object named 'receipt' that has ID, date, and name with their respective get() methods):
public class ReadFile {
private static List<Receipt> readFile() {
List<Receipt> receipts = new ArrayList<>();
try {
BufferedReader reader = new BufferedReader(new FileReader("dataset.csv"));
// Move past the first title line
reader.readLine();
String line = reader.readLine();
// Start reading from second line till EOF, split each string at ","
while (line != null) {
String[] attributes = line.split(",");
Receipt attribute = getAttributes(attributes);
receipts.add(attribute);
line = reader.readLine();
}
reader.close();
} catch (IOException e) {
e.printStackTrace();
}
return receipts;
}
private static Receipt getAttributes(String[] attributes) {
// Get ID located before the first ","
long memberNumber = Long.parseLong(attributes[0]);
// Get date located after the first ","
String date = attributes[1];
// Get name located after the second ","
String name = attributes[2];
return new Receipt(memberNumber, date, name);
}
// Parse the data into new file after sorting
private static void parse(List<Receipt> receipts) {
PrintWriter output = null;
try {
output = new PrintWriter("output.txt");
} catch (FileNotFoundException e) {
e.printStackTrace();
}
// For each receipts, assert the text output stream is not null, print line.
for (Receipt p : receipts) {
assert output != null;
output.println(p.getMemberNumber() + "," + p.getDate() + "," + p.getName());
}
assert output != null;
output.close();
}
// Main method, accept input file, sort and parse
public static void main(String[] args) {
List<Receipt> receipts = readFile();
QuickSort q = new QuickSort();
q.quickSort(receipts);
parse(receipts);
}
}

The easiest way is to use a map.
Sample data from your file.
String[] lines = {
"123,1-1-2020,Apple",
"123,1-2-2020,Pineapple",
"234,1-2-2020,Beer",
"345,1-3-2020,Bacon",
"345,1-4-2020,Cheese",
"345,1-4-2020,Sausage",
"456,1-4-2020,Lemon"};
Create a map
as you read the lines, split them and add them to the map using the compute method. This will put the line in if the key (number and date) doesn't exist. Otherwise it simply appends the last item to the existing entry.
the file does not have to be sorted but the values will be added to the end as they are encountered.
Map<String, String> map = new LinkedHashMap<>();
for (String line : lines) {
String[] vals = line.split(",");
// if v is null, add the line
// if v exists, take the existing line and append the last value
map.compute(vals[0]+vals[1], (k,v)->v == null ? line : v +","+vals[2]);
}
for (String line : map.values()) {
String[] fields = line.split(",",3);
int count = fields[2].split(",").length;
System.out.printf("%s,%s,%s,%s%n", fields[0],fields[1],count,fields[2]);
}
For this sample run prints
123,1-1-2020,1,Apple
123,1-2-2020,1,Pineapple
234,1-2-2020,1,Beer
345,1-3-2020,1,Bacon
345,1-4-2020,2,Cheese,Sausage
456,1-4-2020,1,Lemon

Scanner restarting in Java

My task is to read a text file in chunks of 64 characters, and use 2 different processes called Substitution and Column Transposition to encrypt it. Then, I have to decrypt it and write it out to another file.
I have written and tested out both processes of encrypting and decrypting and it worked wonderfully. But then I tried to loop the processes in case more than 64 characters were in the input file.
As a test case, I tried a 128 character input file. Unfortunately, the result only gives me the first 64 characters twice. I've tracked the scanner position and it goes beyond 64, but the characters read start back from 0. I'm not sure what the problem is.
Here is the relevant part of my code:
public static void main(String[] args) {
//Declare variables
Scanner console = new Scanner(System.in);
String inputFileName = null;
File inputFile = null;
Scanner in = null;
do
{
//Check if there are enough arguments
try
{
inputFileName = args[1];
}
catch (IndexOutOfBoundsException exception)
{
System.out.println("Not enough arguments.");
System.exit(1);
}
catch (Exception exception)
{
System.out.println("There was an error. Please try again.");
System.exit(1);
}
//Check if Input File is valid
try
{
inputFile = new File(inputFileName);
in = new Scanner(inputFile);
outputFile = new File(outputFileName);
out = new Scanner(outputFile);
}
catch (FileNotFoundException exception)
{
System.out.println("Could not find input file.");
System.exit(1);
}
catch (Exception exception)
{
System.out.println("There was an error. Please try again.");
System.exit(1);
}
} while (outputFileName != null && !inputFile.exists());
//Encryption
//Prepare patterns
String subPattern = CreateSubstitutionPattern(hash);
int[] transPattern = CreateTranspositionPattern(hash);
//Apply patterns
String textContent = "";
String applySub = "";
String applyTrans = "";
do
{
textContent = Read64Chars(in);
applySub = applySub + ApplySubstitutionPattern(textContent, subPattern);
applyTrans = applyTrans + ApplyTranspositionPattern(applySub, transPattern);
} while (in.hasNext());
//Decryption
String encryptContent = "";
Scanner encrypt = new Scanner(applyTrans);
String removeTrans = "";
String removeSub = "";
do
{
encryptContent = Read64Chars(encrypt);
System.out.println(applyTrans);
removeTrans = removeTrans + RemoveTranspositionPattern(encryptContent, transPattern);
removeSub = removeSub + RemoveSubstitutionPattern(removeTrans, subPattern);
} while (encrypt.hasNext());
console.close();
in.close();
encrypt.close();
System.out.println(removeSub); //For temporary testing
}
public static String Read64Chars (Scanner in)
{
String textContent = "";
in.useDelimiter("");
for (int x=0; x<64; x++)
{
if (in.hasNext())
{
textContent = textContent + in.next().charAt(0);
}
}
return textContent;
}
Do note that I have more variables to fill in args[0] and args[2] but I removed them for simplicity.
I would like to know if it is true that once a scanner reads a portion of it's input, it "consumes" it, and that portion gets removed. Does the scanner reset itself when declared again through a method? For example, does the declaration only point to the input source of the original scanner, or the actual scanner with its current properties?

encrypt is a diffrent Scanner from in, which you advance by 64 characters when you first call Read64Chars. So, encrypt starts at the first character when you call Read64Chars(encrypt). It seems like you want to use the same Scanner both times.
Also, in the future please name your functions starting with a lowercase letter. I felt dirty typing that... :)

A proper solution to get the whole encrypted text would be a code like this
public static String encryptedTextFile (Scanner in)
{
//ArrayList<String> stringBlocksOf64Chars = new ArrayList<String>();
StringBuilder encryptedTxt = new StringBuilder();
String currentTxt = "";
while (in.hasNextLine()) {
String line = currentTxt + in.nextLine();
currentTxt = "";
int i = 0;
for( ; i < line.length()/64 ; i++){
currentTxt = line.substring(i * 64, (i+1)*64);
//TODO - encrypt the text before adding it to the list
encryptedTxt.append(currentTxt);//encryptedTxt.append(encrypt(currentTxt));
}
currentTxt = line.substring(i * 64, line.length());
}
encryptedTxt.append(currentTxt);
/*for(String str : stringBlocksOf64Chars)
System.out.println(str);*/
return encryptedTxt.toString();
}
Your loop for (int x=0; x<64; x++) makes sure that you read only first 64 characters always and not the complete file. To get around that you should actually read whole file line by line.
The above code block follows this idea.
Steps to break down the logic.
Read the file line by line using scanner.
Break each line into chunks of 64 characters and encrypt the block 64 characters at a time
Generate encrypted text adding the encrypted 64 characters.
Whatever you do first break down the logic/steps you want to use in your code to make it simpler to understand or code.
Break the lines into 64 characters

Using a loop I want to find the value of a column from my CSV file java

Forgive me if this is a basic (or not very well explained) question, I am fairly new to Java and have been reading extensive material as well as trying to understand the relevant Javadoc but to no avail.
To give a brief background as to what I am trying to create, I have created a reader class which reads data in from a csv file (4 lines long) including fields such as Item ID, price, description etc. I have created a separate demo class that displays the details of this csv file (through creating an instance of my reader class) and am now trying to create a method that asks the user to input an Item ID that then displays the corresponding Item, based on the ID input by the user. The part I am stuck on is accessing specific rows/columns in a csv file and then comparing these with a given string (entered by the user which corresponds to a specific field in the csv file)
This is what I have come up with thus far:
input = new Scanner(System.in);
System.out.println("Enter a product code");
String prodC = input.next();
//Here I want to know if there is a way of accessing a field in a csv file
Any ideas would be greatly appreciated.
UPDATE
Thank you for quick responses, am currently reading through and seeing how I can try to implement the various techniques. In response to the comment asking about the file reader, this is how I have set that out:
public CatalogueReader(String filename) throws FileNotFoundException {
this.filename = filename;
this.catalogue = new Catalogue();
Scanner csvFile;
try {
csvFile = new Scanner(new File(filename));
} catch (FileNotFoundException fnf) {
throw new FileNotFoundException("File has not been found!");
}
csvFile.useDelimiter("\n");
boolean first = true;
String productCode;
double price;
String description;
double weight;
int rating;
String category;
boolean ageRestriction;
String csvRows;
while (csvFile.hasNextLine()) {
csvRows = csvFile.nextLine();
if (first) {
first = false;
continue;
}
System.out.println(csvRows);
String[] fields = csvRows.split(",");
productCode = (fields[0].trim());
price = Double.parseDouble(fields[1].trim());
description = fields[2].trim();
weight = Double.parseDouble(fields[3].trim());
rating = Integer.parseInt(fields[4].trim());
category = fields[5].trim();
ageRestriction = Boolean.parseBoolean(fields[6].trim());
catalogue.addAProduct(new Item(productCode, price, description, weight, rating, category, ageRestriction));
}
csvFile.close();
}
}

ok so for a CSV file like this:
"1.0.0.0","1.0.0.255","16777216","16777471","AU","Australia"
"1.0.1.0","1.0.3.255","16777472","16778239","CN","China"
"1.0.4.0","1.0.7.255","16778240","16779263","AU","Australia"
"1.0.8.0","1.0.15.255","16779264","16781311","CN","China"
"1.0.16.0","1.0.31.255","16781312","16785407","JP","Japan"
"1.0.32.0","1.0.63.255","16785408","16793599","CN","China"
"1.0.64.0","1.0.127.255","16793600","16809983","JP","Japan"
"1.0.128.0","1.0.255.255","16809984","16842751","TH","Thailand"
here is a sample of how to read using Java Native Libraries
import java.io.BufferedReader;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.IOException;
public class CSVReader {
public static void main(String[] args) {
CSVReader obj = new CSVReader();
obj.run();
}
public void run() {
String csvFile = YOURFILEPATHHERE ;
BufferedReader br = null;
String line = "";
String cvsSplitBy = ",";
try {
br = new BufferedReader(new FileReader(csvFile));
while ((line = br.readLine()) != null) {
// use comma as separator
String[] country = line.split(cvsSplitBy);
System.out.println("Country [code= " + country[4]
+ " , name=" + country[5] + "]");
}
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
} finally {
if (br != null) {
try {
br.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
System.out.println("Done");
}
}
does this help?

If you are just doing a single look-up and then exiting then just remember the String you are looking for. As you parse the lines compare to see if you have a match and if you do then return that line.
For repeated searches that would be very inefficient though. Assuming your data set is not too large for memory you would be better off parsing the file and putting it into a Map:
Map<String, Data> dataMap = new HashMap<>();
Parse the file, putting all the lines into the map
Then the lookup just becomes:
Data d = dataMap.get(lineKey);
If d is null then there is no matching line. If it not null then you have found your line.

You can create an array list of object. An object for each line in the CSV. Then search the array object with your search criteria.
User CSVReader framework to read the csv file. Sample code (not exactly what you want)
FileInputStream fis = new FileInputStream(file);
CSVReader reader = new CSVReader(new BufferedReader( new InputStreamReader(fis, "UTF-8" )));
ArrayList<String> row = new ArrayList<String>();
ArrayList<Entry> entries = new ArrayList<Entry>();
// a line = ID, Name, Price, Description
while (!reader.isEOF()) {
reader.readFields(row);
if( row.size() >= 4)
entries.add(new Entry(row.get(0), row.get(1), row.get(2), row.get(3)));
}
System.out.println("Size : "+entries);

how get values for table from specific row in text files?

i making a program where I would read data from text files and store them in tables in mysql.
In my program the user would give the directory of where the files are, then the program would find only the .txt files and would continue. Afterwards a table would be created and it would have 2 fields and in these fields I would insert the values from the text file.
My issue is that i don't know how! I would explain you what I mean! In my program I would create table with fields (ID, Name). The values of these fields must be taken from the text file. All the files are as the below:
As you can see the ID is in the third row of the file and the Name is in the fifth. Could anyone help me how can I import the values for ID and Name in the table?How can i get only these values each time from the files?
The code for doing the first steps is:
public static void main(String args[]) throws Exception {
Class.forName("com.mysql.jdbc.Driver");
Connection con = (Connection) DriverManager.getConnection(
"jdbc:mysql://localhost:3306/mydb", "", "");
String dirpath = "";
Scanner scanner1 = new Scanner(System.in);
while (true) {
System.out.println("Please give the directory:");
dirpath = scanner1.nextLine();
File fl = new File(dirpath);
if (fl.canRead())
break;
System.out.println("Error:Directory does not exists");
}
try {
String files;
File folder = new File(dirpath);
File[] listOfFiles = folder.listFiles();
for (int i = 0; i < listOfFiles.length; i++) {
if (listOfFiles[i].isFile()) {
files = listOfFiles[i].getName();
if (files.endsWith(".txt") || files.endsWith(".TXT")) {
List<File> txtFiles = new ArrayList<File>();
txtFiles.add(listOfFiles[i]);
String[] parts = files.split("\\.");
String tablename = parts[0];
for (File txtFile : txtFiles) {
List sheetData = new ArrayList();
try {
FileReader in = new FileReader(txtFile);
BufferedReader br = new BufferedReader(in);
String line = br.readLine();
while (line != null) {
System.out.println(line);
line = br.readLine();
}
in.close();
} catch (Exception e) {
System.err.println("Error: " + e.getMessage());
}
getCreateTable1(con, tablename);
importData(con, txtFile, tablename);
}
}
}
}
} catch (Exception e) {
System.out.println();
}
}
private static String getCreateTable1(Connection con, String tablename) {
try {
Class.forName("com.mysql.jdbc.Driver");
Statement stmt = con.createStatement();
String createtable = "CREATE TABLE "
+ tablename
+ " ( ID INT , name VARCHAR(255)";
System.out.println("Create a new table in the database");
stmt.executeUpdate(createtable);
} catch (Exception e) {
System.out.println(((SQLException) e).getSQLState());
System.out.println(e.getMessage());
e.printStackTrace();
}
return null;
}

BufferedReader br = new BufferedReader(new FileReader(new File("path/to/file")));
String currentLine = br.readLine();
Map<Integer, String> nameByID = new HashMap<Integer, String>();
while (currentLine != null) {
String[] tokens = currentLine.split("\t");
int id = Integer.parseInt(tokens[2]);
String name = tokens[4];
nameByID.put(id, name);
currentLine = br.readLine();
}
br.close();
nameByID will have the names and IDs you need.
Note that some exception handling is required for calls to create a new BufferedReader, for calls to readLine(), and to close the BufferedReader. I didn't insert this because I couldn't remember it off the top of my head but your IDE should prompt you to insert if you're using something like Netbeans or Eclipse

You should try not to reinvent the wheel.
Use a FileNameExtensionFilter to filter the .txt files, this class is from swing but it's fine to use in plain java.
Check if each line matches a regex pattern, that way you can digest the line at the same time as verifying it.
Create a Person object that holds this information and return a Collection of Person - that way you encapsulate your file reading behavior away from your database access layer.
Put all this in a class called, say, FileReader and you have something like the following:
public class FileReader {
private final Pattern linePattern = Pattern.compile("^(\\w++)\\s++(\\w++)\\s*+$");
private final Pattern lineBreakPattern = Pattern.compile("\r?\n");
private final FileFilter txtFilter = new FileNameExtensionFilter("*.txt", "txt");
private final File txtFolder;
public FileReader(File txtFolder) {
this.txtFolder = txtFolder;
}
public List<Person> readFiles() {
final List<Person> people = new LinkedList<>();
for (final File txtFile : txtFolder.listFiles()) {
if (txtFilter.accept(txtFile)) {
people.add(readFile(txtFile));
}
}
return people;
}
private Person readFile(File txtFile) {
try (final Scanner scanner = new Scanner(txtFile)) {
scanner.useDelimiter(lineBreakPattern);
final Person person = new Person();
while (scanner.hasNext()) {
final String line = scanner.next();
final Matcher matcher = linePattern.matcher(line);
if (matcher.matches()) {
switch (matcher.group(1).toUpperCase()) {
case "ID":
person.setId(Integer.parseInt(matcher.group(2)));
break;
case "NAME":
person.setName(matcher.group(2));
break;
default:
throw new IOException("Illegal line '" + matcher.group() + "'.");
}
}
}
return person;
} catch (IOException ex) {
throw new RuntimeException(ex);
}
}
public static final class Person {
private int id;
private String name;
public int getId() {
return id;
}
public void setId(int id) {
this.id = id;
}
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
}
}
So you would create a FileReader with the folder that contains the files and then call readFiles, you then save the returned List<Person> in the database.
Lets go through this class.
The readFiles method loops over all files in the directory and checks whether each one of them matches the txtFilter - this filters out any non .txt file.
The readFiles method also creates and returns a List<Person, this is the result of reading the files. The List is populated by the readFile(File txtFile) method. That method is responsible for reading the individual files and parsing them to a Person.
The Person class is a very simple data transfer object, holding on properties and accessors. No logic.
The readFile method creates a Scanner in a Java 7 try-with-resources construct. It sets the delimiter to a platform independent linebreak pattern (\r?\n means that it matches \r\n or \n) and then loops over the scanner output.
Each line is processed with the linePattern, this probably warrants some explanation:
^(\\w++)\\s++(\\w++)\\s*+$
^ is the "start anchor", i.e. the line starts here
(\\w++) means capture any number of word characters
\\s++ means skip any number of whitespace characters
(\\w++) same as above
\\s*+ means skip zero or more whitespace characters
$ is the "end anchor", i.e. the end of the line
So, if the pattern matches we have a valid line. Moreover, when verifying we grabbed to "groups" of characters, these are our key and value.
Next we use a switch on the first group, this is using Java 7 switches with Strings. We populate the person depending on the value of the key, parsing the int where needed.
Finally we return the populated person.
This class should get you well on your way to accomplishing you goal - the sql insertion of the Person objects into a database is trivial.
You may want to add more verification during the file reading process, for example check that both a NAME and ID were found. I leave this as an exercise.