Related
I have a csv file that doesn't always have the same number of lines. However, I want a method to only read me the last line, so I can access the first column of that last line. So far I haven't found a solution, that does exactly that.
Right now I'm just at the point were I would read every single line with BufferedReader and save it into an Array.
public void readPreviousEntryID(){
String path = "csvs/cartEntries.csv";
try {
BufferedReader br = new BufferedReader((new FileReader(path)));
String line;
while ((line = br.readLine() != null)) {
String[] values = line.split(",");
}
} catch (FileNotFoundException e) {
throw new RuntimeException(e);
}
}
Normally I would then access the first entry of every line by using values[0]. But I just want the first value of the last line.
I thought about counting the number of lines in the while loop by incrementing an integer and then using the final value of that integer to access the corresponding line, but I'm not sure if this would work or how I would implement that.
I hope I included enough information to make the problem understandable. This is my first question here and I'm quite new to Java.
Simply read the lines of the file in a loop and save the values of the last line read. After the loop terminates, values contains the contents of the last line in the file.
public void readPreviousEntryID() throws IOException {
String path = "csvs/cartEntries.csv";
try (FileReader fr = new FileReader(path);
BufferedReader br = new BufferedReader(fr)) {
String[] values = null;
String line = br.readLine();
while (line != null) {
values = line.split(",");
line = br.readLine();
}
if (values == null) {
throw new IOException("File is empty.");
}
// Handle 'values[0]'
}
}
The advantage of the above code is that you don't need to store the entire file contents in the computer memory. If the CSV file is very large, you may get OutOfMemoryError.
Note that is important to close a file after you have finished reading it. Since Java 7 you can use try-with-resources.
Rather than catch the IOException and wrap it in a RuntimeException, I suggest that you simply declare that method readPreviousEntryID may throw an IOException. Refer to Unchecked Exceptions — The Controversy.
It is probably also a good idea to check, after the loop terminates, that values contains the expected number of elements, e.g.
if (values.length == 5) {
// Handle 'values[0]'
}
else {
throw new IOException("Invalid last line.");
}
Edit
Alternatively, no need to split every line. Just save the last line read and split that last line after the loop terminates.
public void readPreviousEntryID() throws IOException {
String path = "csvs/cartEntries.csv";
try (FileReader fr = new FileReader(path);
BufferedReader br = new BufferedReader(fr)) {
String lastLine = null;
String line = br.readLine();
while (line != null) {
lastLine = line;
line = br.readLine();
}
if (lastLine == null) {
throw new IOException("File is empty.");
}
String[] values = lastLine.split(",");
// Handle 'values[0]'
}
}
Why not stored all lines into List and get last line details such as follows
private List<String[]> readLines = new ArrayList<>();
public void readPreviousEntryID(){
String path = "csvs/cartEntries.csv";
try {
String line;
BufferedReader br = new BufferedReader(new FileReader(path));
while ((line = br.readLine()) != null) {
String[] values = line.split(",");
readLines.add(values);
}
br.close();
} catch (IOException e) {
e.printStackTrace();
}
}
public String[] getLastLine() {
return readLines.get(readLines.size()-1);
}
This above function will gives the last row of csv file.
In Linux one would use the tail command to print the n last lines. Search java tail will get you some implementations.
A good fast implementation for large files would use a RandomAccessFile, maybe a MemoryMappedByteBuffer, and search back from the end for a \n.
In your case you can keep it simple.
public String readPreviousEntryID(){
Path path = Paths.get("csvs/cartEntries.csv");
try (Stream<String> lines = Files.lines(path, Charset.defaultCharset())) {
return lines
.filter(line -> !line.isEmpty())
.reduce("", (acc, line) -> line);
} catch (FileNotFoundException e) {
// You gave a relative path, show the actual full path:
System.out.println("File not found: " + Files.toAbsolutePath());
throw new RuntimeException(e);
} // Automatically closes the Stream lines, even on exception or return.
}
Try-with-resources try (... declarations of AutoCloseables ...) { ... } ensuring the call to .close().
Stream, the newest walk through items. Here skipping empty lines and "accumulating" just the last line. You need not keep all lines in memory.
Lambdas, x -> ... or (x, y) -> ... declare an anonymous function with 1 resp. 2 parameter declarations.
Path is a generalisation of disk file only File. Path can also be from an URL, inside a zip file etcetera.
Files is a worthwile utility class providing many Path related goodies. Files.
Some good Answers have been posted. Here is a variation using streams.
Also, learn about NIO.2 as the modern way to work with files in Java.
Some untested code to try:
Path path = Paths.get( "/csvs" , "cartEntries.csv" ) ;
Optional < String > lastLine =
Files
.lines( path )
.reduce( ( previousLine , currentLine ) -> currentLine ) ;
if( lastLine.isPresent() ) {
String[] parts = lastLine.get().split( "," ) ;
…
}
Or, re-organized into a one-liner:
String[] parts =
Files
.lines(
Paths.get( "/csvs" , "cartEntries.csv" )
)
.reduce(
( previousLine , currentLine ) -> currentLine
)
.map(
line -> line.split.( "," )
)
.orElse(
String[] :: new
)
;
Can someone tell me how to read every second line from a file in java?
BufferedReader br = new BufferedReader(new FileReader(file));
String line = br.readLine();
while(line != null){
//Do something ..
line = br.readLine()
}
br.close
One simple way would be to just maintain a counter of number of lines read:
int count = 0;
String line;
while ((line = br.readLine()) != null) {
if (count % 2 == 0) {
// do something with this line
}
++count;
}
But this still technically reads every line in the file, only choosing to process every other line. If you really only want to read every second line, then something like RandomAccessFile might be necessary.
You can do it in Java 8 fashion with very few lines :
static final int FIRST_LINE = 1;
Stream<String> lines = Files.lines(path);
String secondLine = lines.limit(2).skip(FIST_LINE).collect(Collectors.joining("\n"));
First you stream your file lines
You keep only the two first lines
Skip the first line
Note : In java 8, when using Files.lines(), you are supposed to close the stream afterwards or use it in a try-with-resource block.
This is similar to #Tim Biegeleisen's approach, but I thought I would show an alternative to get every other line using a boolean instead of a counter:
boolean skipOddLine = true;
String line;
while ((line = br.readLine()) != null) {
if (skipOddLine = !skipOddLine) {
//Use the String line here
}
}
This will toggle the boolean value every loop iteration, skipping every odd line. If you want to skip every even line instead you just need to change the initial condition to boolean skipOddLine = false;.
Note: This approach only works if you do not need to extend functionality to skip every 3rd line for example, where an approach like Tim's would be easier to modify. It also has the downside of being harder to read than the modulo approach.
This will help you to do it very well
You can use try with resource
You can use stream api java 8
You can use stream api supplier to use stream object again and again
I already hane added comment area to understand you
try (BufferedReader reader =
new BufferedReader(
new InputStreamReader(
new ByteArrayInputStream(x.getBytes()),
"UTF-8"))) { //this will help to you for various languages reading files
Supplier<Stream<String>> fileContentStream = reader::lines; // this will help you to use stream object again and again
if (FilenameUtils.getExtension(x.getOriginalFilename()).equals("txt")) { this will help you to various files extension filter
String secondLine = lines.limit(2).skip(FIST_LINE).collect(Collectors.joining("\n"));
String secondLine =
fileContentStream
.get()
.limit(2)
.skip(1)// you can skip any line with this action
.collect(Collectors.joining("\n"));
}
else if (FilenameUtils.getExtension(x.getOriginalFilename()).equals("pdf")) {
} catch (Exception ex) {
}
I have noticed that using java.util.Scanner is very slow when reading large files (in my case, CSV files).
I want to change the way I am currently reading files, to improve performance. Below is what I have at the moment. Note that I am developing for Android:
InputStreamReader inputStreamReader;
try {
inputStreamReader = new InputStreamReader(context.getAssets().open("MyFile.csv"));
Scanner inputStream = new Scanner(inputStreamReader);
inputStream.nextLine(); // Ignores the first line
while (inputStream.hasNext()) {
String data = inputStream.nextLine(); // Gets a whole line
String[] line = data.split(","); // Splits the line up into a string array
if (line.length > 1) {
// Do stuff, e.g:
String value = line[1];
}
}
inputStream.close();
} catch (IOException e) {
e.printStackTrace();
}
Using Traceview, I managed to find that the main performance issues, specifically are: java.util.Scanner.nextLine() and java.util.Scanner.hasNext().
I've looked at other questions (such as this one), and I've come across some CSV readers, like the Apache Commons CSV, but they don't seem to have much information on how to use them, and I'm not sure how much faster they would be.
I have also heard about using FileReader and BufferedReader in answers like this one, but again, I do not know whether the improvements will be significant.
My file is about 30,000 lines in length, and using the code I have at the moment (above), it takes at least 1 minute to read values from about 600 lines down, so I have not timed how long it would take to read values from over 2,000 lines down, but sometimes, when reading information, the Android app becomes unresponsive and crashes.
Although I could simply change parts of my code and see for myself, I would like to know if there are any faster alternatives I have not mentioned, or if I should just use FileReader and BufferedReader. Would it be faster to split the huge file into smaller files, and choose which one to read depending on what information I want to retrieve? Preferably, I would also like to know why the fastest method is the fastest (i.e. what makes it fast).
uniVocity-parsers has the fastest CSV parser you'll find (2x faster than OpenCSV, 3x faster than Apache Commons CSV), with many unique features.
Here's a simple example on how to use it:
CsvParserSettings settings = new CsvParserSettings(); // many options here, have a look at the tutorial
CsvParser parser = new CsvParser(settings);
// parses all rows in one go
List<String[]> allRows = parser.parseAll(new FileReader(new File("your/file.csv")));
To make the process faster, you can select the columns you are interested in:
parserSettings.selectFields("Column X", "Column A", "Column Y");
Normally, you should be able to parse 4 million rows around 2 seconds. With column selection the speed will improve by roughly 30%.
It is even faster if you use a RowProcessor. There are many implementations out-of-the box for processing conversions to objects, POJOS, etc. The documentation explains all of the available features. It works like this:
// let's get the values of all columns using a column processor
ColumnProcessor rowProcessor = new ColumnProcessor();
parserSettings.setRowProcessor(rowProcessor);
//the parse() method will submit all rows to the row processor
parser.parse(new FileReader(new File("/examples/example.csv")));
//get the result from your row processor:
Map<String, List<String>> columnValues = rowProcessor.getColumnValuesAsMapOfNames();
We also built a simple speed comparison project here.
Your code is good to load big files. However, when an operation is going to be longer than you're expecting, it's good practice to execute it in a task and not in UI Thread, in order to prevent any lack of responsiveness.
The AsyncTask class help to do that:
private class LoadFilesTask extends AsyncTask<String, Integer, Long> {
protected Long doInBackground(String... str) {
long lineNumber = 0;
InputStreamReader inputStreamReader;
try {
inputStreamReader = new
InputStreamReader(context.getAssets().open(str[0]));
Scanner inputStream = new Scanner(inputStreamReader);
inputStream.nextLine(); // Ignores the first line
while (inputStream.hasNext()) {
lineNumber++;
String data = inputStream.nextLine(); // Gets a whole line
String[] line = data.split(","); // Splits the line up into a string array
if (line.length > 1) {
// Do stuff, e.g:
String value = line[1];
}
}
inputStream.close();
} catch (IOException e) {
e.printStackTrace();
}
return lineNumber;
}
//If you need to show the progress use this method
protected void onProgressUpdate(Integer... progress) {
setYourCustomProgressPercent(progress[0]);
}
//This method is triggered at the end of the process, in your case when the loading has finished
protected void onPostExecute(Long result) {
showDialog("File Loaded: " + result + " lines");
}
}
...and executing as:
new LoadFilesTask().execute("MyFile.csv");
You should use a BufferedReader instead:
BufferedReader reader = null;
try {
reader = new BufferedReader( new InputStreamReader(context.getAssets().open("MyFile.csv"))) ;
reader.readLine(); // Ignores the first line
String data;
while ((data = reader.readLine()) != null) { // Gets a whole line
String[] line = data.split(","); // Splits the line up into a string array
if (line.length > 1) {
// Do stuff, e.g:
String value = line[1];
}
}
} catch (IOException e) {
e.printStackTrace();
} finally {
if (reader != null) {
try {
reader.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
I must do an application, that compares some very big csv files, each one having 40,000 records. I have done an application, that works properly, but it spends a lot of time in doing that comparison, because the two files could be disordenated or have different records - for that I must iterate (40000^2)*2 times.
Here is my code:
if (nomFich.equals("CAR"))
{
while ((linea = br3.readLine()) != null)
{
array =linea.split(",");
spliteado = array[0]+array[1]+array[2]+array[8];
FileReader fh3 = new FileReader(cadena + lista2[0]);
BufferedReader bh3 = new BufferedReader(fh3);
find=0;
while (((linea2 = bh3.readLine()) != null))
{
array2 =linea2.split(",");
spliteado2 = array2[0]+array2[1]+array2[2]+array2[8];
if (spliteado.equals(spliteado2))
{
find =1;
}
}
if (find==0)
{
bw3.write("+++++++++++++++++++++++++++++++++++++++++++");
bw3.newLine();
bw3.write("Se han incorporado los siguientes CGI en la nueva lista");
bw3.newLine();
bw3.write(linea);
bw3.newLine();
aparece=1;
}
bh3.close();
}
I think that using a Set in Java is a good option, like the following post suggests:
Comparing two csv files in Java
But before I try it this way, I would like to know, if there are any better options.
Thanks for all.
As far as I can interpret your code, you need to find out which lines in the first CSV file do not have an equal line in the second CSV file. Correct?
If so, you only need to put all lines of the second CSV file into a HashSet. Like so (Java 7 code):
Set<String> linesToCompare = new HashSet<>();
try (BufferedReader reader = new BufferedReader(new FileReader(cadena + lista2[0]))) {
String line;
while ((line = reader.readLine()) != null) {
String[] splitted = line.split(",");
linesToCompare.add(splitted[0] + splitted[1] + splitted[2] + splitted[8]);
}
}
Afterwards you can simply iterate over the lines in the first CSV file and compare:
try (BufferedReader reader = new BufferedReader(new FileReader(...))) {
String line;
while ((line = reader.readLine()) != null) {
String[] splitted = line.split(",");
String joined = splitted[0] + splitted[1] + splitted[2] + splitted[8];
if (!linesToCompare.contains(joined)) {
// handle missing line here
}
}
}
Does that fit your needs?
HashMap<String, String> file1Map = new HashMap<String, String>();
while ((String line = file1.readLine()) != null) {
array =line.split(",");
key = array[0]+array[1]+array[2]+array[8];
file1Map.put(key, key);
}
while ((String line = file2.readLine()) != null) {
array =line.split(",");
key = array[0]+array[1]+array[2]+array[8];
if (file1Map.containsKey(key)) {
//if file1 has same line in file2
}
else {
//if file1 doesn't have line like in file2
}
}
Assuming this all won't fit in memory I would first convert the files to their stripped down versions (el0, el1, el2, el8, orig-file-line-nr-for-reference-afterwards) and then sort said files. After that you can stream through both files simultaneously and compare the records as you go... Taking the sorting out of the equation you only need to compare them 'about once'.
But I'm guessing you could do the same using some List/Array object that allows for sorting and storing in memory; 40k records really doesn't sound all that much to me, unless the elements are very big of course. And it's going to be magnitudes faster.
I have a text file that I want to edit using Java. It has many thousands of lines. I basically want to iterate through the lines and change/edit/delete some text. This will need to happen quite often.
From the solutions I saw on other sites, the general approach seems to be:
Open the existing file using a BufferedReader
Read each line, make modifications to each line, and add it to a StringBuilder
Once all the text has been read and modified, write the contents of the StringBuilder to a new file
Replace the old file with the new file
This solution seems slightly "hacky" to me, especially if I have thousands of lines in my text file.
Anybody know of a better solution?
I haven't done this in Java recently, but writing an entire file into memory seems like a bad idea.
The best idea that I can come up with is open a temporary file in writing mode at the same time, and for each line, read it, modify if necessary, then write into the temporary file. At the end, delete the original and rename the temporary file.
If you have modify permissions on the file system, you probably also have deleting and renaming permissions.
if the file is just a few thousand lines you should be able to read the entire file in one read and convert that to a String.
You can use apache IOUtils which has method like the following.
public static String readFile(String filename) throws IOException {
File file = new File(filename);
int len = (int) file.length();
byte[] bytes = new byte[len];
FileInputStream fis = null;
try {
fis = new FileInputStream(file);
assert len == fis.read(bytes);
} catch (IOException e) {
close(fis);
throw e;
}
return new String(bytes, "UTF-8");
}
public static void writeFile(String filename, String text) throws IOException {
FileOutputStream fos = null;
try {
fos = new FileOutputStream(filename);
fos.write(text.getBytes("UTF-8"));
} catch (IOException e) {
close(fos);
throw e;
}
}
public static void close(Closeable closeable) {
try {
closeable.close();
} catch(IOException ignored) {
}
}
You can use RandomAccessFile in Java to modify the file on one condition:
The size of each line has to be fixed otherwise, when new string is written back, it might override the string in the next line.
Therefore, in my example, I set the line length as 100 and padding with space string when creating the file and writing back to the file.
So in order to allow update, you need to set the length of line a little larger than the longest length of the line in this file.
public class RandomAccessFileUtil {
public static final long RECORD_LENGTH = 100;
public static final String EMPTY_STRING = " ";
public static final String CRLF = "\n";
public static final String PATHNAME = "/home/mjiang/JM/mahtew.txt";
/**
* one two three
Text to be appended with
five six seven
eight nine ten
*
*
* #param args
* #throws IOException
*/
public static void main(String[] args) throws IOException
{
String starPrefix = "Text to be appended with";
String replacedString = "new text has been appended";
RandomAccessFile file = new RandomAccessFile(new File(PATHNAME), "rw");
String line = "";
while((line = file.readLine()) != null)
{
if(line.startsWith(starPrefix))
{
file.seek(file.getFilePointer() - RECORD_LENGTH - 1);
file.writeBytes(replacedString);
}
}
}
public static void createFile() throws IOException
{
RandomAccessFile file = new RandomAccessFile(new File(PATHNAME), "rw");
String line1 = "one two three";
String line2 = "Text to be appended with";
String line3 = "five six seven";
String line4 = "eight nine ten";
file.writeBytes(paddingRight(line1));
file.writeBytes(CRLF);
file.writeBytes(paddingRight(line2));
file.writeBytes(CRLF);
file.writeBytes(paddingRight(line3));
file.writeBytes(CRLF);
file.writeBytes(paddingRight(line4));
file.writeBytes(CRLF);
file.close();
System.out.println(String.format("File is created in [%s]", PATHNAME));
}
public static String paddingRight(String source)
{
StringBuilder result = new StringBuilder(100);
if(source != null)
{
result.append(source);
for (int i = 0; i < RECORD_LENGTH - source.length(); i++)
{
result.append(EMPTY_STRING);
}
}
return result.toString();
}
}
If the file is large, you might want to use a FileStream for output, but that seems pretty much like it is the simplest process to do what you're asking (and without more specificity i.e. on what types of changes / edits / deletions you're trying to do, it's impossible to determine what more complicated way might work).
No reason to buffer the entire file.
Simply write each line as your read it, insert lines when necessary, delete lines when necessary, replace lines when necessary.
Fundamentally, you will not get around having to recreate the file wholesale, especially if it's just a text file.
What kind of data is it? Do you control the format of the file?
If the file contains name/value pairs (or similar), you could have some luck with Properties, or perhaps cobbling together something using a flat file JDBC driver.
Alternatively, have you considered not writing the data so often? Operating on an in-memory copy of your file should be relatively trivial. If there are no external resources which need real time updates of the file, then there is no need to go to disk every time you want to make a modification. You can run a scheduled task to write periodic updates to disk if you are worried about data backup.
In general you cannot edit the file in place; it's simply a very long sequence of characters, which happens to include newline characters. You could edit in place if your changes don't change the number of characters in each line.
Can't you use regular expressions, if you know what you want to change ? Jakarta Regexp should probably do the trick.
Although this question was a time ago posted, I think it is good to put my answer here.
I think that the best approach is to use FileChannel from java.nio.channels package in this scenario. But this, only if you need to have a good performance! You would need to get a FileChannel via a RandomAccessFile, like this:
java.nio.channels.FileChannel channel = new java.io.RandomAccessFile("/my/fyle/path", "rw").getChannel();
After this, you need a to create a ByteBuffer where you will read from the FileChannel.
this looks something like this:
java.nio.ByteBuffer inBuffer = java.nio.ByteBuffer.allocate(100);
int pos = 0;
int aux = 0;
StringBuilder sb = new StringBuilder();
while (pos != -1) {
aux = channel.read(inBuffer, pos);
pos = (aux != -1) ? pos + aux : -1;
b = inBuffer.array();
sb.delete(0, sb.length());
for (int i = 0; i < b.length; ++i) {
sb.append((char)b[i]);
}
//here you can do your stuff on sb
inBuffer = ByteBuffer.allocate(100);
}
Hope that my answer will help you!
I think, FileOutputStream.getFileChannel() will help a lot, see FileChannel api
http://java.sun.com/javase/6/docs/api/java/nio/channels/FileChannel.html
private static void modifyFile(String filePath, String oldString, String newString) {
File fileToBeModified = new File(filePath);
StringBuilder oldContent = new StringBuilder();
try (BufferedReader reader = new BufferedReader(new FileReader(fileToBeModified))) {
String line = reader.readLine();
while (line != null) {
oldContent.append(line).append(System.lineSeparator());
line = reader.readLine();
}
String content = oldContent.toString();
String newContent = content.replaceAll(oldString, newString);
try (FileWriter writer = new FileWriter(fileToBeModified)) {
writer.write(newContent);
}
} catch (IOException e) {
e.printStackTrace();
}
}
You can change the txt file to java by saving on clicking "Save As" and saving *.java extension.