Develop Reference

How do I input data from a CSV file into my code?

I have a CSV file that is like:
Orbit,a,e,i
LEO,7168000,0,90
MEO,20200000,0,54
What I want is that using scanner, I will be able to choose one of the two orbits I have in the CSV file. Let's say for example, using scanner if I put 1 in the console, it chooses the row of the CSV file: LEO,7168000,0,90 and if I put 2 the line: MEO,20200000,0,54.
After that from the row chosen it saves each parameter in a variable skipping the name (LEO,MEO). For example if I choose LEO orbit, it saves the variables like:
double a = 7168000;
double e = 0;
double i = 90;
So in the end I can use those parameters in my program. Thank you for your answers.

Csv file is a plain text file and separates with , character for each cell and \n for each row.
The easy way, you only need using FileInputStream to read and split \n and , character for using.
File file = new File("file.csv");
FileInputStream fis = null;
String dataStr = "";
try {
fis = new FileInputStream(file);
int content;
while ((content = fis.read()) != -1) {
dataStr += (char) content;
}
} catch (IOException e) {
e.printStackTrace();
}
// convert to array rows string
String[] dataRows = dataStr.split("\n");
// loop rows to get cells string
for (int i = 0; i < dataRows.length; i++) {
String[] dataCells = rowData[i].split(",");
//do what ever you want with dataCells
}
Thanks for read.

comparing data line by line from two large files

I need to analyze differences between two large data files which should each have identical structures. Each file is a couple of gigabytes in size, with perhaps 30 million lines or text data. The data files are so large that I hesitate to load each into its own array, when it might be easier to just iterate through the lines in order. Each line has the structure:
topicIdx, recordIdx, other fields...
topicIdx and recordIdx are sequential, starting at zero and incrementing +1 with each iteration, so it is easy to find them in the files. (No searching around required; just increment forward in order).
I need to do something like:
for each line in fileA
store line in String itemsA
get topicIdx and recordIdx
find line in fileB with same topicIdx and recordIdx
if exists
store this line in string itemsB
for each item in itemsA
compare value with same index in itemsB
if these two items are not virtually equal
//do something
else
//do something else
I wrote the following code with FileReader and BufferedReader, but the apis for these do not seem to provide the functionality that I need. Can anyone show me how to fix the code below so that it accomplishes what I desire?
void checkData(){
FileReader FileReaderA;
FileReader FileReaderB;
int topicIdx = 0;
int recordIdx = 0;
try {
int numLines = 0;
FileReaderA = new FileReader("B:\\mypath\\fileA.txt");
FileReaderB = new FileReader("B:\\mypath\\fileB.txt");
BufferedReader readerA = new BufferedReader(FileReaderA);
BufferedReader readerB = new BufferedReader(FileReaderB);
String lineA = null;
while ((lineA = readerA.readLine()) != null) {
if (lineA != null && !lineA.isEmpty()) {
List<String> itemsA = Arrays.asList(lineA.split("\\s*,\\s*"));
topicIdx = Integer.parseInt(itemsA.get(0));
recordIdx = Integer.parseInt(itemsA.get(1));
String lineB = null;
//lineB = readerB.readLine();//i know this syntax is wrong
setB = rows from FileReaderB where itemsB.get(0).equals(itemsA.get(0));
for each lineB in setB{
List<String> itemsB = Arrays.asList(lineB.split("\\s*,\\s*"));
for(int m = 0;m<itemsB.size();m++){}
for(int j=0;j<itemsA.size();j++){
double myDblA = Double.parseDouble(itemsA.get(j));
double myDblB = Double.parseDouble(itemsB.get(j));
if(Math.abs(myDblA-myDblB)>0.0001){
//do something
}
}
}
}
readerA.close();
} catch (IOException e) {e.printStackTrace();}
}

You need both files sorted by your search keys (recordIdx and topicIdx), so you can do kind of a merge operation like this
open file 1
open file 2
read lineA from file1
read lineB from file2
while (there is lineA and lineB)
if (key lineB < key lineA)
read lineB from file 2
continue loop
if (key lineB > key lineA)
read lineA from file 1
continue
// at this point, you have lineA and lineB with matching keys
process your data
read lineB from file 2
Note that you'll only ever have two records in memory.

If you really need this in Java, why not use java-diff-utils ? It implements a well known diff algorithm.

Consider https://code.google.com/p/java-diff-utils/. Let someone else do the heavy lifting.

Read a definite number of lines in a text file, using java

I have a text file with data. The file has information from all months. Imagine that the information for January occupy 50 lines. Than February starts and it occupies 40 more lines. Than I have March and so on... Is it possible to read only part of the file? Can I say "read from line X to line Y"? or is there a better way to accomplish this? I only want to print the data correspondent to one month not the all file. Here is my code
public static void readFile()
{
try
{
DataInputStream inputStream =
new DataInputStream(new FileInputStream("SpreadsheetDatabase2013.txt"));
while(inputStream.available() != 0)
{
System.out.println("AVAILABLE: " + inputStream.available());
System.out.println(inputStream.readUTF());
System.out.println(inputStream.readInt());
for (int i = 0; i < 40; i++)
{
System.out.println(inputStream.readUTF());
System.out.println(inputStream.readUTF());
System.out.println(inputStream.readUTF());
System.out.println(inputStream.readUTF());
System.out.println(inputStream.readUTF());
System.out.println(inputStream.readDouble());
System.out.println(inputStream.readUTF());
System.out.println(inputStream.readBoolean());
System.out.println();
}
}// end while
inputStream.close();
}// end try
catch (Exception e)
{
System.out.println("An error has occurred.");
}//end catch
}//end method
Thank you for your time.

My approach to this would be to read the entire contents of the text file and store it in a ArrayList and read only the lines for the requested month.
Example:
Use this function to read the all the lines from the file.
/**
* Read from a file specified by the filePath.
*
* #param filePath
* The path of the file.
* #return List of lines in the file.
* #throws IOException
*/
public static ArrayList<String> readFromFile(String filePath)
throws IOException {
ArrayList<String> temp = new ArrayList<String>();
File file = new File(filePath);
if (file.exists()) {
BufferedReader brin;
brin = new BufferedReader(new FileReader(filePath));
String line = brin.readLine();
while (line != null) {
if (!line.equals(""))
temp.add(line);
line = brin.readLine();
}
brin.close();
}
return temp;
}
Then read only the ones you need from ArrayList temp.
Example:
if you want to read February month's data assuming its 50 lines of data and starts from 40th line.
for(int i=40;i<90;i++)
{
System.out.println(temp.get(i));
}
Note: This is only just one way of doing this. I am not certain if there is any other way!

I would use the scanner class.
Scanner scanner = new Scanner(filename);
Use scanner.nextLine() to get each of the lines of the file. If you only want from line x to line y you can use a for loop to scan each of the lines that you don't need before going through the scanner for the lines you do need. Be careful not to hit an exception without throwing it though.
Or you can go through the scanner and for each line, add the String contents of the line to an ArrayList. Good luck.

Based on how you said your data was organized, I would suggest doing something like this
ArrayList<String> temp = new ArrayList<String>();
int read = 0;
File file = new File(filePath);
if (file.exists()) {
BufferedReader brin;
brin = new BufferedReader(new FileReader(filePath));
String line = brin.readLine();
while (line != null) {
if (!line.equals("")){
if(line.equals("March"))
read = 1;
else if(line.equals("April"))
break;
else if(read == 1)
temp.add(line);
}
line = brin.readLine();
}
brin.close();
Just tried it myself, that'll take in all the data between March and April. You can adjust them as necessary or make them variables. Thanks to ngoa for the foundation code. Credit where credit is due

If you have Java 7, you can use Files.readAllLines(Path path, Charset cs), e.g.
Path path = // Path to "SpreadsheetDatabase2013.txt"
Charset charset = // "UTF-8" or whatever charset is used
List<String> allLines = Files.readAllLines(path, charset);
List<String> relevantLines = allLines.subList(x, y);
Where x (inclusive) and y (exclusive) indicates the line numbers that are of interest, see List.subList(int fromIndex, int toIndex).
One benefit of this solution, as stated in the JavaDoc of readAllLines():
This method ensures that the file is closed when all bytes have been read or an I/O error, or other runtime exception, is thrown.

Java : Read last n lines of a HUGE file

I want to read the last n lines of a very big file without reading the whole file into any buffer/memory area using Java.
I looked around the JDK APIs and Apache Commons I/O and am not able to locate one which is suitable for this purpose.
I was thinking of the way tail or less does it in UNIX. I don't think they load the entire file and then show the last few lines of the file. There should be similar way to do the same in Java too.

I found it the simplest way to do by using ReversedLinesFileReader from apache commons-io api.
This method will give you the line from bottom to top of a file and you can specify n_lines value to specify the number of line.
import org.apache.commons.io.input.ReversedLinesFileReader;
File file = new File("D:\\file_name.xml");
int n_lines = 10;
int counter = 0;
ReversedLinesFileReader object = new ReversedLinesFileReader(file);
while(counter < n_lines) {
System.out.println(object.readLine());
counter++;
}

If you use a RandomAccessFile, you can use length and seek to get to a specific point near the end of the file and then read forward from there.
If you find there weren't enough lines, back up from that point and try again. Once you've figured out where the Nth last line begins, you can seek to there and just read-and-print.
An initial best-guess assumption can be made based on your data properties. For example, if it's a text file, it's possible the line lengths won't exceed an average of 132 so, to get the last five lines, start 660 characters before the end. Then, if you were wrong, try again at 1320 (you can even use what you learned from the last 660 characters to adjust that - example: if those 660 characters were just three lines, the next try could be 660 / 3 * 5, plus maybe a bit extra just in case).

RandomAccessFile is a good place to start, as described by the other answers. There is one important caveat though.
If your file is not encoded with an one-byte-per-character encoding, the readLine() method is not going to work for you. And readUTF() won't work in any circumstances. (It reads a string preceded by a character count ...)
Instead, you will need to make sure that you look for end-of-line markers in a way that respects the encoding's character boundaries. For fixed length encodings (e.g. flavors of UTF-16 or UTF-32) you need to extract characters starting from byte positions that are divisible by the character size in bytes. For variable length encodings (e.g. UTF-8), you need to search for a byte that must be the first byte of a character.
In the case of UTF-8, the first byte of a character will be 0xxxxxxx or 110xxxxx or 1110xxxx or 11110xxx. Anything else is either a second / third byte, or an illegal UTF-8 sequence. See The Unicode Standard, Version 5.2, Chapter 3.9, Table 3-7. This means, as the comment discussion points out, that any 0x0A and 0x0D bytes in a properly encoded UTF-8 stream will represent a LF or CR character. Thus, simply counting the 0x0A and 0x0D bytes is a valid implementation strategy (for UTF-8) if we can assume that the other kinds of Unicode line separator (0x2028, 0x2029 and 0x0085) are not used. You can't assume that, then the code would be more complicated.
Having identified a proper character boundary, you can then just call new String(...) passing the byte array, offset, count and encoding, and then repeatedly call String.lastIndexOf(...) to count end-of-lines.

The ReversedLinesFileReader can be found in the Apache Commons IO java library.
int n_lines = 1000;
ReversedLinesFileReader object = new ReversedLinesFileReader(new File(path));
String result="";
for(int i=0;i<n_lines;i++){
String line=object.readLine();
if(line==null)
break;
result+=line;
}
return result;

I found RandomAccessFile and other Buffer Reader classes too slow for me. Nothing can be faster than a tail -<#lines>. So this it was the best solution for me.
public String getLastNLogLines(File file, int nLines) {
StringBuilder s = new StringBuilder();
try {
Process p = Runtime.getRuntime().exec("tail -"+nLines+" "+file);
java.io.BufferedReader input = new java.io.BufferedReader(new java.io.InputStreamReader(p.getInputStream()));
String line = null;
//Here we first read the next line into the variable
//line and then check for the EOF condition, which
//is the return value of null
while((line = input.readLine()) != null){
s.append(line+'\n');
}
} catch (java.io.IOException e) {
e.printStackTrace();
}
return s.toString();
}

CircularFifoBuffer from apache commons . answer from a similar question at How to read last 5 lines of a .txt file into java
Note that in Apache Commons Collections 4 this class seems to have been renamed to CircularFifoQueue

package com.uday;
import java.io.File;
import java.io.RandomAccessFile;
public class TailN {
public static void main(String[] args) throws Exception {
long startTime = System.currentTimeMillis();
TailN tailN = new TailN();
File file = new File("/Users/udakkuma/Documents/workspace/uday_cancel_feature/TestOOPS/src/file.txt");
tailN.readFromLast(file);
System.out.println("Execution Time : " + (System.currentTimeMillis() - startTime));
}
public void readFromLast(File file) throws Exception {
int lines = 3;
int readLines = 0;
StringBuilder builder = new StringBuilder();
try (RandomAccessFile randomAccessFile = new RandomAccessFile(file, "r")) {
long fileLength = file.length() - 1;
// Set the pointer at the last of the file
randomAccessFile.seek(fileLength);
for (long pointer = fileLength; pointer >= 0; pointer--) {
randomAccessFile.seek(pointer);
char c;
// read from the last, one char at the time
c = (char) randomAccessFile.read();
// break when end of the line
if (c == '\n') {
readLines++;
if (readLines == lines)
break;
}
builder.append(c);
fileLength = fileLength - pointer;
}
// Since line is read from the last so it is in reverse order. Use reverse
// method to make it correct order
builder.reverse();
System.out.println(builder.toString());
}
}
}

A RandomAccessFile allows for seeking (http://download.oracle.com/javase/1.4.2/docs/api/java/io/RandomAccessFile.html). The File.length method will return the size of the file. The problem is determining number of lines. For this, you can seek to the end of the file and read backwards until you have hit the right number of lines.

I had similar problem, but I don't understood to another solutions.
I used this. I hope thats simple code.
// String filePathName = (direction and file name).
File f = new File(filePathName);
long fileLength = f.length(); // Take size of file [bites].
long fileLength_toRead = 0;
if (fileLength > 2000) {
// My file content is a table, I know one row has about e.g. 100 bites / characters.
// I used 1000 bites before file end to point where start read.
// If you don't know line length, use #paxdiablo advice.
fileLength_toRead = fileLength - 1000;
}
try (RandomAccessFile raf = new RandomAccessFile(filePathName, "r")) { // This row manage open and close file.
raf.seek(fileLength_toRead); // File will begin read at this bite.
String rowInFile = raf.readLine(); // First readed line usualy is not whole, I needn't it.
rowInFile = raf.readLine();
while (rowInFile != null) {
// Here I can readed lines (rowInFile) add to String[] array or ArriyList<String>.
// Later I can work with rows from array - last row is sometimes empty, etc.
rowInFile = raf.readLine();
}
}
catch (IOException e) {
//
}

Here is the working for this.
private static void printLastNLines(String filePath, int n) {
File file = new File(filePath);
StringBuilder builder = new StringBuilder();
try {
RandomAccessFile randomAccessFile = new RandomAccessFile(filePath, "r");
long pos = file.length() - 1;
randomAccessFile.seek(pos);
for (long i = pos - 1; i >= 0; i--) {
randomAccessFile.seek(i);
char c = (char) randomAccessFile.read();
if (c == '\n') {
n--;
if (n == 0) {
break;
}
}
builder.append(c);
}
builder.reverse();
System.out.println(builder.toString());
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}

Here is the best way I've found to do it. Simple and pretty fast and memory efficient.
public static void tail(File src, OutputStream out, int maxLines) throws FileNotFoundException, IOException {
BufferedReader reader = new BufferedReader(new FileReader(src));
String[] lines = new String[maxLines];
int lastNdx = 0;
for (String line=reader.readLine(); line != null; line=reader.readLine()) {
if (lastNdx == lines.length) {
lastNdx = 0;
}
lines[lastNdx++] = line;
}
OutputStreamWriter writer = new OutputStreamWriter(out);
for (int ndx=lastNdx; ndx != lastNdx-1; ndx++) {
if (ndx == lines.length) {
ndx = 0;
}
writer.write(lines[ndx]);
writer.write("\n");
}
writer.flush();
}

(See commend)
public String readFromLast(File file, int howMany) throws IOException {
int numLinesRead = 0;
StringBuilder builder = new StringBuilder();
try (RandomAccessFile randomAccessFile = new RandomAccessFile(file, "r")) {
try (ByteArrayOutputStream baos = new ByteArrayOutputStream()) {
long fileLength = file.length() - 1;
/*
* Set the pointer at the end of the file. If the file is empty, an IOException
* will be thrown
*/
randomAccessFile.seek(fileLength);
for (long pointer = fileLength; pointer >= 0; pointer--) {
randomAccessFile.seek(pointer);
byte b = (byte) randomAccessFile.read();
if (b == '\n') {
numLinesRead++;
// (Last line often terminated with a line separator)
if (numLinesRead == (howMany + 1))
break;
}
baos.write(b);
fileLength = fileLength - pointer;
}
/*
* Since line is read from the last so it is in reverse order. Use reverse
* method to make it ordered correctly
*/
byte[] a = baos.toByteArray();
int start = 0;
int mid = a.length / 2;
int end = a.length - 1;
while (start < mid) {
byte temp = a[end];
a[end] = a[start];
a[start] = temp;
start++;
end--;
}// End while
return new String(a).trim();
} // End inner try-with-resources
} // End outer try-with-resources
} // End method

I tried RandomAccessFile first and it was tedious to read the file backwards, repositioning the file pointer upon every read operation. So, I tried #Luca solution and I got the last few lines of the file as a string in just two lines in a few minutes.
InputStream inputStream = Runtime.getRuntime().exec("tail " + path.toFile()).getInputStream();
String tail = new BufferedReader(new InputStreamReader(inputStream)).lines().collect(Collectors.joining(System.lineSeparator()));

Code is 2 lines only
// Please specify correct Charset
ReversedLinesFileReader rlf = new ReversedLinesFileReader(file, StandardCharsets.UTF_8);
// read last 2 lines
System.out.println(rlf.toString(2));
Gradle:
implementation group: 'commons-io', name: 'commons-io', version: '2.11.0'
Maven:
<dependency>
<groupId>commons-io</groupId><artifactId>commons-io</artifactId><version>2.11.0</version>
</dependency>

Modify a .txt file in Java

I have a text file that I want to edit using Java. It has many thousands of lines. I basically want to iterate through the lines and change/edit/delete some text. This will need to happen quite often.
From the solutions I saw on other sites, the general approach seems to be:
Open the existing file using a BufferedReader
Read each line, make modifications to each line, and add it to a StringBuilder
Once all the text has been read and modified, write the contents of the StringBuilder to a new file
Replace the old file with the new file
This solution seems slightly "hacky" to me, especially if I have thousands of lines in my text file.
Anybody know of a better solution?

I haven't done this in Java recently, but writing an entire file into memory seems like a bad idea.
The best idea that I can come up with is open a temporary file in writing mode at the same time, and for each line, read it, modify if necessary, then write into the temporary file. At the end, delete the original and rename the temporary file.
If you have modify permissions on the file system, you probably also have deleting and renaming permissions.

if the file is just a few thousand lines you should be able to read the entire file in one read and convert that to a String.
You can use apache IOUtils which has method like the following.
public static String readFile(String filename) throws IOException {
File file = new File(filename);
int len = (int) file.length();
byte[] bytes = new byte[len];
FileInputStream fis = null;
try {
fis = new FileInputStream(file);
assert len == fis.read(bytes);
} catch (IOException e) {
close(fis);
throw e;
}
return new String(bytes, "UTF-8");
}
public static void writeFile(String filename, String text) throws IOException {
FileOutputStream fos = null;
try {
fos = new FileOutputStream(filename);
fos.write(text.getBytes("UTF-8"));
} catch (IOException e) {
close(fos);
throw e;
}
}
public static void close(Closeable closeable) {
try {
closeable.close();
} catch(IOException ignored) {
}
}

You can use RandomAccessFile in Java to modify the file on one condition:
The size of each line has to be fixed otherwise, when new string is written back, it might override the string in the next line.
Therefore, in my example, I set the line length as 100 and padding with space string when creating the file and writing back to the file.
So in order to allow update, you need to set the length of line a little larger than the longest length of the line in this file.
public class RandomAccessFileUtil {
public static final long RECORD_LENGTH = 100;
public static final String EMPTY_STRING = " ";
public static final String CRLF = "\n";
public static final String PATHNAME = "/home/mjiang/JM/mahtew.txt";
/**
* one two three
Text to be appended with
five six seven
eight nine ten
*
*
* #param args
* #throws IOException
*/
public static void main(String[] args) throws IOException
{
String starPrefix = "Text to be appended with";
String replacedString = "new text has been appended";
RandomAccessFile file = new RandomAccessFile(new File(PATHNAME), "rw");
String line = "";
while((line = file.readLine()) != null)
{
if(line.startsWith(starPrefix))
{
file.seek(file.getFilePointer() - RECORD_LENGTH - 1);
file.writeBytes(replacedString);
}
}
}
public static void createFile() throws IOException
{
RandomAccessFile file = new RandomAccessFile(new File(PATHNAME), "rw");
String line1 = "one two three";
String line2 = "Text to be appended with";
String line3 = "five six seven";
String line4 = "eight nine ten";
file.writeBytes(paddingRight(line1));
file.writeBytes(CRLF);
file.writeBytes(paddingRight(line2));
file.writeBytes(CRLF);
file.writeBytes(paddingRight(line3));
file.writeBytes(CRLF);
file.writeBytes(paddingRight(line4));
file.writeBytes(CRLF);
file.close();
System.out.println(String.format("File is created in [%s]", PATHNAME));
}
public static String paddingRight(String source)
{
StringBuilder result = new StringBuilder(100);
if(source != null)
{
result.append(source);
for (int i = 0; i < RECORD_LENGTH - source.length(); i++)
{
result.append(EMPTY_STRING);
}
}
return result.toString();
}
}

If the file is large, you might want to use a FileStream for output, but that seems pretty much like it is the simplest process to do what you're asking (and without more specificity i.e. on what types of changes / edits / deletions you're trying to do, it's impossible to determine what more complicated way might work).

No reason to buffer the entire file.
Simply write each line as your read it, insert lines when necessary, delete lines when necessary, replace lines when necessary.
Fundamentally, you will not get around having to recreate the file wholesale, especially if it's just a text file.

What kind of data is it? Do you control the format of the file?
If the file contains name/value pairs (or similar), you could have some luck with Properties, or perhaps cobbling together something using a flat file JDBC driver.
Alternatively, have you considered not writing the data so often? Operating on an in-memory copy of your file should be relatively trivial. If there are no external resources which need real time updates of the file, then there is no need to go to disk every time you want to make a modification. You can run a scheduled task to write periodic updates to disk if you are worried about data backup.

In general you cannot edit the file in place; it's simply a very long sequence of characters, which happens to include newline characters. You could edit in place if your changes don't change the number of characters in each line.

Can't you use regular expressions, if you know what you want to change ? Jakarta Regexp should probably do the trick.

Although this question was a time ago posted, I think it is good to put my answer here.
I think that the best approach is to use FileChannel from java.nio.channels package in this scenario. But this, only if you need to have a good performance! You would need to get a FileChannel via a RandomAccessFile, like this:
java.nio.channels.FileChannel channel = new java.io.RandomAccessFile("/my/fyle/path", "rw").getChannel();
After this, you need a to create a ByteBuffer where you will read from the FileChannel.
this looks something like this:
java.nio.ByteBuffer inBuffer = java.nio.ByteBuffer.allocate(100);
int pos = 0;
int aux = 0;
StringBuilder sb = new StringBuilder();
while (pos != -1) {
aux = channel.read(inBuffer, pos);
pos = (aux != -1) ? pos + aux : -1;
b = inBuffer.array();
sb.delete(0, sb.length());
for (int i = 0; i < b.length; ++i) {
sb.append((char)b[i]);
}
//here you can do your stuff on sb
inBuffer = ByteBuffer.allocate(100);
}
Hope that my answer will help you!

I think, FileOutputStream.getFileChannel() will help a lot, see FileChannel api
http://java.sun.com/javase/6/docs/api/java/nio/channels/FileChannel.html

private static void modifyFile(String filePath, String oldString, String newString) {
File fileToBeModified = new File(filePath);
StringBuilder oldContent = new StringBuilder();
try (BufferedReader reader = new BufferedReader(new FileReader(fileToBeModified))) {
String line = reader.readLine();
while (line != null) {
oldContent.append(line).append(System.lineSeparator());
line = reader.readLine();
}
String content = oldContent.toString();
String newContent = content.replaceAll(oldString, newString);
try (FileWriter writer = new FileWriter(fileToBeModified)) {
writer.write(newContent);
}
} catch (IOException e) {
e.printStackTrace();
}
}

You can change the txt file to java by saving on clicking "Save As" and saving *.java extension.