my problem is to read non primes from txt file and write prime factors in same file.
i actually dont know how BufferedReader works.from my understanding i am trying to read the file data to buffer(8kb) and write prime factors to file.(by creating a new one)
class PS_Task2
{
public static void main(String[] args)
{
String line=null;
int x;
try
{
FileReader file2 = new FileReader("nonprimes.txt");
BufferedReader buff2=new BufferedReader(file2);
File file1 = new File("nonprimes.txt");
file1.createNewFile();
PrintWriter d=new PrintWriter(file1);
while((line = buff2.readLine()) != null)
{
x=Integer.parseInt(line);
d.printf ("%d--> ", x);
while(x%2==0)
{
d.flush();
d.print("2"+"*");
x=x/2;
}
for (int i = 3; i <= Math.sqrt(x); i = i+2)
{
while (x%i == 0)
{
d.flush();
d.printf("%d*", i);
x = x/i;
}
}
if (x > 2)
{
d.flush();
d.printf ("%d ", x);
}
d.flush();//FLUSING THE STREAM TO FILE
d.println("\n");
}
d.close(); // CLOSING FILE
}
feel free to give detailed explanation. :D thanks ~anirudh
reading and writing to a file in java doesnt EDIT the file, but clear the old one and creates a new one, you can use many approachesfor example, to get your data, modify it, and either save it on memory in a StringBuilder or a collection or what ever and re-write it again
well i created fileOne.txt containing the following data :
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
and i want to multiply all those numbers by 10, then re-write them again :
public static void main(String [] args) throws Exception{ // just for the example
// locate the file
File fileOne = new File("fileOne.txt");
FileReader inputStream = new FileReader(fileOne);
BufferedReader reader = new BufferedReader(inputStream);
// create a LinkedList to hold the data read
List<Integer> numbers = new LinkedList<Integer>();
// prepare variables to refer to the temporary objects
String line = null;
int number = 0;
// start reading
do{
// read each line
line = reader.readLine();
// check if the read data is not null, so not to use null values
if(line != null){
number = Integer.parseInt(line);
numbers.add(number*10);
}
}while(line != null);
// free resources
reader.close();
// check the new numbers before writing to file
System.out.println("NEW NUMBERS IN MEMORY : "+numbers);
// assign a printer
PrintWriter writer = new PrintWriter(fileOne);
// write down data
for(int newNumber : numbers){
writer.println(newNumber);
}
// free resources
writer.flush();
writer.close();
}
this approach is not very good when dealing with massive data
As per your problem statement, you need to take input from a file, do some processing and write back the processed data in the same file. For this, please note the below points:
You may not create a file with same name in a directory, so you must create the new file at some other location; or write the content into different file and later rename it after deleting original one.
While your file is open for reading, modifying the same file is not a good idea. you could use below approach:
Read the content of the file and store in a data structure liek Arrays, ArrayList.
Close the file.
Process the data stored in the data structure.
Open the file in write mode (over-write mode rather than append mode)
Write back the processed data into the file.
Related
This question already has answers here:
How do I sort very large files
(10 answers)
Closed 1 year ago.
I'm having a problem working with a 1.3 Gb CSV file (it contains 3 million rows). The problem is that I want to sort the file according to a field called "Timestamp" and I can't split the file into multiple reads because otherwise the sorting won't work properly. I get the following error at one point :
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
This is my code:
public class createCSV {
public static BufferedReader br = null;
public static String csvFile = "/Scrivania/dataset";
public static String newcsvFile = "/Scrivania/ordinatedataset";
public static String extFile = ".csv";
public static void main(String[] args) {
try {
List<List<String>> csvLines = new ArrayList<>();
br = new BufferedReader(new FileReader(csvFile+extFile));
CSVWriter writer = new CSVWriter(new FileWriter(newcsvFile+extFile));
String line = br.readLine();
String[] fields = line.split(",");
writer.writeNext(fields);
line = br.readLine();
while(line!=null) {
csvLines.add(Arrays.asList(line.split(",")));
line = br.readLine();
}
csvLines.sort(new Comparator<List<String>>() {
#Override
public int compare(List<String> o1, List<String> o2) {
return o1.get(8).compareTo(o2.get(8));
}
});
for(List<String>lin:csvLines){
writer.writeNext(lin.toArray(new String[0]));
}
writer.close();
}catch(IOException e) {
e.printStackTrace();
}
}
}
I have tried increasing the heap size to the maximum, 2048, in particular: -Xms512M -Xmx2048M in Run->Run Configuratins but it still gives me an error. How could I solve and sort the whole file? Thank you in advance
The approach of reading file with FileReader will keep data of file in-memory this leads to exhaustion of memory. What you need is streaming through file. You can achieve this with Scanner class of Apache commons library.
With Scanner:
List<List<String>> csvLines = new ArrayList<>();
FileInputStream inputStream = null;
Scanner sc = null;
try {
inputStream = new FileInputStream(path);
sc = new Scanner(inputStream, "UTF-8");
while (sc.hasNextLine()) {
String line = sc.nextLine();
csvLines.add(Arrays.asList(line.split(",")));
}
// note that Scanner suppresses exceptions
if (sc.ioException() != null) {
throw sc.ioException();
}
} finally {
if (inputStream != null) {
inputStream.close();
}
if (sc != null) {
sc.close();
}
}
Apache Commons:
LineIterator it = FileUtils.lineIterator(theFile, "UTF-8");
try {
while (it.hasNext()) {
String line = it.nextLine();
// do something with line
}
} finally {
LineIterator.closeQuietly(it);
}
Hopefully you can find an existing library that will do this for you, or use a command line tool called from Java to do this instead. If you need to code this yourself, here's a suggestion as to a pretty simple approach you might code up...
There's a simple general approach to sorting a large file like this. I call it a "shard sort". Here's what you do:
Pick a number N that is the number of shards you'll have and a function that will produce a value between 1 and N for each input entry such that you get roughly the same number of entries in each shard. For example, you could choose N to be 10, and you could use the seconds part of your timestamp and have the shard id be id = seconds % 10. This should "randomly" spread your entries across the 10 shards.
Now open the input file and 10 output files, one for each shard. Read each entry from the input file, compute its shard id, and write it to the file for that shard id.
Now read each shard file into memory, sort it bases on on each entry's timestamp, and write it back out to file. For this example, this will take 10% of the memory needed to sort the whole file.
Now open the 10 shard files for reading and a new result file to contain the final result. Read in the next entry in all 10 input files. Write out the earliest entry timestamp-wise of those 10 entries to the output file. When you write out a value, you read a new one from the shard file it came from. Repeat this process this until all the shard files are empty and all entries in memory have been written.
If your file is so big that 10 shards isn't enough, use more. You could, for example, use 60 shard files and use the entire seconds value from your timestamp for the shard id.
I am looking at the performance of a particular stock over a period of 30 days. I downloaded the data from Yahoo finance which appears as a CSV file. If I would like to create a new column in my dataset to show the daily percentage change between open and close using java refer to column H as where the output should appear, how should I do so?
Thanks in advance!
You can just edit your file line-by-line and add a separator character using the concat() function.
Open the file with a FileReader or BufferedReader and then start parsing.
Official Javadoc: https://docs.oracle.com/javase/7/docs/api/java/lang/String.html#concat(java.lang.String)
Please see also this.
With OpenCSV: http://opencsv.sourceforge.net/
public class OpenCSVTest {
public static void main(String[] args) {
StringWriter target = new StringWriter();
try(CSVReader reader = new CSVReader(new StringReader("my,test"));
CSVWriter writer = new CSVWriter(target);) {
for(String[] line : reader) {
String[] added = Arrays.copyOf(line, line.length + 1);
added[added.length-1] = "addition";
writer.writeNext(added);
}
} catch (IOException e) {
e.printStackTrace();
}
System.out.println(target.toString());
}
}
You can replace the StringReader and StringWriter with your input and output. The reader will then iterate through each line in your csv for you. You can make a copy of the original line and add your new column and write it out into the target.
Breakdown of the code:
try(CSVReader reader = new CSVReader(new StringReader("my,test"));
CSVWriter writer = new CSVWriter(target);) { ... }
This is a try-with-resources block. It makes sure that the reader and writer are closed after they are done.
for(String[] line : reader) {
String[] added = Arrays.copyOf(line, line.length + 1);
added[added.length-1] = "addition";
writer.writeNext(added);
}
This is a standard for loop. The CSVReader implements the Iterable interface, which enables us to use it in the for loop directly.
Arrays.copyOf(line, line.length + 1); is a function that creates a copy of the array passed to it, but with a new size. Because you want to add a column, we make a copy of the original array my,test and add 1 more space at the end of it, where we can then assign the new value to it by doing: added[added.length-1] = "addition";
Finally, we just pass that to the writer which will then correctly write the values into the target, in this case a StringWriter, in your case likely a file.
The following code reads a bunch of .csv files and then combines them into one .csv file. I tried to system.out.println ... all datapoints are correct, however when i try to use the PrintWriter I get:
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space.
I tried to use FileWriter but got the same error. How should I correct my code?
public class CombineCsv {
public static void main(String[] args) throws IOException {
PrintWriter output = new PrintWriter("C:\\User\\result.csv");
final File file = new File("C:\\Users\\is");
int i = 0;
for (final File child: file.listFiles()) {
BufferedReader CSVFile = new BufferedReader( new FileReader( "C:\\Users\\is\\"+child.getName()));
String dataRow = CSVFile.readLine();
while (dataRow != null) {
String[] dataArray = dataRow.split(",");
for (String item:dataArray) {
System.out.println(item + "\t");
output.append(item+","+child.getName().replaceAll(".csv", "")+",");
i++;
}
dataRow = CSVFile.readLine(); // Read next line of data.
} // Close the file once all data has been read.
CSVFile.close();
}
output.close();
System.out.println(i);
}
}
I can only think of two scenarios in which that code could result in an OOME:
If the file directory has a very large number of elements, then file.listFiles() could create a very large array of File objects.
If one of the input files includes a line that is very long, then CSVFile.readLine() could use a lot of memory in the process of reading it. (Up to 6 times the number of bytes in the line.)
The simplest approach to solving both of these issues is to increase the Java heap size using the -Xmx JVM option.
I can see no reason why your use of a PrintWriter would be the cause of the problem.
Try
boolean autoFlush = true;
PrintWriter output = new PrintWriter(myFileName, autoFlush);
It creates a PrintWriter instance which flushes content everytime when there is a new line or format.
This question is unlikely to help any future visitors; it is only relevant to a small geographic area, a specific moment in time, or an extraordinarily narrow situation that is not generally applicable to the worldwide audience of the internet. For help making this question more broadly applicable, visit the help center.
Closed 9 years ago.
I guess this comes down to reading and writing to the same file. I would like to be able to return the same text file as is input, but with all integer values quadrupled. Should I even be attempting this with Java, or is it better to write to a new file and overwrite the original .txt file?
In essence, I'm trying to transform This:
12
fish
55 10 yellow 3
into this:
48
fish
220 40 yellow 12
Here's what I've got so far. Currently, it doesn't modify the .txt file.
import java.io.*;
import java.util.Scanner;
public class CharacterStretcher
{
public static void main(String[] args)
{
Scanner keyboard = new Scanner( System.in );
System.out.println("Copy and paste the path of the file to fix");
// get which file you want to read and write
File file = new File(keyboard.next());
File file2 = new File("temp.txt");
BufferedReader reader;
BufferedWriter writer;
try {
// new a writer and point the writer to the file
FileInputStream fstream = new FileInputStream(file);
// Use DataInputStream to read binary NOT text.
reader = new BufferedReader(new InputStreamReader(fstream));
writer = new BufferedWriter(new FileWriter(file2, true));
String line = "";
String temp = "";
int var = 0;
int start = 0;
System.out.println("000");
while ((line = reader.readLine()) != null)
{
System.out.println("a");
if(line.contains("="))
{
System.out.println("b");
var = 0;
temp = line.substring(line.indexOf('='));
for(int x = 0; x < temp.length(); x++)
{
System.out.println(temp.charAt(x));
if(temp.charAt(x)>47 && temp.charAt(x)<58) //if 0<=char<=9
{
if(start==0)
start = x;
var*=10;
var+=temp.indexOf(x)-48; //converts back into single digit
}
else
{
if(start!=0)
{
temp = temp.substring(0, start) + var*4 + temp.substring(x);
//writer.write(line.substring(0, line.indexOf('=')) + temp);
//TODO: Currently writes a bunch of garbage to the end of the file, how to write in the middle?
//move x if var*4 has an extra digit
if((var<10 && var>2)
|| (var<100 && var>24)
|| (var<1000 && var>249)
|| (var<10000 && var>2499))
x++;
}
//start = 0;
}
System.out.println(temp + " " + start);
}
if(start==0)
writer.write(line);
else
writer.write(temp);
}
}
System.out.println("end");
// writer the content to the file
//writer.write("I write something to a file.");
// always remember to close the writer
writer.close();
//writer = null;
file2.renameTo(file); //TODO: Not sure if this works...
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
}
Given that this is a pretty quick and simple hack of a formatted text file, I don't think you need to be too clever about it.
Your logic for deciding whether you are looking at a number is pretty complex and I'd say it's overkill.
I've written up a basic outline of what I'd do in this instance.
It's not very clever or impressive, but should get the job done I think.
I've left out the overwriting and reading the input form the console so you get to do some of the implementation yourself ;-)
import java.io.*;
public class CharacterStretcher {
public static void main(String[] args) {
//Assumes the input is at c:\data.txt
File inputFile = new File("c:\\data.txt");
//Assumes the output is at c:\temp.txt
File outputFile = new File("c:\\temp.txt");
try {
//Construct a file reader and writer
final FileInputStream fstream = new FileInputStream(inputFile);
final BufferedReader reader = new BufferedReader(new InputStreamReader(fstream));
final BufferedWriter writer = new BufferedWriter(new FileWriter(outputFile, false));
//Read the file line by line...
String line;
while ((line = reader.readLine()) != null) {
//Create a StringBuilder to build our modified lines that will
//go into the output file
StringBuilder newLine = new StringBuilder();
//Split each line from the input file by spaces
String[] parts = line.split(" ");
//For each part of the input line, check if it's a number
for (String part : parts) {
try {
//If we can parse the part as an integer, we assume
//it's a number because it almost certainly is!
int number = Integer.parseInt(part);
//We add this to out new line, but multiply it by 4
newLine.append(String.valueOf(number * 4));
} catch (NumberFormatException nfEx) {
//If we couldn't parse it as an integer, we just add it
//to the new line - it's going to be a String.
newLine.append(part);
}
//Add a space between each part on the new line
newLine.append(" ");
}
//Write the new line to the output file remembering to chop the
//trailing space off the end, and remembering to add the line
//breaks
writer.append(newLine.toString().substring(0, newLine.toString().length() - 1) + "\r\n");
writer.flush();
}
//Close the file handles.
reader.close();
writer.close();
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
}
You may want to consider one of these:
Build the new file in memory, rather than trying to write to the same file you are reading from. You could use StringBuilder for this.
Write to a new file, then overwrite the old file with the new one. This SO Question may help you there.
With both of these, you will be able to see your whole output, separate from the input file.
Additionally, with option (2), you don't have the risk of the operation failing in the middle and giving you a messed up file.
Now, you certainly can modify the file in-place. But it seems like unnecessary complexity for your case, unless you have really huge input files.
At the very least, if you try it this way first, you can narrow down on why the more complicated version is failing.
You cannot read and simultaneously write to the same file, because this would modify the text you currently read. This means, you must first write a modified new file and later rename it to the original one. You probably need to remove the original file before renameing.
For renaming, you can use File.renameTo or see one of the many SO's questions
You seem to parse integers in your code by collecting single digits and adding them up. You should consider using either a Scanner.nextInt or employ Integer.parseInt.
You can read your file line by line, split the words at white space and then parse them and check if it is either an integer or some other word.
I have a text file with data. The file has information from all months. Imagine that the information for January occupy 50 lines. Than February starts and it occupies 40 more lines. Than I have March and so on... Is it possible to read only part of the file? Can I say "read from line X to line Y"? or is there a better way to accomplish this? I only want to print the data correspondent to one month not the all file. Here is my code
public static void readFile()
{
try
{
DataInputStream inputStream =
new DataInputStream(new FileInputStream("SpreadsheetDatabase2013.txt"));
while(inputStream.available() != 0)
{
System.out.println("AVAILABLE: " + inputStream.available());
System.out.println(inputStream.readUTF());
System.out.println(inputStream.readInt());
for (int i = 0; i < 40; i++)
{
System.out.println(inputStream.readUTF());
System.out.println(inputStream.readUTF());
System.out.println(inputStream.readUTF());
System.out.println(inputStream.readUTF());
System.out.println(inputStream.readUTF());
System.out.println(inputStream.readDouble());
System.out.println(inputStream.readUTF());
System.out.println(inputStream.readBoolean());
System.out.println();
}
}// end while
inputStream.close();
}// end try
catch (Exception e)
{
System.out.println("An error has occurred.");
}//end catch
}//end method
Thank you for your time.
My approach to this would be to read the entire contents of the text file and store it in a ArrayList and read only the lines for the requested month.
Example:
Use this function to read the all the lines from the file.
/**
* Read from a file specified by the filePath.
*
* #param filePath
* The path of the file.
* #return List of lines in the file.
* #throws IOException
*/
public static ArrayList<String> readFromFile(String filePath)
throws IOException {
ArrayList<String> temp = new ArrayList<String>();
File file = new File(filePath);
if (file.exists()) {
BufferedReader brin;
brin = new BufferedReader(new FileReader(filePath));
String line = brin.readLine();
while (line != null) {
if (!line.equals(""))
temp.add(line);
line = brin.readLine();
}
brin.close();
}
return temp;
}
Then read only the ones you need from ArrayList temp.
Example:
if you want to read February month's data assuming its 50 lines of data and starts from 40th line.
for(int i=40;i<90;i++)
{
System.out.println(temp.get(i));
}
Note: This is only just one way of doing this. I am not certain if there is any other way!
I would use the scanner class.
Scanner scanner = new Scanner(filename);
Use scanner.nextLine() to get each of the lines of the file. If you only want from line x to line y you can use a for loop to scan each of the lines that you don't need before going through the scanner for the lines you do need. Be careful not to hit an exception without throwing it though.
Or you can go through the scanner and for each line, add the String contents of the line to an ArrayList. Good luck.
Based on how you said your data was organized, I would suggest doing something like this
ArrayList<String> temp = new ArrayList<String>();
int read = 0;
File file = new File(filePath);
if (file.exists()) {
BufferedReader brin;
brin = new BufferedReader(new FileReader(filePath));
String line = brin.readLine();
while (line != null) {
if (!line.equals("")){
if(line.equals("March"))
read = 1;
else if(line.equals("April"))
break;
else if(read == 1)
temp.add(line);
}
line = brin.readLine();
}
brin.close();
Just tried it myself, that'll take in all the data between March and April. You can adjust them as necessary or make them variables. Thanks to ngoa for the foundation code. Credit where credit is due
If you have Java 7, you can use Files.readAllLines(Path path, Charset cs), e.g.
Path path = // Path to "SpreadsheetDatabase2013.txt"
Charset charset = // "UTF-8" or whatever charset is used
List<String> allLines = Files.readAllLines(path, charset);
List<String> relevantLines = allLines.subList(x, y);
Where x (inclusive) and y (exclusive) indicates the line numbers that are of interest, see List.subList(int fromIndex, int toIndex).
One benefit of this solution, as stated in the JavaDoc of readAllLines():
This method ensures that the file is closed when all bytes have been read or an I/O error, or other runtime exception, is thrown.