I am using below piece of code for reading the 150MB CSV file and getting GC error
Same code which was causing the problem
public List<String[]> readCsvFile(String ipFilePath) {
logger.info("Start executing readCsvFile method !!! on file " + ipFilePath);
CSVReader csvReader = null;
List<String[]> allRecrods = null;
Reader reader = null;
try {
reader = Files.newBufferedReader(Paths.get(ipFilePath));
csvReader = new CSVReader(reader);
allRecrods = csvReader.readAll();
} catch (Exception e) {
logger.error("Error in CsvFileReader !!!");
e.printStackTrace();
logger.error("Exception : ", e);
} finally {
try {
reader.close();
csvReader.close();
} catch (IOException e) {
logger.error("Error while closing fileReader/csvFileParser !!!");
e.printStackTrace();
logger.error("IOException : ", e);
}
}
return allRecrods;
}
I am getting error on the method : csvReader.readAll() as mentioned above.
I am not sure what is the problem which the code, and how to solve this, as the same code is working fine with 20-30 MB files.
The simplest solution is to increase heap size with flag "-Xmx" for example:
"-Xmx1024m". First you should use some heap size monitoring tool to see if the usage is expected.
You should not read all the lines but instead process the file line by line and size won't matter and is way more memory efficient.
Example from here: http://www.baeldung.com/java-read-lines-large-file
FileInputStream inputStream = null;
Scanner sc = null;
try {
inputStream = new FileInputStream(path);
sc = new Scanner(inputStream, "UTF-8");
while (sc.hasNextLine()) {
String line = sc.nextLine();
// System.out.println(line);
}
// note that Scanner suppresses exceptions
if (sc.ioException() != null) {
throw sc.ioException();
}
} finally {
if (inputStream != null) {
inputStream.close();
}
if (sc != null) {
sc.close();
}
}
Edit: Also if you are looking for a framework I can recommend Spring Batch https://projects.spring.io/spring-batch/
Related
Here's what I need to happen: I've got a text file with some values in them that I need to read (Let's say it is example.txt). I need to read that file like i would using ex. FileInputStream or BufferedReader). How would I go about doing it?
PS - This is what I tried doing, but it didn't help. I would always get an error saying the file has to be .xml
try {
BufferedReader bufferReader = new BufferedReader(new FileReader(file);
try {
String line = bufferReader.readLine();
while(line != null) {
//do something
}
} catch (IOException e) {
e.printStackTrace();
}
} catch (FileNotFoundException e) {
e.printStackTrace();
}
I have a problem on my code; basically I have an array containing some key:
String[] ComputerScience = { "A", "B", "C", "D" };
And so on, containing 40 entries.
My code reads 900 pdf from 40 folder corresponding to each element of ComputerScience, manipulates the extracted text and stores the output in a file named A.txt , B.txt, ecc ...
Each folder "A", "B", ecc contains 900 pdf.
After a lot of documents, an exception "Too many open files" is thrown.
I'm supposing that I am correctly closing files handler.
static boolean writeOccurencesFile(String WORDLIST,String categoria, TreeMap<String,Integer> map) {
File dizionario = new File(WORDLIST);
FileReader fileReader = null;
FileWriter fileWriter = null;
try {
File cat_out = new File("files/" + categoria + ".txt");
fileWriter = new FileWriter(cat_out, true);
} catch (IOException e) {
e.printStackTrace();
}
try {
fileReader = new FileReader(dizionario);
} catch (FileNotFoundException e) { }
try {
BufferedReader bufferedReader = new BufferedReader(fileReader);
if (dizionario.exists()) {
StringBuffer stringBuffer = new StringBuffer();
String parola;
StringBuffer line = new StringBuffer();
int contatore_index_parola = 1;
while ((parola = bufferedReader.readLine()) != null) {
if (map.containsKey(parola) && !parola.isEmpty()) {
line.append(contatore_index_parola + ":" + map.get(parola).intValue() + " ");
map.remove(parola);
}
contatore_index_parola++;
}
if (! line.toString().isEmpty()) {
fileWriter.append(getCategoryID(categoria) + " " + line + "\n"); // print riga completa documento N x1:y x2:a ...
}
} else { System.err.println("Dictionary file not found."); }
bufferedReader.close();
fileReader.close();
fileWriter.close();
} catch (IOException e) { return false;}
catch (NullPointerException ex ) { return false;}
finally {
try {
fileReader.close();
fileWriter.close();
} catch (IOException e) {
e.printStackTrace();
}
}
return true;
}
But the error still comes. ( it is thrown at:)
try {
File cat_out = new File("files/" + categoria + ".txt");
fileWriter = new FileWriter(cat_out, true);
} catch (IOException e) {
e.printStackTrace();
}
Thank you.
EDIT: SOLVED
I found the solution, there was, in the main function in which writeOccurencesFile is called, another function that create a RandomAccessFile and doesn't close it.
The debugger sais that Exception has thrown in writeOccurencesFile but using Java Leak Detector i found out that the pdf were already opened and not close after parsing to pure text.
Thank you!
Try using this utility specifically designed for the purpose.
This Java agent is a utility that keeps track of where/when/who opened files in your JVM. You can have the agent trace these operations to find out about the access pattern or handle leaks, and dump the list of currently open files and where/when/who opened them.
When the exception occurs, this agent will dump the list, allowing you to find out where a large number of file descriptors are in use.
i have tried using try-with resources; but the problem remains.
Also running in system macos built-in console print out a FileNotFound exception at the line of FileWriter fileWriter = ...
static boolean writeOccurencesFile(String WORDLIST,String categoria, TreeMap<String,Integer> map) {
File dizionario = new File(WORDLIST);
try (FileWriter fileWriter = new FileWriter( "files/" + categoria + ".txt" , true)) {
try (FileReader fileReader = new FileReader(dizionario)) {
try (BufferedReader bufferedReader = new BufferedReader(fileReader)) {
if (dizionario.exists()) {
StringBuffer stringBuffer = new StringBuffer();
String parola;
StringBuffer line = new StringBuffer();
int contatore_index_parola = 1;
while ((parola = bufferedReader.readLine()) != null) {
if (map.containsKey(parola) && !parola.isEmpty()) {
line.append(contatore_index_parola + ":" + map.get(parola).intValue() + " ");
map.remove(parola);
}
contatore_index_parola++;
}
if (!line.toString().isEmpty()) {
fileWriter.append(getCategoryID(categoria) + " " + line + "\n"); // print riga completa documento N x1:y x2:a ...
}
} else {
System.err.println("Dictionary file not found.");
}
}
}
} catch (IOException e) {
e.printStackTrace();
}
return true;
}
This is the code that i am using now, although the bad managing of Exception, why the files seem to be not closed?
Now i am making a test with File Leak Detector
Maybe your code raises another exception that you are not handling. Try add catch (Exception e) before finally block
You also can move BufferedReader declaration out the try and close it in finally
I want to learn my phone cpu model name and I tryed to use /proc/cpuinfo and a lot of code but I failed. Can anyone help me?
Run
$ adb shell cat /proc/cpuinfo
Here is my code
public static String getCpuName() {
try {
FileReader fr = new FileReader("/proc/cpuinfo");
BufferedReader br = new BufferedReader(fr);
String text = br.readLine();
br.close();
String[] array = text.split(":\\s+", 2);
if (array.length >= 2) {
return array[1];
}
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
return null;
}
What about your code?
[Update]Problem is different, nothing about csv file format. Question
would be "While using File Writer to write a csv file last few records
are missing.
In my java application i need to append more than 65535 rows in a csv file. but it only writes 65535 rows in a sheet. I haven't used any libraries. some final records missing. how to resolve this..........
public void writeSubmission(){
try {
writer = new FileWriter("res/sample.csv");
writer.append("PhraseId");
writer.append(',');
writer.append("Sentiment");
writer.append('\n');
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
public void readTestData(){
String path="res/test.tsv";
Calculation cal=new Calculation();
int counter=0;
try {
BufferedReader bReader = new BufferedReader(new FileReader(path));
String line;
writeSubmission();
bReader.readLine();
while ((line = bReader.readLine()) != null) {
String datavalue[] = line.split("\t");
writer.append(datavalue[0]);
writer.append(',');
try {
double value=cal.calculate(datavalue[2]);
System.out.println(value);
String val;
if(value<-0.4)
{
val="0";
}
else if(value>-0.4 && value<-0.1)
{
val="1";
}
else if(value>-0.1 && value<+0.1)
{
val="2";
}
else if(value>+0.1 && value<+0.40)
{
val="3";
}
else{
val="4";
}
counter++;
writer.append(val);
} catch (Exception e) {
// TODO Auto-generated catch block
System.out.println(e);
}
writer.append('\n');
}
System.out.println(counter);
bReader.close();
} catch (FileNotFoundException e) {
e.printStackTrace();
System.out.println(e);
} catch (IOException e){
e.printStackTrace();
System.out.println(e);
}
}
I think your issue is probably that the tool you're opening up the CSVs with on the other end doesn't want more than 65535 rows, not that Java's doing anything wrong. It's a bug on the other end, not your Java code, almost certainly. (FileWriter wouldn't care at all about 65535 lines, for example.)
If you're using Excel 2003, for example, you'd see this issue: How to get around 64k row limit in Excel
Finally i fixed the error.File Writer should be flush after used in the code push existing stream. OR should be close the writer it automatically flush and close
The flush method flushes the output stream and forces any buffered
output bytes to be written out. The general contract of flush is that
calling it is an indication that, if any bytes previously written have
been buffered by the implementation of the output stream, such bytes
should immediately be written to their intended destination.
writer = new FileWriter("res/sample.csv");
writer.flush();
writer.close();
I'm currently trying to store json data I get on the Internet in the internal storage of the phone. In order to do so, I have get the data from an API as a string (no issue in this part, nor during the parsing afterwards). Then comes the part where I try to store it in a file... And the problems that come along!
The ultimate goal is to update data when an Internet connection is available, and use the data previously stored when no connection can be found.
Here is my code (data is in the String jsonString):
// IN CASE OF INTERNET CONNECTION (ie "jsonString != null")
if (jsonString != null) {
try {
FileOutputStream fos = new FileOutputStream(json_InternalFile);
fos.write(jsonString.getBytes());
fos.close();
} catch (FileNotFoundException e) {
Log.e("Response: ", "> " + "File not found");
e.printStackTrace();
} catch (IOException e) {
Log.e("Response: ", "> " + "IO Exception");
e.printStackTrace();
}
}
// IF NO CONNECTION AVAILABLE (ie "jsonString == null)
if (jsonString == null) {
try {
FileInputStream fis = new FileInputStream(json_InternalFile);
DataInputStream dis = new DataInputStream(fis);
BufferedReader br = new BufferedReader(new InputStreamReader(dis));
String strLine;
while ((strLine = br.readLine()) != null) {
jsonString = jsonString + strLine;
}
dis.close();
} catch (FileNotFoundException e) {
Log.e("Response: ", "> " + "File not found");
e.printStackTrace();
} catch (IOException e) {
Log.e("Response: ", "> " + "File not found");
e.printStackTrace();
}
}
None of the parts works individually. When I run the app, I got this message: "Unfortunately, the app "APP_NAME" stopped running" (whichever part I comment)
I also tried to replace the line:
FileOutputStream fos = new FileOutputStream(json_InternalFile);
with:
FileOutputStream fos = openFileOutput(jsonStorage_FileName, 0);
... Without success ^^'
I used - and abused ;) - the following code I found:
http://www.mysamplecode.com/2012/06/android-internal-external-storage.html
Thank you in advance, I really can't see the problem here (and it's turning me slightly mad ><)