I have a problem on my code; basically I have an array containing some key:
String[] ComputerScience = { "A", "B", "C", "D" };
And so on, containing 40 entries.
My code reads 900 pdf from 40 folder corresponding to each element of ComputerScience, manipulates the extracted text and stores the output in a file named A.txt , B.txt, ecc ...
Each folder "A", "B", ecc contains 900 pdf.
After a lot of documents, an exception "Too many open files" is thrown.
I'm supposing that I am correctly closing files handler.
static boolean writeOccurencesFile(String WORDLIST,String categoria, TreeMap<String,Integer> map) {
File dizionario = new File(WORDLIST);
FileReader fileReader = null;
FileWriter fileWriter = null;
try {
File cat_out = new File("files/" + categoria + ".txt");
fileWriter = new FileWriter(cat_out, true);
} catch (IOException e) {
e.printStackTrace();
}
try {
fileReader = new FileReader(dizionario);
} catch (FileNotFoundException e) { }
try {
BufferedReader bufferedReader = new BufferedReader(fileReader);
if (dizionario.exists()) {
StringBuffer stringBuffer = new StringBuffer();
String parola;
StringBuffer line = new StringBuffer();
int contatore_index_parola = 1;
while ((parola = bufferedReader.readLine()) != null) {
if (map.containsKey(parola) && !parola.isEmpty()) {
line.append(contatore_index_parola + ":" + map.get(parola).intValue() + " ");
map.remove(parola);
}
contatore_index_parola++;
}
if (! line.toString().isEmpty()) {
fileWriter.append(getCategoryID(categoria) + " " + line + "\n"); // print riga completa documento N x1:y x2:a ...
}
} else { System.err.println("Dictionary file not found."); }
bufferedReader.close();
fileReader.close();
fileWriter.close();
} catch (IOException e) { return false;}
catch (NullPointerException ex ) { return false;}
finally {
try {
fileReader.close();
fileWriter.close();
} catch (IOException e) {
e.printStackTrace();
}
}
return true;
}
But the error still comes. ( it is thrown at:)
try {
File cat_out = new File("files/" + categoria + ".txt");
fileWriter = new FileWriter(cat_out, true);
} catch (IOException e) {
e.printStackTrace();
}
Thank you.
EDIT: SOLVED
I found the solution, there was, in the main function in which writeOccurencesFile is called, another function that create a RandomAccessFile and doesn't close it.
The debugger sais that Exception has thrown in writeOccurencesFile but using Java Leak Detector i found out that the pdf were already opened and not close after parsing to pure text.
Thank you!
Try using this utility specifically designed for the purpose.
This Java agent is a utility that keeps track of where/when/who opened files in your JVM. You can have the agent trace these operations to find out about the access pattern or handle leaks, and dump the list of currently open files and where/when/who opened them.
When the exception occurs, this agent will dump the list, allowing you to find out where a large number of file descriptors are in use.
i have tried using try-with resources; but the problem remains.
Also running in system macos built-in console print out a FileNotFound exception at the line of FileWriter fileWriter = ...
static boolean writeOccurencesFile(String WORDLIST,String categoria, TreeMap<String,Integer> map) {
File dizionario = new File(WORDLIST);
try (FileWriter fileWriter = new FileWriter( "files/" + categoria + ".txt" , true)) {
try (FileReader fileReader = new FileReader(dizionario)) {
try (BufferedReader bufferedReader = new BufferedReader(fileReader)) {
if (dizionario.exists()) {
StringBuffer stringBuffer = new StringBuffer();
String parola;
StringBuffer line = new StringBuffer();
int contatore_index_parola = 1;
while ((parola = bufferedReader.readLine()) != null) {
if (map.containsKey(parola) && !parola.isEmpty()) {
line.append(contatore_index_parola + ":" + map.get(parola).intValue() + " ");
map.remove(parola);
}
contatore_index_parola++;
}
if (!line.toString().isEmpty()) {
fileWriter.append(getCategoryID(categoria) + " " + line + "\n"); // print riga completa documento N x1:y x2:a ...
}
} else {
System.err.println("Dictionary file not found.");
}
}
}
} catch (IOException e) {
e.printStackTrace();
}
return true;
}
This is the code that i am using now, although the bad managing of Exception, why the files seem to be not closed?
Now i am making a test with File Leak Detector
Maybe your code raises another exception that you are not handling. Try add catch (Exception e) before finally block
You also can move BufferedReader declaration out the try and close it in finally
Related
I am using below piece of code for reading the 150MB CSV file and getting GC error
Same code which was causing the problem
public List<String[]> readCsvFile(String ipFilePath) {
logger.info("Start executing readCsvFile method !!! on file " + ipFilePath);
CSVReader csvReader = null;
List<String[]> allRecrods = null;
Reader reader = null;
try {
reader = Files.newBufferedReader(Paths.get(ipFilePath));
csvReader = new CSVReader(reader);
allRecrods = csvReader.readAll();
} catch (Exception e) {
logger.error("Error in CsvFileReader !!!");
e.printStackTrace();
logger.error("Exception : ", e);
} finally {
try {
reader.close();
csvReader.close();
} catch (IOException e) {
logger.error("Error while closing fileReader/csvFileParser !!!");
e.printStackTrace();
logger.error("IOException : ", e);
}
}
return allRecrods;
}
I am getting error on the method : csvReader.readAll() as mentioned above.
I am not sure what is the problem which the code, and how to solve this, as the same code is working fine with 20-30 MB files.
The simplest solution is to increase heap size with flag "-Xmx" for example:
"-Xmx1024m". First you should use some heap size monitoring tool to see if the usage is expected.
You should not read all the lines but instead process the file line by line and size won't matter and is way more memory efficient.
Example from here: http://www.baeldung.com/java-read-lines-large-file
FileInputStream inputStream = null;
Scanner sc = null;
try {
inputStream = new FileInputStream(path);
sc = new Scanner(inputStream, "UTF-8");
while (sc.hasNextLine()) {
String line = sc.nextLine();
// System.out.println(line);
}
// note that Scanner suppresses exceptions
if (sc.ioException() != null) {
throw sc.ioException();
}
} finally {
if (inputStream != null) {
inputStream.close();
}
if (sc != null) {
sc.close();
}
}
Edit: Also if you are looking for a framework I can recommend Spring Batch https://projects.spring.io/spring-batch/
I am writing a method that takes in a List of Twitter Status objects as a parameter, opens a log file containing String represenatations of Tweets, checks if any of the String representations of the Status objects are already written to the file - if so, it removes them from the list, if not it appends the Status to the file.
Everything is working up until I attempt to write to the file. Nothing is being written at all. I am led to believe that it is due to the method having the file open in two different places: new File("tweets.txt") and new FileWriter("tweets.txt, true).
Here is my method:
private List<Status> removeDuplicates(List<Status> mentions) {
File mentionsFile = new File("tweets.txt");
try {
mentionsFile.createNewFile();
} catch (IOException e1) {
// Print error + stacktrace
}
List<String> fileLines = new ArrayList<>();
try {
Scanner scanner = new Scanner(mentionsFile);
while (scanner.hasNextLine()) {
fileLines.add(scanner.nextLine());
}
scanner.close();
} catch (FileNotFoundException e) {
// Print error + stacktrace
}
List<Status> duplicates = new ArrayList<>();
for (Status mention : mentions) {
String mentionString = "#" + mention.getUser().getScreenName() + " \"" + mention.getText() + "\" (" + mention.getCreatedAt() + "\")";
if (fileLines.contains(mentionString)) {
duplicates.add(mention);
} else {
try {
Writer writer = new BufferedWriter(new FileWriter("tweets.txt", true));
writer.write(mentionString);
} catch (IOException e) {
// Print error + stacktrace
}
}
}
mentions.removeAll(duplicates);
return mentions;
}
I wrote here few thoughts looking your code.
Remember to always close the object Reader and Writer.
Have a look at try-with-resources statement :
try (Writer writer = new BufferedWriter(new FileWriter("tweets.txt", true))) {
writer.write(mentionString);
} catch (IOException e) {
// Print error + stacktrace
}
To read an entire file in a List<String>:
List<String> lines = Files.readAllLines(Paths.get("tweets.txt"), StandardCharsets.UTF_8);
And again, I think it's a bad practice write in the same file you're reading of.
I would suggest to write in a different file if you don't have a particular constraint.
But if you really want have this behavior there are few alternative.
Create a temporary file as output and, when you process is successfully completed, than move it to the old one using Files.move(from, to).
So I'm trying to import a CSV file into my MySQL database through my Java program. The program imports everything that's in the file, like it's suppose to, but the first row, it send to the end of the table, and the program see it's there, but if I search for that nr, it says it doesn't exists. And if I go directly to the database table and edit the nr(if the nr is 137, and I edit and write 137 again) the program recognize that nr, and if I search for it, it will find, and the database table organizes itself and sends that entry where is suppose to be.
I just don't see any logic in this. I someone could help me out, I'd appreciated.
LOAD DATA INFILE 'C:\\Users\\carla.DESKTOP-9364K9K\\Desktop\\Alunos_1.csv'
INTO TABLE utentes character set utf8
FIELDS TERMINATED BY ','
(NrProcesso, Nome, #Nome_Resumido, Ano, Turma, #Subsidio, #Nome_EE, #NIF, #email, #Obs)
SET
Subsidio = IF(#Subsidio='','Nenhum',#Subsidio),
Nome_Resumido = IF(#Nome_Resumido='',NULL,#Nome_Resumido),
Nome_EE = IF(#Nome_EE='',NULL,#Nome_EE),
NIF = IF(#NIF = '', NULL,#NIF),
email = IF(#email='',NULL,#email),
Obs = IF(#Obs='',NULL,#Obs);
Thanks in advance.
You have do do something to check cell/column value and form a sql to inject in MySQL.
public List<Object> getRecordingsListFromCsv(String csvFileLocation, String mp3FileLocation, String mp3FileLocation2, String saveFileLocation, ChannelSftp sftp) {
Map<String, File> recordingsFilesMap = null;
BufferedReader br = null;
List<String> errorFilesList = new ArrayList<>();
List<Object> tempList = new LinkedList<>();
try {
csvRows = 0;
recordingsFilesMap = new LinkedHashMap<>();
br = new BufferedReader(new FileReader(csvFileLocation));
String line = br.readLine();
scriptLog.info("\n" + csvFileLocation + " loaded. Parsing File...");
while ((line = br.readLine()) != null) {
String[] csvArray = parseCsvLineToArray(line);
// System.out.println(Arrays.asList(csvArray) + "\n\n");
if (csvArray[0].trim().isEmpty()) {
continue;
}
/* Do your stuff here */
csvRows++;
}
} catch (FileNotFoundException e) {
scriptLog.error("\n---ERROR---\n FILE NOT FOUND: " + csvFileLocation);
String errorStr = "Type=" + e.toString();
errorStr += "StackTrace=" + Arrays.toString(e.getStackTrace());
scriptLog.error(errorStr);
} catch (IOException e) {
String errorStr = "Type=" + e.toString();
errorStr += "StackTrace=" + Arrays.toString(e.getStackTrace());
scriptLog.error(errorStr);
} finally {
if (br != null) {
try {
br.close();
} catch (IOException e) {
System.out.println(e.toString());
}
}
}
Hope it will help you at some extent!!
I made a wrapper ConfigurationFile class to help handle Gdx.files stuff, and it worked fine for a long time, but now it's not working, and I don't know why.
I have two of the following two methods: internal(...) and local(...). The only difference between the two is handling the load from arguments from (File folder, String name) and (String path).
-Snip Now Unnecessary Information-
UPDATE
After more configuring, I came to find out that they're not behaving the same. I have an assets/files/ folder that Gdx.files.internal(...) will access fine, but ConfigurationFile.internal(...) will access files/, and they're set up the same way. I'll give you the two pieces of code that I used for testing.
Using Gdx.files.internal(...) directly (works as expected):
FileHandle handle = Gdx.files.internal("files/virus_data");
BufferedReader reader = null;
try {
reader = new BufferedReader(handle.reader());
String c = "";
while ((c = reader.readLine()) != null) {
System.out.println(c); // prints out all 5 lines on the file.
}
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
if (reader != null) reader.close();
} catch (IOException e) {
e.printStackTrace();
}
}
Using ConfigurationFile.internal(...):
// First part, calls ConfigurationFile#internal(String path)
ConfigurationFile config = ConfigurationFile.internal("files/virus_data");
// ConfigurationFile#internal(String path)
public static ConfigurationFile internal(String path) {
ConfigurationFile config = new ConfigurationFile();
// This is literally calling Gdx.files.internal("files/virus_data");
config.handle = Gdx.files.internal(path);
config.file = config.handle.file();
config.folder = config.file.getParentFile();
config.init();
return config;
}
// ConfigurationFile#init()
protected void init() {
// File not found.
// Creates a new folder as a sibling of "assets"
// Creates a new file called "virus_data"
if (!folder.exists()) folder.mkdirs();
if (!file.exists()) {
try {
file.createNewFile();
} catch (IOException e) {
e.printStackTrace();
}
} else loadFile();
}
// ConfigurationFile#loadFile()
protected void loadFile() {
BufferedReader reader = null;
try {
reader = new BufferedReader(handle.reader());
String c = "";
while ((c = reader.readLine()) != null) {
System.out.println(c);
if (!c.contains(":")) continue;
String[] values = c.split(":");
String key = values[0];
String value = values[1];
if (values.length > 2) {
for (int i = 2; i < values.length; i++) {
value += ":" + values[i];
}
}
key = key.trim();
value = value.trim();
mapValues.put(key, value);
}
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
if (reader != null) reader.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
What I'm having trouble understanding is what's the difference between these two ways that it is causing my ConfigurationFile to create a new File in a folder that is a sibling of assets. Could someone tell me why this is happening?
My suggestion is not to use
Gdx.files.internal(folder + "/" + name);
If you have to use the File api, do it this way:
Gdx.files.internal(new File(folder, name).toString());
This way you avoid weird things that could be happening with path separators.
If Gdx maybe needs relative paths for some reason (perhaps relative to some Gdx internal home directory), you could use NIO to do something like
final Path gdxHome = Paths.get("path/to/gdx/home");
//...
File combined = new File(folder, name);
String relativePath = gdxHome.relativize(combined.toPath()).toString();
Okay, so after intense testing, I found out the problem, which I found to be ridiculous.
Since the file is Internal, that means a new File(...) reference can't be properly made to it, but instead it's an InputStream (if I'm correct), but anyways, using the method FileHandle#file() on an Internal file causes some kind of conversion for the path, so after removing anything that dealed with FileHandle#file() for an Internal file fixed it.
I'm currently trying to store json data I get on the Internet in the internal storage of the phone. In order to do so, I have get the data from an API as a string (no issue in this part, nor during the parsing afterwards). Then comes the part where I try to store it in a file... And the problems that come along!
The ultimate goal is to update data when an Internet connection is available, and use the data previously stored when no connection can be found.
Here is my code (data is in the String jsonString):
// IN CASE OF INTERNET CONNECTION (ie "jsonString != null")
if (jsonString != null) {
try {
FileOutputStream fos = new FileOutputStream(json_InternalFile);
fos.write(jsonString.getBytes());
fos.close();
} catch (FileNotFoundException e) {
Log.e("Response: ", "> " + "File not found");
e.printStackTrace();
} catch (IOException e) {
Log.e("Response: ", "> " + "IO Exception");
e.printStackTrace();
}
}
// IF NO CONNECTION AVAILABLE (ie "jsonString == null)
if (jsonString == null) {
try {
FileInputStream fis = new FileInputStream(json_InternalFile);
DataInputStream dis = new DataInputStream(fis);
BufferedReader br = new BufferedReader(new InputStreamReader(dis));
String strLine;
while ((strLine = br.readLine()) != null) {
jsonString = jsonString + strLine;
}
dis.close();
} catch (FileNotFoundException e) {
Log.e("Response: ", "> " + "File not found");
e.printStackTrace();
} catch (IOException e) {
Log.e("Response: ", "> " + "File not found");
e.printStackTrace();
}
}
None of the parts works individually. When I run the app, I got this message: "Unfortunately, the app "APP_NAME" stopped running" (whichever part I comment)
I also tried to replace the line:
FileOutputStream fos = new FileOutputStream(json_InternalFile);
with:
FileOutputStream fos = openFileOutput(jsonStorage_FileName, 0);
... Without success ^^'
I used - and abused ;) - the following code I found:
http://www.mysamplecode.com/2012/06/android-internal-external-storage.html
Thank you in advance, I really can't see the problem here (and it's turning me slightly mad ><)