Develop Reference

Java too many open files exception

I have a problem on my code; basically I have an array containing some key:
String[] ComputerScience = { "A", "B", "C", "D" };
And so on, containing 40 entries.
My code reads 900 pdf from 40 folder corresponding to each element of ComputerScience, manipulates the extracted text and stores the output in a file named A.txt , B.txt, ecc ...
Each folder "A", "B", ecc contains 900 pdf.
After a lot of documents, an exception "Too many open files" is thrown.
I'm supposing that I am correctly closing files handler.
static boolean writeOccurencesFile(String WORDLIST,String categoria, TreeMap<String,Integer> map) {
File dizionario = new File(WORDLIST);
FileReader fileReader = null;
FileWriter fileWriter = null;
try {
File cat_out = new File("files/" + categoria + ".txt");
fileWriter = new FileWriter(cat_out, true);
} catch (IOException e) {
e.printStackTrace();
}
try {
fileReader = new FileReader(dizionario);
} catch (FileNotFoundException e) { }
try {
BufferedReader bufferedReader = new BufferedReader(fileReader);
if (dizionario.exists()) {
StringBuffer stringBuffer = new StringBuffer();
String parola;
StringBuffer line = new StringBuffer();
int contatore_index_parola = 1;
while ((parola = bufferedReader.readLine()) != null) {
if (map.containsKey(parola) && !parola.isEmpty()) {
line.append(contatore_index_parola + ":" + map.get(parola).intValue() + " ");
map.remove(parola);
}
contatore_index_parola++;
}
if (! line.toString().isEmpty()) {
fileWriter.append(getCategoryID(categoria) + " " + line + "\n"); // print riga completa documento N x1:y x2:a ...
}
} else { System.err.println("Dictionary file not found."); }
bufferedReader.close();
fileReader.close();
fileWriter.close();
} catch (IOException e) { return false;}
catch (NullPointerException ex ) { return false;}
finally {
try {
fileReader.close();
fileWriter.close();
} catch (IOException e) {
e.printStackTrace();
}
}
return true;
}
But the error still comes. ( it is thrown at:)
try {
File cat_out = new File("files/" + categoria + ".txt");
fileWriter = new FileWriter(cat_out, true);
} catch (IOException e) {
e.printStackTrace();
}
Thank you.
EDIT: SOLVED
I found the solution, there was, in the main function in which writeOccurencesFile is called, another function that create a RandomAccessFile and doesn't close it.
The debugger sais that Exception has thrown in writeOccurencesFile but using Java Leak Detector i found out that the pdf were already opened and not close after parsing to pure text.
Thank you!

Try using this utility specifically designed for the purpose.
This Java agent is a utility that keeps track of where/when/who opened files in your JVM. You can have the agent trace these operations to find out about the access pattern or handle leaks, and dump the list of currently open files and where/when/who opened them.
When the exception occurs, this agent will dump the list, allowing you to find out where a large number of file descriptors are in use.

i have tried using try-with resources; but the problem remains.
Also running in system macos built-in console print out a FileNotFound exception at the line of FileWriter fileWriter = ...
static boolean writeOccurencesFile(String WORDLIST,String categoria, TreeMap<String,Integer> map) {
File dizionario = new File(WORDLIST);
try (FileWriter fileWriter = new FileWriter( "files/" + categoria + ".txt" , true)) {
try (FileReader fileReader = new FileReader(dizionario)) {
try (BufferedReader bufferedReader = new BufferedReader(fileReader)) {
if (dizionario.exists()) {
StringBuffer stringBuffer = new StringBuffer();
String parola;
StringBuffer line = new StringBuffer();
int contatore_index_parola = 1;
while ((parola = bufferedReader.readLine()) != null) {
if (map.containsKey(parola) && !parola.isEmpty()) {
line.append(contatore_index_parola + ":" + map.get(parola).intValue() + " ");
map.remove(parola);
}
contatore_index_parola++;
}
if (!line.toString().isEmpty()) {
fileWriter.append(getCategoryID(categoria) + " " + line + "\n"); // print riga completa documento N x1:y x2:a ...
}
} else {
System.err.println("Dictionary file not found.");
}
}
}
} catch (IOException e) {
e.printStackTrace();
}
return true;
}
This is the code that i am using now, although the bad managing of Exception, why the files seem to be not closed?
Now i am making a test with File Leak Detector

Maybe your code raises another exception that you are not handling. Try add catch (Exception e) before finally block
You also can move BufferedReader declaration out the try and close it in finally

GC overhead limit exceeded while read from CSV file

I am using below piece of code for reading the 150MB CSV file and getting GC error
Same code which was causing the problem
public List<String[]> readCsvFile(String ipFilePath) {
logger.info("Start executing readCsvFile method !!! on file " + ipFilePath);
CSVReader csvReader = null;
List<String[]> allRecrods = null;
Reader reader = null;
try {
reader = Files.newBufferedReader(Paths.get(ipFilePath));
csvReader = new CSVReader(reader);
allRecrods = csvReader.readAll();
} catch (Exception e) {
logger.error("Error in CsvFileReader !!!");
e.printStackTrace();
logger.error("Exception : ", e);
} finally {
try {
reader.close();
csvReader.close();
} catch (IOException e) {
logger.error("Error while closing fileReader/csvFileParser !!!");
e.printStackTrace();
logger.error("IOException : ", e);
}
}
return allRecrods;
}
I am getting error on the method : csvReader.readAll() as mentioned above.
I am not sure what is the problem which the code, and how to solve this, as the same code is working fine with 20-30 MB files.

The simplest solution is to increase heap size with flag "-Xmx" for example:
"-Xmx1024m". First you should use some heap size monitoring tool to see if the usage is expected.

You should not read all the lines but instead process the file line by line and size won't matter and is way more memory efficient.
Example from here: http://www.baeldung.com/java-read-lines-large-file
FileInputStream inputStream = null;
Scanner sc = null;
try {
inputStream = new FileInputStream(path);
sc = new Scanner(inputStream, "UTF-8");
while (sc.hasNextLine()) {
String line = sc.nextLine();
// System.out.println(line);
}
// note that Scanner suppresses exceptions
if (sc.ioException() != null) {
throw sc.ioException();
}
} finally {
if (inputStream != null) {
inputStream.close();
}
if (sc != null) {
sc.close();
}
}
Edit: Also if you are looking for a framework I can recommend Spring Batch https://projects.spring.io/spring-batch/

Append a string in front of line in java?

I am creating a pattern lock based project in android.
I have a file called category.txt
The content of the file is as below
Sports:Race:Arcade:
No what i want is that whenever the user draw a pattern for a specific games category the pattern should get append in front of that category.
eg :
Sports:Race:"string/pattern string to be appended here for race"Arcade:
i have used following code but it is not working.
private void writefile(String getpattern,String category)
{
String str1;
try {
file = new RandomAccessFile(filewrite, "rw");
while((str1 = file.readLine()) != null)
{
String line[] = str1.split(":");
if(line[0].toLowerCase().equals(category.toLowerCase()))
{
String colon=":";
file.write(category.getBytes());
file.write(colon.getBytes());
file.write(getpattern.getBytes());
file.close();
Toast.makeText(getActivity(),"In Writefile",Toast.LENGTH_LONG).show();
}
}
}
catch (FileNotFoundException e)
{
e.printStackTrace();
}
catch(IOException io)
{
io.printStackTrace();
}
}
please help !

Using RandomAccessFile you have to calculate the position. I think it's much easier to just replace the file content with a little help from apache-commons-io FileUtils. This might be not the best idea if you have a very large file but it's quite simple.
String givenCategory = "Sports";
String pattern = "stringToAppend";
final String colon = ":";
try {
List<String> lines = FileUtils.readLines(new File("someFile.txt"));
String modifiedLine = null;
int index = 0;
for (String line : lines) {
String[] categoryFromLine = line.split(colon);
if (givenCategory.equalsIgnoreCase(categoryFromLine[0])) {
modifiedLine = new StringBuilder().append(pattern).append(colon).append(givenCategory).append(colon).toString();
break;
}
index++;
}
if (modifiedLine != null) {
lines.set(index, modifiedLine);
FileUtils.writeLines(new File("someFile.txt"), lines);
}
} catch (IOException e1) {
// do something
}

Java Program keeps inserting the same line into my file

so I am trying to read a file and add two lines of code to it at the top. So far that hadn't been working so I tried just reading the lines and writing them back. This was a major failure. It only writes to one of the files in the directory and just keeps filling it with the xmlopentag even though its been commented out, over and over. If anyone has any ideas it would be appreciated.
ArrayList<String> lines = new ArrayList();
BufferedReader reader = null;
String xmlstylesheet = "<?xml-stylesheet type=\"text/xsl\" href=\""+stylefilename+"\"?>";
String xmlopentag = "<?xml version=\"1.0\" encoding=\"UTF-8\"?>";
for(int i=0;i<files.length;i++) {
lines.clear();
try {
reader = new BufferedReader(new FileReader(files[i]));
String text = null;
while ((text = reader.readLine()) != null) {
lines.add(text);
}
} catch (FileNotFoundException ex) {
Logger.getLogger(MainWindow.class.getName()).log(Level.SEVERE, null, ex);
} catch (IOException ex) {
Logger.getLogger(MainWindow.class.getName()).log(Level.SEVERE, null, ex);
}
//lines.add(1, xmlstylesheet);
//lines.add(0, xmlopentag);
try {
PrintWriter writer = new PrintWriter(new BufferedWriter(new FileWriter(files[i])));
for(int j=0;j<lines.size();j++) {
writer.write(lines.get(i));
}
} catch (IOException ex) {
Logger.getLogger(MainWindow.class.getName()).log(Level.SEVERE, null, ex);
}
}

writer.write(lines.get(i));
should probably be
writer.write(lines.get(j));

for loop for PrintWriter operation is having an index j for iteration but using i to get the list value
please check it properly and change below code
for(int j=0;j<lines.size();j++) {
writer.write(lines.get(i));
}
to
for(int j=0;j<lines.size();j++) {
writer.write(lines.get(j));
}

How to use java program to run command prompt commands?

this is my first time posting here, so I'm not really sure what to say/ask.
Anyways, I am trying to make a simple java program that runs command prompt commands from the java program, mainly used for ping flood (ping flooding myself).
Here is my current code
public class Core extends JFrame {
JTextField ipTextField;
int packets = 0;
boolean running = false;
public Core() {
super("Fatique");
Container container = getContentPane();
JButton bAttack = new JButton("Start Attack");
JButton bStop = new JButton("Stop Attack");
JPanel jPanel = new JPanel();
container.setLayout(new FlowLayout());
ipTextField = new JTextField("IP Address", 30);
container.add(ipTextField);
bAttack.addActionListener(new ActionListener()
{
public void actionPerformed(ActionEvent e)
{
String input = ipTextField.getText();
String[] value = input.split(":");
int amountOfPackets = Integer.parseInt(value[1]);
exec("cmd /c" + input + " -t -n " + amountOfPackets);
running = true;
}
});
bStop.addActionListener(new ActionListener()
{
public void actionPerformed(ActionEvent e)
{
stop();
}
});
if(!running) {
jPanel.add(bAttack);
} else {
jPanel.add(bStop);
}
add(jPanel);
}
public void exec(String cmd) {
try {
Process p = Runtime.getRuntime().exec(cmd);
System.out.println(getOutput(p) + " - " + getPacketsSent());
} catch (IOException e) {
e.printStackTrace();
}
}
public String getOutput(Process p) {
String output = null;
try {
BufferedReader in = new BufferedReader(new InputStreamReader(p.getInputStream()));
String line = null;
while ((line = in.readLine()) != null) {
output = line;
packets++;
}
return output;
} catch (IOException e) {
System.err.println(e.getStackTrace());
}
return null;
}
public int getPacketsSent() {
return packets;
}
public void stop() {
exec("cmd /c break");
running = false;
}
public static void main(String[] args) {
Core c = new Core();
c.setSize(500, 300);
c.setVisible(true);
c.setResizable(false);
c.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
c.setLocationRelativeTo(null);
}
I'm quite new at java, so that might not do what I want it to do.
What I want it to do is I enter an ip address in the textfield, and split it with ":", and after that the amount of packets, for instance
127.0.0.1:100
Though now when I try to use that ip and packet amount, it returns "null - 0" (from exec method), and I'm not even sure if it did anything related to ping.
What I am trying to accomplish is as I already said, ping flood myself, and then output whatever I get as response, though I have no idea if this code does anything even related to that, I mostly use logic when coding java.
public String getOutput(Process p) {
String output = null;
try {
BufferedReader in = new BufferedReader(new InputStreamReader(p.getInputStream()));
String line = null;
while ((line = in.readLine()) != null) {
output = line;
packets++;
}
return output;
} catch (IOException e) {
System.err.println(e.getStackTrace());
}
return null;
}
Could someone explain me why my code code is not working how I want it to work? Please don't judge, as I already said, I'm quite new to java programming.
EDIT: Here is a quick "informative" explanation of what I am trying to accomplish.
I type in an ip address and how many packets I want to send. In this explanation, I am using localhost ip, and 5 packets.
I start the attack. At this part, I want the program to run cmd prompt command
ping 127.0.0.1 -t -n 5
127.0.0.1 being the ip that I put in the textfield in my program, and 5 is the amount of packets I put in the textfield.
I started the attack, so this is what should happen in the command prompt:
The language is Finnish, but still the same thing.
This is the basic explanation of what I am trying to accomplish, hopefully someone understood and can help/tell why my code is not working, or is working but not printing the proper lines in eclipse console.

There is a problem with your getOutput method. It looks like you intend to collect every line of output. But in fact, since you are assigning line to output, you will only return the last line before the end of stream.
To fix this, change
output = line;
to
output += line + "\n";
Or to be more correct:
output += line + LINE_SEPARATOR;
where you previously declared the latter as:
final String LINE_SEPARATOR = System.getProperty("line.separator");
That doesn't directly explain why you are getting null, but that might be because the command you are running is writing output to the 'error' stream rather than the 'output' stream.

Try something like this:
try {
Runtime rt = Runtime.getRuntime();
Process p = rt.exec("ping 192.168.16.67");
InputStream in = p.getInputStream();
OutputStream out = p.getOutputStream ();
InputStream err = p.getErrorStream();
p.destroy();
} catch(Exception exc) {}
Then, you'll have to read the out variable to parse the ping command output continuously.

bAttack.addActionListener(new ActionListener()
{
public void actionPerformed(ActionEvent e)
{
String input = ipTextField.getText();
String[] value = input.split(":");
int amountOfPackets = Integer.parseInt(value[1]);
try {
p=Runtime.getRuntime().exec("ping -n "+amountOfPackets+" "+value[0]);
} catch (IOException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}
running = true;
}
Just a small modification of your code. get output is as:
public String getOutput(Process p) {
String output = null;
try {
BufferedReader in = new BufferedReader(new InputStreamReader(p.getInputStream()));
String line = null;
while ((line = in.readLine()) != null) {
output =output+ line+"\n";
packets++;
}
return output;
} catch (IOException e) {
System.err.println(e.getStackTrace());
}
return null;
}
Here output is JTextArea I have taken to display the output of PING process. I cannot show you the output because I lack reputation.
I don't know why first line is null. Anyway, it works.
Hope this help you. Have good time coding.