Introduction
I've found plenty of information about the Too many open files exception on the Web but I couldn't solve this strange case. As I've read, the exception is thrown when the number of opened file descriptors by process defined in the OS is exceeded. The nature of these files is diverse. Files could be sockets, documents, etc. And I've found robust and secure ways to open files that I have implemented in my Java application.
The application is a short program that downloads Web pages using the Boilerpipe algorithm. This way I get the most representative contents of that site. Then, I write it in an appropriate format (TREC format) to disk. The URLs of these websites are taken from a MySQL database that I access using the JDBC connector.
So, I think that the exception can be thrown form three different places:
Connection to database
HTTP Connection to the web sites
Opening and writing files
Although, as I said, I think that I use a correct way of opening and writing those files.
Problem
There are thousands of URL's to process and the exception is thrown after a while (what makes it also very difficult to debug...). I don't know if that matters, but URLs are classified into different categories and I run different instances of the program to speed up the whole process. Categories don't overlap so there shouldn't be any problem.
Code
To make it more readable I'm going to show just those three parts of my code simplified:
Database access
// Connect to database
Connection dbconn = null;
try {
String dbUrl = "jdbc:mysql://" + dbServer + "/" + dbName;
Class.forName ("com.mysql.jdbc.Driver").newInstance ();
dbconn = DriverManager.getConnection(dbUrl, dbUser, dbPass);
System.out.println ("Database connection established");
} catch (Exception e) {
e.printStackTrace();
System.err.println ("Cannot connect to database server");
System.exit(-1);
}
System.out.println(" Downloading category: " + category);
Statement s = null;
try {
s = dbconn.createStatement();
} catch (SQLException e) {
System.err.println ("Error on creating the statement");
System.exit(-1);
e.printStackTrace();
}
String q = "SELECT resource,topic FROM " +
"content_links " +
"WHERE topic LIKE 'Top/" + category + "%';";
try {
s.executeQuery(q);
} catch(Exception e) {
System.err.println ("Error on executing the SQL statement");
System.exit(-1);
e.printStackTrace();
}
ResultSet rs = null;
try {
rs = s.getResultSet ();
} catch (SQLException e) {
System.err.println ("Error on getting the result set");
System.exit(-1);
e.printStackTrace();
}
int count = 0, webError = 0;
// work with the result set
try {
while (rs.next ()) {
// MAIN LOOP
}
} catch (SQLException e) {
System.err.println ("Error on getting next item");
System.exit(-1);
e.printStackTrace();
}
// Close connection to database
if (dbconn != null) {
try {
dbconn.close ();
System.out.println (" Database connection terminated");
} catch (Exception e) { /* ignore close errors */ }
}
HTTP connection, extract site's title and boilerpipe filter
try {
String title = "";
org.jsoup.nodes.Document doc = Jsoup.connect(urlVal).get();
for (Element element : doc.select("*")) {
if (element.tagName().equalsIgnoreCase("title")) {
title = element.text();
}
if (!element.hasText() && element.isBlock()) {
element.remove();
}
}
String contents = "";
contents = NumWordsRulesExtractor.INSTANCE.getText(doc.text());
storeFile(id, urlVal, catVal, title, contents);
}
} catch (BoilerpipeProcessingException e) {
System.err.println("Connection failed to: " + urlVal);
} catch (MalformedURLException e1) {
System.err.println("Malformed URL: " + urlVal);
} catch(Exception e2) {
System.err.println("Exception: " + e2.getMessage());
e2.getStackTrace();
}
Writing file
private static void storeFile(String id, String url, String cat, String title, String contents) {
BufferedWriter out = null;
try {
out = new BufferedWriter(
new OutputStreamWriter(
new FileOutputStream(
new File(path + "/" + id + ".webtrec")),"UTF8"));
// write in TREC format
out.write("...");
} catch (IOException e) {
System.err.println("Error: " + e.getMessage());
e.printStackTrace();
} finally {
try {
out.close();
} catch (IOException e) {
System.err.println("Error: " + e.getMessage());
e.printStackTrace();
}
}
Yup. You are leaking file descriptors.
In the first case you open a DB connection and never close it. The connection will typically use a socket or something to talk to the database. Since you don't close the connection, the socket won't be closed, and you will leak a file descriptor.
In the second case, I suspect that the call to Jsoup.connect(urlVal) is opening a connection, which you don't then close. That will result in a file descriptor leak.
Correction - there is no close() method on the Connection interface. It looks like the actual connection must be created and then closed internally by the get method. Assuming that is so, there is no file descriptor leak in the second case.
The third case does not leak file descriptors. However, if you fail to open the file, out.close(); statement will attempt to call a method on null ... and will throw a NPE.
The solution is to find all of the places where you open files, database connection, http connections, and make sure that the handle is always closed.
One way to do it is to put the close() call (or equivalent) in a finally block ... but make sure that you don't accidentally call close() on null.
The other way to do it is to use the Java 7 "try with resource" syntax. For example:
private static void storeFile(String id, String url, String cat,
String title, String contents) {
try (BufferedWriter out = new BufferedWriter(
new OutputStreamWriter(
new FileOutputStream(
new File(path + "/" + id + ".webtrec")),"UTF8"))) {
// write in TREC format
out.write("...");
out.close();
} catch (IOException e) {
System.err.println("Error: " + e.getMessage());
e.printStackTrace();
}
}
(Note however that the Java 7 syntax can only be used with resources that implement the new Closeable interface.)
To add to Stephen's comprehensive analysis. I recommend using a connection pool for the database, although, as Stephen has pointed, unless you're closing these connections, you'll drain the pool close, but at least it will be easier to discover why...
I've not seen any evidence, but you should be using some kind of Thread pool to download the pages, this will help to maximize the resources of the system. Some of executor service would suffice. Like I say, you're probably already doing this, but you didn't show any code (or comment) for it.
Related
I am having an java package, which connects with a database and fetches some data. At some rare case, I am getting heap memory exception, since the fetched query data size is exceeding the java heap space. Increasing the java heap space is not something the business can think for now.
Other option is to catch the exception and continue the flow with stopping the execution. ( I know catching OOME is not a good idea but here only me local variables are getting affected). My code is below:
private boolean stepCollectCustomerData() {
try {
boolean biResult = generateMetricCSV();
} catch (OutOfMemoryError e) {
log.error("OutOfMemoryError while collecting data ");
log.error(e.getMessage());
return false;
}
return true;
}
private boolean generateMetricCSV(){
// Executing the PAC & BI cluster SQL queries.
try (Connection connection = DriverManager.getConnection("connectionURL", "username", "password")) {
connection.setAutoCommit(false);
for (RedshiftQueryDefinition redshiftQueryDefinition: redshiftQueryDefinitions){
File csvFile = new File(dsarConfig.getDsarHomeDirectory() + dsarEntryId, redshiftQueryDefinition.getCsvFileName());
log.info("Running the query for metric: " + redshiftQueryDefinition.getMetricName());
try( PreparedStatement preparedStatement = createPreparedStatement(connection,
redshiftQueryDefinition.getSqlQuery(), redshiftQueryDefinition.getArgumentsList());
ResultSet resultSet = preparedStatement.executeQuery();
CSVWriter writer = new CSVWriter(new FileWriter(csvFile));) {
if (resultSet.next()) {
resultSet.beforeFirst();
log.info("Writing the data to CSV file.");
writer.writeAll(resultSet, true);
log.info("Metric written to csv file: " + csvFile.getAbsolutePath());
filesToZip.put(redshiftQueryDefinition.getCsvFileName(), csvFile);
} else {
log.info("There is no data for the metric " + redshiftQueryDefinition.getCsvFileName());
}
} catch (SQLException | IOException e) {
log.error("Exception while generating the CSV file: " + e);
e.printStackTrace();
return false;
}
}
} catch (SQLException e){
log.error("Exception while creating connection to the Redshift cluster: " + e);
return false;
}
return true;
}
We are getting exception in the line "ResultSet resultSet = preparedStatement.executeQuery()" in the later method and i am catching this exception in the parent method. Now, i need to make sure when the exception is caught in the former method, is the GC already triggered and cleared the local variables memory? (such as connection and result set variable) If not, when that will be happen?
I am worried about the java heap space because, this is continuous flow and I need to keep on fetching the data for another users.
The code that i have provided is only to explain the underlying issue and flow and kindly ignore syntax, etc.., I am using JDK8
Thanks in advance.
I am connecting to a website using Jsoup. I have inserted a .timeout to check if the website times out.
Because I already handle IOException, my IDE is not allowing me to put another catch for SocketTimeOutException, which is fair, I guess.
Is there a way to tell from IOException what is the error?
I'd like to have specific debugging output for different errors, so that I can keep the code clean and efficient.
Here is my code (yes, some variables are missing, they are pretty simple though and not really necessary):
Document doc = null;
try
{
doc = Jsoup.connect(Url).proxy(proxy).userAgent(userAgent).timeout(10000).get();
}
catch (IOException e)
{
if (...)
{
if(specificThread.running == false) {
System.out.println("Thread got interrupted! Link visited -> "+ Url);
Thread.currentThread().interrupt();
}
try
{
System.out.println("Localhost detected! If this happens, you're stupid");
doc = Jsoup.connect(Url).userAgent(userAgent).get();
}
catch (IOException e1)
{
System.out.println("Couldn't connect to " + Url + " , localhost was detected.");
e1.printStackTrace();
System.exit(-1);
}
catch (java.lang.IllegalArgumentException error)
{
System.out.println("Malformed URL detected -> " + Url) ;
}
}
else
{
System.out.println("Couldn't connect to " + Url);
e.printStackTrace();
}
}
catch (java.lang.IllegalArgumentException error)
{
System.out.println("Malformed URL detected -> " + Url);
}
catch (java.net.SocketTimeoutException error) //IDE Is blocking this
{
//Handle error here
}
Is it possible to do this?
Put the catch for the SocketTimeoutException before the catch for the IOException.
Exception handling works by looking through a table until a matching exception type is found. As such, more specific exceptions have to come before more general exceptions, in order that they can be matched.
This is described in JLS Sec 14.21, "Unreachable Statements":
It is a compile-time error if a statement cannot be executed because it is unreachable.
...
A catch block C is reachable iff both of the following are true:
...
There is no earlier catch block A in the try statement such that the type of C's parameter is the same as or a subclass of the type of A's parameter.
So, you just need to make sure there is no earlier catch block catching a superclass parameter.
I am attempting a client/server type chat box (using GUI's). I won't get into details of the multi-threading I used in the program since I believe it is not part of the problem (I hope not) and it will be good amount of code to post. Anyways, for both my client and my server I create a socket, and some other stream classes within a try block, and some reason the sockets close after the catch blocks. PS I do NOT call socket.close() method anywhere that could end if early
Server, this is ran into a constructor of one of my class. It breaks down into, my main has the actually server stuff on a different thread, (like my previous post) it is a fix so that the gui can load and run the server stuff without one waiting on the other. Anyways, without all that detail, here is my code
public ChatAppProtocol(Socket sock)
{
super("ChatAppServer");
// this also has a clas var of Socket
this.sock = sock;
try (
PrintWriter output = new PrintWriter(this.sock.getOutputStream(), true);
BufferedReader input = new BufferedReader(new InputStreamReader(this.sock.getInputStream())) ;
)
{
// first stream of a string is the username loging in from client
String name = input.readLine();
// this returns false, so its not closed
System.out.println("closed?: " + this.sock.isClosed());
}
catch (IOException e)
{
e.printStackTrace();
}
// PROBLEM!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
// closed after the catch blocks before methods even ends
// p.s. i also plan on using the socket in another method but can't since it closes
System.out.println("closed?: " +this.sock.isClosed());
}
now my client
#FXML
private void login()
{
this.name = this.username.getText().trim();
this.portnum = Integer.parseInt(this.port.getText());
this.name = this.username.getText().trim();
this.ipaddr = this.ip.getText().trim();
try (t
Socket socket = new Socket(this.ipaddr, this.portnum);
PrintWriter output = new PrintWriter(socket.getOutputStream(), true);
)
{
this.sock = socket;
output.println(this.name);
// this returns false, not closed
System.out.println("closed?: " +this.sock.isClosed());
}
catch (UnknownHostException e)
{
System.err.println("Problem at ip: " + this.ipaddr);
System.exit(1);
}
// PROBLEM!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
// returns true here, closes before methods end and i cant reuse it
System.out.println("IS IT CLOSED!!!!!! " + this.sock.isClosed());
}
}
so, any reason why both this different class, different files, different project sockets close after try-catch blocks? Can't find answer online, and been on it for a while and I am stuck. I found out about this problem after seeing this on the server side console
java.net.SocketException: Socket is closed
at java.net.Socket.getOutputStream(Socket.java:943)
at chatappserver.ChatAppProtocol.run(ChatAppProtocol.java:62)
Because you're creating socket with the brackets of the try block, it is automatically closed upon exiting the block. Instead, try creating it inside the block itself and it shouldn't be closed:
try {
this.sock = new Socket(this.ipaddr, this.portnum);
PrintWriter output = new PrintWriter(socket.getOutputStream(), true);
output.println(this.name);
} catch (UnknownHostException e) {
System.err.println("Problem at ip: " + this.ipaddr);
System.exit(1);
}
// this.sock should still be open at this point.
Have a read of the Java tutorial on try-with-resources for more information on why you're getting your current behaviour.
You are using try-with-resources, which is roughly an equivalent of:
try
{
this.sock = new Socket(this.ipaddr, this.portnum));
output.println(this.name);
// this returns false, not closed
System.out.println("closed?: " +this.sock.isClosed());
}
catch (UnknownHostException e)
{
System.err.println("Problem at ip: " + this.ipaddr);
System.exit(1);
} finally {
if (this.sock != null)
this.sock.close();
}
Just initialize the socket outside the resources clause of try (...) and it won't get closed
System.out.println("======================================");
System.out.println("List of Ships and their revenue");
System.out.println("======================================");
try
{
inputStream = new ObjectInputStream (new FileInputStream (fileName));
}
catch (Exception e)
{
System.out.println("Problem");
}
try
{
while (true)
{
Ship copyObject = (Ship)inputStream.readObject();
System.out.println(copyObject.getShipName() + " " + copyObject.getRevenue());// fix format later
}
}
catch (EOFException e)
{
}
catch (Exception e)
{
}
I want to retrieve data from a binary file, which I definitely know that I has data as a objects. But the result of the program is only the below:
=================================
List of Ships and their revenue
=================================
It probably means that there is something wrong with the try block. Note that this is just part of my program, which relates to the result I want to get. Ship is a superclass of the classes to which the object in the file belong. (update: I just print the exception, but no exception is thrown).
By the looks of it you are in an infinite loop which dies with an exception which results in no output. Try doing the following instead of the giant second try catch block.
while(true) {
try{
Ship copyObject = (Ship)inputStream.readObject();
System.out.println(copyObject.getShipName() + " " + copyObject.getRevenue());
}
catch(Exception ex) {
System.out.println("No more ships");
break;
}
}
So I have tried using the stock Play! 2.2 configuration for the MySql database connection. Unfortunately the guides out there are less than helpful when using the stock database (h2) alongside a MySql. SO, I coded a separate model to handle the MySql connection. It works intermittently, and I'm trying to figure out why it doesn't work all of the time.
this is the "connect" function
String sourceSchema = "db";
String databaseHost = "host";
String databaseURLSource = "jdbc:mysql://" + databaseHost + "/" + sourceSchema;
String databaseUserIDSource = "userid";
String databasePWDSource = "password";
try {
Class.forName("com.mysql.jdbc.Driver").newInstance();
conn = DriverManager.getConnection(databaseURLSource,
databaseUserIDSource, databasePWDSource);
return true;
} catch (InstantiationException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (IllegalAccessException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (ClassNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (SQLException e) {
Logger.error("SQLException: " + e.getMessage());
}
All of my credentials are correct (here obviously they are changed) Next, in my lib folder, I have the
mysql-connector-java-5.1.21-bin.jar
in place.
Next, in my Build.scala, I have this under appDependencies:
"mysql" % "mysql-connector-java" % "5.1.21"
when I try to validate the connection, using:
public boolean isConnected() {
return conn != null;
}
The connection fails (intermittantly) and then gives me:
SQLException: Before start of result set
and sometimes:
SQLException: No Suitable driver found for mysql ...
This is how my query is executed:
String qs = String.format("SELECT * FROM community_hub.alert_journal LIMIT("+ from +","+ to +")");
String qscount = String.format("SELECT COUNT(*) AS count FROM community_hub.alert_journal");
try {
if (isConnected()) {
Statement stmt = conn.createStatement();
//obtain count of rows
ResultSet rs1 = stmt.executeQuery(qscount);
//returns the number of pages to draw on index
int numPages = returnPages(rs1.getInt("count"),rpp);
NumPages(numPages);
ResultSet rs = stmt.executeQuery(qs);
while (rs.next())
{
AlertEntry ae = new AlertEntry(
rs.getTimestamp("date"),
rs.getString("service_url"),
rs.getString("type"),
rs.getString("offering_id"),
rs.getString("observed_property"),
rs.getString("detail")
);
list.add(ae);
}
rs.close();
disconnect();
} else {
System.err.println("Connection was null");
}
}
catch (Exception e)
{
e.printStackTrace();
}
Help?
Thanks!
does the mysql error tell you anything?
the first error "SQLException: Before start of result set" looks like its incomplete. Maybe the error log contains the full message or you can
the second one "SQLException: No Suitable driver found for mysql" clearly indicates a classpath issue.
usually connection pools like c3p0 or BoneCP recommed to use a validation query to determine if a connection is valid (something like "select 1" for mysql). That may help to make sure the connection is ok and not rely on the driver?