Develop Reference

How to automatically collapse repetitive log output in log4j

Every once in a while, a server or database error causes thousands of the same stack trace in the server log files. It might be a different error/stacktrace today than a month ago. But it causes the log files to rotate completely, and I no longer have visibility into what happened before. (Alternately, I don't want to run out of disk space, which for reasons outside my control right now is limited--I'm addressing that issue separately). At any rate, I don't need thousands of copies of the same stack trace--just a dozen or so should be enough.
I would like it if I could have log4j/log4j2/another system automatically collapse repetitive errors, so that they don't fill up the log files. For example, a threshold of maybe 10 or 100 exceptions from the same place might trigger log4j to just start counting, and wait until they stop coming, then output a count of how many more times they appeared.
What pre-made solutions exist (a quick survey with links is best)? If this is something I should implement myself, what is a good pattern to start with and what should I watch out for?
Thanks!

Will the BurstFilter do what you want? If not, please create a Jira issue with the algorithm that would work for you and the Log4j team would be happy to consider it. Better yet, if you can provide a patch it would be much more likely to be incorporated.

Log4j's BurstFilter will certainly help prevent you filling your disks. Remember to configure it so that it applies in as limited a section of code as you can, or you'll filter out messages you might want to keep (that is, don't use it on your appender, but on a particular logger that you isolate in your code).
I wrote a simple utility class at one point that wrapped a logger and filtered based on n messages within a given Duration. I used instances of it around most of my warning and error logs to protect the off chance that I'd run into problems like you did. It worked pretty well for my situation, especially because it was easier to quickly adapt for different situations.
Something like:
...
public DurationThrottledLogger(Logger logger, Duration throttleDuration, int maxMessagesInPeriod) {
...
}
public void info(String msg) {
getMsgAddendumIfNotThrottled().ifPresent(addendum->logger.info(msg + addendum));
}
private synchronized Optional<String> getMsgAddendumIfNotThrottled() {
LocalDateTime now = LocalDateTime.now();
String msgAddendum;
if (throttleDuration.compareTo(Duration.between(lastInvocationTime, now)) <= 0) {
// last one was sent longer than throttleDuration ago - send it and reset everything
if (throttledInDurationCount == 0) {
msgAddendum = " [will throttle future msgs within throttle period]";
} else {
msgAddendum = String.format(" [previously throttled %d msgs received before %s]",
throttledInDurationCount, lastInvocationTime.plus(throttleDuration).format(formatter));
}
totalMessageCount++;
throttledInDurationCount = 0;
numMessagesSentInCurrentPeriod = 1;
lastInvocationTime = now;
return Optional.of(msgAddendum);
} else if (numMessagesSentInCurrentPeriod < maxMessagesInPeriod) {
msgAddendum = String.format(" [message %d of %d within throttle period]", numMessagesSentInCurrentPeriod + 1, maxMessagesInPeriod);
// within throttle period, but haven't sent max messages yet - send it
totalMessageCount++;
numMessagesSentInCurrentPeriod++;
return Optional.of(msgAddendum);
} else {
// throttle it
totalMessageCount++;
throttledInDurationCount++;
return emptyOptional;
}
}
I'm pulling this from an old version of the code, unfortunately, but the gist is there. I wrote a bunch of static factory methods that I mainly used because they let me write a single line of code to create one of these for that one log message:
} catch (IOException e) {
DurationThrottledLogger.error(logger, Duration.ofSeconds(1), "Received IO Exception. Exiting current reader loop iteration.", e);
}
This probably won't be as important in your case; for us, we were using a somewhat underpowered graylog instance that we could hose down fairly easily.

Temporary tablespace of CLOB not freed

I have the Problem that my Java-Application is exporting a larger amount of clobs from a database, but always runs out of temporary tablespace as the old clobs are not freed.
A simplified code example how I do it would be:
public void getClobAndDoSomething (oracle.jdbc.OracleCallableStatement pLSQLCodeReturningClob) {
try (OracleCallableStatement statement = pLSQLCodeReturningClob) {
statment.registerOutParameter(1, Types.CLOB);
statement.execute();
oracle.sql.CLOB clob = statement.getCLOB(1);
clob.open(CLOB.MODE_READONLY);
Reader reader = clob.getCharacterStream();
BufferedReader bufferedReader = new BufferedReader(reader);
doSomethingWithClob(bufferedReader);
bufferedReader.close();
reader.close();
clob.close();
clob.freeTemporary();
} catch (SQLException e) {
if (e.getErrorCode() == 1652) {
//Server ran out of temporary tablespace
} else
handleException(e);
} catch (IOException e) {
handleException(e);
}
}
If this method is called in a loop it will always end up running out of temporary table space at some point.
The only reliable way to free the space is by closing the connection and opening a new one (for example by using clob.getInternalConnection.close()) but this would slow down the application and make the current multi-threaded approach unusable.
Sadly the oracle documentation on ojdbc where not really helpful and google only found articles telling me to use the free() method of lobs which is not even implemented by oracles temporary clobs.
Additional Note:
This issue does also occur when using oracles APEXExport.class to export a big workspace.
Driver and System specifics:
OS: Windows 7 Professional x64
Java: 1.8.0_45 64-Bit
ojdbc: 6 (Are there more specific versions?)
Database: Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
Test code if you have a APEX-Application:
java.sql.Connection con = getConnection();
String gStmtGetAppClob = "begin ? := wwv_flow_utilities.export_application_to_clob(?, ?, ?, ?); end;";
int appId = 100;
while (true) {
OracleCallableStatement exportApplicationToClob = (OracleCallableStatement) con.prepareCall(gStmtGetAppClob);
exportApplicationToClob.setString(3, "Y"); //Public reports
exportApplicationToClob.setString(4, "N"); //Saved reports
exportApplicationToClob.setString(5, "N"); //Interactive report notifications
exportApplicationToClob.setBigDecimal(2, new BigDecimal(appId));
getClobAndDoSomething(exportApplicationToClob);
try {
Thread.sleep(50);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
break;
}
}
con.close();
Update:
After more testing I found out that the clobs are getting freed at some point without closing the connection. So it seems like the free() is actually a lazyFree(). But this can take more than a minute.
I can also convert the CLOB to Clob, don't know what I was doing wrong earlier. Problem stays unchanged if using Clob.

In pl/sql world this would have been handled through temporary CLOB and reusing it inside loop.
Assuming that you are using java.sql.CLOB., it does not seem to have createTemporary CLOB option, but oracle.sql.CLOB does. It also has freeTemporary() method to clear temp space.
https://docs.oracle.com/cd/E18283_01/appdev.112/e13995/oracle/sql/CLOB.html
Your calling routine can create a temporary CLOB and pass it as a parameter (lets say p_clob) to this method. Assign the return value of query to p_clob every time instead of creating new CLOB (e.g. CLOB clob = statement.getCLOB).
Short of time right now, but will edit a detailed code later. If you can work with above, then good.

neo4j : Limited queries sent in java

For a website developed under Tomcat, I connect my program to a database neo4j. Connection is done through jdbc.
My program is right now local, and the database is on a remote server.
When I start Tomcat, it first check if specifics nodes are presents, and if not, it creates them.
There are about 135 nodes.
Problem : After about ten, the program stops, and stay in something like an infinite loop.
I assume I should close something, but what?
Here is my code :
private ResultSet sendCommand(String command) throws SQLException
{
try(Statement statement = _neo4jConnection.createStatement())
{
return statement.executeQuery(command);
}
}
and a function to call this code (all functions are based on the same structure)
public static Long createNode(NodeLabel labelName)
{
try
{
ResultSet rs = getInstance().sendCommand("CREATE (n:"+labelName+") RETURN id(n)");
Long result= rs.next()?rs.getLong("id(n)"):null;
rs.close();
return result;
} catch (SQLException e) {
e.printStackTrace();
return null;
}
}

in my latest experiment I reused the same statement multiple times, not sure that is the best way but it seems to work good, too.
https://github.com/peterneubauer/blogs/tree/master/csv_jdbc, code at https://github.com/peterneubauer/blogs/blob/master/csv_jdbc/src/test/java/org/neo4j/jdbctest/JDBCTest.java

The only solution I found was to regularly disconnect and reconnect to the database (After about 20 statements). This is awful, but it works.
Well, however, I gave up with jdbc for neo4j, too much issues, and catastrophical performances (about 1s to get one simple data), what occures still more issues!
Anyway, Thanks for your help
Niko

Create a database / execute a bunch of mysql statements from Java

I have a library that needs to create a schema in MySQL from Java. Currently, I have a dump of the schema that I just pipe into the mysql command. This works okay, but it is not ideal because:
It's brittle: the mysql command needs to be on the path: usually doesn't work on OSX or Windows without additional configuration.
Also brittle because the schema is stored as statements, not descriptively
Java already can access the mysql database, so it seems silly to depend on an external program to do this.
Does anyone know of a better way to do this? Perhaps...
I can read the statements in from the file and execute them directly from Java? Is there a way to do this that doesn't involve parsing semicolons and dividing up the statements manually?
I can store the schema in some other way - either as a config file or directly in Java, not as statements (in the style of rails' db:schema or database.yml) and there is a library that will create the schema from this description?
Here is a snippet of the existing code, which works (when mysql is on the command line):
if( db == null ) throw new Exception ("Need database name!");
String userStr = user == null ? "" : String.format("-u %s ", user);
String hostStr = host == null ? "" : String.format("-h %s ", host);
String pwStr = pw == null ? "" : String.format("-p%s ", pw);
String cmd = String.format("mysql %s %s %s %s", hostStr, userStr, pwStr, db);
System.out.println(cmd + " < schema.sql");
final Process pr = Runtime.getRuntime().exec(cmd);
new Thread() {
public void run() {
try (OutputStream stdin = pr.getOutputStream()) {
Files.copy(f, stdin);
}
catch (IOException e) { e.printStackTrace(); }
}
}.start();
new Thread() {
public void run() {
try (InputStream stdout = pr.getInputStream() ) {
ByteStreams.copy(stdout, System.out);
}
catch (IOException e) { e.printStackTrace(); }
}
}.start();
int exitVal = pr.waitFor();
if( exitVal == 0 )
System.out.println("Create db succeeded!");
else
System.out.println("Exited with error code " + exitVal);

The short answer (as far as i know) is no.
You will have to do some parsing of the file into separate statements.
I have faced the same situation and you can find many questions on this topic here on SO.
some like here will show a parser. others can direct to tools Like this post from apache that can convert the schema to an xml format and then can read it back.
My main intention when writing this answer is to tell that I chose to use the command line in the end.
extra configuration: maybe it is an additional work but you can do it by config or at runtime based on the system you are running inside. you do the effort one time and you are done
depending on external tool: it is not as bad as it seems. you have some benefits too.
1- you don't need to write extra code or introduce additional libraries just for parsing the schema commands.
2- the tool is provided by the vendor. it is probably more debugged and tested than any other code that will do the parsing.
3- it is safer on the long run. any additions or changes in the format of dump that "might" break the parser will most probably be supported with the tool that comes with the database release. you won't need to do any change in your code.
4- the nature of the action where you are going to use the tool (creating schema) does not suggest frequent usage, minimizing the risk of it becoming a performance bottle neck.
I hope you can find the best solution for your needs.

Check out Yank, and more specifically the code examples linked to on that page. It's a light-weight persistence layer build on top of DBUtils, and hides all the nitty-gritty details of handling connections and result sets. You can also easily load a config file like you mentioned. You can also store and load SQL statements from a properties file and/or hard code the SQL statements in your code.

Java Heap Space Exception, with big ammount of data, any solution?

I've a litle big problem with java heap memory
I'm trying to migrate from oracle database 11g to access file 2007
This is not a problem below 65.000 records, now from there...
The aplication is throwing java heap exception, the memory consumption is raising over 600m and the CPU usage over 50% until the exeption.
As far as i know the rset.next() don't get all data (over 50 colums x +65000 row), but some records x time
i've try to set fetch size too, nothing happened
rset.setFetchSize(1000);
i've erase my code and show a output, same error
while (rset.next()) {
if (cont % 5000 == 0) {
System.out.println(cont + " proccesed and counting ...");
}
}
please don't give me the answer of using -xm(s, x)512, 1024, etc...
this could solved, not in my particulary case (i've tryied to set this even higher xD, nothing happend, i got the same exception at 65.000 records too)
Is there any other options i could try??,
meaby changing some driver configurations or string conections ??
please help
sorry aboubt my english
this is my connection:
Class.forName("oracle.jdbc.driver.OracleDriver");
this.conn = DriverManager.getConnection("jdbc:oracle:thin:#" + getServer() + ":1521:orcl", getUser(), getPassword());
this.stmt = this.conn.createStatement(java.sql.ResultSet.TYPE_SCROLL_INSENSITIVE, java.sql.ResultSet.CONCUR_UPDATABLE);

It looks like the problem is that you are using a Scrollable ResultSet and that is going to use more memory.