Develop Reference

Getting CPU 100 percent when I am trying to downloading CSV in Spring

I am getting CPU performance issue on server when I am trying to download CSV in my project, CPU goes 100% but SQL returns the response within 1 minute. In the CSV we are writing around 600K records for one user it is working fine but for concurrent users we are getting this issue.
Environment
Spring 4.2.5
Tomcat 7/8 (RAM 2GB Allocated)
MySQL 5.0.5
Java 1.7
Here is the Spring Controller code:-
#RequestMapping(value="csvData")
public void getCSVData(HttpServletRequest request,
HttpServletResponse response,
#RequestParam(value="param1", required=false) String param1,
#RequestParam(value="param2", required=false) String param2,
#RequestParam(value="param3", required=false) String param3) throws IOException{
List<Log> logs = service.getCSVData(param1,param2,param3);
response.setHeader("Content-type","application/csv");
response.setHeader("Content-disposition","inline; filename=logData.csv");
PrintWriter out = response.getWriter();
out.println("Field1,Field2,Field3,.......,Field16");
for(Log row: logs){
out.println(row.getField1()+","+row.getField2()+","+row.getField3()+"......"+row.getField16());
}
out.flush();
out.close();
}}
Persistance Code:- I am using spring JDBCTemplate
#Override
public List<Log> getCSVLog(String param1,String param2,String param3) {
String sql =SqlConstants.CSV_ACTIVITY.toString();
List<Log> csvLog = JdbcTemplate.query(sql, new Object[]{param1, param2, param3},
new RowMapper<Log>() {
#Override
public Log mapRow(ResultSet rs, int rowNum)
throws SQLException {
Log log = new Log();
log.getField1(rs.getInt("field1"));
log.getField2(rs.getString("field2"));
log.getField3(rs.getString("field3"));
.
.
.
log.getField16(rs.getString("field16"));
}
return log;
}
});
return csvLog;
}

I think you need to be specific on what you meant by "100% CPU usage" whether it's the Java process or MySQL server. As you have got 600K records, trying to load everything in to memory would easily end up in OutOfMemoryError. Given that this works for one user means that you've got enough heap space to process this number of records for just one user and symptoms surface when there are multiple users trying to use the same service.
First issue I can see in your posted code is that you try to load everything into one big list and the size of the list varies based on the content of the Log class. Using a list like this also means that you have to have enough memory to process JDBC result set and generate new list of Log instances. This can be a major problem with a growing number of users. This type of short-lived objects will cause frequent GC and once GC cannot keep up with the amount of garbage being collected it fails obviously. To solve this major issue my suggestion is to use ScrollableResultSet. Additionally you can make this result set read-only, for example below is code fragment for creating a scrollable result set. Take a look at the documentation for how to use it.
Statement st = conn.createStatement(ResultSet.TYPE_SCROLL_SENSITIVE, ResultSet.CONCUR_READ_ONLY);
Above option is suitable if you're using pure JDBC or SpringJDBC template. If Hibernate is already used in your project you can still achieve the same this with the below code fragment. Again please check the documentation for more information and you have a different JPA provider.
StatelessSession session = sessionFactory.openStatelessSession();
Query query = session.createSQLQuery(queryStr).setCacheable(false).setFetchSize(Integer.MIN_VALUE).setReadOnly(true);
query.setParameter(query_param_key, query_paramter_value);
ScrollableResults resultSet = query.scroll(ScrollMode.FORWARD_ONLY);
This way you're not loading all the records to Java process in one go, instead you they're loaded on demand and will have small memory footprint at any given time. Note that JDBC connection will be open until you're done with processing the entire record set. This also means that your DB connection pool can be exhausted if many users are going to download CSV files from this endpoint. You need to take measures to overcome this problem (i.e use of an API manager to rate limit the calls to this endpoint, reading from a read-replica or whatever viable option).
My other suggestion is to stream data which you have already done, so that any records fetched from the DB are processed and sent to client before the next set of records are processed. Again I would suggest you to use a CSV library such as SuperCSV to handle this as these libraries are designed to handle a good load of data.
Please note that this answer may not exactly answer your question as you haven't provided necessary parts of your source such as how to retrieve data from DB but will give the right direction to solve this issue

Your problem in loading all data on application server from database at once, try to run query with limit and offset parameters (with mandatory order by), push loaded records to client and load next part of data with different offset. It help you decrease memory footprint and will not required keep connection to database open all the time. Of course, database will loaded a bit more, but maybe whole situation will better. Try different limit values, for example 5K-50K and monitor cpu usage - on both app server and database.
If you can allow keep many open connection to database #Bunti answer is very good.
http://dev.mysql.com/doc/refman/5.7/en/select.html

OrmLite: Difference between Dao.callBatchTasks() and TransactionManager.callInTransaction()

Which is the difference between these methods? I have readed the docs but I don't understand what callBatchTasks method do. Documentation says:
This will turn off what databases call "auto-commit" mode, run the
call-able and then re-enable "auto-commit".
Is't it a transaction?
Thanks.

Difference between Dao.callBatchTasks() and TransactionManager.callInTransaction()
The difference depends on the database you are using. Under Android, there is no difference. The javadocs for callBatchTasks(...) says:
Call the call-able that will perform a number of batch tasks. This is for performance when you want to run a number of database operations at once -- maybe loading data from a file. This will turn off what databases call "auto-commit" mode, run the call-able, and then re-enable "auto-commit". If auto-commit is not supported then a transaction will be used instead.
Android's SQLite is one of the databases. Inside the internal ORMLite code you see:
private <CT> CT doCallBatchTasks(DatabaseConnection connection, boolean saved,
Callable<CT> callable) throws SQLException {
if (databaseType.isBatchUseTransaction()) {
return TransactionManager.callInTransaction(connection, saved, databaseType,
callable);
}
...
So internally, when using under Android, dao.callBatchTasks(...) is a call through to TransactionManager.callInTransaction(...).

How to read and write the records in EhCache?

Hi All,
My current requirement is to store and read the records using EhCache. I am new to EhCache Implementation. I have read the EhCache Documentation and started to implement. I have done the records insert part and also read part. While the records are inserted, there will be *.data nd *.index files are created. Following is Code.
public class Driver
{
public static void main(String[] args) {
CacheManager cm = CacheManager.create("ehcache.xml");
Cache cache = cm.getCache("test");
// I do a couple of puts
for(int i=0;i<10;i++){
cache.put(new Element("key1", "val1"));
cache.flush();
}
System.out.println(cache.getKeys());
for(int i=0;i<10;i++){
Element el = cache.get("key"+i);
System.out.println(el.getObjectValue());
}
cm.shutdown();
}
}
Now what the issue is cm.shutdown(). If I am commenting this line and comment out the insert part and run the program means, Not able to retrieve the records and also *.index file is deleted. So In real scenario if the program is stopped abruptly means we can't read the records after startup. I want to know why the file is deleted and why I cant read the records in this situation... The Exception coming in the console is
net.sf.ehcache.util.SetAsList#b66cc
Exception in thread "main" java.lang.NullPointerException
at Driver.main(Driver.java:29)...
Any Input is needed Please..

What you are doing is correct and the expected behaviour is correct too. Caches are typically used to enhance application performance by providing frequently used data quickly, while avoiding costly trips to datastore.
Not all applications need to persist cache after the system is shutdown- and that's the default behaviour you are seeing (Most applications will build cache on application startup or as requests start coming in). The data you are caching is in heap - and as soon as your JVM dies- the cache is gone. Now you want to persist it beyond restart? There are options available. Loook up here
And I am copying the code snippet right from the same page:
DiskStoreConfiguration diskStoreConfiguration = new DiskStoreConfiguration();
diskStoreConfiguration.setPath("/my/path/dir");
// Already created a configuration object ...
configuration.addDiskStore(diskStoreConfiguration);
// By adding configuration for storing the cache in a file - you are not using default cache manager
CacheManager mgr = new CacheManager(configuration);
In addition, you will have to also configure the persistence options as explained here
Again copying code snippet from link:
<cache>
<persistence strategy=”localRestartable” synchronousWrites=”true”/>
</cache>
Hope this helps!

Should JDBC Blob (not) be free()'d after use?

Just whacking together an export from an old DB that contains binary data, I stumbled over an exception in one of our utility methods:
java.lang.AbstractMethodError: net.sourceforge.jtds.jdbc.BlobImpl.free()
After checking our codebase, I found that utility method was never used until now, bascially it looks like this:
public BinaryHolder getBinary(final int columnIndex) throws SQLException {
Blob blob = null;
try {
blob = resultSet.getBlob(columnIndex);
final BinaryHolder binary = BinaryHolderUtil.create(blob);
return binary;
} finally {
if (blob != null)
blob.free();
}
}
BinaryHolder is just a wrapper that holdes the binary data (and before you ask, the code executes fine until it reaches the finally clause - BinaryHolderUtil.create(blob) does not attempt to free the blob).
Investigating further I found that everywhere else we access Blob's, the blob is just obtained using getBlob() and not free'd at all (The Javadoc says it will be automatically disposed of when the result set is closed).
Question now: Should the blob be free()'d manually (after all the ResultSet may be held for more than just accessing the blob), and if yes how can it be free()'d in a way that works even with a driver that does not implement it?
(We are using SQL-Server with JTDS1.25, if that wasn't already obvious from the exception)

The Blob.free() was introduced in JDBC 4.0 / Java 6. So you are most likely using a JDBC 3.0 or earlier JDBC driver.
As with most (JDBC) resources, closing them as soon as possible has its advantages (eg GC can collect it earlier, database resources are freed etc). That is also why you can close a ResultSet even though it is closed when you close the statement (or execute the statement again), just like you can close a Statement even though it is closed when the Connection is closed.
So a Blob does not need to be freed, but it is - in general - a good idea to free it when you are done with it.
BTW: JTDS is only JDBC 3.0, you would be better off using the Microsoft SQL Server JDBC driver of Microsoft itself.

how to copy a schema in mysql using java

in my application i need to copy a schema with its tables and store procedures from a base schemn to a new schema.
i am looking for a way to implement this.
i looked into exacting the mysqldump using cmd however it is not a good solution because i have a client side application and this requires an instillation of the server on the client side.
the other option is my own implantation using show query.
the problem here is that i need t implement it all from scratch and the must problematic part is that i will need to arrange the order of the tables according to there foreign key (because if there is a foreign key in the table, the table i am pointing to needs to be created first).
i also thought of creating a store procedure to do this but store procedures in my SQL cant access the disk.
perhaps someone has an idea on how this can be implemented in another way?

You can try using the Apache ddlutils. There is a way to export the ddls from a database to an xml file and re-import it back.
The api usage page has examples on how to export schema to an xml file, read from xml file and apply it to a new database. I have reproduced those functions below along with a small snippet on how to use it to accomplish what you are asking for. You can use this as starting point and optimize it further.
DataSource sourceDb;
DataSource targetDb;
writeDatabaseToXML(readDatabase(sourceDb), "database-dump.xml");
changeDatabase(targetDb,readDatabaseFromXML("database-dump.xml"));
public Database readDatabase(DataSource dataSource)
{
Platform platform = PlatformFactory.createNewPlatformInstance(dataSource);
return platform.readModelFromDatabase("model");
}
public void writeDatabaseToXML(Database db, String fileName)
{
new DatabaseIO().write(db, fileName);
}
public Database readDatabaseFromXML(String fileName)
{
return new DatabaseIO().read(fileName);
}
public void changeDatabase(DataSource dataSource,
Database targetModel)
{
Platform platform = PlatformFactory.createNewPlatformInstance(dataSource);
platform.createTables(targetModel, true, false);
}

You can use information_schema to fetch the foreign key information and build a dependency tree. Here is an example.
But I think you are trying to solve something that has been solved many times before. I'm not familiar with Java, but there are ORM tools (for Python at least) that can inspect your current database and create a complementing model in Java (or Python). Then you can deploy that model into another database.