Develop Reference

Java Completable Future - better to have loop of futures or future of a loop?

I need to run a method hundreds of times for different data, at various points of the method its waiting on data from DB or response from web call. It seems to make sense to run this asynchronously so that processing can occur during wait times, however I must wait on the result returning from all the runs before moving on, my question is what's the difference between:
Creating a single completableFuture and running the loop within this. Then ensuring the completableFuture has finished before moving on.
OR
Creating a loop of completable futures each with a single method call, then using allOf to then wait on the last one finishing.
Thanks

Running the for loop inside CompletableFututre is not a good idea because that for loop will be executed synchronously. Having multiple CompletableFuture that call the same method multiple times is a better idea but you should make sure that all methods that are blocking are executed asynchronously.
List<CompletableFuture> futures = Arrays.asList("1", "2", "3")
.stream()
.map(a -> CompletableFuture.supplyAsync(() -> method1(),
executorService))
.map(a -> a.thenCompose(b -> CompletableFuture.supplyAsync(() -> dbcall(b),
dbExecutorService)))
.collect(Collectors.toList());
This way method1 and dbcall are executed on different ExecutorService and blocking call to DB on dbcall method in dbExecutorService does not lead to threads being exhausted in executorService.

Project Loom
This work will be simpler when Project Loom technology arrives in Java.
This project is adding virtual threads (fibers) to the Java concurrency toolbox. Many running virtual threads can be mapped to run on top of platform/kernel threads. When a virtual thread blocks, it is “parked”, and another virtual thread is assigned to execute on the platform/kernel thread. This switching between virtual threads is done very quickly, making thread-blocking extremely cheap in terms of its impact on performance.
Virtual threads are also extremely cheap in terms of its use of memory. Whereas platform/kernel threads are allocated rather large stack sizes no matter the need, virtual threads have a stack that expands as needed… and shrinks when no longer needed.
Virtual threads promise to eliminate the risks of using thread pools. Every thread is fresh with its own ThreadLocal values.
Experimental builds based on early-access Java 17 are available now. The Project Loom team is soliciting feedback.
AutoCloseable
In Loom, the ExecutorService interface becomes AutoCloseable. So we can use try-with-resources syntax. The flow-of-control leaves the try block only after all submitted tasks are done/failed/canceled. When leaving the try block, the executor service is automatically closed.
No need for CompletableFuture
You can simply launch many virtual threads, millions even, and let them run. Most of the need for the many methods on CompletableFuture evaporates. For more info, see the more recent presentations and interviews with Ron Pressler of Oracle.
We can simply spin off all the tasks on virtual threads, collecting Future objects along the way. Then simply wait for all those tasks to finish.
Example code
Establish an ExecutorService instance. Submit to that executor service your Callable tasks. Capture the returned Future objects to track successful completion.
int countTasks = 1_000 ; // Number of tasks to spin off into threads.
List < Future < YourResultClass > > futures = new ArrayList <>( countTasks );
try (
ExecutorService executorService = Executors.newVirtualThreadExecutor() ;
)
{
for ( int i = 0 ; i < countTasks ; i++ )
{
// Submit a Callable object to the executor service.
Future < YourResultClass > future = executorService.submit( ( ) -> {
// The work to be done in each thread.
YourResultClass yourResultObject = … ;
return yourResultObject;
} );
futures.add( future );
}
}
// At this point, flow-of-control blocks until all submitted tasks are done/failed/canceled.
// After this point, the executor service is automatically closed.
After the work is done, we can example the collected Future objects to verify results.
// Report on all the futures, all the results of the thousand tasks.
for ( Future < YourResultClass > future : futures )
{
try
{
System.out.println(
"future.isDone(): " + ( future.isDone() + " | future.isCompletedNormally(): " + future.isCompletedNormally() + " | future.isCancelled(): " + future.isCancelled() + " | result: " + future.get().toString() )
);
}
catch ( InterruptedException e )
{
e.printStackTrace();
}
catch ( ExecutionException e )
{
e.printStackTrace();
}
}
Example app
Here is a complete app. Not what I would do in production of course, but it makes for a decent demonstration I hope.
This code spins off a thousand tasks. Each task makes a REST call to ask Wikipedia for a random page. The contents of that page are then written to a H2 database. We collect the Future objects returned when submitting Callable tasks to the executor service, and we examine those after the work is done.
I configured an in-memory database, but you could just as well put the database in storage.
For simplicity, I defined my WikipediaPage class as a record. This new feature in Java 16 is a brief way to write a class whose main purpose is to immutable and transparently carry data. The compiler implicitly creates the constructor, getters, equals & hashCode, and toString. Not important to this Answer; you could just as well use a conventional class.
package work.basil.example.loopingfutures;
import org.h2.jdbcx.JdbcDataSource;
import javax.sql.DataSource;
import java.io.IOException;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.sql.*;
import java.time.*;
import java.util.ArrayList;
import java.util.List;
import java.util.Objects;
import java.util.UUID;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
public class App
{
public static void main ( String[] args )
{
System.out.println( Runtime.version() );
System.out.println( Runtime.getRuntime().maxMemory() );
App app = new App();
app.demo();
}
private void demo ( )
{
DataSource dataSource = this.configureDataSource();
this.destroyDatabaseContents( dataSource );
this.createDatabase( dataSource );
this.work( dataSource , 1_000 );
this.dumpTable( dataSource );
}
private DataSource configureDataSource ( )
{
JdbcDataSource dataSource = Objects.requireNonNull( new JdbcDataSource() ); // Implementation of `DataSource` bundled with H2.
dataSource.setURL( "jdbc:h2:mem:looping_loom_example_db;DB_CLOSE_DELAY=-1" ); // Set `DB_CLOSE_DELAY` to `-1` to keep in-memory database in existence after connection closes.
dataSource.setUser( "scott" );
dataSource.setPassword( "tiger" );
return dataSource;
}
private void destroyDatabaseContents ( DataSource dataSource )
{
try (
Connection conn = dataSource.getConnection() ;
)
{
String sql = """
DROP TABLE IF EXISTS wikipedia_page_
;
""";
System.out.println( "sql: \n" + sql );
try ( Statement stmt = conn.createStatement() ; )
{
stmt.execute( sql );
}
}
catch ( SQLException e )
{
e.printStackTrace();
}
}
private void createDatabase ( final DataSource dataSource )
{
try (
Connection conn = dataSource.getConnection() ;
)
{
String sql = """
CREATE TABLE wikipedia_page_
(
id_ UUID NOT NULL PRIMARY KEY ,
url_ VARCHAR NOT NULL ,
content_ CLOB NOT NULL ,
when_fetched_ TIMESTAMP WITH TIME ZONE NOT NULL ,
row_created_ TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP()
)
;
""";
System.out.println( "sql: \n" + sql );
try ( Statement stmt = conn.createStatement() ; )
{
stmt.execute( sql );
}
}
catch ( SQLException e )
{
e.printStackTrace();
}
}
private void work ( final DataSource dataSource , final int countPagesToFetchFromWikipedia )
{
List < Future < WikipediaPage > > futures = new ArrayList <>( countPagesToFetchFromWikipedia );
long start = System.nanoTime();
try (
ExecutorService executorService = newVirtualThreadExecutor() ;
)
{
for ( int i = 0 ; i < countPagesToFetchFromWikipedia ; i++ )
{
// Submit a Callable object to the executor service.
Future < WikipediaPage > future = executorService.submit( ( ) -> {
// To meet Wikipedia's limit of 200 requests per second, let's throttle by sleeping the worker thread.
try {Thread.sleep( Duration.ofMillis( 100 ) ); } catch ( InterruptedException e ) { e.printStackTrace(); }
WikipediaPage page = this.fetchPage();
this.persistPage( dataSource , page );
return page;
} );
futures.add( future );
}
}
// At this point, flow-of-control blocks until all submitted tasks are done/failed/canceled. The executor service is automatically closed.
Duration duration = Duration.ofNanos( System.nanoTime() - start );
System.out.println( "duration = " + duration + " for a count of " + countPagesToFetchFromWikipedia );
// Report on all the futures, all the results of the thousand tasks.
for ( Future < WikipediaPage > future : futures )
{
try
{
System.out.println(
"future.isDone(): " + ( future.isDone() + " | future.isCompletedNormally(): " + future.isCompletedNormally() + " | future.isCancelled(): " + future.isCancelled() + " | result: " + future.get().toString() )
);
}
catch ( InterruptedException e )
{
e.printStackTrace();
}
catch ( ExecutionException e )
{
e.printStackTrace();
}
}
System.out.println( "INFO - End of `work` method. Message # 26893b25-b09c-40d5-8cee-60e6a1d53852." );
}
private WikipediaPage fetchPage ( )
{
WikipediaPage page = null; // To be returned.
HttpClient client =
HttpClient
.newBuilder()
.followRedirects( HttpClient.Redirect.NORMAL )
.build();
HttpRequest request =
HttpRequest
.newBuilder()
.uri( URI.create( "https://en.wikipedia.org/api/rest_v1/page/random/summary" ) )
.build();
try
{
HttpResponse < String > response = client.send( request , HttpResponse.BodyHandlers.ofString() );
OffsetDateTime whenFetched = OffsetDateTime.now( ZoneOffset.UTC );
URI uri = response.uri();
String content = response.body();
System.out.println( "response = " + response );
System.out.println( "content = " + content );
page = new WikipediaPage( UUID.randomUUID() , uri.toString() , content , whenFetched );
}
catch ( IOException e )
{
e.printStackTrace();
}
catch ( InterruptedException e )
{
e.printStackTrace();
}
return Objects.requireNonNull( page );
}
private void persistPage ( final DataSource dataSource , final WikipediaPage wikipediaPage )
{
Objects.requireNonNull( dataSource );
Objects.requireNonNull( wikipediaPage );
String sql = """
INSERT INTO wikipedia_page_ ( id_ , url_ , content_ , when_fetched_ )
VALUES ( ? , ? , ? , ? )
;
""";
System.out.println( "sql: \n" + sql );
try (
Connection conn = dataSource.getConnection() ;
PreparedStatement pstmt = Objects.requireNonNull( conn ).prepareStatement( sql ) ;
)
{
pstmt.setObject( 1 , wikipediaPage.id );
pstmt.setString( 2 , wikipediaPage.url );
pstmt.setString( 3 , wikipediaPage.content );
pstmt.setObject( 4 , wikipediaPage.whenFetched );
int countRowsAffected = pstmt.executeUpdate();
if ( countRowsAffected != 1 )
{
System.out.println( "ERROR - Failed to insert row. Message # 4c3503d9-8cad-4e21-a625-c58054d9ca78." );
}
}
catch ( SQLException e )
{
e.printStackTrace();
}
}
private void dumpTable ( final DataSource dataSource )
{
String sql = "SELECT * FROM wikipedia_page_ ;";
System.out.println( "sql: \n" + sql );
try (
Connection conn = dataSource.getConnection() ;
Statement stmt = conn.createStatement() ;
ResultSet rs = stmt.executeQuery( sql ) ;
)
{
while ( rs.next() )
{
//Retrieve by column name
UUID id = rs.getObject( "id_" , UUID.class );
String url = rs.getString( "url_" );
String content = rs.getString( "content_" );
OffsetDateTime whenFetched = rs.getObject( "when_fetched_" , OffsetDateTime.class );
WikipediaPage wp = new WikipediaPage( id , url , content , whenFetched );
System.out.println( "wp = " + wp );
}
}
catch ( SQLException e )
{
e.printStackTrace();
}
}
record WikipediaPage(UUID id , String url , String content , OffsetDateTime whenFetched)
{
#Override
public String toString ( )
{
// Omitting the `content` field for brevity.
return "WikipediaPage{ " +
"id=" + id +
" | url='" + url + '\'' +
" | whenFetched=" + whenFetched +
" }";
}
}
}
Results
All of these results are for 1,000 tasks. Run on a Mac mini (2018) 3 GHz Intel Core i5 with 32 gigs of RAM, macOS Mojave 10.14.6, Java 17-loom+2-42 assigned maximum memory of 8 gigs (8589934592).
When running with Executors.newSingleThreadExecutor(), takes 7 minutes.
When running with Executors.newFixedThreadPool( 10 ), takes 1 minute.
When running with Executors.newFixedThreadPool( 100 ), takes half a minute.
When running with Executors.newVirtualThreadExecutor(), takes a quarter minute.

Java flight recorder profiling summary extraction

I would like to be able to extract, in a textual format, some kind of a summary of the 'hot spots' detected by analyzing a JFR recording. To be precise, I would like to extract the hot-spots resulting from running some specific functions or some specific class. Something like this:
<some tool> myfile.jfr --look-at-class='org/me/MyClass' --limit 10 --order-by self-time
And obtain a table of the most time-consuming 10 methods methods called from MyClass in org.me. I tried looking at the jfr command-line tool, but it does not have such functionality. Alternatively, JMC only has a graphical interface, but not a command-line interface. Is there another way to obtain such a result?

It's easy to create such a tool using JFR Parsing API.
import jdk.jfr.consumer.RecordingFile;
import java.nio.file.Paths;
import java.util.HashMap;
public class JfrTop {
public static void main(String[] args) throws Exception {
var fileName = args[0];
var packageName = args.length > 1 ? args[1] : "";
var top = args.length > 2 ? Integer.parseInt(args[2]) : 10;
var hotMethods = new HashMap<String, Long>();
long total = 0;
try (var recording = new RecordingFile(Paths.get(fileName))) {
while (recording.hasMoreEvents()) {
var event = recording.readEvent();
if (event.getEventType().getName().equals("jdk.ExecutionSample")) {
var stackTrace = event.getStackTrace();
if (stackTrace != null && stackTrace.getFrames().size() > 0) {
var method = stackTrace.getFrames().get(0).getMethod();
var className = method.getType().getName();
if (className.startsWith(packageName)) {
var fullName = className + '.' + method.getName() + method.getDescriptor();
hotMethods.compute(fullName, (key, val) -> val == null ? 1L : (val + 1));
}
}
total++;
}
}
}
double percent = 100.0 / total;
hotMethods.entrySet().stream()
.sorted((e1, e2) -> Long.compare(e2.getValue(), e1.getValue()))
.limit(top)
.forEachOrdered(e -> System.out.printf("%5.2f%% %s\n", e.getValue() * percent, e.getKey()));
}
}
How to run:
java JfrTop idea.jfr com.intellij.openapi.editor 10
Sample output:
20,35% com.intellij.openapi.editor.impl.RangeHighlighterTree$RHNode.recalculateRenderFlags()V
4,20% com.intellij.openapi.editor.impl.IntervalTreeImpl.maxEndOf(Lcom/intellij/openapi/editor/impl/IntervalTreeImpl$IntervalNode;I)I
3,19% com.intellij.openapi.editor.impl.IntervalTreeImpl.assertAllDeltasAreNull(Lcom/intellij/openapi/editor/impl/IntervalTreeImpl$IntervalNode;)V
2,19% com.intellij.openapi.editor.impl.IntervalTreeImpl$IntervalNode.computeDeltaUpToRoot()I
1,94% com.intellij.openapi.editor.impl.IntervalTreeImpl.pushDelta(Lcom/intellij/openapi/editor/impl/IntervalTreeImpl$IntervalNode;)Z
1,63% com.intellij.openapi.editor.impl.IntervalTreeImpl$IntervalNode.hasAliveKey(Z)Z
1,50% com.intellij.openapi.editor.impl.IntervalTreeImpl.correctMax(Lcom/intellij/openapi/editor/impl/IntervalTreeImpl$IntervalNode;I)V
1,31% com.intellij.openapi.editor.impl.IntervalTreeImpl$1.hasNext()Z
0,88% com.intellij.openapi.editor.impl.IntervalTreeImpl$IntervalNode.tryToSetCachedValues(IZI)Z
0,63% com.intellij.openapi.editor.impl.IntervalTreeImpl.findOrInsert(Lcom/intellij/openapi/editor/impl/IntervalTreeImpl$IntervalNode;)Lcom/intellij/openapi/editor/impl/IntervalTreeImpl$IntervalNode;

Java CSV Writing

Im currently trying to write data into excel for a report. I can write data to the csv file however its not coming out in excel in the order I want. I need the data to print under best and worst fitness in each column instead of it all print under Average. Here is the relevant code, any help would be appreciated:
String [] Fitness = "Average fitness#Worst fitness #Best Fitness".split("#");
writer.writeNext(Fitness);
//takes data from average fitness and stores as an int
int aFit = myPop.individuals[25].getFitness();
//converts int to string
String aFit1 = Integer.toString(aFit);
//converts string to string array
String aFit2 [] = aFit1.split(" ");
//writes to csv
writer.writeNext(aFit2);
//String [] nextCol = "#".split("#");
int wFit = myPop.individuals[49].getFitness();
String wFit1 = Integer.toString(wFit);
String wFit2 [] = wFit1.split(" ");
writer.writeNext(wFit2);
int bFit = myPop.individuals[1].getFitness();
String bFit1 = Integer.toString(bFit);
String bFit2 [] = bFit1.split(" ");
writer.writeNext(bFit2);
enter image description here

I think you should call your "writeNext" method once per line of datas:
String [] Fitness = "Average fitness#Worst fitness #Best Fitness".split("#");
writer.writeNext(Fitness);
int aFit = myPop.individuals[25].getFitness();
String aFit1 = Integer.toString(aFit);
int wFit = myPop.individuals[49].getFitness();
String wFit1 = Integer.toString(wFit);
int bFit = myPop.individuals[1].getFitness();
String bFit1 = Integer.toString(bFit);
writer.writeNext(new String[]{aFit1, wFit1, bFit1});

From the docs at
CSVWriter.html#writeNext(java.lang.String[])
public void writeNext(String[] nextLine)
- Writes the next line to the file.
The String array to provide is
A string array with each comma-separated element as a separate entry.
You are writing 3 separate lines instead of 1 and each line you write contains an Array with a single entry.
writer.writeNext(aFit2);
writer.writeNext(wFit2);
writer.writeNext(bFit2);
Solution:
Create a single Array with all 3 entries (column values) and write that once on a single line.

I am assuming you are using CSVWriter to write to a CSV file. Please make sure to mention as much details as possible in a question, it makes it much more readable to others.
As you can see from the documentation of CSVWriter:
void writeNext(String[] nextLine)
Writes the next line to the file.
The writeNext method actually writes the array to the an individual line of the file. From your code:
writer.writeNext(aFit2);
writer.writeNext(wFit2);
writer.writeNext(bFit2);
So, instead of doing this `String aFit2 [] = aFit1.split(" ");
Create an array of the values and then pass that array to writeNext
As an example, you can consider you own example of passing the array of column names, which gets written in a single line:
writer.writeNext(Fitness);

Apache Commons CSV
Here is the same kind of solution, but using the Apache Commons CSV library. This library specifically supports the Microsoft Excel variant of CSV format, so you may find it particularly useful.
CSVFormat.Predefined.EXCEL
Your data, both read and written in this example.
The Commons CSV library can read the first row as header names.
Here is a complete example app in a single .java file. First the app reads from an existing WorstBest.csv data file:
Average,Worst,Best
10,5,15
11,5,16
10,6,16
11,6,15
10,5,16
10,5,16
10,4,16
Each row is represented as a List of three String objects, a List< String >. We add each row to a collection, a list of lists, a List< List< String > >.
Then we write out that imported data to another file. Each written file is name WorstBest_xxx.csv where xxx is the current moment in UTC.
package com.basilbourque.example;
import org.apache.commons.csv.CSVFormat;
import org.apache.commons.csv.CSVPrinter;
import org.apache.commons.csv.CSVRecord;
import java.io.BufferedReader;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.time.Instant;
import java.time.temporal.ChronoUnit;
import java.util.ArrayList;
import java.util.List;
import java.util.Objects;
public class WorstBest {
public static void main ( String[] args ) {
WorstBest app = new WorstBest();
List < List < String > > data = app.read();
app.write( data );
}
private List < List < String > > read ( ) {
List < List < String > > listOfListsOfStrings = List.of();
try {
// Locate file to import and parse.
Path path = Paths.get( "/Users/basilbourque/WorstBest.csv" );
if ( Files.notExists( path ) ) {
System.out.println( "ERROR - no file found for path: " + path + ". Message # 3cf416de-c33b-4c39-8507-5fbb72e113f2." );
}
// Hold data read from file.
int initialCapacity = ( int ) Files.lines( path ).count();
listOfListsOfStrings = new ArrayList <>( initialCapacity );
// Read CSV file.
BufferedReader reader = Files.newBufferedReader( path );
Iterable < CSVRecord > records = CSVFormat.RFC4180.withFirstRecordAsHeader().parse( reader );
for ( CSVRecord record : records ) {
// Average,Worst,Best
// 10,5,15
// 11,5,16
String average = record.get( "Average" ); // Must use annoying zero-based index counting.
String worst = record.get( "Worst" );
String best = record.get( "Best" );
// Collect
listOfListsOfStrings.add( List.of( average , worst , best ) ); // For real work, you would define a class to hold these values.
}
} catch ( IOException e ) {
e.printStackTrace();
}
return listOfListsOfStrings;
}
private void write ( List < List < String > > listOfListsOfStrings ) {
Objects.requireNonNull( listOfListsOfStrings );
// Determine file in which to write data.
String when = Instant.now().truncatedTo( ChronoUnit.SECONDS ).toString().replace( ":" , "•" ); // Colons are forbidden in names by some file systems such as HFS+.
Path path = Paths.get( "/Users/basilbourque/WorstBest_" + when + ".csv" );
// Loop collection of data (a list of lists of strings).
try ( final CSVPrinter printer = CSVFormat.EXCEL.withHeader( "Average" , "Worst" , "Best" ).print( path , StandardCharsets.UTF_8 ) ; ) {
for ( List < String > list : listOfListsOfStrings ) {
printer.printRecord( list.get( 1 - 1 ) , list.get( 2 - 1 ) , list.get( 3 - 1 ) ); // Annoying zero-based index counting.
}
} catch ( IOException e ) {
e.printStackTrace();
}
}
}

How to report memory leaks in Android/Java using PhantomReference

After reconstructing an old piece of code I once wrote, then forgot, now rewritten... I am putting it here as a wiki for all to use :-)
So, basically: If you got memory leaks in a complex Android app, containing images and cross-references. How would you go and find which (type of) objects are leaking? There are a few (very hard to learn and use) tools provided with the Android SDK. Probably there are more which I don't know. Yet, Java do provide PhantomReference as a mean to do this, even though going through the mess required to set up the required classes can be much work (And nasty too... JDK-8034946).
But what is the most simple/effective way of doing so? My solution below.

LeakCanary is a 3rd party library which automatically detects memory leaks, after adding the dependency you can add the following line to your application class:
LeakCanary.install(this);
The library provides a nice notification & trace of the leak, you can also define your own reference watchers (although the default ones seem to work fairly well).

My solution in one class: "MemCheck"
To monitor any object, just call:
MemCheck.add( this ); // in any class constructor
This will "monitor" the number of objects allocated, and most importantly - Deallocated.
To log the leaks at any required time, call:
MemCheck.countAndLog();
As an alternative, set MemCheck.periodic = 5 (number of seconds).
This will report the number of monitored objects in memory every 5 seconds. It will also conveniently log the used/free memory.
So, MemCheck.java:
package com.xyz.util;
import java.lang.ref.PhantomReference;
import java.lang.ref.ReferenceQueue;
import java.util.HashSet;
import java.util.Iterator;
import java.util.Map.Entry;
import java.util.TreeMap;
import android.os.Handler;
import android.util.Log;
public class MemCheck
{
private static final boolean enabled = true;
private static final int periodic = 0; // seconds, 0 == disabled
private static final String tag = MemCheck.class.getName();
private static TreeMap<String, RefCount> mObjectMap = new TreeMap<String, RefCount>();
private static Runnable mPeriodicRunnable = null;
private static Handler mPeriodicHandler = null;
public static void add( Object object )
{
if( !enabled )
return;
synchronized( mObjectMap )
{
String name = object.getClass().getName();
RefCount queue = mObjectMap.get( name );
if( queue == null )
{
queue = new RefCount();
mObjectMap.put( name, queue );
queue.add( object );
}
else
queue.add( object );
}
}
public static void countAndLog()
{
if( !enabled )
return;
System.gc();
Log.d( tag, "Log report starts" );
Iterator<Entry<String, RefCount>> entryIter = mObjectMap.entrySet().iterator();
while( entryIter.hasNext() )
{
Entry<String, RefCount> entry = entryIter.next();
String name = entry.getKey();
RefCount refCount = entry.getValue();
Log.d( tag, "Class " + name + " has " + refCount.countRefs() + " objects in memory." );
}
logMemoryUsage();
Log.d( tag, "Log report done" );
}
public static void logMemoryUsage()
{
if( !enabled )
return;
Runtime runtime = Runtime.getRuntime();
Log.d( tag, "Max Heap: " + runtime.maxMemory() / 1048576 + " MB, Used: " + runtime.totalMemory() / 1048576 +
" MB, Free: " + runtime.freeMemory() / 1024 + " MB" );
if( periodic > 0 )
{
if( mPeriodicRunnable != null )
mPeriodicHandler.removeCallbacks( mPeriodicRunnable );
if( mPeriodicHandler == null )
mPeriodicHandler = new Handler();
mPeriodicRunnable = new Runnable()
{
#Override
public void run()
{
mPeriodicRunnable = null;
countAndLog();
logMemoryUsage(); // this will run the next
}
};
mPeriodicHandler.postDelayed( mPeriodicRunnable, periodic * 1000 );
}
}
private static class RefCount
{
private ReferenceQueue<Object> mQueue = new ReferenceQueue<Object>();
private HashSet<Object> mRefHash = new HashSet<Object>();
private int mRefCount = 0;
public void add( Object o )
{
synchronized( this )
{
mRefHash.add( new PhantomReference<Object>( o, mQueue ) ); // Note: References MUST be kept alive for their references to be enqueued
mRefCount++;
}
}
public int countRefs()
{
synchronized( this )
{
Object ref;
while( ( ref = mQueue.poll() ) != null )
{
mRefHash.remove( ref );
mRefCount--;
}
return mRefCount;
}
}
}
}

Fast CSV parsing

I have a java server app that download CSV file and parse it. The parsing can take from 5 to 45 minutes, and happens each hour.This method is a bottleneck of the app so it's not premature optimization. The code so far:
client.executeMethod(method);
InputStream in = method.getResponseBodyAsStream(); // this is http stream
String line;
String[] record;
reader = new BufferedReader(new InputStreamReader(in), 65536);
try {
// read the header line
line = reader.readLine();
// some code
while ((line = reader.readLine()) != null) {
// more code
line = line.replaceAll("\"\"", "\"NULL\"");
// Now remove all of the quotes
line = line.replaceAll("\"", "");
if (!line.startsWith("ERROR"){
//bla bla
continue;
}
record = line.split(",");
//more error handling
// build the object and put it in HashMap
}
//exceptions handling, closing connection and reader
Is there any existing library that would help me to speed up things? Can I improve existing code?

Apache Commons CSV
Have you seen Apache Commons CSV?
Caveat On Using split
Bear in mind is that split only returns a view of the data, meaning that the original line object is not eligible for garbage collection whilst there is a reference to any of its views. Perhaps making a defensive copy will help? (Java bug report)
It also is not reliable in grouping escaped CSV columns containing commas

opencsv
Take a look at opencsv.
This blog post, opencsv is an easy CSV parser, has example usage.

The problem of your code is that it's using replaceAll and split which are very costly operation. You should definitely consider using a csv parser/reader that would do a one pass parsing.
There is a benchmark on github
https://github.com/uniVocity/csv-parsers-comparison
that unfortunately is ran under java 6. The number are slightly different under java 7 and 8. I'm trying to get more detail data for different file size but it's work in progress
see https://github.com/arnaudroger/csv-parsers-comparison

Apart from the suggestions made above, I think you can try improving your code by using some threading and concurrency.
Following is the brief analysis and suggested solution
From the code it seems that you are reading the data over the network (most possibly apache-common-httpclient lib).
You need to make sure that bottleneck that you are saying is not in the data transfer over the network.
One way to see is just dump the data in some file (without parsing) and see how much does it take. This will give you an idea how much time is actually spent in parsing (when compared to current observation).
Now have a look at how java.util.concurrent package is used. Some of the link that you can use are (1,2)
What you ca do is the tasks that you are doing in for loop can be executed in a thread.
Using the threadpool and concurrency will greatly improve your performance.
Though the solution involves some effort, but at the end this will surly help you.

opencsv
You should have a look at OpenCSV. I would expect that they have performance optimizations.

A little late here, there is now a few benchmarking projects for CSV parsers. Your selection will depend on the exact use-case (i.e. raw data vs data binding etc).
SimpleFlatMapper
uniVocity
sesseltjonna-csv (disclaimer: I wrote this parser)

Quirk-CSV
The new kid on the block. It uses java annotations and is built on apache-csv which one of the faster libraries out there for csv parsing.
This library is also thread safe as well if you wanted to re-use the CSVProcessor you can and should.
Example:
Pojo
#CSVReadComponent(type = CSVType.NAMED)
#CSVWriteComponent(type = CSVType.ORDER)
public class Pojo {
#CSVWriteBinding(order = 0)
private String name;
#CSVWriteBinding(order = 1)
#CSVReadBinding(header = "age")
private Integer age;
#CSVWriteBinding(order = 2)
#CSVReadBinding(header = "money")
private Double money;
#CSVReadBinding(header = "name")
public void setA(String name) {
this.name = name;
}
#Override
public String toString() {
return "Name: " + name + System.lineSeparator() + "\tAge: " + age + System.lineSeparator() + "\tMoney: "
+ money;
}}
Main
import java.io.IOException;
import java.io.StringReader;
import java.io.StringWriter;
import java.util.*;
public class SimpleMain {
public static void main(String[] args) {
String csv = "name,age,money" + System.lineSeparator() + "Michael Williams,34,39332.15";
CSVProcessor processor = new CSVProcessor(Pojo.class);
List<Pojo> list = new ArrayList<>();
try {
list.addAll(processor.parse(new StringReader(csv)));
list.forEach(System.out::println);
System.out.println();
StringWriter sw = new StringWriter();
processor.write(list, sw);
System.out.println(sw.toString());
} catch (IOException e) {
}
}}
Since this is built on top of apache-csv you can use the powerful tool CSVFormat. Lets say the delimiter for the csv are pipes (|) instead of commas(,) you could for Example:
CSVFormat csvFormat = CSVFormat.DEFAULT.withDelimiter('|');
List<Pojo> list = processor.parse(new StringReader(csv), csvFormat);
Another benefit are inheritance is also consider.
For other examples on handling reading/writing non-primitive data

For speed you do not want to use replaceAll, and you don't want to use regex either. What you basically always want to do in critical cases like that is making a state-machine character by character parser. I've done that having rolled the whole thing into an Iterable function. It also takes in the stream and parses it without saving it out or caching it. So if you can abort early that's likely going to go fine as well. It should also be short enough and well coded enough to make it obvious how it works.
public static Iterable<String[]> parseCSV(final InputStream stream) throws IOException {
return new Iterable<String[]>() {
#Override
public Iterator<String[]> iterator() {
return new Iterator<String[]>() {
static final int UNCALCULATED = 0;
static final int READY = 1;
static final int FINISHED = 2;
int state = UNCALCULATED;
ArrayList<String> value_list = new ArrayList<>();
StringBuilder sb = new StringBuilder();
String[] return_value;
public void end() {
end_part();
return_value = new String[value_list.size()];
value_list.toArray(return_value);
value_list.clear();
}
public void end_part() {
value_list.add(sb.toString());
sb.setLength(0);
}
public void append(int ch) {
sb.append((char) ch);
}
public void calculate() throws IOException {
boolean inquote = false;
while (true) {
int ch = stream.read();
switch (ch) {
default: //regular character.
append(ch);
break;
case -1: //read has reached the end.
if ((sb.length() == 0) && (value_list.isEmpty())) {
state = FINISHED;
} else {
end();
state = READY;
}
return;
case '\r':
case '\n': //end of line.
if (inquote) {
append(ch);
} else {
end();
state = READY;
return;
}
break;
case ',': //comma
if (inquote) {
append(ch);
} else {
end_part();
break;
}
break;
case '"': //quote.
inquote = !inquote;
break;
}
}
}
#Override
public boolean hasNext() {
if (state == UNCALCULATED) {
try {
calculate();
} catch (IOException ex) {
}
}
return state == READY;
}
#Override
public String[] next() {
if (state == UNCALCULATED) {
try {
calculate();
} catch (IOException ex) {
}
}
state = UNCALCULATED;
return return_value;
}
};
}
};
}
You would typically process this quite helpfully like:
for (String[] csv : parseCSV(stream)) {
//<deal with parsed csv data>
}
The beauty of that API there is worth the rather cryptic looking function.

Apache Commons CSV ➙ 12 seconds for million rows
Is there any existing library that would help me to speed up things?
Yes, the Apache Commons CSV project works very well in my experience.
Here is an example app that uses Apache Commons CSV library to write and read rows of 24 columns: An integer sequential number, an Instant, and the rest are random UUID objects.
For 10,000 rows, the writing and the read each take about half a second. The reading includes reconstituting the Integer, Instant, and UUID objects.
My example code lets you toggle on or off the reconstituting of objects. I ran both with a million rows. This creates a file of 850 megs. I am using Java 12 on a MacBook Pro (Retina, 15-inch, Late 2013), 2.3 GHz Intel Core i7, 16 GB 1600 MHz DDR3, Apple built-in SSD.
For a million rows, ten seconds for reading plus two seconds for parsing:
Writing: PT25.994816S
Reading only: PT10.353912S
Reading & parsing: PT12.219364S
Source code is a single .java file. Has a write method, and a read method. Both methods called from a main method.
I opened a BufferedReader by calling Files.newBufferedReader.
package work.basil.example;
import org.apache.commons.csv.CSVFormat;
import org.apache.commons.csv.CSVParser;
import org.apache.commons.csv.CSVPrinter;
import org.apache.commons.csv.CSVRecord;
import java.io.*;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.time.Duration;
import java.time.Instant;
import java.util.UUID;
public class CsvReadingWritingDemo
{
public static void main ( String[] args )
{
CsvReadingWritingDemo app = new CsvReadingWritingDemo();
app.write();
app.read();
}
private void write ()
{
Instant start = Instant.now();
int limit = 1_000_000; // 10_000 100_000 1_000_000
Path path = Paths.get( "/Users/basilbourque/IdeaProjects/Demo/csv.txt" );
try (
Writer writer = Files.newBufferedWriter( path, StandardCharsets.UTF_8 );
CSVPrinter printer = new CSVPrinter( writer , CSVFormat.RFC4180 );
)
{
printer.printRecord( "id" , "instant" , "uuid_01" , "uuid_02" , "uuid_03" , "uuid_04" , "uuid_05" , "uuid_06" , "uuid_07" , "uuid_08" , "uuid_09" , "uuid_10" , "uuid_11" , "uuid_12" , "uuid_13" , "uuid_14" , "uuid_15" , "uuid_16" , "uuid_17" , "uuid_18" , "uuid_19" , "uuid_20" , "uuid_21" , "uuid_22" );
for ( int i = 1 ; i <= limit ; i++ )
{
printer.printRecord( i , Instant.now() , UUID.randomUUID() , UUID.randomUUID() , UUID.randomUUID() , UUID.randomUUID() , UUID.randomUUID() , UUID.randomUUID() , UUID.randomUUID() , UUID.randomUUID() , UUID.randomUUID() , UUID.randomUUID() , UUID.randomUUID() , UUID.randomUUID() , UUID.randomUUID() , UUID.randomUUID() , UUID.randomUUID() , UUID.randomUUID() , UUID.randomUUID() , UUID.randomUUID() , UUID.randomUUID() , UUID.randomUUID() , UUID.randomUUID() , UUID.randomUUID() );
}
} catch ( IOException ex )
{
ex.printStackTrace();
}
Instant stop = Instant.now();
Duration d = Duration.between( start , stop );
System.out.println( "Wrote CSV for limit: " + limit );
System.out.println( "Elapsed: " + d );
}
private void read ()
{
Instant start = Instant.now();
int count = 0;
Path path = Paths.get( "/Users/basilbourque/IdeaProjects/Demo/csv.txt" );
try (
Reader reader = Files.newBufferedReader( path , StandardCharsets.UTF_8) ;
)
{
CSVFormat format = CSVFormat.RFC4180.withFirstRecordAsHeader();
CSVParser parser = CSVParser.parse( reader , format );
for ( CSVRecord csvRecord : parser )
{
if ( true ) // Toggle parsing of the string data into objects. Turn off (`false`) to see strictly the time taken by Apache Commons CSV to read & parse the lines. Turn on (`true`) to get a feel for real-world load.
{
Integer id = Integer.valueOf( csvRecord.get( 0 ) ); // Annoying zero-based index counting.
Instant instant = Instant.parse( csvRecord.get( 1 ) );
for ( int i = 3 - 1 ; i <= 22 - 1 ; i++ ) // Subtract one for annoying zero-based index counting.
{
UUID uuid = UUID.fromString( csvRecord.get( i ) );
}
}
count++;
if ( count % 1_000 == 0 ) // Every so often, report progress.
{
//System.out.println( "# " + count );
}
}
} catch ( IOException e )
{
e.printStackTrace();
}
Instant stop = Instant.now();
Duration d = Duration.between( start , stop );
System.out.println( "Read CSV for count: " + count );
System.out.println( "Elapsed: " + d );
}
}