Develop Reference

Java serialization vs MsgPack vs Bson

I am working with websockets, i want the process of sending/recieving data be as fast as possible. I have come across BSON and MsgPack libraries for binary serialization. However, using simple tests:
#Message
class MessageTemplate implements Serializable {
public String msg;
}
public class test {
static void start(){
MessageTemplate x = new MessageTemplate();
for (int i = 0; i < 1000; ++i)
x.msg += UUID.randomUUID().toString();
System.out.println("===============================================================");
{
long startTime = System.nanoTime();
byte[] bytes = SerializationUtils.serialize(x);
MessageTemplate x1 = (MessageTemplate) SerializationUtils.deserialize(bytes);
long endTime = System.nanoTime();
long duration = (endTime - startTime);
System.out.println("TIME1:" + String.valueOf(duration) + ", SIZE: " + bytes.length);
}
System.out.println("===============================================================");
MessagePack msgpack = new MessagePack();
{
long startTime = System.nanoTime();
try {
byte[] b = msgpack.write(x);
MessageTemplate dst = msgpack.read(b, MessageTemplate.class);
long endTime = System.nanoTime();
long duration = (endTime - startTime);
System.out.println("TIME1:" + String.valueOf(duration) + ", SIZE: " + b.length);
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
Output:
starting===============================================================
TIME1:2560388, SIZE: 36171
===============================================================
TIME1:93729732, SIZE: 36013
It seems that serialization is way faster than MsgPack. However searching i have found not any mentioning of java serialization as "serializating library/format".
What are the drawbacks of using it? Why is it or isnt used? The only drawback i see is that mobile app will have ios/android client, so there wont be java on both sides in every case.
Thanks for help and answers.

How to Wait for Completion of ANY Worker Thread?

I want a dispatcher thread that executes and retrieves results from a pool of worker threads. The dispatcher needs to continuously feed work to the worker threads. When ANY of the worker thread completes, the dispatcher needs to gather its results and re-dispatch (or create a new) worker thread. It seems to me like this should be obvious but I have been unable to find an example of a suitable pattern. A Thread.join() loop would be inadequate because that is really "AND" logic and I am looking for "OR" logic.
The best I could come up with is to have the dispatcher thread wait() and have the worker threads notify() when they are done. Though seems like I would have to guard against two worker threads that end at the same time causing the dispatcher thread to miss a notify(). Plus, this seems a little bit inelegant to me.
Even less elegant is the idea of the dispatcher thread periodically waking up and polling the worker thread pool and checking each thread to see if it has completed via isAlive().
I took a look at java.util.concurrent and didn't see anything that looked like it fit this pattern.
I feel that to implement what I mention above would involve a lot of defensive programming and reinventing the wheel. There's got to be something that I am missing. What can I leverage to implement this pattern?
This is the single-threaded version. putMissingToS3() would become the dispatcher thread and the capability represented in the uploadFileToBucket() would become the worker thread.
private void putMissingToS3()
{
int reqFilesToUpload = 0;
long reqSizeToUpload = 0L;
int totFilesUploaded = 0;
long totSizeUploaded = 0L;
int totFilesSkipped = 0;
long totSizeSkipped = 0L;
int rptLastFilesUploaded = 0;
long rptSizeInterval = 1000000000L;
long rptLastSize = 0L;
StopWatch rptTimer = new StopWatch();
long rptLastMs = 0L;
StopWatch globalTimer = new StopWatch();
StopWatch indvTimer = new StopWatch();
for (FileSystemRecord fsRec : fileSystemState.toList())
{
String reqKey = PathConverter.pathToKey(PathConverter.makeRelativePath(fileSystemState.getRootPath(), fsRec.getFullpath()));
LocalS3MetadataRecord s3Rec = s3Metadata.getRecord(reqKey);
// Just get a rough estimate of what the size of this upload will be
if (s3Rec == null)
{
++reqFilesToUpload;
reqSizeToUpload += fsRec.getSize();
}
}
long uploadTimeGuessMs = (long)((double)reqSizeToUpload/estUploadRateBPS*1000.0);
printAndLog("Estimated upload: " + natFmt.format(reqFilesToUpload) + " files, " + Utils.readableFileSize(reqSizeToUpload) +
", Estimated time " + Utils.readableElapsedTime(uploadTimeGuessMs));
globalTimer.start();
rptTimer.start();
for (FileSystemRecord fsRec : fileSystemState.toList())
{
String reqKey = PathConverter.pathToKey(PathConverter.makeRelativePath(fileSystemState.getRootPath(), fsRec.getFullpath()));
if (PathConverter.validate(reqKey))
{
LocalS3MetadataRecord s3Rec = s3Metadata.getRecord(reqKey);
//TODO compare and deal with size mismatches. Maybe go and look at last-mod dates.
if (s3Rec == null)
{
indvTimer.start();
uploadFileToBucket(s3, syncParms.getS3Bucket(), fsRec.getFullpath(), reqKey);
indvTimer.stop();
++totFilesUploaded;
totSizeUploaded += fsRec.getSize();
logOnly("Uploaded: Size=" + fsRec.getSize() + ", " + indvTimer.stopDeltaMs() + " ms, File=" + fsRec.getFullpath() + ", toKey=" + reqKey);
if (totSizeUploaded > rptLastSize + rptSizeInterval)
{
long invSizeUploaded = totSizeUploaded - rptLastSize;
long nowMs = rptTimer.intervalMs();
long invElapMs = nowMs - rptLastMs;
long remSize = reqSizeToUpload - totSizeUploaded;
double progessPct = (double)totSizeUploaded/reqSizeToUpload*100.0;
double mbps = (invElapMs > 0) ? invSizeUploaded/1e6/(invElapMs/1000.0) : 0.0;
long remMs = (long)((double)remSize/((double)invSizeUploaded/invElapMs));
printOnly("Progress: " + d2Fmt.format(progessPct) + "%, " + Utils.readableFileSize(totSizeUploaded) + " of " +
Utils.readableFileSize(reqSizeToUpload) + ", Rate " + d3Fmt.format(mbps) + " MB/s, " +
"Time rem " + Utils.readableElapsedTime(remMs));
rptLastMs = nowMs;
rptLastFilesUploaded = totFilesUploaded;
rptLastSize = totSizeUploaded;
}
}
}
else
{
++totFilesSkipped;
totSizeSkipped += fsRec.getSize();
logOnly("Skipped (Invalid chars): Size=" + fsRec.getSize() + ", " + fsRec.getFullpath() + ", toKey=" + reqKey);
}
}
globalTimer.stop();
double mbps = 0.0;
if (globalTimer.stopDeltaMs() > 0)
mbps = totSizeUploaded/1e6/(globalTimer.stopDeltaMs()/1000.0);
printAndLog("Actual upload: " + natFmt.format(totFilesUploaded) + " files, " + Utils.readableFileSize(totSizeUploaded) +
", Time " + Utils.readableElapsedTime(globalTimer.stopDeltaMs()) + ", Rate " + d3Fmt.format(mbps) + " MB/s");
if (totFilesSkipped > 0)
printAndLog("Skipped Files: " + natFmt.format(totFilesSkipped) + " files, " + Utils.readableFileSize(totSizeSkipped));
}
private void uploadFileToBucket(AmazonS3 amazonS3, String bucketName, String filePath, String fileKey)
{
File inFile = new File(filePath);
ObjectMetadata objectMetadata = new ObjectMetadata();
objectMetadata.addUserMetadata(Const.LAST_MOD_KEY, Long.toString(inFile.lastModified()));
objectMetadata.setLastModified(new Date(inFile.lastModified()));
PutObjectRequest por = new PutObjectRequest(bucketName, fileKey, inFile).withMetadata(objectMetadata);
// Amazon S3 never stores partial objects; if during this call an exception wasn't thrown, the entire object was stored.
amazonS3.putObject(por);
}

I think you are at right package. you should use ExecutorService API.
This removes burden of waiting and watching for thread's notification.
Example:
import java.util.concurrent.ExecutorService;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.Executors;
public class ExecutorEx{
static class ThreadA implements Runnable{
int id;
public ThreadA(int id){
this.id = id;
}
public void run(){
//To simulate some work
try{Thread.sleep(Math.round(Math.random()*100));}catch(Exception e){}
// to show message
System.out.println(this.id + "--Test Message" + System.currentTimeMillis());
}
}
public static void main(String args[]) throws Exception{
int poolSize = 10;
ExecutorService pool = Executors.newFixedThreadPool(poolSize);
int i=0;
while(i<100){
pool.submit(new ThreadA(i));
i++;
}
pool.shutdown();
while(!pool.isTerminated()){
pool.awaitTermination(60, TimeUnit.SECONDS);
}
}
}
And if you want to return something from your thread will need to implement Callable instead of Runnable(call() instead of run()) and collect returned values in Future object array, that you can iterate over later.

Extract icons from files takes too long

I need to display lots of files with filename and icon in my program.
Therefor I am extracting the icons from the files, but it takes too long.
I have tried 2 different methods to extract the icons, but both are really slow (in my case REALLY slow, because I get the files from a networkdrive).
Here is an example, where I extract the icons and count the number of icons (do nothing with the files/icons)
public class Main {
public static void main(String[] args) {
File folder = new File("C:\\Windows\\System32\\");
File[] list = folder.listFiles();
for(int i = 0; i< 3; i++) {
long startTime = System.currentTimeMillis();
System.out.println("Method 1: " + getIconNumber1(list)+ " Icons");
long stopTime = System.currentTimeMillis();
long elapsedTime = stopTime - startTime;
System.out.println("Finished Method 1 in " + (float) elapsedTime / 1000 + "sec");
long startTime2 = System.currentTimeMillis();
System.out.println("Method 2: " + getIconNumber2(list)+ " Icons");
long stopTime2 = System.currentTimeMillis();
long elapsedTime2 = stopTime2 - startTime2;
System.out.println("Finished Method 2 in " + (float) elapsedTime2 / 1000 + "sec");
System.out.println("-----------------");
}
}
private static int getIconNumber1(File[] list) {
int nr = 0;
for(File f : list) {
try {
ShellFolder sf = ShellFolder.getShellFolder(f);
ImageIcon icon = new ImageIcon(sf.getIcon(true));
nr++;
} catch (Exception e) {
e.printStackTrace();
}
}
return nr;
}
private static int getIconNumber2(File[] list) {
int nr = 0;
for(File f : list){
FileSystemView view = FileSystemView.getFileSystemView();
Icon icon = view.getSystemIcon(f);
nr++;
}
return nr;
}
}
Is there a faster way to do this?

How to insert a stop watch before the polling starts

May I know How to insert a stop watch for this piece of code from Poll() method...I have to make start count such that before the database starts and amount of time it took for polling.
public void poll() throws Exception {
st = conn.createStatement();
for (int i=0; i<10; i++)
{
Timestamp start;
rs = st.executeQuery( "select * from msg_new_to_bde" );
Timestamp end;
//speed = end - start;
Collection<KpiMessage> pojoCol = new ArrayList<KpiMessage>();
while (rs.next()) {
KpiMessage filedClass = convertRecordsetToPojo(rs);
pojoCol.add(filedClass);
}
for (KpiMessage pojoClass : pojoCol) {
System.out.println("=== Iteratioin Nr. " + i + "====");
System.out.print(pojoClass.getSequence());
System.out.print(pojoClass.getTableName());
System.out.print(pojoClass.getEntryTime());
System.out.print(pojoClass.getProcessingTime());
System.out.println(pojoClass.getStatus());
// System.out.println(pojoClass.getprocessDuration());
}
System.out.print(pojoCol.size());
}
}

You have to use currentTimeMillis() function:
Before launch polling:
long start = System.currentTimeMillis();
After Polling execution:
long stop= System.currentTimeMillis();
Execution time is stop - start in milliseconds.

I believe System.currentTimeMillis is what you looking for.
long startTime = System.currentTimeMillis();
//
long endTime = System.currentTimeMillis();
System.out.println((endTime - startTime) + "ms");

java.util.Date date = new java.util.Date();
Timestamp start = new Timestamp(date.getTime());
//process
java.util.Date date1 = new java.util.Date();
Timestamp end = new Timestamp(date1.getTime());

long start = System.currentTimeMillis();
rs = st.executeQuery( "select * from msg_new_to_bde" );
long stop= System.currentTimeMillis();
System.out.println("execution time: " +stop-start + " ms");

long start = System.nanoTime();
timeThisMethod();
long end = System.nanoTime();
long howLongDidItTake = end - start;
This method is more precise then System.currentTimeMillis()
Citation from java API :
Returns the current value of the most precise available system timer,
in nanoseconds.

Difference in duration of audio file converted from WAVE to MP3

I think there should be no difference in total time duration of play back of an audio file if we convert it between different formats.
For example if I record wave file of total time duration for 2 seconds, its size now is 20.3 MB . now i convert this wave file to mp3 file using ffmpeg latest build it becomes 1.35 mb in sizes. Now I get the time duration of the same converted MP3 file using below code.
public static String getDurationWithMp3Spi(File file)
throws UnsupportedAudioFileException, IOException, Exception {
AudioFileFormat fileFormat = AudioSystem.getAudioFileFormat(file);
System.out.println(" File for duration MP3 " + file.getAbsolutePath());
if (fileFormat instanceof TAudioFileFormat) {
Map<?, ?> properties = ((TAudioFileFormat) fileFormat).properties();
String key = "duration";
Long microseconds = (Long) properties.get(key);
int mili = (int) (microseconds / 1000);
int sec = (mili / 1000) % 60;
int min = (mili / 1000) / 60;
String mp3Len = null;
String mins = null;
String secs = null;
if (min == 0) {
mins = "00";
}
if (min < 10) {
mins = "0" + min;
}else{
mins = "" + min;
}
if (sec == 0) {
secs = "00";
}
if (sec < 10) {
secs = "0" + sec;
}else{
secs = ""+secs;
}
mp3Len = mins + ":" + secs;
System.out.println("time = " + min + ":" + sec);
return mp3Len;
} else {
throw new UnsupportedAudioFileException();
}
}
If I see in windows media player or any other player it will show the same duration as it is for original wave but when I get from this method it is different from originals and the difference is very big.
Is there in difference in time duration of audio file if it is converted from WAVE to MP3 or vice versa?
Any help please. the above code uses MP3SPI plugin.
the above method does the conversion and get the duration.
for (Iterator<FileItem> fileIter = fileList.iterator(); fileIter
.hasNext();) {
FileItem fileItem = fileIter.next();
// write file to disk to specified path
if (!fileItem.isFormField()) {
String fileName = fileItem.getName();
System.out.println(" file Name " + fileName);
// save file to desired destination
waveFileSavePath = processFolderAppendee(waveFileSavePath,
fileName);
File waveFile = new File(waveFileSavePath);
fileItem.write(waveFile);
Thread.sleep(100);
// do conversion
String mp3FileName = fileName.replace("wav", "mp3");
mp3Path = mp3Path + "/" + mp3FileName;
convertToMP3(servletContext, waveFileSavePath, mp3Path);
Thread.sleep(100);
// prepare data(s)
GuestMessagesForm guestMessageForm = prepareGuestMessageData(
accountId, waveFileSavePath, mp3Path);
PlayListMessagesForm playListMessageForm = preparePlayListMessageData(accountId);
// save data(s)
// this method calls duration
saveGuestMessage(guestMessageForm);
savePlayListMessage(playListMessageForm);
} else {
// do nothing
}
waveFileSavePath = servletContext.getRealPath(recordDir);
}