Develop Reference

Serialization/deserialization java adding objects to existing array list in saved file [duplicate]

Is it not possible to append to an ObjectOutputStream?
I am trying to append to a list of objects. Following snippet is a function that is called whenever a job is finished.
FileOutputStream fos = new FileOutputStream
(preferences.getAppDataLocation() + "history" , true);
ObjectOutputStream out = new ObjectOutputStream(fos);
out.writeObject( new Stuff(stuff) );
out.close();
But when I try to read it I only get the first in the file.
Then I get java.io.StreamCorruptedException.
To read I am using
FileInputStream fis = new FileInputStream
( preferences.getAppDataLocation() + "history");
ObjectInputStream in = new ObjectInputStream(fis);
try{
while(true)
history.add((Stuff) in.readObject());
}catch( Exception e ) {
System.out.println( e.toString() );
}
I do not know how many objects will be present so I am reading while there are no exceptions. From what Google says this is not possible. I was wondering if anyone knows a way?

Here's the trick: subclass ObjectOutputStream and override the writeStreamHeader method:
public class AppendingObjectOutputStream extends ObjectOutputStream {
public AppendingObjectOutputStream(OutputStream out) throws IOException {
super(out);
}
#Override
protected void writeStreamHeader() throws IOException {
// do not write a header, but reset:
// this line added after another question
// showed a problem with the original
reset();
}
}
To use it, just check whether the history file exists or not and instantiate either this appendable stream (in case the file exists = we append = we don't want a header) or the original stream (in case the file does not exist = we need a header).
Edit
I wasn't happy with the first naming of the class. This one's better: it describes the 'what it's for' rather then the 'how it's done'
Edit
Changed the name once more, to clarify, that this stream is only for appending to an existing file. It can't be used to create a new file with object data.
Edit
Added a call to reset() after this question showed that the original version that just overrode writeStreamHeader to be a no-op could under some conditions create a stream that couldn't be read.

As the API says, the ObjectOutputStream constructor writes the serialization stream header to the underlying stream. And this header is expected to be only once, in the beginning of the file. So calling
new ObjectOutputStream(fos);
multiple times on the FileOutputStream that refers to the same file will write the header multiple times and corrupt the file.

Because of the precise format of the serialized file, appending will indeed corrupt it. You have to write all objects to the file as part of the same stream, or else it will crash when it reads the stream metadata when it's expecting an object.
You could read the Serialization Specification for more details, or (easier) read this thread where Roedy Green says basically what I just said.

The easiest way to avoid this problem is to keep the OutputStream open when you write the data, instead of closing it after each object. Calling reset() might be advisable to avoid a memory leak.
The alternative would be to read the file as a series of consecutive ObjectInputStreams as well. But this requires you to keep count how many bytes you read (this can be implementd with a FilterInputStream), then close the InputStream, open it again, skip that many bytes and only then wrap it in an ObjectInputStream().

I have extended the accepted solution to create a class that can be used for both appending and creating new file.
import java.io.DataOutputStream;
import java.io.IOException;
import java.io.ObjectOutputStream;
import java.io.OutputStream;
public class AppendableObjectOutputStream extends ObjectOutputStream {
private boolean append;
private boolean initialized;
private DataOutputStream dout;
protected AppendableObjectOutputStream(boolean append) throws IOException, SecurityException {
super();
this.append = append;
this.initialized = true;
}
public AppendableObjectOutputStream(OutputStream out, boolean append) throws IOException {
super(out);
this.append = append;
this.initialized = true;
this.dout = new DataOutputStream(out);
this.writeStreamHeader();
}
#Override
protected void writeStreamHeader() throws IOException {
if (!this.initialized || this.append) return;
if (dout != null) {
dout.writeShort(STREAM_MAGIC);
dout.writeShort(STREAM_VERSION);
}
}
}
This class can be used as a direct extended replacement for ObjectOutputStream.
We can use the class as follows:
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
public class ObjectWriter {
public static void main(String[] args) {
File file = new File("file.dat");
boolean append = file.exists(); // if file exists then append, otherwise create new
try (
FileOutputStream fout = new FileOutputStream(file, append);
AppendableObjectOutputStream oout = new AppendableObjectOutputStream(fout, append);
) {
oout.writeObject(...); // replace "..." with serializable object to be written
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
}

How about before each time you append an object, read and copying all the current data in the file and then overwrite all together to file.

Java Program doesn't work after exported from eclipse

When I open the runnable jar file, it still could be opened but it get stuck after half a second like this.
(I cannot post an image, so I post the image here: https://pbs.twimg.com/media/B4RN2_MCYAAi1DD.png)
It works well in eclipse.
When I run it in CMD, it says:
Exception in thread "game" java.lang.NullPointerException: in
at javazoom.jl.decoder.Bitstream.<init>(Unknown Source)
at javazoom.jl.player.Player.<init>(Unknown Source)
at javazoom.jl.player.Player.<init>(Unknown Source)
at lian.xiangru.game.AudioHandler.<init>(AudioHandler.java:12)
at lian.xiangru.game.GameBoard.playSound(GameBoard.java:410)
at lian.xiangru.game.GameBoard.move(GameBoard.java:224)
at lian.xiangru.game.GameBoard.moveTiles(GameBoard.java:271)
at lian.xiangru.game.GameBoard.checkKeys(GameBoard.java:340)
at lian.xiangru.game.GameBoard.update(GameBoard.java:146)
at lian.xiangru.game.Game.update(Game.java:42)
at lian.xiangru.game.Game.run(Game.java:77)
at java.lang.Thread.run(Unknown Source)
It seems something goes wrong with my resources.
This is my playSound method:
private void playSound() {
// how to play mp3 https://www.youtube.com/watch?v=f-7cgX_I220
AudioHandler sound = new AudioHandler(
SOUND_LIST[(int) Math.round((Math.log(highestValue) / Math.log(2))) - 1]);
sound.start();
}
This is my AudioHandler class:
import javazoom.jl.decoder.JavaLayerException;
import javazoom.jl.player.Player;
class AudioHandler extends Thread {
private Player playMP3;
public AudioHandler(String mp3) {
try {
playMP3 = new Player(getClass().getResourceAsStream(mp3));
} catch (JavaLayerException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
public void run() {
try {
playMP3.play();
} catch (JavaLayerException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
The lists are:
public static final String[] SOUND_LIST = { "/mayuri.mp3", "/mikoto.mp3",
"/gai.mp3", "/shougo.mp3", "/gintoki.mp3", "/inori.mp3",
"/yuzuru.mp3", "/misaki.mp3", "/armin.mp3", "/alphonse.mp3",
"/alphonse.mp3", "/akane.mp3", "/armin.jpg" };
public static final String[] QUOTE_LIST = { "/mayuri.txt", "/mikoto.txt",
"/gai.txt", "/shougo.txt", "/gintoki.txt", "/inori.txt",
"/yuzuru.txt", "/misaki.txt", "/armin.txt", "/alphonse.txt",
"/alphonse.txt", "/akane.txt", "/armin.txt" };
public static final String[] ICON_LIST = {"/mayuri.jpg", "/mikoto.jpg",
"/gai.jpg", "/shougo.jpg", "/gintoki.jpg", "/inori.jpg",
"/yuzuru.jpg", "/misaki.jpg", "/armin.jpg", "/alphonse.jpg",
"/alphonse.jpg", "/akane.jpg", "/armin.jpg"
};
Thank you !!

NullPointerException when loading a resource usually means that the resource could not be found. This behavior is different than the normal file open, no exception is thrown.
Are your audio files included in your jar? If not, make sure that when you export the Jar, you set the right options so they are included. A better alternative might be to introduce a real build process into your application if you need repeatability.

I just found the reason!
There is a file called Mikoto.mp3 in my resources folder and I typed it as mikoto.mp3 in my code. This could be allowed when I use eclipse to run it. But when I run the runnable jar file, it fails because it is case sensitive.
When I change the file name to mikoto.mp3, it works!
Thanks for your answer!

How to watch file for new content and retrieve that content

I have a file with name foo.txt. This file contains some text. I want to achieve following functionality:
I launch program
write something to the file (for example add one row: new string in foo.txt)
I want to get ONLY NEW content of this file.
Can you clarify the best solution of this problem? Also I want resolve related issues: in case if I modify foo.txt I want to see diff.
The closest tool which I found in Java is WatchService but if I understood right this tool can only detect type of event happened on filesystem (create file or delete or modify).

Java Diff Utils is designed for that purpose.
final List<String> originalFileContents = new ArrayList<String>();
final String filePath = "C:/Users/BackSlash/Desktop/asd.txt";
FileListener fileListener = new FileListener() {
#Override
public void fileDeleted(FileChangeEvent paramFileChangeEvent)
throws Exception {
// use this to handle file deletion event
}
#Override
public void fileCreated(FileChangeEvent paramFileChangeEvent)
throws Exception {
// use this to handle file creation event
}
#Override
public void fileChanged(FileChangeEvent paramFileChangeEvent)
throws Exception {
System.out.println("File Changed");
//get new contents
List<String> newFileContents = new ArrayList<String> ();
getFileContents(filePath, newFileContents);
//get the diff between the two files
Patch patch = DiffUtils.diff(originalFileContents, newFileContents);
//get single changes in a list
List<Delta> deltas = patch.getDeltas();
//print the changes
for (Delta delta : deltas) {
System.out.println(delta);
}
}
};
DefaultFileMonitor monitor = new DefaultFileMonitor(fileListener);
try {
FileObject fileObject = VFS.getManager().resolveFile(filePath);
getFileContents(filePath, originalFileContents);
monitor.addFile(fileObject);
monitor.start();
} catch (InterruptedException ex) {
ex.printStackTrace();
} catch (FileNotFoundException e) {
//handle
e.printStackTrace();
} catch (IOException e) {
//handle
e.printStackTrace();
}
Where getFileContents is :
void getFileContents(String path, List<String> contents) throws FileNotFoundException, IOException {
contents.clear();
BufferedReader reader = new BufferedReader(new InputStreamReader(new FileInputStream(path), "UTF-8"));
String line = null;
while ((line = reader.readLine()) != null) {
contents.add(line);
}
}
What I did:
I loaded the original file contents in a List<String>.
I used Apache Commons VFS to listen for file changes, using FileMonitor. You may ask, why? Because WatchService is only available starting from Java 7, while FileMonitor works with at least Java 5 (personal preference, if you prefer WatchService you can use it). note: Apache Commons VFS depends on Apache Commons Logging, you'll have to add both to your build path in order to make it work.
I created a FileListener, then I implemented the fileChanged method.
That method load new contents form the file, and uses Patch.diff to retrieve all differences, then prints them
I created a DefaultFileMonitor, which basically listens for changes to a file, and I added my file to it.
I started the monitor.
After the monitor is started, it will begin listening for file changes.

Getting StreamCorruptedException after closing OutputStream [duplicate]

Is it not possible to append to an ObjectOutputStream?
I am trying to append to a list of objects. Following snippet is a function that is called whenever a job is finished.
FileOutputStream fos = new FileOutputStream
(preferences.getAppDataLocation() + "history" , true);
ObjectOutputStream out = new ObjectOutputStream(fos);
out.writeObject( new Stuff(stuff) );
out.close();
But when I try to read it I only get the first in the file.
Then I get java.io.StreamCorruptedException.
To read I am using
FileInputStream fis = new FileInputStream
( preferences.getAppDataLocation() + "history");
ObjectInputStream in = new ObjectInputStream(fis);
try{
while(true)
history.add((Stuff) in.readObject());
}catch( Exception e ) {
System.out.println( e.toString() );
}
I do not know how many objects will be present so I am reading while there are no exceptions. From what Google says this is not possible. I was wondering if anyone knows a way?

Here's the trick: subclass ObjectOutputStream and override the writeStreamHeader method:
public class AppendingObjectOutputStream extends ObjectOutputStream {
public AppendingObjectOutputStream(OutputStream out) throws IOException {
super(out);
}
#Override
protected void writeStreamHeader() throws IOException {
// do not write a header, but reset:
// this line added after another question
// showed a problem with the original
reset();
}
}
To use it, just check whether the history file exists or not and instantiate either this appendable stream (in case the file exists = we append = we don't want a header) or the original stream (in case the file does not exist = we need a header).
Edit
I wasn't happy with the first naming of the class. This one's better: it describes the 'what it's for' rather then the 'how it's done'
Edit
Changed the name once more, to clarify, that this stream is only for appending to an existing file. It can't be used to create a new file with object data.
Edit
Added a call to reset() after this question showed that the original version that just overrode writeStreamHeader to be a no-op could under some conditions create a stream that couldn't be read.

As the API says, the ObjectOutputStream constructor writes the serialization stream header to the underlying stream. And this header is expected to be only once, in the beginning of the file. So calling
new ObjectOutputStream(fos);
multiple times on the FileOutputStream that refers to the same file will write the header multiple times and corrupt the file.

Because of the precise format of the serialized file, appending will indeed corrupt it. You have to write all objects to the file as part of the same stream, or else it will crash when it reads the stream metadata when it's expecting an object.
You could read the Serialization Specification for more details, or (easier) read this thread where Roedy Green says basically what I just said.

The easiest way to avoid this problem is to keep the OutputStream open when you write the data, instead of closing it after each object. Calling reset() might be advisable to avoid a memory leak.
The alternative would be to read the file as a series of consecutive ObjectInputStreams as well. But this requires you to keep count how many bytes you read (this can be implementd with a FilterInputStream), then close the InputStream, open it again, skip that many bytes and only then wrap it in an ObjectInputStream().

I have extended the accepted solution to create a class that can be used for both appending and creating new file.
import java.io.DataOutputStream;
import java.io.IOException;
import java.io.ObjectOutputStream;
import java.io.OutputStream;
public class AppendableObjectOutputStream extends ObjectOutputStream {
private boolean append;
private boolean initialized;
private DataOutputStream dout;
protected AppendableObjectOutputStream(boolean append) throws IOException, SecurityException {
super();
this.append = append;
this.initialized = true;
}
public AppendableObjectOutputStream(OutputStream out, boolean append) throws IOException {
super(out);
this.append = append;
this.initialized = true;
this.dout = new DataOutputStream(out);
this.writeStreamHeader();
}
#Override
protected void writeStreamHeader() throws IOException {
if (!this.initialized || this.append) return;
if (dout != null) {
dout.writeShort(STREAM_MAGIC);
dout.writeShort(STREAM_VERSION);
}
}
}
This class can be used as a direct extended replacement for ObjectOutputStream.
We can use the class as follows:
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
public class ObjectWriter {
public static void main(String[] args) {
File file = new File("file.dat");
boolean append = file.exists(); // if file exists then append, otherwise create new
try (
FileOutputStream fout = new FileOutputStream(file, append);
AppendableObjectOutputStream oout = new AppendableObjectOutputStream(fout, append);
) {
oout.writeObject(...); // replace "..." with serializable object to be written
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
}

How about before each time you append an object, read and copying all the current data in the file and then overwrite all together to file.

Possible bug with load() and parse() methods in PDFBox?

I tried to use PDFBox on regular .pdf files and it worked fine.
However when I encountered a corrupted .pdf , the code would "freeze" .. not throwing errors or something .. simply the load or parse function take forever to execute
Here is the corrupted file (i have zipped it so that everybody could download it), it is probably not a native pdf file but it was saved as a .pdf extension and it is only 4 Kb.
I am not an expert at all, but I think that this is a bug with PDFBox. According to documentation, both load() and parse() methods are supposed to throw exceptions if they fail. However in case with my file, the code would take forever to execute and not throw exception.
I tried using only load, one can try parse() .. the result is the same
import java.io.FileNotFoundException;
import java.io.IOException;
import org.apache.pdfbox.pdmodel.PDDocument;
import org.apache.pdfbox.util.PDFTextStripper;
public class TestTest {
public static void main(String[] args) throws FileNotFoundException, IOException {
System.out.println(pdfToText("C:\\..............MYFILE.pdf"));
System.out.println("done ! ! !");
}
private static String pdfToText(String fileName) throws IOException {
PDDocument document = null;
document = PDDocument.load(new File(fileName)); // THIS TAKES FOREVER
PDFTextStripper stripper = new PDFTextStripper();
document.close();
return stripper.getText(document);
}
}
How to force this code throw an exception or stop executing if the .pdf file is corrupted?
Thanks

Try this solution:
private static String pdfToText(String fileName) {
PDDocument document = null;
try {
document = PDDocument.load(fileName);
PDFTextStripper stripper = new PDFTextStripper();
return stripper.getText(document);
} catch (IOException e) {
System.err.println("Unable to open PDF Parser. " + e.getMessage());
return null;
} finally {
if (document != null) {
try {
document.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}

For implementing simple timeouts for 3rd party libs I often use an implementation like Apache Commons ThreadMonitor:
long timeoutInMillis = 1000;
try {
Thread monitor = ThreadMonitor.start(timeoutInMillis);
// do some work here
ThreadMonitor.stop(monitor);
} catch (InterruptedException e) {
// timed amount was reached
}
Example code is from Apache's ThreadMonitor Javadoc.
I only use this when the 3rd party API does not provide some timeout mechanism, of course.
However I was forced to tweak this a bit some weeks ago, because this solution does not work well with (3rd party) code that is using Exception masking.
In particular we run into problems with c3p0 which masks all Exceptions (and in particular InterruptedExceptions). Our solution was to tweak the implementation to also check the exception's cause chain for InterruptedExceptions.