Develop Reference

Serialization/deserialization java adding objects to existing array list in saved file [duplicate]

Is it not possible to append to an ObjectOutputStream?
I am trying to append to a list of objects. Following snippet is a function that is called whenever a job is finished.
FileOutputStream fos = new FileOutputStream
(preferences.getAppDataLocation() + "history" , true);
ObjectOutputStream out = new ObjectOutputStream(fos);
out.writeObject( new Stuff(stuff) );
out.close();
But when I try to read it I only get the first in the file.
Then I get java.io.StreamCorruptedException.
To read I am using
FileInputStream fis = new FileInputStream
( preferences.getAppDataLocation() + "history");
ObjectInputStream in = new ObjectInputStream(fis);
try{
while(true)
history.add((Stuff) in.readObject());
}catch( Exception e ) {
System.out.println( e.toString() );
}
I do not know how many objects will be present so I am reading while there are no exceptions. From what Google says this is not possible. I was wondering if anyone knows a way?

Here's the trick: subclass ObjectOutputStream and override the writeStreamHeader method:
public class AppendingObjectOutputStream extends ObjectOutputStream {
public AppendingObjectOutputStream(OutputStream out) throws IOException {
super(out);
}
#Override
protected void writeStreamHeader() throws IOException {
// do not write a header, but reset:
// this line added after another question
// showed a problem with the original
reset();
}
}
To use it, just check whether the history file exists or not and instantiate either this appendable stream (in case the file exists = we append = we don't want a header) or the original stream (in case the file does not exist = we need a header).
Edit
I wasn't happy with the first naming of the class. This one's better: it describes the 'what it's for' rather then the 'how it's done'
Edit
Changed the name once more, to clarify, that this stream is only for appending to an existing file. It can't be used to create a new file with object data.
Edit
Added a call to reset() after this question showed that the original version that just overrode writeStreamHeader to be a no-op could under some conditions create a stream that couldn't be read.

As the API says, the ObjectOutputStream constructor writes the serialization stream header to the underlying stream. And this header is expected to be only once, in the beginning of the file. So calling
new ObjectOutputStream(fos);
multiple times on the FileOutputStream that refers to the same file will write the header multiple times and corrupt the file.

Because of the precise format of the serialized file, appending will indeed corrupt it. You have to write all objects to the file as part of the same stream, or else it will crash when it reads the stream metadata when it's expecting an object.
You could read the Serialization Specification for more details, or (easier) read this thread where Roedy Green says basically what I just said.

The easiest way to avoid this problem is to keep the OutputStream open when you write the data, instead of closing it after each object. Calling reset() might be advisable to avoid a memory leak.
The alternative would be to read the file as a series of consecutive ObjectInputStreams as well. But this requires you to keep count how many bytes you read (this can be implementd with a FilterInputStream), then close the InputStream, open it again, skip that many bytes and only then wrap it in an ObjectInputStream().

I have extended the accepted solution to create a class that can be used for both appending and creating new file.
import java.io.DataOutputStream;
import java.io.IOException;
import java.io.ObjectOutputStream;
import java.io.OutputStream;
public class AppendableObjectOutputStream extends ObjectOutputStream {
private boolean append;
private boolean initialized;
private DataOutputStream dout;
protected AppendableObjectOutputStream(boolean append) throws IOException, SecurityException {
super();
this.append = append;
this.initialized = true;
}
public AppendableObjectOutputStream(OutputStream out, boolean append) throws IOException {
super(out);
this.append = append;
this.initialized = true;
this.dout = new DataOutputStream(out);
this.writeStreamHeader();
}
#Override
protected void writeStreamHeader() throws IOException {
if (!this.initialized || this.append) return;
if (dout != null) {
dout.writeShort(STREAM_MAGIC);
dout.writeShort(STREAM_VERSION);
}
}
}
This class can be used as a direct extended replacement for ObjectOutputStream.
We can use the class as follows:
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
public class ObjectWriter {
public static void main(String[] args) {
File file = new File("file.dat");
boolean append = file.exists(); // if file exists then append, otherwise create new
try (
FileOutputStream fout = new FileOutputStream(file, append);
AppendableObjectOutputStream oout = new AppendableObjectOutputStream(fout, append);
) {
oout.writeObject(...); // replace "..." with serializable object to be written
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
}

How about before each time you append an object, read and copying all the current data in the file and then overwrite all together to file.

Java - Overriding StdOut and writing Japanese to a File

I have been trying to create a class that can be set as Standard Out and that writes to a File. So far, no problem. However, when I try to write some Japanese into the output it won't show up in the files. All I get are lines of '?' characters.
This is how my OutputStream class looks like (I create an instance of it and hands that over to "System.setOut(OutputStream)" in my main method).
public class MyStdOutStream extends OutputStream {
private OutputStreamWriter out;
public MyStdOutStream(File file) throws IOException {
out = new OutputStreamWriter(new FileOutputStream(file), StandardCharsets.UTF_8);
}
#Override
public void close() throws IOException {
out.flush();
out.close();
super.close(); //No idea if this line does anything...
}
#Override
public void write(int nr) throws IOException {
out.write(nr);
}
}
I then accept input through a JTextField (while testing, it will be a different source later), and print that both with "System.out.println(String)" and also appending it to a JTextArea. It shows up as proper Japanese in the JTextArea, but not in the File.
Does anyone know what I am doing wrong? I have tested with all available Charsets in the StandardCharsets import without success.
EDIT: It is an assignment, and I am not allowed to use anything other than the Standard Java Library (so, no imported JARs)

Why does findbugs not detect when streams are not closed?

I've a usecase where I'm creating an InputStream in one class & passing it to another. If I remove the finally block where I close the stream, it does not get detected in findbugs. Why is that?
Class A {
public static void methodA(InputStream is) {
// Do something.
// The stream is NOT closed.
}
}
Class B {
public void methodB(Sting filePath) {
FileInputStream fis = new FileInputStream(new File(filePath));
A.methodA(fis);
}
}
Ideally, findbugs should have detected that the stream is not closed in this use case. But, it doesn't & I'm curious to know why!

Overriding writeStreamHeader() to append objects to a single file

I need to append multiple objects in a single file in multiple sessions.
I searched for a fair amount of time and I got two solutions from here.
1) Using List to get already written objects from the file, add new objects to the List and rewriting the file.
2) Overriding writeStreamHeader()
I followed the second method i.e., overriding writeStreamHeader().
He stated that
A workaround is to subclass ObjectOutputStream and override
writeStreamHeader(). The overriding writeStreamHeader() should call
the super writeStreamHeader method if it is the first write to the
file and it should call ObjectOutputStream.reset() if it is appending
to a pre-existing ObjectOutputStream within the file.
So I tried this
class ObjectOutput extends ObjectOutputStream
{
protected ObjectOutput(OutputStream os) throws IOException, SecurityException {
super(os);
}
protected void writeStreamHeader() throws IOException
{
File file = new File("abc.txt");
if(file.exists())
{
reset();
}
else
{
file.createNewFile();
super.writeStreamHeader();
}
}
}
When I try to read the objects from the file there is an exception
Exception in thread "main" java.io.StreamCorruptedException: invalid stream header: 79737200
at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:806)
at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299)
at Get.main(Get.java:11)
Then I tried this solution
Now, it worked perfectly!
So, what's wrong with the first code? I called the super.writeStreamHeader() when there is no file and the second method also calls the same method in the same scenario.
So,is there anything I am missing?
Thank you.

You can do this too and still it will work fine
import java.io.IOException;
import java.io.ObjectOutputStream;
import java.io.OutputStream;
public class MyObjectOutputStream extends ObjectOutputStream {
public MyObjectOutputStream() throws IOException{
super();
}
public MyObjectOutputStream(OutputStream outputStream) throws IOException{
super(outputStream);
}
public void writeStreamHeader(){}
}
The explanation for this is the same as the link to the solution you provided in the question itself.

How can I recreate a chained OutputStream with only filename modified

I have got an OutputStream which can be initialized as a chain of OutputStreams. There could be any level of chaining .Only thing guaranteed is that at the end of the chain is a FileOutputStream.
I need to recreate this chained outputStream with a modified Filename in FileOutputStream. This would have been possible if out variable (which stores the underlying chained outputStream) was accessible ; as shown below.
public OutputStream recreateChainedOutputStream(OutputStream os) throws IOException {
if(os instanceof FileOutputStream) {
return new FileOutputStream("somemodified.filename");
} else if (os instanceof FilterOutputStream) {
return recreateChainedOutputStream(os.out);
}
}
Is there any other way of achieving the same?

You can use reflection to access the os.out field of the FilterOutputStream, this has however some drawbacks:
If the other OutputStream is also a kind of RolloverOutputStream, you can have a hard time reconstructing it,
If the other OutputStream has custom settings, like GZip compression parameter, you cannot reliable read this
If there is a
A quick and dirty implementation of recreateChainedOutputStream( might be:
private final static Field out;
{
try {
out = FilterInputStream.class.getField("out");
out.setAccessible(true);
} catch(Exception e) {
throw new RuntimeException(e);
}
}
public OutputStream recreateChainedOutputStream(OutputStream out) throws IOException {
if (out instanceof FilterOutputStream) {
Class<?> c = ou.getClass();
COnstructor<?> con = c.getConstructor(OutputStream.class);
return con.invoke(this.out.get(out));
} else {
// Other output streams...
}
}
While this may be ok in your current application, this is a big no-no in the production world because the large amount of different kind of OutputStreams your application may recieve.
A better way to solve would be a kind of Function<String, OutputStream> that works as a factory to create OutputStreams for the named file. This way the external api keeps its control over the OutputStreams while your api can adress multiple file names. An example of this would be:
public class MyApi {
private final Function<String, OutputStream> fileProvider;
private OutputStream current;
public MyApi (Function<String, OutputStream> fileProvider, String defaultFile) {
this.fileProvider = fileProvider;
selectNewOutputFile(defaultFile);
}
public void selectNewOutputFile(String name) {
OutputStream current = this.current;
this.current = fileProvider.apply(name);
if(current != null) current.close();
}
}
This can then be used in other applications as:
MyApi api = new MyApi(name->new FileOutputStream(name));
For simple FileOutputStreams, or be used as:
MyApi api = new MyApi(name->
new GZIPOutputStream(
new CipherOutputStream(
new CheckedOutputStream(
new FileOutputStream(name),
new CRC32()),
chipper),
1024,
true)
);
For a file stream that stored checksummed using new CRC32(), chipped using chipper, gzip according to a 1024 buffer with sync write mode.