I have a collection of objects:
Map<BufferedImage, Map<ImageTransform, Set<Point>>> map
I want to write those to a file, and then be able to read them back in the same struct.
I can't just write the collection as it is, because BufferedImage doesn't implement the Serializable (nor the Externalizable) interface. So I need to use the methods from the ImageIO class to write the image.
ImageTransform is a custom object that implements Serializable. So, I believe the value part of my map collection, should be writeable as it is.
Here is what I do to write to the file:
ObjectOutputStream out = new ObjectOutputStream(new FileOutputStream(file));
for (BufferedImage image : map.keySet()) {
ImageIO.write(image, "PNG", out); // write the image to the stream
out.writeObject(map.get(image)); // write the 'value' part of the map
}
Here is what I do to read back from the file:
ObjectInputStream in = new ObjectInputStream(new FileInputStream(file));
while(true) {
try {
BufferedImage image = ImageIO.read(in);
Map<ImageTransform, Set<Point>> value =
(Map<ImageTransform, Set<Point>>) in.readObject(); // marker
map.put(image, value);
} catch (IOException ioe) {
break;
}
}
However, this doesn't work. I get a java.io.OptionalDataException at marker.
java.io.OptionalDataException
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1300)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:368)
My question is, firstly, is the writing concept correct ? is ImageIO#write good for this case, or should I think about using/storing the BufferedImage#getRgb int[] array ? is the array more compact (as in, takes up less space in the file) ?
Secondly, how should I be reading the object back from the file ? How do I know when the EOF is reached ? Why doesn't the above work ?
I hope the info provided is enough, if you need more info on something, please tell me.
Thanks in advance.
It's not working as ObjectOutputStream and ObjectInputStream write/expect a certain file format that is violated when you write an image out of order. To use ObjectStreams successfully you will need to observe the contract that is specifed by ObjectStreams.
To do this you will need to create a holding class, and use this class as the key to your map instead of BufferedImages. This holding class should implement Serializable and a three methods (not in any actual interface) that mark the Class as needing special handling during reading and writing. The method signatures must be exactly as specified or serialization won't work.
For more information have a look at the documentation on ObjectOutputStream.
public class ImageHolder implements Serializable {
BufferedImage image;
public ImageHolder(BufferedImage image) {
this.image = image;
}
private void readObject(ObjectInputStream stream)
throws IOException, ClassNotFoundException {
image = ImageIO.read(stream);
}
private void writeObject(ObjectOutputStream stream)
throws IOException {
ImageIO.write(image, "PNG", stream);
}
private void readObjectNoData() throws ObjectStreamException {
// leave image as null
}
And then serialsation should be as simple as outputStream.writeObject(map). Though you will need to check that the implementing class of ImageTransform is serialisable too.
One way to 'cheat' and only have a single object to serialize is to add the group of objects to an expandable, serializable list. Then serialize the list.
BTW - I would tend to use XMLEncoder over serialized Objects because they can be restored in later JVMs. There is no such guarantee for serialized Objects.
#Ivan c00kiemon5ter V Kanak: "I'm trying to keep the file as small in size as possible,..
That is often wasted effort, given disk space is so cheap.
*.. so I guess Serialization is better for that. *
Don't guess. Measure.
..I'll try using a List and see how that goes. ..
Cool. Note that if using the XMLEncoder, I'd recommend Zipping it in most cases. That would reduce the file size of the cruft of XML. This situation is different in storing images.
Image formats typically incorporate compression of a type that is not conducive to being further compressed by Zip. That can be side-stepped by storing the XML compressed, and the images as 'raw' in separate entries in the Zip. OTOH I think you'll find the amount of bytes saved by compressing the XML alone is not worth the effort - given the final file size of the image entries.
Related
I'm creating a game where all locations of 'blocks' are stored in the variable block_area - an object of class Area. My game has been running correctly for a week now, and I've decided to implement a save and load feature where I save block_area to a file Drifter, with this as my code:
Area block_area; // Later initialized
void saveArea()
{
try
{
FileOutputStream fos = new FileOutputStream(savefile);
ObjectOutputStream oos = new ObjectOutputStream(fos);
oos.writeObject(block_area);
oos.close();
}
catch(Exception e)
{
e.printStackTrace();
}
}
void loadArea()
{
try
{
FileInputStream fis = new FileInputStream(savefile);
ObjectInputStream ois = new ObjectInputStream(fis);
block_area = (Area)ois.readObject();
ois.close();
}
catch(Exception e)
{
e.printStackTrace();
}
}
However, this is my very first time writing and reading an OBJECT to a file, so I don't know much about it. When I try to save the object to the file, it gives me this error:
java.io.NotSerializableException: java.awt.geom.Area
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
at Drifter.saveArea(Drifter.java:58)
at Drifter.keyPressed(Drifter.java:315)
...
If anyone can tell me how I can go about writing and reading an object with a file, the help will be greatly appreciated.
TL;DR How do I write the contents of an Area object to a file and read it?
ALSO I have a few follow-up questions:
Is ObjectInputStream the best course of action here? I have seen a few answers where people recommend using XML, and JSON, but I can never find the time to learn about them, and would prefer to stick to a pure Java method (without any third party tools)
Is there any other method of saving an object's information to an external source that I can use instead of file handling?
EDIT - I should also mention that my class implements Serializable
The exception is pretty self explanatory NotSerializableException: java.awt.geom.Area . Any object that you want to serialize must implement the Serializable interface. java,awt.geom.Area does not. Any attributes of that class must also implement Serializable, be a primitive, or be defined as transient.
I'd suggest either Figuring out a way to read Area into an object that does implement Serializable. When you read it back out, you can construct a new Area object. This is probably what the JSON/XML method mentioned in the comments is doing. The added benefit of a human readable storage format is that you can edit it in a text editor. You won't be able to do that with the binary output of a serialized object`.
Uploading an image to hbase using Java program, after retrieving the image I found there is difference in file size eventually increased and most of Exif and Meta data loss
(GPS location data, camera details, etc..)
Code :
public ArrayList<Object> uploadImagesToHbase(MultipartFile uploadedFileRef){
byte[] bytes =uploadedFileRef.getBytes();
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
ImageIO.write(image, "jpg", outputStream);
HBaseAdmin admin = new HBaseAdmin(configuration);
HTable table = new HTable(configuration, "sample");
Put image = new Put(Bytes.toBytes("1"));
image.add(Bytes.toBytes("DataColumn"), Bytes.toBytes(DataQualifier), bytes);
table.put(image);
How to store and retrieve a Image with out any change / loss?
Please try using SerializationUtils from Apache Commons Lang.
Below are methods
static Object clone(Serializable object) //Deep clone an Object using serialization.
static Object deserialize(byte[] objectData) //Deserializes a single Object from an array of bytes.
static Object deserialize(InputStream inputStream) //Deserializes an Object from the specified stream.
static byte[] serialize(Serializable obj) //Serializes an Object to a byte array for storage/serialization.
static void serialize(Serializable obj, OutputStream outputStream) //Serializes an Object to the specified stream.
While storing in to hbase you can store byte[] which was returned from serialize.
While getting the Object you can type cast to corresponding object for ex: File object and can get it back.
Most likely you are just over-complicating things. :-)
The reason why you are losing the Exif and other metadata, is that the ImageIO convenience methods ImageIO.read(...) and ImageIO.write(...) does not preserve metadata. The good news is, they are not needed.
As you seem to already have the image data from the MultipartFile, you should simply store that data (the byte array) in the database, and you will store exactly what the user uploaded. No difference in file size, and metadata will be untouched.
Your code above doesn't compile for me, and I'm no HBase expert, so I just leave that out (as you have already been able to store an image, to see the size/quality difference and metadata loss, I assume you know how to do that :-) ). But here's the basics:
public ArrayList<Object> uploadImagesToHbase(MultipartFile uploadedFileRef) {
byte[] bytes = uploadedFileRef.getBytes();
// Store the above "bytes" byte array in HBase *as is* (no ImageIO)
}
I have an ArrayList of a Serializable, which I can serialize then save and load it from a file. But what if I want to add an object to the arraylist, without loading the whole list, then saving the whole thing again? I don't feel like loading the whole list, then adding an object to it and then save it again, as it would impact my performance.
These are the two method I've made for saving and loading the file. The Deck class of course implements Serializable.
public static List<Deck> loadDeckDatabase() throws IOException, ClassNotFoundException {
FileInputStream fis = new FileInputStream("decks");
ObjectInputStream ois = new ObjectInputStream(fis);
List decList = (List) ois.readObject();
ois.close();
return decList;
}
public static void saveDeckDatabase(List<Deck> decks) throws IOException, ClassNotFoundException {
FileOutputStream fos = new FileOutputStream("decks");
ObjectOutputStream oos = new ObjectOutputStream(fos);
oos.writeObject(decks);
oos.close();
}
I hope someone can help me. Thanks!
Either:
You have to load and save, as you don't know how the Deck is serialized.
You can to write your own serialization so you actually know how to append.
See also here:
https://stackoverflow.com/a/7290812/461499
Why don't you just use SQLite database? It is light, local (stored just in file) database supported by Java. The way you are using it is same that using common database.
Look at the tutorial here: http://www.tutorialspoint.com/sqlite/sqlite_java.htm
If you don't want to use the database I see two ways to dealt with you problem:
Keep every array object in other file and keep a files counter in some place - which is not very elegant solution and I guess it will increase I/O operations count
Serialize your structure to JSON and write your own method to add element. Since JSON's structure is very simple it seems to be quite easy to just add new element with simple file and string operations
I'm creating an installer and there are some resource files (.xmls, .zip files, a .jar file, etc) that must be read during installation, but I'd like to pack them into a custom file (i.e., a .dat file) so that when distributed, users don't get to mess around with them too much. The problem is that the installer must be written in Java and I've never done this sort of thing before in any programming language. Is it even possible? If so, how can I pack it in a way that can be read by my Java app afterwards and how can I make my Java app read it?
There are a lot of questions you'll need to answer for yourself about the requirements of this filetype. Does it need to be compressed? Encrypted? Does it need to support random access reading, or is stream-reading good enough?
I could be wrong, but I don't think that's what you're asking in this question. If I'm reading you correctly, I think you're asking "how do I read & write arbitrary file data?"
So that's the question I'll answer. Update your question if that's not quite what you're looking for.
Custom filetypes can easily be implemented using the DataInputStream and DataOutputStream classes. These will let you read & write primitives (boolean, char, byte, int, long, float, double) to the stream. There are also some convenience methods for reading & writing UTF-8 encoded Strings, byte-arrays, and a few other goodies.
Let's get started.
For the sake of argument, let's pretend that all my data elements are byte arrays. And each of them has a name. So my filetype can be modeled logically as a Map<String, byte[]>. I'd implement my custom filetype reader/writer class like this:
public class MyFileTypeCodec {
public static void writeToFile(File f, Map<String, byte[]> map)
throws IOException {
// Create an output stream
DataOutputStream stream = new DataOutputStream(
new BufferedOutputStream(new FileOutputStream(f))
);
// Delegate writing to the stream to a separate method
writeToStream(stream, map);
// Always be sure to flush & close the stream.
stream.flush();
stream.close();
}
public static Map<String, byte[]> readFromFile(File f)
throws IOException {
// Create an input stream
DataInputStream stream = new DataInputStream(
new BufferedInputStream(new FileInputStream(f))
);
// Delegate reading from the stream to a separate method
Map<String, byte[]> map = readFromStream(stream);
// Always be sure to close the stream.
stream.close();
return map;
}
public static void writeToStream(DataOutputStream stream, Map<String, byte[]> map)
throws IOException {
// First, write the number of entries in the map.
stream.writeInt(map.size());
// Next, iterate through all the entries in the map
for (Map.Entry<String, byte[]> entry : map.entrySet()) {
// Write the name of this piece of data.
stream.writeUTF(entry.getKey());
// Write the data represented by this name, making sure to
// prefix the data with an integer representing its length.
byte[] data = entry.getValue();
stream.writeInt(data.length);
stream.write(data);
}
}
public static Map<String, byte[]> readFromStream(DataInputStream stream)
throws IOException {
// Create the data structure to contain the data from my custom file
Map<String, byte[]> map = new HashMap<String, byte[]>();
// Read the number of entries in this file
int entryCount = stream.readInt();
// Iterate through all the entries in the file, and add them to the map
for (int i = 0; i < entryCount; i++) {
// Read the name of this entry
String name = stream.readUTF();
// Read the data associated with this name, remembering that the
// data has an integer prefix representing the array length.
int dataLength = stream.readInt();
byte[] data = new byte[dataLength];
stream.read(data, 0, dataLength);
// Add this entry to the map
map.put(name, data);
}
return map;
}
}
The basic idea is that you can write any data to an output stream (and read it back again) if you can represent that data as some combination of primitives. Arrays (or other collections) can be prefixed with their length, like I've done here. Or you can avoid writing the length prefix if you put a TERMINUS sentinel at the end (kind of like null-terminated strings).
I always use this kind of setup when I implement a custom filetype codec, with file IO methods delegating down into stream IO methods. Usually, I discover later that the object I'm reading & writing from this stream could be just as easily written into some larger & more complex file.
So I might have a SuperFancyCodec for reading/writing the data for my whole system, and it calls down into my TinySpecialPurposeCodec. As long as the stream reading & writing methods are public, then I can assemble new filetypes using a component-oriented methodology.
The extension usually have very little to do with how the file is interpreted.
If you'd like to have just config.dat instead of config.xml you just rename the file. (You'd typically give an xml-parser an InputStream or a Reader as input, which may read any file, regardless of extension)
If the problem you're describing is about combining multiple files, (.zip, .jar, etc) into a single .dat file, you could for instance zip them together, and name the zip file with a .dat extension. Java has good support for zip-files and can handle the zip file regardless of filename / extension.
Related link: Reading the Contents of a ZIP File
When creating/reading files in Java (or anything else), the file extension is not strictly tyed to the actual structure of the file's data. If I wanted, I could make an XML files file.gweebz. OS's and applications would not know what to do with it, but once opened, it would be clear that it is XML.
That being said, it is often good to follow the conventions already established and usually .dat files are files in a binary format. You can use .dat for what you want, but be warned that some users may have OS bindings for the file type and clicking on your file may cause different-than-expected behavior on their systems.
As for how to do it in Java. Grabbing a file handle in Java is easy...
File myFile = new File("/dir/file.gweebz");
It is as simple as that and you can name it whatever you want. You will need other classes to write and read from the file or to do compression, but I will assume you know how to do that. If not, this site will have the answer.
I have a file that contains bytes, chars, and an object, all of which need to be written then read. What would be the best way to utilize Java's different IO streams for writing and reading these data types? More specifically, is there a proper way to add delimiters and recognize those delimiters, then triggering what stream should be used? I believe I need some clarification on using multiple streams in the same file, something I have never studied before. A thorough explanation would be a sufficient answer. Thanks!
As EJP already suggested, use ObjectOutputStream and ObjectInputStream an0d wrap your other elements as an object(s). I'm giving as an answer so I could show an example (it's hard to do it in comment) EJP - if you want to embed it in your question, please do and I'll delete the answer.
class MyWrapedData implements serializeable{
private String string1;
private String string2;
private char char1;
// constructors
// getters setters
}
Write to file:
ObjectOutputStream out = new ObjectOutputStream(new FileOutputStream(fileName));
out.writeObject(myWrappedDataInstance);
out.flush();
Read from file
ObjectInputStream in = new ObjectInputStream(new FileInputStream(fileName));
Object obj = in.readObject();
MyWrapedData wraped = null;
if ((obj != null) && (obj instanceof MyWrappedData))
wraped = (MyWrapedData)obj;
// get the specific elements from the wraped object
see very clear example here: Read and Write
Redesign the file. There is no sensible way of implementing it as presently designed. For example the object presupposes an ObjectOutputStream, which has a header - where's that going to go? And how are you going to know where to switch from bytes to chars?
I would probably use an ObjectOutputStream for the whole thing and write everything as objects. Then Serialization solves all those problems for you. After all you don't actually care what's in the file, only how to read and write it.
Can you change the structure of the file? It is unclear because the first sentence of your question contradicts being able to add delineators. If you can change the file structure you could output the different data types into separate files. I would consider this the 'proper' way to delineate the data streams.
If you are stuck with the file the way it is then you will need to write an interface to the file's structure which in practice is a shopping list of read operations and a lot of exception handling. A hackish way to program because it will require a hex editor and a lot of trial and error but it works in certain cases.
Why not write the file as XML, possibly with a nice simple library like XSTream. If you are concerned about space, wrap it in gzip compression.
If you have control over the file format, and it's not an exceptionally large file (i.e. < 1 GiB), have you thought about using Google's Protocol Buffers?
They generate code that parses (and serializes) file/byte[] content. Protocol buffers use a tagging approach on every value that includes (1) field number and (2) a type, so they have nice properties such as forward/backward compatability with optional fields etc. They are fairly well optimized for both speed and file size, adding only ~2 bytes of overhead for a short byte[], with ~2-4 additional bytes to encode the length on larger byte[] fields (VarInt encoded lengths).
This could be overkill, but if you have a bunch of different fields & types, protobuf is really helpful. See: http://code.google.com/p/protobuf/.
An alternative is Thrift by Facebook, with support for a few more languages although possibly less use in the wild last I checked.
If the structure of your file is not fixed, consider using a wrapper per type. First you need to create the interface of your wrapper classes….
interface MyWrapper extends Serializable {
void accept(MyWrapperVisitor visitor);
}
Then you create the MyWrapperVisitor interface…
interface MyWrapperVisitor {
void visit(MyString wrapper);
void visit(MyChar wrapper);
void visit(MyLong wrapper);
void visit(MyCustomObject wrapper);
}
Then you create your wrapper classes…
class MyString implements MyWrapper {
public final String value;
public MyString(String value) {
super();
this.value = value;
}
#Override
public void accept(MyWrapperVisitor visitor) {
visitor.visit(this);
}
}
.
.
.
And finally you read your objects…
final InputStream in = new FileInputStream(myfile);
final ObjectInputStream objIn = new ObjectInputStream(in);
final MyWrapperVisitor visitor = new MyWrapperVisitor() {
#Override
public void visit(MyString wrapper) {
//your logic here
}
.
.
.
};
//loop over all your objects here
final MyWrapper wrapper = (MyWrapper) objIn.readObject();
wrapper.accept(visitor);