In different languages I need to provide users with a stream of JSON objects with an interface similar to the following:
JSONObject json = stream.nextJSON();
Since it is a stream, each call will block until a full object has been retrieved. This means it makes no sense to try and encapsulate each JSON object inside a big array. An extra layer of structure and processing has to be added to the stream.
I have thought of two options:
Segmenting the stream with the null-termination character.
Writing a primitive parser that understand JSON scope so can detect the end of an object.
Each of the above have a number of potential issues to discuss: How will null-termination interact with the file system, socket or underlying streams in C++, Java and other languages? What edge cases would we need to take in to account when parsing? (different types of quote symbol might confuse a parser, for example). Furthermore, there might be alternatives to the two above.
So the question is: What is the best way to provide a JSON InputStream?
Well Google already thought about it apparently:
http://sites.google.com/site/gson/streaming
Related
In Python i have a class with some string attributes and a function that returns an object of this class with atttributes set (sometimes can return an array of objects).
Theres any way to get this return in Java? Where i can see the strings of the object?
I Tried Jython but couldnt make it work!
Use json.dump function in Python to serialize your object into json format. Then use something like json.org library in Java to parse this object into Java object, some example over here.
Mind that not every object might be serializable, in general data structures like dictionaries or lists are easily serializable, from your description it seems like you want to move an instance of an object from one program into another, which is not possible to be done automatically and requires human work in rewriting the code as instances of classes contain not only data but also functions (methods).
Good luck!
I need to parse untrusted Java serialized objects. The data is given to me as a byte array (written at some point by ObjectOutputStream).
I do not want to simply call ObjectInputStream.readObject() and/or load the actual object. I am looking for a way to safely parse the bytes and grab field names & values.
--
Here's a little summary of my attempt so far, after taking a look at the ObjectInputStream procedure for deserializing objects.
I have tried to extract field types/names (as unicode strings) recursively based on expected stream constants. I end up with a list of field names whose values should appear in the byte array in order. I am uneasy about this approach because it is probably buggy. Especially accommodating for what seems to be individual serialization protocols followed by HashMap, ArrayList, etc. But it might work, if I can figure out a way to read the bytes that represent field values:
I can try to read and store primitives based on size/offset, but when I encounter my first object, it gets a bit more complicated -- there is no clear way to distinguish between which bytes are associated with which values anymore (without actually loading the object in the way that ObjectInputStream probably does?).
--
Can anyone suggest either a potential solution that I'm obviously looking past, or a trusted library that can help parse the serialized data without loading objects?
Thank you for reading, and for all comments/suggestions!!! I apologize if something is unclear and I would be happy to clarify if you bear with me.
You can't do this in principle. Any Java class can take over its own Serialization and write arbitrary data to the stream that only it knows how to parse and reconstruct, via code that is only invoked during deserialization.
I needed to save some data in Java in various ways, to a File, to a String, to System.out... And I ended up with 3 methods doing pretty much the same thing. So I changed them into a single method with an OutputStream as a parameter. I wrote a few things to a single OutputStream, e.g. some text, a serialized object, another serialized object, some numerical data ...
But now I'm stuck. I overlooked the fact that I cannot distinguish between the different things that have been written. I create an InputStream for the data. I use a Scanner on that stream to read the text first, and then I tried using an ObjectInputStream to read the serialized objects, but I get an EOFException.
I guess that the Scanner reads ahead. How can I prevent the scanner to read ahead.
Or rather, how can I read each group of data using an appropriate InputStream for each of them.
You really don't want to try using different readers to read from the same stream. Even if you manage to get it working on your machine, it might break when you run it on a different OS or with a different JVM implementation.
You should choose a single method of reading and writing data. Since you're using serialized objects in the stream you're probably best off using that for everything. You already pointed out in your comments that it would be very difficult to read binary data in through a string and interpret it correctly. However, it's not hard to take a String object, write it out on the output stream, read it back in and cast it as a String.
Now there's the problem of interpreting your data. I suggest writing everything out in tag-data pairs. You write out an Integer first (maybe the ordinal of an enum to make them easier to use in your program), then you write out your data. The integer represents the type of data that's coming next in the stream (e.g. either Text or Object), and then the next object you read in is the data and you know what type it is. If it is Text you can cast the object to a String, and pass it into a Scanner, and if it's an object then you just do whatever you need to do with the object.
To make things a bit cleaner you could build a wrapper around the stream with a method for each data type. Maybe you could have a getNextObject() method and a getNextTextScanner() method. Each would first check the next Integer tag in the stream to make sure it's reading the right data (throwing an exception if it finds a mismatch), and then would either return the next Object or return a new Scanner for processing a String of data.
Really, it would be better if you could use separate streams for the two different types of data. But, if you're really stuck using the same stream then that's how I'd do it.
Yes, I know it's bad practice and I should instead normalize my tables. That put aside, is it possible to serialize a String [] array and store it in the database?
I am from the lenient and forgiving world of PHP, where invoking the serialize() function and would convert the array into a string.
Is there an equivalent of doing such heresy in Java?
Apart from normalization, are there more elegant ways of storing String Arrays in the database?
In case it's applicable, I am using the jdbc driver for my MySQL connections.
Yes. You can serialize any Java objects and store the serialized data into MySQL.
If you use the regular serialization (ObjectOutputStream), the output is always binary. Even String is serialized into binary data. So you have to Base64 encode the stream or use a binary column like BLOB.
This is different from PHP, whose serialize() converts everything into text.
You can also use the XML serialization in Java (XMLEncoder) but it's very verbose.
If you're thinking in terms of raw arrays, you're still writing PHP in Java.
Java's an object-oriented language. An array of Strings really isn't much of an abstraction.
You'll get perfectly good advice here telling you that it's possible to serialize that array of Strings into a BLOB that you can readily store in MySQL, and you can tell yourself that leniency is a virtue.
But I'll going to remind you that you're losing something by not thinking in terms of objects. They're really about abstraction and encapsulation and dealing with things at a higher level than bare metal ints, Strings, and arrays.
It'd be a good exercise to try and design an object that might encapsulate an array or another more sophisticated data structure of child objects that were more than Strings. There'd be a 1:m relationship between parent and child that would better reflect the problem you were really trying to solve. That would be a far more object-oriented design than the one you're proposing here.
There are various good serialization/deserialization libraries that automatically convert JavaBean objects to/from XML and JSON strings. One I've had good experience with is XStream.
Java's built-in support for serialization can do the same thing, and you can write custom serialization/deserialization methods for Java to call.
You can roll your own serialization methods too, eg converting to and from a comma-separated value (CSV) format.
I'd opt for a library like XStream first, assuming there's a very compelling reason not to normalize the data.
You don't want to serialize the array. I'm not sure why you'd serialize it in PHP either, because implode() and explode() would be more appropriate. You really should normalize your data, but aside from that, you could very easily Google a solution for converting an array to a string.
But surely the more logical thing to do would be to save each string as its own record with a suitable identifier. That would probably be less coding than serializing -- a simple loop through the elements of the array -- and would result in a clean database design, rather than some gooey mess.
If you really don't want to normalize this values into a separate table where each string would be in its own row, then just convert your array to a list of comma separated values (possibly escaping commas somehow). Maybe quoting each string so that "str1","str2".
Google for CSV RFC for spec on how this should be properly escaped.
What is the commonly accepted method for converting arbitrary objects to and from their String representations, assuming that the exact class of the object is known ? In other words, I need to implement some methods similar to the following:
public interface Converter {
/**
* Convert this object to its String representation.
*/
public String asString(Object obj);
/**
* Take the String representation of an object produced by asString,
* and convert it back to an object of the appropriate class.
*/
public Object asObject(String stringRepresentation, Class clazz);
}
Ideally, the solution should:
Use the object's built-in toString() functionality, if possible. Thus, converter.asString(new Integer(5)) should return "5", and converter.asObject("5", Integer.class) should return an Integer with the value of 5.
Produce output that is human-readable whenever possible.
Deal with all common Java data types, including java.util.Date .
Allow me to plug in conversion functionality for my own, custom classes.
Be reasonably light-weight and efficient.
I understand that there are any number of ready-made solutions that do this (such as Google's protocol buffers, for example), and that I could easily implement a one-off solution myself. My question is not, "how do I solve this problem", but rather, "which one of the many ready-made solutions is the current industry standard ?".
My question is not, "how do I solve this problem", but rather, "which one of the many ready-made solutions is the current industry standard ?".
None of them have emerged as defacto standard.
The closest you can get it "default" XML serialization mechanism which BTW sucks if you pretend to write them by hand ( and It is good enough when you use them automatically )
The next thing closest to an standard and that would be for daily usage, would be JSON to Java, but, well, you know, it is not Java Java
I would vote for Json as well and then particularly Gson. It handles generic/parameterized objects very well.
Alternatively, you can also write a generic object converter which does all of the needed conversions with a little help of reflection, such as this example. But if your "API" require that this converter is to be published as an interface to the enduser, then I would only suggest to replace
public Object asObject(String stringRepresentation, Class clazz);
by for example
public <T extends Object> T asObject(String stringRepresentation, Class<T> clazz);
so that one doesn't need to cast it afterwards.
You can look at the svenson library, it converts java objects to json. Its pretty quick and uses annotations to introduce custom converters.
http://code.google.com/p/svenson/
Not long ago I would have proposed an xml serializer, but after playing with couchdb for a couple of days, I serve a new master... json.
Although it is tempting to use or attempt to implement "toString()" as a reversible operation, the purpose of "toString()" is to generate a user-friendly and easily understandable representation of an object, and this goal is often at odds with including enough state information to truly restore the original object.
If you are looking to persist an object, using XML, JSON, or binary serialization is probably the best way to go. The "toString()" function should report a human-friendly representation of an object (e.g. "5", "(3,0,2)", "5+6i", "{1, 2, 3, 4, 5, 6}", "{x => y, z => 3}", etc.). Even in cases where it is possible to completely restore the object from the generated string, the time to write a function to parse each type of (potentially unstructured) text is best conserved via automated XML persistence in favor of time to write the actual application.
I agree with Oscar that XML might be the preferable form here, if you can tolerate large uncompressed file sizes. To elaborate on his answer, in my experience if you write a fairly straightforward utility class you can serialize your objects into XML with not too much work. To read them back, I would recommend Apache Digester which does a great job of rule-based interpretation.
I would only opt for other file formats if I cared about performance or file sizes, though I eprsonally in most cases prefer the flexibility of XML.