i'm generating several large Hashmaps and putting them into 1 object on Windows (using Java). Now i want to transfer this object to my Android app by giving it as an Asset.
I have tried using the Java serialization, but when i unpack the object in my App, all hashmaps have null values.
My custom classes all implement Serialzable and contain basic types or other custom classes. (which contain basic types or other custom classes, and so on, the chain does end, its not a circle).
Does anybody know what could be the problem?
Or does anyone know a XMLSerializer(or other) that supports flattening private fields without getters and setters?
Google Protocol Buffers is probably today the most efficient and convenient serialization/deserialization solution on Android and can be connected to many languages platforms.
As the alternatives mentioned here are java serialization and XML, I suggest you look at the chapter in the FAQ answering Why use protobuf.
This article offers a small example of protobuf usage on android (and a kind of dubious benchmark comparison).
Related
I'm new to this Apache Avro(serialization framework). I know what serialization is but why there are separate frameworks lik avro, thrift, protocol buffers and
Why cant we use java serialization api's instead of these separate frameworks, are there any flaws in java serializatio api's.
What is the meaning of below phrase
"does not require running a code-generation program when a schema changes" in avro or in any other serializatio framework.
Please help me to understand all these!!
Why cant we use java serialization api's instead of these separate frameworks, are there any flaws in java serializatio api's.
I would assume you can use Java Serialization unless you know otherwise.
The main reasons not to use it are
you know there is a performance problem.
you need to exchange data across languages. Java Serialization is only for Java.
does not require running a code-generation program when a schema changes
I am guessing this means it can read serialized data with an older or newer model without having to re-generate and compile the code. i.e. it is tolerant of changes in the model.
BTW: As the data models I work with are usually a) very simple b) require maximum performance, I write my own Serialization without using a framework (or write my own framework) This is fine provided your model is very simple and won't change often.
In short, unless you know you can't, try Java Serialization first.
A comparison I did on different Serialization Methods
1.
The problem with java serialization is that it's not agnostic of your code. Meaning that is tightly coupled to the structure of you classes. Other serialization frameworks provide you with some flexibility/control that it's useful to bypass this kind of situations. Even though there is a way in java standard mechanism to control serialization through the writeObject readObject methods, it is a problem that other fwks have addressed in a more elegant way.
Second, you cannot interexchange the output of your java serialization with other language - platforms.
Last, but not least. Java serialization does not produce the more compact result possible, which might lead to performance degradation if you perform things like transfer data over a network. Other protocols (like Oracle's POF or protocol buffers) are more optimized to produce an smaller output.
2.
Regarding your second question I guess that what that means is that you don't need to run any precompile job that generates code in the case that the structure of your serialized classes changes. I personally hate frameworks that force some kind of compile-time code generation. I hate the hassle of having to even have to look at generated code, but that is just me and my ocd.
Two principle things Avro does well: Hadoop's MapReduce and communication protocol structures. I use it for MapReduce where I put numerous data instances in a single file all conforming to a particular schema; each record is stored very efficiently and markers delineate each individual record. Hadoop also uses it to communicate data between the Map and Reduce tasks. Much better than storing field names alongside data. These files are easy to split into multiple parts for processing in a distributed computing environment. Since the schema is embedded into the file, a reader doesn't have to know what the data looks like. Avro is not tied to any language and there are several language APIs for reading Avro data. If you want to write out a single complex object, then Java's serialization OR Avro will work. If you want more power and efficiency and are using millions of individual instances, then Avro is a good alternative. I am sure you can do this with the Java API, but why work that hard.
There are mechanisms to evolve schemas thru the schema resolution rules. There are also tools that will turn your java objects into schemas for you.
The best place to start is here: http://avro.apache.org/docs/current/spec.html It may take a couple of reads to get the gist. Read it again after trying to use some of the tools that come with the Avro package. Avro takes a while to get the hang of. JSON is only used as a data specification language it isn't used to store the data. You can generate schemas using the API or using a JSON file. Lots of flexibility and enough rope to easily get into trouble with -- well worth it tho.
I need a framework to transfer POJOs between two (or more in a client/server model) Java programs over TCP/IP. I need it to be as simple as possible but it must support several clients per server, and easy implementation of encryption is a plus.
So far I have looked at Java RMI, JRemoting, AltRMI and NinjaRMI. Right now JRemoting looks like the best choice as it is simple and don't require strange and seemingly unnecessary extends and implements as most of the others do. No active development seems to be going on on any of them except a little on Java RMI. I don't know if that is because they are stable and don't need more development, or because these kinds of frameworks are just not "cool" any more.
The POJOs are just bags of properties. I need the server to hold a static list of objects, and the clients must be able to (1) Read the list, (2) Add an object to the list, and (3) Remove an object from the list.
Any suggestions?
You could probably use any serialization technology, for example you could use JSON and add on encryption and compression later in order to cutdown the amount of traffic you are sending. JSON has the advantage of being language agnostic, so you don't restrict the implementation of either side of the connection.
Many JSON libraries are available; see json.org.
Do you need to do remote method calls, or are your POJOs just bags of properties? If the latter, it would probably be easiest to just use plain Java serialization.
look here: http://code.google.com/p/google-gson/
You can have a look on Protocol Buffers. I think Google uses Protocol Buffers internally.
I'm looking for a super simple JSON or YAML library (not particularly bothered which one) written in Java, and can be used in both GWT on the client, and in its original Java form on the server.
What I'm trying to do is this: I have my models, which are shared between the client and the server, and these are the primary source of data interchange. I want to design the web service in between to be as simple as possible, and decided to take the RESTful approach.
My problem is that I know our application will grow substantially in the future, and writing all the getters, setters, serialization, factories, etc. by hand fills me with absolute dread. So in order to avoid it, I decided to implement annotations to keep track of attributes on the models.
The reason I can't just serialize everything directly, using GWT's own one, or one which works through reflection, is because we need a certain amount of logic going on in the serialization process. I.e. whether references to other models get serialized during the serialization of the original model, or whether an ID is just passed, and general simple things like that. I've then written an annotation processor to preprocess my shared models and generate an implementing class with all the getters, setters, serialization, lazy-loading, etc.
To make a long story short, I need some type of simple YAML or JSON library, which allows me to encode and decode manually, so I can generate this code through my annotation processor. I have had a look around the interwebs, but every single one I ran into supported some reflection which, while all fine and dandy, make it pretty much useless for GWT. And in the case of GWT's own JSON library, it uses JSNI for speed purposes, making it useless server side.
One solution I did think about involved writing writing two sets of serialization methods on the models, one for the client and one for the server, but I'd rather not do that.
Also, I'm pretty new to GWT, and even though I have done a lot of Java, it was back in the 1.2 days, so it's a bit rusty. So if you think I'm going about this problem completely the wrong way, I'm open to suggestions.
Have you looked into itemscript? Some excerpts from the description on the webpage:
A cross-platform GWT & standard Java JSON library, with convenient classes, parsers, and utilities.
A RESTful connector API for retrieval of data (JSON, text & small binary files) over a variety of protocols.
The same JSON API can be used in both standard Java and in GWT Java.
I am creating a WCF Service that will be consumed by both .NET and Java client applications.
We do not have any Java experience in the team, so are looking for guidelines or rules to follow to ensure that we do not accidentally include any types in our WCF Service interface or do anything else that would preclude it from being consumed by a Java client application.
Are our worries well-founded? If so, what should we be wary of?
Edit
One example of a concern is whether a .NET DateTime value is represented in the service interface in a manner that can be correctly understood by a Java client.
Edit2
A second example of a concern is the use of any nullable value types (bool?, int? etc).
Edit3
At present some of our development teams are hand-writing .xsd files to define the various objects that the WCF interface methods will take as arguments and return as return values. They are then using xsd.exe to auto-generate C# classes from these.
The rationale behind this is that it guarantees that the generated classes won't contain anything that is .NET-specific.
The downside is that this adds a development burden and also precludes us from documenting these classes using <summary> tags (.NET equivalent of javadoc comments).
The recommendation to start with XSD is a good one. That will not guarantee compatibility on each side, as XML Schema is really big and no web services stack supports all of it. (Example: lists).
So, start with XSD, but confine yourself to mainstream types. Primitives, complextypes composed of primitives, arrays of same. You can safely nest complextypes and arrays. (arrays of complextypes, complextypes that contain arrays or complextypes, etc).
Stay away from restrictions, substitution groups, lists, derivations, and any other XSD esoterica. Even XSD enumerations should be avoided.
About dateTime:
It's not enough to use a nullable datetime. There are formatting concerns as well. The .NET DateTime is a higher resolution quantity than a Java Calendar and as a result, shipping a .NET time to Java can result in de-serialization exceptions on the Java side. (EDIT: using the DataType="dateTime" decorator in the XmlElement attribute on the .NET side can make sure you serialize properly)
Some old advice on that.
Finally, it is not true that you cannot use in-code XML doc on the classes that get generated. With C#'s partial classes, you can write separate code from the generated classes with the in-code doc you want. Even if you re-gen the code, your partial class code will remain unchanged. EDIT: When you compile, the doc will appear on the classes.
EDIT: Someone asked, if using XSD-first is not enough to guarantee interop, why use it? My answer: it is not a guarantee but it is a good step, it helps. It keeps you away from designing interfaces in code (either Java or C# or VB, etc) that expose platform-specific things like .NET DataSets, generic Dictionaries, Java ResultSets, etc, all of which present interop problems. There are still pitfalls in the more marginal parts of XSD, but you can usually avoid those with thoughtful design.
I should have mentioned in my original answer that you can apply an iterative process to the development of the interface. Design in XSD, then generate (client) stub and (server) skeleton code from the XSD+WSDL, then adjust and do it again.
Using basicHttpBinding endpoint will guarantee that any SOAP 1.1 compatible client will be able to consume your service.
Use DateTime?, i.e. a nullable struct. Java does not have the notional of complex value types (i.e. structs). I believe there are ways to specify in the WSDL not to allow nulls, but I think WCF doesn't do it out of the box.
Use arrays instead of collections. If I recall correctly, the collection types do not translate very well over SOAP, but array types do quite well.
You should get a copy of Eclispe or Netbeans. After you create a prototype WCF service, run the web service wizard to create your proxies. Examine the object model for any major defects with particular emphasis on complex objects or non-primitive types (treat string as a primitive).
The learning curve to do that with Netbeans or Eclipse is pretty flat, so it's not a huge burden.
Edit: The other potential problems are with the bindings. If you stick with HTTP(S), you should be good... start going to the alternatives like TCP or MSMQ, you'll have to do a lot of work in Java. Also, some of the security features don't interop in all cases, such as using NTLM tokens... Take an iterative approach. Start with a simple HTTP w/SOAP binding with no security and go from there.
Does WCF provide standard SOAP interfaces? If it does, getting Java to talk to it should be a doddle.
Re: Edit1: The WSDL / XSD will use a standard date time format (Calendar at the Java end, a formatted datetime string in the SOAP, DateTime in .NET), or you could force it into a string format of your own choosing.
Read up on the Apache Axis (1.4 and 2.0) documentation for Java web services. It's very easy to use to get a Java web service client set up from the wsdl/xsd that your web service will provide.
Edit3: In Java you would define a Java model (with all your favoured documentation), then run Java2WSDL (preferably as an ANT/Maven task) to create your WSDL (however I have found you need to hand-reorder the fields in it). Axis 2 supports Collections and Enums just fine, Axis 1.4 likes Arrays and hand-rolled Java 1.4 style enumerations. From this WSDL you would create a server-side skeleton using WSDL2Java in which the only thing you need to do is implement your business logic.
I'm currently working on a project which needs to persist any kind of object (of which implementation we don't have any control) so these objects could be recovered afterwards.
We can't implement an ORM because we can't restrict the users of our library at development time.
Our first alternative was to serialize it with the Java default serialization but we had a lot of trouble recovering the objects when the users started to pass different versions of the same object (attributes changed types, names, ...).
We have tried with the XMLEncoder class (transforms an object into a XML), but we have found that there is a lack of functionality (doesn't support Enums for example).
Finally, we also tried JAXB but this impose our users to annotate their classes.
Any good alternative?
It's 2011, and in a commercial grade REST web services project we use the following serializers to offer clients a variety of media types:
XStream (for XML but not for JSON)
Jackson (for JSON)
Kryo (a fast, compact binary serialization format)
Smile (a binary format that comes with Jackson 1.6 and later).
Java Object Serialization.
We experimented with other serializers recently:
SimpleXML seems solid, runs at 2x the speed of XStream, but requires a bit too much configuration for our situation.
YamlBeans had a couple of bugs.
SnakeYAML had a minor bug relating to dates.
Jackson JSON, Kryo, and Jackson Smile were all significantly faster than good old Java Object Serialization, by about 3x to 4.5x. XStream is on the slow side. But these are all solid choices at this point. We'll keep monitoring the other three.
http://x-stream.github.io/ is nice, please take a look at it! Very convenient
of which implementation we don't have any control
The solution is don't do this. If you don't have control of a type's implementation you shouldn't be serialising it. End of story. Java serialisation provides serialVersionUID specifically for managing serialisation incompatibilities between different versions of a type. If you don't control the implementation you cannot be sure that IDs are being changed correctly when a developer changes a class.
Take a simple example of a 'Point'. It can be represented by either a cartesian or a polar coordinate system. It would be cost prohibitive for you to build a system that could cope dynamically with these sorts of corrections - it really has to be the developer of the class who designs the serialisation.
In short it's your design that's wrong - not the technology.
The easiest thing for you to do is still to use serialization, IMO, but put more thought into the serialized form of the classes (which you really ought to do anyway). For instance:
Explicitly define the SerialUID.
Define your own serialized form where appropriate.
The serialized form is part of the class' API and careful thought should be put into its design.
I won't go into a lot of details, since pretty much everything I have said comes from Effective Java. I'll instead, refer you to it, specifically the chapters about Serialization. It warns you about all the problems you're running into, and provides proper solutions to the problem:
http://www.amazon.com/Effective-Java-2nd-Joshua-Bloch/dp/0321356683
With that said, if you're still considering a non-serialization approach, here are a couple:
XML marshalling
As many has pointed out is an option, but I think you'll still run into the same problems with backward compatibility. However, with XML marshalling, you'll hopefully catch these right away, since some frameworks may do some checks for you during initialization.
Conversion to/from YAML
This is an idea I have been toying with, but I really liked the YAML format (at least as a custom toString() format). But really, the only difference for you is that you'd be marshalling to YAML instead of XML. The only benefit is that that YAML is slightly more human readable than XML. The same restrictions apply.
google came up with a binary protocol -- http://code.google.com/apis/protocolbuffers/ is faster, has a smaller payload compared to XML -- which others have suggested as alternate.
One of the advanteages of protocol buffers is that it can exchange info with C, C++, python and java.
Try serializing to json with Gson for example.
Also a very fast JDK serialization drop-in replacement:
http://ruedigermoeller.github.io/fast-serialization/
If serialization speed is important to you then there is a comprehensive benchmark of JVM serializers here:
https://github.com/eishay/jvm-serializers/wiki
Personally, I use Fame a lot, since it features interoperability with Smalltalk (both VW and Squeak) and Python. (Disclaimer, I am the main contributor of the Fame project.)
Possibly Castor?
Betwixt is a good library for serializing objects - but it's not going to be an automatic kind of thing. If the number of objects you have to serialize is relatively fixed, this may be a good option for you, but if your 'customer' is going to be throwing new classes at you all the time, it may be more effort than it's worth (Definitely easier than XMLEncoder for all the special cases, though).
Another approach is to require your customer to provide the appropriate .betwixt files for any objects they throw at you (that effectively offloads the responsibility to them).
Long and short - serialization is hard - there is no completely brain dead approach to it. Java serialization is as close to a brain dead solution as I've ever seen, but as you've found, incorrect use of the version uid value can break it. Java serialization also requires use of the marker 'Serializable' interface, so if you can't control your source, you are kind of out of luck on that one.
If the requirement is truly as arduous as you describe, you may have to resort to some sort of BCE (Byte code modification) on the objects / aspects / whatever. This is getting way outside the realm of a small development project, and into the realm of Hibernate, Casper or an ORM....
SBE is an established library for fast, bytebuffer based serialization library and capable of versioning. However it is a bit hard to use as you need to write length wrapper classes over it.
In light of its shortcomings, I have recently made a Java-only serialization library inspired by SBE and FIX-protocol (common financial market protocol to exchange trade/quote messages), that tries to keep the advantages of both while overcoming their weaknesses. You can take a look at https://github.com/iceberglet/anymsg
Another idea: Use cache. Caches provide much better control, scalability and robustness to the application. Still need to serialize, though, but the management becomes much easier with within a caching service framework. Cache can be persisted in memory, disk, database or array - or all of the options - with one being overflow, stand by, fail-over for the other . Commons JCS and Ehcache are two java implementations, the latter is an enterprise solution free up to 32 GB of storage (disclaimer: I don't work for ehcache ;-)).