What is the right way to persist data defined using protobuf3. I am using golang and Java, both place with support of ORMs. In java with Hibernate and golang with gorm. Both place i need to convert the Generated code to corresponding Entity model. I feel that is more pain full to maintain same object structure in order to be understandable by ORM. Is there any Database which i can use along with protobuf objects as is. Or i can define the relations between objects in the protobuf itself.
Any helps really appreciated.
There is a not-straightforward solution to this problem.
Protobuf 3 standardises JSON mapping for the messages. Once you serialise your message to JSON, you have multiple options for storing it in a database.
The following (and many more) databases can store JSON data:
MariaDB
PostgreSQL
MongoDB
Your ORM is dealing with objects, by definition. It should not know or care about serialization on the network. I'd suggest deserializing the protobuf message into objects that your ORM is used to and letting it persist them. There's no good reason to couple your persistence tier to the network protocol.
It might make sense to store the protobuf serialization directly if you get rid of JPA and go with a document based solution.
You have to decide how much value JPA is providing for you.
Although this question is quite old, things have happened since then and the FoundationDB Record Layer, released by Apple in 2018, stores Protocol Buffer natively.
In Go, I don't know about gorm, but it seems that with Ent (a competing ORM) Protobufs can be deserialized into exactly the same objects which are used for DB tables/relations. Ent's official tutorial for that.
The caveat is that you specify your Protobuf with Ent's Golang structures, not via the standard proto3 language.
Related
Does hibernate internally uses Serialization for persisting POJO classes? If yes, How does it use it in persisting data? If No, then how does it persist data to DB?
Hibernate persists data to the database using SQL. Java serialization is not used at all. (SQL) databases are language-agnostic. As such, they cannot depend on language-specific technology such as Java serialization.
Serialization is only relevant when you need to send a POJO over the wire to other servers running Java. For example, if you have some sort of cache of POJOs that spans multiple machines, you could use serialization to send copies of the POJO over the wire.
See https://stackoverflow.com/a/2726387/14731 for a related discussion.
I want to transfer hibernate objects with GWT-RPC to the frontend. Of course i can not transfer the annotated class because the annotations can not be compiled to javascript. So i did the hibernate mapping purely in the ".hbm.xml". This worked fine for very simple objects. But as soon as i add more complex things like a oneToMany relationship realized with e.g. a set, the compiler complains about some serialization issues with the set (But the objects in the set are serializable as well).
I guess it does't work because hibernate creates some kind of special set that can not be interpreted by GWT?
Is there any way to get around this or do i need another approach to get my objects to the frontend?
Edit: It seems that my approach is not possible with RPC because hibernate changes the objects. (see answer from thanos). There is a newer approach from google to transfer objects to the the frontend: The request factory. It looks really good and i will try this now.
Edit2: Request factory works perfectly and is much more convenient than RPC!
This is a quote from GWT documentation. It says that hibernate changes the object from the original form in order to make it persistent.
What this means for GWT RPC is that by the time the object is ready to be transferred over the wire, it actually isn't the same object that the compiler thought was going to be transferred, so when trying to deserialize, the GWT RPC mechanism no longer knows what the type is and refuses to deserialize it.
Unfortunately the only way to implement the solution is by making DTOs and their appropriate converters.
Using Gilead is a cleaner approach (no need for all this DTO code), but DTOs are more ligtweight and thus produce less traffic through the wire.
Anyhow there is also Dozer, that will generate the DTOs for you so there will not be much need for yo to actually write the code.
Either way as mchq08 said the link he provided will solve many of questions.
I would also make another suggestion! Separate the projects. Create a new one as a model for your application and include the jar into the GWT. In this way your GWT project will be almost in its' entirety the GUI and the jar library can be re-used for other projects too.
When I created my RPC to Hibernate I used this example as a framework. I would recommend downloading their source code and reading the section called "Integration Strategies" since I felt the "Basic" section did not justify DTO. One thing this tutorial did not go over as well is the receiving and sending part from the web page(which converts to JS) so thats why I am recommending you downloading their source code and looking at how they send/receive each the DTOs.
Post the stack trace and some code that you believe will be useful to solving this error.
Google's GWT & Hibernate
Reading this (and the source code) can take some time but really helps understands their logic.
I used the next approatch: for each hibernate entity class I had client replica without any hibernate stuff. Also I had mechanism for copy data between client <-> server clases.
This was working, but I belive current GWT version should work with hibernate-annotated classes..
On a client project, I use Moo (which I wrote) to translate Hibernate-enhanced domain objects into DTOs relatively painlessly.
I'm currently searching for a really easy way to get simple Java Objects persistent in Databases and/or XML and/or other types of data stores.
For big projects in the company i would use hibernate, ibatis, datanucleus or something like that. But with small private projects this will take over 80% of the worktime.
I also found "simpleORM" but this one requires to code data-related stuff pretty hardly into the data-model classes. I don't really like that style so this is no option for me.
Do you have a suggestion for some library which simply takes my objects and saves / loads them as they are or with very little configuration?
You could try my ORMLite library, which was designed as a simple replacement for hibernate and iBatis. I'm the main author. It supports a number of JDBC databases and has an Android backend. Here is the getting started section of the manual which has some code examples. Here also are working examples of simple usage patterns.
Try Norm. It's a lightweight layer over JDBC. It adds almost zero overhead to JDBC calls and is very easy to learn.
You could just serialize your objects into a file/database whatsoever.
If you want to define the mapping then you'd have to go for more configuration and the standard OR mappers out there (like Hibernate) don't really add that much on top.
You could try xstream. It's really simple OXM library working without upfront configuration.
Sample code:
XStream xstream = new XStream();
// marshalling
String xml = xstream.toXML(domainObject);
// unmarshalling
domainObject = xstream.fromXML(xml);
For relational database persistence try one of the JPA implementations, such as OpenJPA.
The setup overhead is minimal. You can let JPA to create your schema & tables for your from your object definitions, so you don't need to hand crank any sql. All you need to supply is some annotations on your entities and a single config file, persistence.xml.
You can also use jEasyORM (http://jeasyorm.sourceforge.net/).
In most cases it automatically maps objects to database tables with no need for configuration.
You may want to consider www.sormula.org. Minimal programming/annotations and simple learning curve. It uses standard SQL and JDBC so will work with any relational db.
U could try SnakeORM http://sourceforge.net/p/selibs/wiki/Home/
It doesnt have many runtime dependencies, uses JPA annotations and follows DAO pattern.
Disclosure: I am the author of this project
Well if you want an ORM, then that implies that you want to map objects to tables, columns to fields etc. In this case, if you want to avoid the hassle of bigger ORM implementations, you could just use plain old JDBC, with simple DataAccessor patterns. But then this does not translated to XML directly.
If you want to just persist the object somewhere, and only care about "understanding" the object in Java, then serialization is a simple effective method, as Thomas mentioned earlier.
You could also try my little ORM library, Java2DB. I created it specifically for small projects that just want quick and easy access to their database. Check it out on GitHub.
Onyx Database is a very feature rich Java NoSQL database alternative. It's pure java with several persisting modes (caching, embedded-database, save-to-remote, and save-to-remote-cluster. It has an embedded ORM, and is probably the easiest persistence API I've used.
I'm having a problem. I would like to create Document object, and I would like to have a user property with com.google.appengine.api.users.User type (on GAE's docs site, they said we should use this object instead of email address or something else, because this object probably will be enchanced to be unique). But now the object can't be compiled by GWT, because I don't have the source for that object.
How can I resolve the problem?
I was searching for documents about DTOs, but I realized that maybe that's not the best pattern I should use.
What do you recommend?
Very thanks for your help!
Regards,
Bálint Kriván
to avoid DTOs for objects with com.google.appengine.api.users.User inside you can probably use the work from
http://www.resmarksystems.com/code/
He has build wrappers for the Core GAE data types (Key, Text, ShortBlob, Blob, Link, User). I've tested it with datastore.Text and it worked well.
There is a lot of debate about whether you should be able to reuse objects from the server on the client. However, the reuse rarely works out well in real applications so I generally recommend creating pure java objects that you copy your data into to send to the client. This allows you to tailor the data to what you need on the client and avoids pitfalls where you accidently send sensitive information over the wire.
So in this case, I would recommend that you create a separate object to send over the wire. BTW if you have the AppEngine SDK for Java (http://code.google.com/appengine/downloads.html), it includes a demo application that I did (sticky) that demonstrates this technique.
this question also addresses the issue:
It links to a semi workable solution for automatically making your persistent objects gwt-rpc compatible.
I had the same question, your answer is interesting, but i am always sad to copy twice a data... Plus, when your dao gets the data, you will have to parse all the results to copy them to the pure java object, isn't it? It seems to be a heavy operation. What's your opinion about those question?
I am developing a Java based desktop application. There are some data generated from the application object model that I need to persist (preferably to a file). There is also a requirement to protect the persisted file so that others can't derive the object model details from the data. What's the best strategy for doing these? I was in the impression that these requirements are very common for desktop apps. However, I haven't been able to found much useful info on it. Any suggestion appreciated.
Your question has two parts. 1st: How to persist data? 2nd: How to protect them?
There is a lot of ways how to persist data. From simple XML, java serialization to own data format. There is no way how to prevent revers engineering data just by "plain text". You can just make it harder, but not impossible. To make it quite impossible you need to use strong encryption and here comes a problem. How to encrypt data and don't reveal secure token. If you are distributing secure token with your application it is just a matter of time to find it and problem is solved. So entering a secure token during installation is not an option. If user has to authenticate to use application it should help, but it is the same problem. The next option is to use custom protected bijection algorithm to obfuscate data. And the last option is to do nothing just keep the data format private and don't publish them and obfuscate your application to prevent from reverse engineering.
At the best value comes simple obfuscation of data (XOR primenumber) with custom data format and obfuscated application.
If you don't need to modify this file you can serialize the object graph to a file. The contents are binary and they could only be read using the classes where they were written.
You can also use Java DB ( shipped with java since 1.5 I think ) and an ORM tool for that such as Hibernate.
EDIT
It is bundled since 1.6 http://developers.sun.com/javadb/
XStream works if you want to do simple xml reading and writing to a file. Xstream allows you to take any java object and write it to and read it from you file.
I think "serialization" is the word:
http://java.sun.com/developer/technicalArticles/Programming/serialization/
If you really need the security implied in your statement ("...protect the persisted file so that others can't derive the object model details from the data."), I'd serialize the data in memory (to Java serialized form, XML, or whatever) and then encrypt that byte-stream to a file.
You can try using an embedded database like Berkeley DB Java Edition (http://www.oracle.com/database/berkeley-db/je/index.html). Their direct persistent layer API will most likely suit your needs. The database contents are synced to files on disk. From just looking at the files directly, it's not easy to figure out the object model from the data. I've had good experiences with it, it's lightning fast and works well with desktop applications.