I am writing multiple protobuf messages to a file in C++.
int fifoPipe = open("/media/my_pipe", O_WRONLY);
MyModel *model = new MyModel();
// Write to fifo pipe.
model->SerializeToFileDescriptor(fifoPipe);
I had read that the message size is written to the file automatically before the message data. Apparently this is not the case:
https://developers.google.com/protocol-buffers/docs/techniques?hl=en
So in Java it should be possible to read the message like so:
MyModel.parseDelimitedFrom(fileInputStream);
However there are questions here:
Cannot deserialize protobuf data from C++ in Java
Are there C++ equivalents for the Protocol Buffers delimited I/O functions in Java?
That say the message size must be written manually.
Since the questions are kind of old this procedure may have changed.
What is the proper way of writing multiple messages in C++ and reading those messages in Java?
Also consider how Java would respond to a half written message. In theory it should wait for data the size of message size to be present before returning.
Unfortunately, the parseDelimitedFrom() and writeDelimitedTo() methods still have not been added to the C++ library.
The code I wrote in the answer to one of the questions you referenced is still the best way to implement this in C++:
https://stackoverflow.com/a/22927149/2686899
The main reason why Google with all its resources hasn't gotten around to adding these in C++ is because Google internally does not use this format at all. For network communications, Google uses its internal RPC protocol (very similar to GRPC, which they open sourced recently), and for storing messages to disk they usually a variety of internal formats that are a bit more featureful than this "delimited" format (you might consider using sqlite, for example).
In fact, when Protobuf was first open sourced, parseDelimitedFrom() didn't exist even in Java. I added it later specifically as a stopgap for users of the open source library -- a lot of people asked us how to write multiple messages to a file, and telling them "you should develop your own framing library" didn't seem very nice.
Google may be a huge company, but at that time I was the only person working full-time on Protocol Buffers. Unfortunately, for reasons that I don't quite remember, I only implemented the functions in Java, and never got around to adding them in C++ as well. In retrospect it seems a bit silly -- writing C++ code wouldn't have been very hard, as you can see at the link above. But I had a lot to do, as you might imagine.
The current Protobuf team took over in 2010, and I moved on to other things (and eventually left Google). I don't know for sure why they haven't added this code to the C++ library, but my guess is that not enough people have asked for it and they are focused on other things. I bet if you file a bug, and link to my code -- or better yet, submit a pull request yourself -- they might just take it. (I would submit a pull request myself, but right now I don't have time to write the necessary unit tests, etc...)
EDIT: OK, against my better judgment I took the time to prepare a pull request: https://github.com/google/protobuf/pull/710
Related
I have been working on a project alone for more than two years for a company. The project is a really big one using rxtx to communicate with a hardware device. I used Java 8 and JAVAFX for the UI. Now it is almost finished and I am starting to search how to deliver the end user application that the company will distribute over its clients.
The problem is that the company I am working with wants the code to be non reachable when the software is between final clients hands because the Java code contains some extremely sensitive information that could have very bad consequences for the company if final clients happened to know them. The clients can literally perform actions they don’t have the right to perform.
So after searching (a lot) and thinking relatively to my case, I understood that giving a JAR obfuscated isn’t the solution. I then tried to generate a JAR and then transform it to an EXE but all I succeeded on was wrapping the JAR into EXE which does not prevent extracting the JAR and then seeing all the code easily. Finally, I found that I should use AoT compilation like GCJ compiler to produce native binary exe from my Java code but here I am stuck because after watching videos and reading articles etc I didn’t manage to find a clear way to produce the native binary exe.
I am now confused since I don’t know if I am on the right path and good direction or if I am totally wrong and there is another way of protecting the code (at least from non professional hackers, I understand that it is not possible to make it 100% safe but I am just searching for a reasonable and good way). How should I manage this final step of my work?
I currently work for a company that has code that we don't want anyone to have access to for the security of our clients and-- less important-- for legal reasons. ;-)
One possible solution you could look into would be to rewrite the code you deem most sensitive into a C/C++ library. It would be possible to compile this into a .so/.dll/.dylib file for the respective OSs and it would make it difficult, not entirely impossible, but difficult to decompile.
The trouble would come from learning how to access native code from Java as much of the documentation is not helpful or just simply nonexistent. This would utilize the Java Native Interface (JNI) which allows Java to, well, interface with the native (compiled C/C++) code. This would make it possible to create a Jar file that would effectively become a Java library for you to access throughout the rest of your project. The native code, however will still need to be loaded at runtime, but that's apart of learning how JNI works. A helpful link I found for JNI is http://jnicookbook.owsiak.org/ (for as long as it's still a functional link).
One of our clients here where I work has a project written in Java and needed to implement our code that is unfortunately all written in C. So we needed a way to access this C/C++ code from Java. This is the way we went about solving this issue without rewriting our code in Java. But we had the benefit (?) of having already written our code in C.
This solution to write a bunch of extra code last minute in another language that I may or may not be familiar with doesn't sound like particularly fun time.
I would be curious to learn what possible problems others might see with this solution.
I am working on IBM RTC and I need to import a .csv file to RTC using JAVA. Is there a way to do this? If yes, can someone help me with the same.
Parsing CSV data is something that you definitely do not want to implement yourself, there are plenty of libraries for that (see here).
RTC offers a wide range of APIs that can be used with, see:
rsjazz.wordpress.com or
jazz.net
In that sense: you can write Java code that reads CSV data, and RTC has a rich API that allows you push "content" into the system.
But a word of warning: I used that java API some years ago to manipulate information within our RTC instance. That was a very painful experience. I found the APIs to be badly documented and extremely hard to use. It took me several days to come to working code that would make just a few small updates to our stories/tasks.
Maybe things have improved since then, but be prepared for, as said ... a painful experience.
EDIT, regarding your comment on "other options":
Well, I dont see them: you want to push data you have in CSV into your RTC instance. So, if you still want to do that, you have to use that means that are available to you! And don't let my words discourage you. A) it was some time back when I did my programming with RTC, so maybe their APIs are better structured and more intuitive today. B) there is some documentation out there (for example here). And I think everybody can register at jazz.net; so when you have further, specific questions, you might find "better" answers there!
All I wanted to say was: I know that other products such as jenkins or sonarqube have great APIs; and you work with that, all nice, easy, fun. You get things working with RTC, too. Just the path there, maybe isnt that nice and easy.
My personal recommendation: start with the RTC part first. Meaning: just try to write a small programm that authenticates against the server; and then push some example data into the system. If that works nicely for you; then spend the time on pulling / transforming the real data that you have in mind!
In my java apps, I always comment my classes and methods to indicate what they are all about. The problem I often have however is documenting a complete process whereby the process is spread over several classes. I need to document how all the pieces work together to give an easy overview of how something works.
What I find problematic is WHERE I write this documentation. I could just as well write it in a separate file such as a Word document but the documentation tends to become alienated from the actual code and it is possible to get out of sync. On the other hand, if I document it in a single java code file, it makes it difficult for someone reading one of the dependent classes to know how that class fits into the overall process unless they are aware of the documentation in the class where I wrote it. A possible solution is just to include a reference in each class that makes up the entire process and just designate one of the classes as the "primary" class where the documentation originates from.
Or is there a better alternative approach on how I should be doing this?
EDIT:
An example is where you have a mobile app that uploads data to the server and then you download data from the server to another device. You have a process that covers three components (a sending mobile device, the server and the receiving mobile device). None of these can be really considered the "starting point in the overall data transfer process, so when documenting this process, where would this documentation go?
I've been doing some programming in Java and some in C but now I need to sort of use both together.
Here's the situation, I'm using Hadoop/Hbase to process and store a lot of data but I'm using C/Cuda to do number crunching on the data. Is there a stable/mature/common way to take data (it's basically a log file) in Java and pass it to a C program, which C processes the data it stores it as a linked list that is then accessible by the Java app?
I might not be searching for the right thing, but so far I found JavaCPP, which is good but seems to involve both programs together. Because Java handles the data flow and C handles the processing of the data, I thought it might be better to keep them as independent programs that can communicate to each other as opposed to a single program that may become confusing. But I'm totally flexible so any suggestions/solutions are welcomed.
You may find it easier to keep the programs testable and clear if you leave them separate and then use a client-server approach, or simply choose a common file format and have the latter steps poll the output directory for new files to process.
To make it easier to define file formats across different languages, consider a package like Apache Thrift or Google Protocol Buffers.
Here what I have on the top of my head
1. run C program using command line from java app.
2. Use JNI/JNA
3. Implement your own "client-server" architecture. It sounds complicated but in some cases it may be the best and the simplest solution.
4. Communicate using Web service, SOAP, REST, whatever.
I hope this is helpful for the beginning.
You are welcome to ask more specific questions once you have.
We have two code bases, one written in C++ (MS VS 6) and another in Java (JDK 6).
Looking for creative ways to make the two talk to each other.
More Details:
Both applications are GUI applications.
Major rewrites or translations are not an option.
Communications needs to be two-way.
Try to avoid anything involving writing files to disk.
So far the options considered are:
zero MG
RPC
CORBA
JNI
Compiling Java to native code, and then linking
Essentially, apart from the last item, this boils down to a choice between various ways to achieve interprocess communication between a Java application and a C++ application. Still open to other creative suggestions!
If you have attempted this, or something similar before please chime in with your suggestions, lessons learnt, pitfalls to avoid, etc.
Someone will no doubt point out shortly, that there is no one correct answer to this question. I thought I would tap on the collective expertise of the SO community anyway, and hope to get many excellent answers.
Well, it depends on how tightly integrated you want these applications to be and how you see them evolving in the future. If you just want to communicate data between the two of them (e.g. you want one to be able to open a file written by the other, or read a stream directly from the other), then I would say that protocol buffers are your best bet. If you want the window rendered by one of these GUI apps to actually be embedded in a panel of the other GUI app, then you probably want to use the JNI approach. With the JNI approach, you can use SWIG to automate a great deal of it, though it is dangerously magical and comes with a number of caveats (e.g. it doesn't do so well with function overloading).
I strongly recommend against CORBA, RMI, and similarly remote-procedure-call implementations, mostly because, in my experience, they tend to be very heavy-weight and consume a lot of resources. If you do want something similar to RMI, I would recommend something lighter weight where you pass messages, but not actual objects (as is the case with RMI). For example, you could use protocol buffers as your message format, and then simply serialize these back and forth across normal sockets.
Kit Ho mentioned XML or JSON, but protocol buffers are significantly more efficient than either of those formats and also have notions of backwards-compatibility built directly into the definition language.
Use Jacob ( http://sourceforge.net/projects/jacob-project ), JCom ( http://sourceforge.net/projects/jcom ), or j-Interop ( http://j-interop.org ) and use COM for communication.
Since you're using Windows, I'd suggest using DDE (Dynamic Data Exchange). There's a Java library available from Java Parts.
Dont' know how much data and what type of data you wanna transfer and communicate.
But to simplify the way, I suggest using XML or Json based on HTTP protocol.
Since there are lots of library for both applications and you won't spend too much effort to implement and understand.
More, if you have additional applications to talk with, it is not hard since both tech. are cross-languages.
correct me if i am wrong