Where I work we are in need of a protocol capable of:
User login/logout
Send/recive instructions
Send/recive files
Send/recive audio stream(could use RTP)
Send/recive small XML files Use
cryptography for all those.
It will be implemented in java. So I have some questions, since I´ve never implemeted a network protocol yet.
Is it possible to use existing protocols to build this one?
What tool can I use to help me design the protocol? for "Modeling"
Is it possible to acomplish all this, doing it alone? I have as much time as I need for this.
I have a pretty good background in Java and C++, but not yet with sockets/networking programming.
Thanks
Take a look a Google Protocol Buffers, which will generate a compact wire protocol as well as autogenerating Java message classes. I wish I'd heard of it before rolling my own message codec using Java NIO ByteBuffers.
I've got a feeling you're trying to reinvent either SIP (if your packet processing is mostly stateless and XML is small enough to go into <3k packets), or XMPP.
If you need a connection oriented login/logout, and stateful commands/instructions, then XMPP is probably closer to the requirements. Also, Jingle extension to XMPP already deals with RTP setup and teardown. XML messages are trivial to embed into custom XMPP packets (which themselves are XML) and there are known XMPP solutions for proxying a file transfer.
I'm pretty sure it meets your requirements quite well (at least the way they're presented here). If you don't have to design a completely new protocol, it's probably easier if you don't. Also reusing an existing XMPP server will allow you to solve the pain of creating your own message broker. There's OpenFire server, which is written in Java.
I dont know if this is bad advice or not, but what I usually do for my networking applications is to make a Message object that holds a TAG string and CONTENT string. The CONTENT part is usually a JSON string and the message itself is also sent to/from the server as a json string.
When the server or client receives a message, it parses the json into a Message object. You can then check the TAG part of the message to see what type of content is held in the CONTENT part of the message and decide what to do with it.
For example, if TAG=="LOGIN" then the CONTENT may be login details or similar. And when the TAG=="MESSAGE" then the CONTENT will perhaps be a json string representing your parameters, for example, who is the recipient/s and what is the content of the message, etc.
You can then do you encryption and decryption on the strings. If this is a stupid way of doing it, please tell me so in a comment so i can learn :)
I also usually implement a state design pattern on both sides, but at least on the server side. For example, the server starts out in a WaitingForLogin state. When the client logs in it switches to a different state that only listens for files and chat messages as an example. In this way I found it is a bit easier to manage.
You could use http or https. The java media framework contains an implementation of rtp.
Writing the protocol from scratch may require a lot of work. Take a look at XMPP.
If you want to write your own protocol, start with learning a form of RPC like JSON or similar, which will make your life a lot easier.
Related
sorry if my english isn't perfect.
I'm trying to make an app and I need to exchange information between more devices.
I thought that could be a solution connect the devices on a server but I really don't have the idea where start.
What language I need to study to make this? There is a better solution?
This highly depends on what you are trying to achieve in the first place. It would be helpful if you could tell what you are trying to do, but I will still outline some general aspects:
You need to decide, what information is going to be exchanged and how this should happen
What information: Figure out, what exactly needs to be sent and received. Generic text messages? Images? Byte Streams?
How should this be done: Generally spoken, there are two approaches of getting information as a client: Polling and subscribing.
Polling: This approach means to periodically check an endpoint for new data. For example, HTTP uses this way: A web browser or any other client (REST-Client for example) periodically requests information from a HTTP-Server, using a connection just for this single request.
Subscribing / Sync / Notification: In some way or another, the client tells the server that it is interested in the information and wants to get notified when there is something new. The connection is initiated at the beginning and held open for further usage. The benefit of this approach is that changes are received immediately, but on the other hand a permanent connection needs to be maintained.
Things to study
At the beginning, get a good understanding of the TCP/IP Protocol, how Sockets work, how common Protocols do their job (e.g. HTTP, WebSockets)
Take a look at specific Protocols working on top of the basic ones
Tip: REST: Most common WebServices Protocol, providing a common way to exchange stateless data. Uses Polling.
WebSockets: Socket connection using Web Browsers. Commonly used to update information without needing to poll.
There is no specific language to learn for connections. It's more about understanding what the difficulties are and what ways have been invented to address this. Once you get to this point and know what you want to do, it's possible in every language.
Recommendation: As you seem to use Java/Android, I would try to use REST. A really great client-side library for REST on Android is Retrofit. For the server side use what fits for you .. common Java way would be to use Jersey, but you are free to choose from a lot of choices. If using Jersey is too hard for the beginning, maybe take a look at the JS/NodeJS world, those guys invented Express, which allows you to create a REST service out of just a database, wihtout having to code a lot.
First you need to decide if you want to go for an Android or an iOS application. There are other various mobile operating systems as well, but these are widely used . If you want to go for android which is most widely used in my opinion, then you need to learn Java. If you want to go for iOS application, then you need to learn swift or objectiveC. These languages provide the API to connect with various types of services such as Facebook, Firebase and Amazon etc. If you want to connect to some other local server who’s IP is known to you, then you can use socket programming to send messages.
There could be many ways you can implement this. One way will be using Web services. Of course REST might be a better option, if you follow this approach. You can implement Your service(server side code) with any language. I will recommend you use java since you are already using android.
Aside from this You might need to go through the basics of REST, its specifications and
some reference implementations for language of your preference.
I was going to use an objectOutputStream but heard this is unreliable because different java versions might deserialize objects differently. Something about 'horrible cross-architecture practice..'
So how else can I send objects and arrays between these devices, where the receiving end can piece back together the proper object or array data?
Edit: Just read what you are doing. You might not need a web server. A lot of people recommend one because of the massive support web servers have. You certainly use TCP or UDP to talk between a server and a client. You'll need to have some protocol if you want data interchange, and most people here would be familiar with XML or JSON
If you need inspiration, try looking at a few protocols like FTP, or even Bittorrent
Web Server case:
I wrote a Java web server for a college homework assignment. A web server actually be quite simple if you have a good grasp on TCP/IP. The code scattered everywhere online to do it though gets a little hard to decipher what exactly is going on, but once you do, it's not bad
You definitely should check out the RFC for HTTP, even though those tend to be worded in legalese. Beyond that, on the server, you basically read strings in line-by-line and you should be able to figure out what to do on the server (example GET /somefile.html HTTP/1.0). Just do System.out.println on those lines and go from there. The same goes for client code. You also can use telnet to see what a web server does
To test, I would actually recommend trying just a regular web browser like Firefox, Chrome, IE, Safari, and even curl scripts. This is an easy test to see if your server is running correctly
As far as data interchange goes, XML or JSON would be recommended, mostly that if you learn how to handle it, you get 100 experience points for your resume. However, to get things started, you can start out just by sending and receiving text like "Wazzzaaap". Web browsers can also grab XML and JSON data.
By 'java server', what kind of protocols are you using?
One option is RPC, which is defined in java.rmi
If you are using http, the simplest choice is to implement a small servlet in tomcat/jetty and use restful services
The data format can be xml, json, bin, etc
I'm looking for a simple java library which would enable to write code in the following fashion:
Remote remote = Remote.connect("some_host:1234");
Future<String> response = remote.request("hello");
// do something else
String reply = response.get();
It should be based on tcp/ip and use plain text messages across the network to be language agnostic, so that non-Java servers are also able to send/receive requests/responses.
(Before telling me to use plain sockets, keep in mind that in that case you need to implement wrappers to delimit payload, care about received messages reordering, thread handling... the example is simple, but it is not that trivial to implement well.)
Is there any existing API out there like this?
PS: ...the simpler, the better!
Take a look at JMS.
Use plain sockets :-)
Sockets use TCP, and TCP takes care of payloads, multiple packets and ordering. You'll still need to handle the threading, but java.util.concurrent has all you need for this. Don't forget to decide of a character encoding for your strings.
TCP implementations usually give to the applications only a stream-like interface, with no access to the individual packets. (Also, some routers/firewalls might want to repack the data in the TCP streams, which means that they do not necessarily arrive in the same blocks as sent.)
Thus we really need to use some packaging protocol on top of TCP (or really on top of any stream-pair).
A really simple protocol would be one line for each request/response, but this will work only with small data sizes (or you need to somehow escape embedded newlines).
If you want it more structured, you could use something XML-based (like XMPP): each request/response would be a complete XML element (including subelements, if necessary).
Also, if you want to use the request–response scheme, you will need to either say responses have to come in the same order as the requests were ordered (which disallows or at least complicates parallel processing on the server side for multiple requests on the same connection), or you will have to define request numbers, and the responses then will somehow include the request number they relate to.
As examples, HTTP uses the first approach (from 1.1 - before there was only one request/response pair for each connection), while the X protocol uses the second one.
For HTTP, there are already implementations around (both on client and on server side), and it can be made totally plain text (depending on the data you are sending).
Alternatively, we could build our protocol directly on a packet-based protocol like UDP. But this has the usual reliability problems of UDP, and we also need to use message-numbers (to relate responses to requests) or such - which could mean that we have to re-implement half of TCP again.
So, sorry, no real answer other than use HTTP.
With Apache Mina you're able to develop your own protocol, but it might be overkill.
I am not aware of any library which gives you a layer quite like this over sockets. The IMAP protocol has a framing layer which is quite like this, but i don't know of any way to use it independently of the rest of IMAP. Such a library would be simple enough to write, and potentially quite useful, so if anyone fancies trying it, i encourage them to do so!
The closest thing i can think of to what you want is ZeroMQ in request-reply mode. ZeroMQ is written in C, but there is a Java binding. It's a pretty good library - there are bindings for many languages, so it's practically language agnostic, and it does indeed take care of delimiting payload, caring about received messages reordering, and thread handling. I don't think it's plain-text, though.
I am writing a client server iPhone app. The server is J2EE based.
I need to communicate the state of my client object (objective C) to the server. It is possible (and feasible) to say encode the objective C object, send the bytes to the J2EE server through a socket and create a Java object out of this stream. If so, can you kindly point me to a starting point.
Thanks in advance
Anything is possible, but that does not make it feasible. Except the technical difficulties these interfaces tend over time to create a lot of management headaches, example when one or both sides perform an upgrade.
I would seriously consider using some encoding to some platform neutral format like protobuf, thrift, JSON, XML or similar.
It sounds like it would be easier to serialize the Objective-C object to XML, JSON, or another text-based representation and ingest and unmarshal that in Java. To return it, reverse the process.
There are myriad ways to skin this cat. If you are using J2EE, you might even consider using a standard means of communication rather than rolling your own. For example, you could use a webservice, REST, etc. Objectice-C has good support for HTTP connections and it is fairly trivial to create an XML or JSON payload. SudzC is a great tool for creating a client proxy from WSDL.
Here are a few tools on the Objective-C side:
json-framework
SudzC
What are possible options to transfer a lot of data from one computer to another not in the same LAN. The amount of data is about 100Mb unzipped and 2Mb zipped? Another requirement is that when I create a server for this (with C#) Java clients should be able to consume it.
Does WCF support something like this? But if Java clients won't be able to consume it I'm not interested.
What could be other strategies here?
I'd just use something common like HTTP or FTP, since there will be plenty of existing libraries to do it and you're pretty much guaranteed not to have compatibility problems. 2MB is not an unreasonably large amount of data for those protocols.
This is an interesting kind of question. The question is fairly simple to answer. But the interesting thing is that this kind of questions are new, they didn't exists before. Let me explain, but first I will answer your question:
You should create a server and clients both using old fashion TCP streams. To not waist bandwidth you need to compress the stream somehow, here use one of the most common compression algorithms you can find (anyone said Zip?). Now you have a language independent protocol. Clients in any language will work, mission accomplished. Also to keep it cross-platform, do not pick the best compression out there, pick the most common one (It will be good enough).
Now to why this kind of questions are interesting, they show something about OOP on the large scale. People understanding and using huge frameworks and asking if this or that framework can perform this or that simple task for them. Here we have lose our roots, we have lost the inner workings of things, it's hitting the nail not with a hammer but with a nuclear missile. It's overshooting the target, and it will produce huge applications, with huge footprint and often poor performance.
I believe that this questions has increased in number since OOP was fully adopted. It's like new programmers only want to learn these new big frameworks and that the framework dim the view of the world. There is absolutely nothing wrong with big frameworks, they are great, but I believe it's wrong to start out using them before one have mastered the basics. It's like learning to fly using a NASA space shuttle instead of a school version of a Cessna private airplane.
In C# you can serialize your object as an XML and transmit, on the other end your can deserialize your XML back to an object.
In terms of files size, you can transmit as zipped or 7z..and on the client decompress it before parsing the xml.
WCF supports SOAP and includes optional JSON serialization for XHTTP. There are other mechanisms but they are MS orientated. You will easily be able to consume the service you create. However you will have to consider how to encode the data as it will hit the wire in a non binary data friendly manner (XML/JSON).
You may wish to instead create a simple http handler that can return the data directly as zip using appropriate mime headers etc. You should then be able to just hit that using your Java client.
XMPP is another option. You need another server, but this could be an advantage: the client wouldn't need to know the servers IP address, server and clients would simply connect to the XMPP server to exchange message and files.
Related links (for Java):
Openfire XMPP server
XMPP library for java (Smack)
You didn't mentioned what type of data do you want to send. So for keeping things simple I will suppose that you have data stream which can be converted to byte array. Content of the stream has to be in format which is understandable to both C# and Java!
The best choice is to compress your data stream with GZip stream. Gzip should be supported on Java. Than you can send that stream converted to byte array as response from your WCF service operation. You can use default text encoding which will convert byte array into Base64 encoded string. If your java client supports MTOM (it is standard which is supported by Java) than you can use MTOM encoding which uses smaller messages.
If you don't have a stream with well known content format you have some sort of custom data. For custom data you have to use interoperable transport format which is XML. Using XML will futher increase size of your data. In that case you should consider dividing your data transfer into several calls. You can also try to host your WCF service in IIS 7.x and take advantage of its build in feature - compression of dynamic content. If your Java client calls the service with HTTP Accept-Encoding header set to compress, gzip it will automatically compress the response. Be aware that only .NET 4.0 WCF clients can work with such service.