What are possible options to transfer a lot of data from one computer to another not in the same LAN. The amount of data is about 100Mb unzipped and 2Mb zipped? Another requirement is that when I create a server for this (with C#) Java clients should be able to consume it.
Does WCF support something like this? But if Java clients won't be able to consume it I'm not interested.
What could be other strategies here?
I'd just use something common like HTTP or FTP, since there will be plenty of existing libraries to do it and you're pretty much guaranteed not to have compatibility problems. 2MB is not an unreasonably large amount of data for those protocols.
This is an interesting kind of question. The question is fairly simple to answer. But the interesting thing is that this kind of questions are new, they didn't exists before. Let me explain, but first I will answer your question:
You should create a server and clients both using old fashion TCP streams. To not waist bandwidth you need to compress the stream somehow, here use one of the most common compression algorithms you can find (anyone said Zip?). Now you have a language independent protocol. Clients in any language will work, mission accomplished. Also to keep it cross-platform, do not pick the best compression out there, pick the most common one (It will be good enough).
Now to why this kind of questions are interesting, they show something about OOP on the large scale. People understanding and using huge frameworks and asking if this or that framework can perform this or that simple task for them. Here we have lose our roots, we have lost the inner workings of things, it's hitting the nail not with a hammer but with a nuclear missile. It's overshooting the target, and it will produce huge applications, with huge footprint and often poor performance.
I believe that this questions has increased in number since OOP was fully adopted. It's like new programmers only want to learn these new big frameworks and that the framework dim the view of the world. There is absolutely nothing wrong with big frameworks, they are great, but I believe it's wrong to start out using them before one have mastered the basics. It's like learning to fly using a NASA space shuttle instead of a school version of a Cessna private airplane.
In C# you can serialize your object as an XML and transmit, on the other end your can deserialize your XML back to an object.
In terms of files size, you can transmit as zipped or 7z..and on the client decompress it before parsing the xml.
WCF supports SOAP and includes optional JSON serialization for XHTTP. There are other mechanisms but they are MS orientated. You will easily be able to consume the service you create. However you will have to consider how to encode the data as it will hit the wire in a non binary data friendly manner (XML/JSON).
You may wish to instead create a simple http handler that can return the data directly as zip using appropriate mime headers etc. You should then be able to just hit that using your Java client.
XMPP is another option. You need another server, but this could be an advantage: the client wouldn't need to know the servers IP address, server and clients would simply connect to the XMPP server to exchange message and files.
Related links (for Java):
Openfire XMPP server
XMPP library for java (Smack)
You didn't mentioned what type of data do you want to send. So for keeping things simple I will suppose that you have data stream which can be converted to byte array. Content of the stream has to be in format which is understandable to both C# and Java!
The best choice is to compress your data stream with GZip stream. Gzip should be supported on Java. Than you can send that stream converted to byte array as response from your WCF service operation. You can use default text encoding which will convert byte array into Base64 encoded string. If your java client supports MTOM (it is standard which is supported by Java) than you can use MTOM encoding which uses smaller messages.
If you don't have a stream with well known content format you have some sort of custom data. For custom data you have to use interoperable transport format which is XML. Using XML will futher increase size of your data. In that case you should consider dividing your data transfer into several calls. You can also try to host your WCF service in IIS 7.x and take advantage of its build in feature - compression of dynamic content. If your Java client calls the service with HTTP Accept-Encoding header set to compress, gzip it will automatically compress the response. Be aware that only .NET 4.0 WCF clients can work with such service.
Related
I want some help on a way to transfer data from a python web application to a java desktop application.
What I am doing is having java listen on a port and receive data. But I have no idea how I would send data from python to an open port on a server.
What my question is how would I send data from a python web app to an open port on a computer. And would there be any problems like data types and any other things?
This is a really large question as there are many ways to send data back and forth between server (your java app) and client (your python app).
Your situation is not quite clear (what exactly is your "python web application"?), but you may want to look into XML-RPC. XML-RPC is extremely simple to use and set up, and takes care of "problems like data types and any other things". You basically just set up some functions on your server that the client can call, and have python call them. Arguments are neatly wrapped up by teh client and unwrapped by the server. Return values are the same. It is a simple and clean interface.
For python making calls to the server, you want to use the xmlrpclib module.
To set up an XMLRPC server in java, you have many options. I'm not a Java guy, but I'm sure it is quite simple on that side as well.
There are many good xml-rpc tutorials. Here is one that covers client and server in python.
Like I said earlier, there are MANY options available to you. XML-RPC is a good and simple way to get your feet wet, without really limiting you very much (eg: it has built in fault handling).
Good luck!
If you use a platform independent data format -- xml, json, yaml, ascii txt, ... -- to represent numbers, you have really nothing to worry about.
If you can not afford the inefficiencies of above, then a binary protocol is required.
Java uses network byte ordering (or Big Endian). Python uses the native host byte ordering, OR, you can specify the byte ordering. Here you want to specify Big Endian (sec 7.3.2.1) in writing your numeric data.
Why not use sockets in python too and send it to the java server. Java does not know that the end client is python, what it reads is just data(bytes). I have done this, and it works seamlessly.
See the python's struct module for more details on converting datatypes
I'm looking for a simple java library which would enable to write code in the following fashion:
Remote remote = Remote.connect("some_host:1234");
Future<String> response = remote.request("hello");
// do something else
String reply = response.get();
It should be based on tcp/ip and use plain text messages across the network to be language agnostic, so that non-Java servers are also able to send/receive requests/responses.
(Before telling me to use plain sockets, keep in mind that in that case you need to implement wrappers to delimit payload, care about received messages reordering, thread handling... the example is simple, but it is not that trivial to implement well.)
Is there any existing API out there like this?
PS: ...the simpler, the better!
Take a look at JMS.
Use plain sockets :-)
Sockets use TCP, and TCP takes care of payloads, multiple packets and ordering. You'll still need to handle the threading, but java.util.concurrent has all you need for this. Don't forget to decide of a character encoding for your strings.
TCP implementations usually give to the applications only a stream-like interface, with no access to the individual packets. (Also, some routers/firewalls might want to repack the data in the TCP streams, which means that they do not necessarily arrive in the same blocks as sent.)
Thus we really need to use some packaging protocol on top of TCP (or really on top of any stream-pair).
A really simple protocol would be one line for each request/response, but this will work only with small data sizes (or you need to somehow escape embedded newlines).
If you want it more structured, you could use something XML-based (like XMPP): each request/response would be a complete XML element (including subelements, if necessary).
Also, if you want to use the request–response scheme, you will need to either say responses have to come in the same order as the requests were ordered (which disallows or at least complicates parallel processing on the server side for multiple requests on the same connection), or you will have to define request numbers, and the responses then will somehow include the request number they relate to.
As examples, HTTP uses the first approach (from 1.1 - before there was only one request/response pair for each connection), while the X protocol uses the second one.
For HTTP, there are already implementations around (both on client and on server side), and it can be made totally plain text (depending on the data you are sending).
Alternatively, we could build our protocol directly on a packet-based protocol like UDP. But this has the usual reliability problems of UDP, and we also need to use message-numbers (to relate responses to requests) or such - which could mean that we have to re-implement half of TCP again.
So, sorry, no real answer other than use HTTP.
With Apache Mina you're able to develop your own protocol, but it might be overkill.
I am not aware of any library which gives you a layer quite like this over sockets. The IMAP protocol has a framing layer which is quite like this, but i don't know of any way to use it independently of the rest of IMAP. Such a library would be simple enough to write, and potentially quite useful, so if anyone fancies trying it, i encourage them to do so!
The closest thing i can think of to what you want is ZeroMQ in request-reply mode. ZeroMQ is written in C, but there is a Java binding. It's a pretty good library - there are bindings for many languages, so it's practically language agnostic, and it does indeed take care of delimiting payload, caring about received messages reordering, and thread handling. I don't think it's plain-text, though.
Where I work we are in need of a protocol capable of:
User login/logout
Send/recive instructions
Send/recive files
Send/recive audio stream(could use RTP)
Send/recive small XML files Use
cryptography for all those.
It will be implemented in java. So I have some questions, since I´ve never implemeted a network protocol yet.
Is it possible to use existing protocols to build this one?
What tool can I use to help me design the protocol? for "Modeling"
Is it possible to acomplish all this, doing it alone? I have as much time as I need for this.
I have a pretty good background in Java and C++, but not yet with sockets/networking programming.
Thanks
Take a look a Google Protocol Buffers, which will generate a compact wire protocol as well as autogenerating Java message classes. I wish I'd heard of it before rolling my own message codec using Java NIO ByteBuffers.
I've got a feeling you're trying to reinvent either SIP (if your packet processing is mostly stateless and XML is small enough to go into <3k packets), or XMPP.
If you need a connection oriented login/logout, and stateful commands/instructions, then XMPP is probably closer to the requirements. Also, Jingle extension to XMPP already deals with RTP setup and teardown. XML messages are trivial to embed into custom XMPP packets (which themselves are XML) and there are known XMPP solutions for proxying a file transfer.
I'm pretty sure it meets your requirements quite well (at least the way they're presented here). If you don't have to design a completely new protocol, it's probably easier if you don't. Also reusing an existing XMPP server will allow you to solve the pain of creating your own message broker. There's OpenFire server, which is written in Java.
I dont know if this is bad advice or not, but what I usually do for my networking applications is to make a Message object that holds a TAG string and CONTENT string. The CONTENT part is usually a JSON string and the message itself is also sent to/from the server as a json string.
When the server or client receives a message, it parses the json into a Message object. You can then check the TAG part of the message to see what type of content is held in the CONTENT part of the message and decide what to do with it.
For example, if TAG=="LOGIN" then the CONTENT may be login details or similar. And when the TAG=="MESSAGE" then the CONTENT will perhaps be a json string representing your parameters, for example, who is the recipient/s and what is the content of the message, etc.
You can then do you encryption and decryption on the strings. If this is a stupid way of doing it, please tell me so in a comment so i can learn :)
I also usually implement a state design pattern on both sides, but at least on the server side. For example, the server starts out in a WaitingForLogin state. When the client logs in it switches to a different state that only listens for files and chat messages as an example. In this way I found it is a bit easier to manage.
You could use http or https. The java media framework contains an implementation of rtp.
Writing the protocol from scratch may require a lot of work. Take a look at XMPP.
If you want to write your own protocol, start with learning a form of RPC like JSON or similar, which will make your life a lot easier.
I am writing a client server iPhone app. The server is J2EE based.
I need to communicate the state of my client object (objective C) to the server. It is possible (and feasible) to say encode the objective C object, send the bytes to the J2EE server through a socket and create a Java object out of this stream. If so, can you kindly point me to a starting point.
Thanks in advance
Anything is possible, but that does not make it feasible. Except the technical difficulties these interfaces tend over time to create a lot of management headaches, example when one or both sides perform an upgrade.
I would seriously consider using some encoding to some platform neutral format like protobuf, thrift, JSON, XML or similar.
It sounds like it would be easier to serialize the Objective-C object to XML, JSON, or another text-based representation and ingest and unmarshal that in Java. To return it, reverse the process.
There are myriad ways to skin this cat. If you are using J2EE, you might even consider using a standard means of communication rather than rolling your own. For example, you could use a webservice, REST, etc. Objectice-C has good support for HTTP connections and it is fairly trivial to create an XML or JSON payload. SudzC is a great tool for creating a client proxy from WSDL.
Here are a few tools on the Objective-C side:
json-framework
SudzC
we have this scenario:
A server which contains needed data and client component which these data wants.
On the server are stored 2 types of data:
- some information - just a couple of strings basically
- binary data
We have a problem with getting binary data. Both sides are written in Java 5 so we have couple of ways....
Web Service is not the best solution because of speed, memory etc...
So, What would you prefer?
I would like to miss low level socket connection if possible...
thanks in advance
Vitek
I think the only way to do LARGE amounts of data is going to be with raw socket access.
You will hit the Out of Memory issues on large files with most other methods.
Socket handling is really pretty straight forward in Java, and it will let you stream the data without loading the entire file into memory (which is what happens behind the scenes without your own buffering).
Using this strategy I managed to build a system that allowed for the transfer of arbitrarily large files (I was using a 7+ GB DVD image to test the system) without hitting memory issues.
Take a look at the W3C standard MTOM to transfer binary data as part of a SOAP service. It is efficient in that it sends as a binary and can also send as buffered chunks. It will also interop with other clients or providers:
How to do MTOM Interop
Server Side - Sending Attachments with SOAP
You might want to have a look at protobuf, this is the library that google uses to exchange data. Its very efficient and extensible. On a sidenote, Never underestimate the bandwidth of a station wagon full of 1TB harddisks!
I've tried converting the binary data to Base64 and then sending it over via SOAP calls and it's worked for me. I don't know if that counts as a web service, but if it does, then you're pretty much stuck with sockets.
Some options:
You could use RMI which will hide the socket level stuff for you, and perhaps gzip the data...but if the connection fails it won't resume for you. Probably will encounter memory issues too.
just HTTP the data with a binary mime type (again perhaps configuring gzip on the webserver). similar problem on resume.
spawn something like wget (I think this can do resume)
if the client already has the data (a previous version of it), rsync would copy only the changes
What about the old, affordable and robust FTP? You can for example easily embed an FTP server in your server-side components and then code a FTP client. FTP was born exactly for that (File Transfer Protocol, isn't it?), while SOAP with attachments was not designed with that stuff in mind and can perform very badly.
For example you could have a look at:
http://mina.apache.org/ftpserver/
But there are other implementations out there, Apache Mina is just the first one I can recall.
Good luck & regards
Is sneakernet an option? :P
RMI is well known for its ease-of-use and its memory leaks. Be warned. Depending on just how much data we're talking about, sneakernet and sockets are both good options.
Consider GridFTP as your transport layer. See also this question.