I have Google'd my butt off, and I can't find anything on this topic.
I am trying to create a download client using Java, and I have figured out how to download files with Java, but I want to accelerate the download speed. I know how this works (opening several connections to the download server), but how can I achieve this?
I am looking for either some detailed explanation of such an algorithm or some code examples.
This is only possible if the server side supports range requests. You can determine that by checking using a HEAD request if the HTTP response header contains Accept-Ranges: bytes. If that is the case, then you can just spawn several threads which downloads the file in parts using the Range header. The URLConnection and ExecutorService are helpful in this.
Keep in mind that you also take the limitation in amount of threads and network bandwidth of your own machine into account.
Related questions:
Reading first part of file using HTTP
How to use URLConnection to fire and handle HTTP requests
Make simultaneous web requests in Java
BalusC described the trick and here is a reference to some source-code you can review and start with:
JDownLoader[Java]: http://svn.jdownloader.org/projects/show/jd
Free Download Manager[CPP]: http://freedownload.svn.sourceforge.net/viewvc/freedownload/
#BalusC Nice Work
I'm a bit unclear, are you writing a Java client that will talk to a server (perhaps a Java servlet?), so you control both sides of the data transfer? If so, you can do nearly anything you want. Java has java.util.zip, which has functions to do the compression.
If you want to download four (or N) files at once, just start up N threads and pass the HTTP requests to the server in parallel. This may not actually improve things, depending on link speed, network congestion, etc.
Writing your own client and making it properly multi-thread safe is a whole lot of work, which is why people just use the Apache HTTP client code. Its rock solid.
Related
I am looking for a sample java code which reads time from a particular server (could be a webservice or could be some bytes using TCP) and sets its on client machine. This could be achieved easily if you dont consider latency and compromise on those millisecond/nanosecond differences. I would like to have a pseudo-code for server and client, or an algorithm to achieve this.
Also I would like to know how NTP servers set time on client servers? Is there any white paper that explains this?
Answers are here. For more results you may google with keywords like "NTP architecture" or similar
https://en.wikipedia.org/wiki/Network_Time_Protocol
http://www.tldp.org/HOWTO/TimePrecision-HOWTO/ntp.html
I was going to use an objectOutputStream but heard this is unreliable because different java versions might deserialize objects differently. Something about 'horrible cross-architecture practice..'
So how else can I send objects and arrays between these devices, where the receiving end can piece back together the proper object or array data?
Edit: Just read what you are doing. You might not need a web server. A lot of people recommend one because of the massive support web servers have. You certainly use TCP or UDP to talk between a server and a client. You'll need to have some protocol if you want data interchange, and most people here would be familiar with XML or JSON
If you need inspiration, try looking at a few protocols like FTP, or even Bittorrent
Web Server case:
I wrote a Java web server for a college homework assignment. A web server actually be quite simple if you have a good grasp on TCP/IP. The code scattered everywhere online to do it though gets a little hard to decipher what exactly is going on, but once you do, it's not bad
You definitely should check out the RFC for HTTP, even though those tend to be worded in legalese. Beyond that, on the server, you basically read strings in line-by-line and you should be able to figure out what to do on the server (example GET /somefile.html HTTP/1.0). Just do System.out.println on those lines and go from there. The same goes for client code. You also can use telnet to see what a web server does
To test, I would actually recommend trying just a regular web browser like Firefox, Chrome, IE, Safari, and even curl scripts. This is an easy test to see if your server is running correctly
As far as data interchange goes, XML or JSON would be recommended, mostly that if you learn how to handle it, you get 100 experience points for your resume. However, to get things started, you can start out just by sending and receiving text like "Wazzzaaap". Web browsers can also grab XML and JSON data.
By 'java server', what kind of protocols are you using?
One option is RPC, which is defined in java.rmi
If you are using http, the simplest choice is to implement a small servlet in tomcat/jetty and use restful services
The data format can be xml, json, bin, etc
I'm developing a distributed application, and I need to connect a client Java based to a server C++ based. Both of them will need to send information to each other, but I need them to be able to do things while waiting for the information, and they don't know when they are gonna get new information, or send information.
How can I achieve this? Now I'm trying to implement a basic communication with Sockets, but I don't really get to communicate them. I have read that using sockets + threads is usually a good approach for client-server apps.
Could you please recommend me some web or book to read about this, or send me some example code to learn?
Do you think that i should use other approach, better than sockets? maybe a higher level library (i would need it for c++ and java) or a totally different way?
EDIT:
I will add some extra information.
What I would love to achieve is the following:
My C++ program has a main loop, where I would like to have a call like GetUpdatedDataFromRemoteDevice() where I read the new values of some numerical variables that previously got updated from the net (the socket, for example).
Eventually, the C++ program will need to send a message to the remote device, to tell him to send other kind of data, and after that, keep getting the updated values.
From the Java program (remote device) the application running is an interactive touchable screen, that cant get blocked by the network transmissions, because it must keep working for the user, so all the networking should be done in a separated thread.
That thread, should connect to the server, and when a button is pushed, start to send the data (4 changing numerical values) in a loop until another event happens.
It would be nice also to be easily re-connectable to the server.
ICE is a modern and good library for distributed applications:
many languages as C++ and Java
many platforms
GNU GPL
good performance
easy to use
First, you define the messages you want to exchange between server and client.
Then, you implement the C++ and Java source code to handle these messages.
More info at http://zeroc.com/ice.html
Have fun ;-)
EDIT: I have to use ACE in some projects. I can tell ACE is very old, maybe mature, but uses outdated C++ coding rules :-(
Therefore ACE is not as easy to use as STL or BOOST. Moreover, ACE is not really efficient... I prefer ICE ;-)
I don't know what your application is but robust client server socket programming is pretty hairy task to do properly. Hardware byte order, String encoding, Network errors, retries, duplicate messages, acks etc.. require lots of good design and careful programming. You need to get it work well as single-threaded before even thinking using multiple threads.
Unless you need instant notifications from server to client I suggest that you use HTTP as protocol between client and server. Client can poll server occasionally for new messages.
Anyway the problem has been solved multiple times already.
http://activemq.apache.org/
http://www.rabbitmq.com/devtools.html
http://www.cs.wustl.edu/~schmidt/ACE-overview.html
I did something of this sort once. In my case it was easier to connect my C++ app to a local Java app using JNI and then have the two Java apps talk to each other.
I am implementing a website using PHP for the front end and a Java service as the back end. The two parts are as follows:
PHP front end listens to http requests and interacts with the database.
The Java back end run continuously and responds to calls from the front end.
More specifically, the back end is a daemon that connects and maintain the link to several IM services (AOL, MSN, Yahoo, Jabber...).
Both of the layers will be deployed on the same system (a CentOS box, I suppose) and introducing a middle layer (for instance: using XML-RPC) will reduce the performance (the resource is also rather limited).
Question: Is there a way to link the two layers directly? (no more web services in between)
Since this is communication between two separate running processes, a "direct" call (as in JNI) is not possible. The easiest ways to do such interprocess communcation are probably named pipes and network sockets. In both cases, you'll have to define a communication protocol and implement it on both sides. Using a standard protocol such as XML-RPC makes this easier, but is not strictly necessary.
There are generally four patterns for application integration:
via Filesystem, ie. one producers writes data to a directory monitored by the consumer
via Database, ie. two applications share a schema or table and use it to swap data
via RMI/RPC/web service/any blocking, sync call from one app to another. For PHP to Java you can pick from the various integration libraries listed above, or use some web services standards like SOAP.
via messaging/any non-blocking, async operation where one app sends a message to another app.
Each of these patterns has pros and cons, but a good rule of thumb is to pick the one with the loosest coupling that you can get away with. For example, if you selected #4 your Java app could crash without also taking down your PHP app.
I'd suggest before looking at specific libraries or technologies listed in the answers here that you pick the right pattern for you, then investigate your specific options.
I have tried PHP-Java bridge(php-java-bridge.sourceforge.net/pjb/) and it works quite well. Basically, we need to run a jar file (JavaBridge.jar) which listens on port(there are several options available like Local socket, 8080 port and so on). Your java class files must be availabe to the JavaBridge in the classpath. You need to include a file Java.inc in your php and you can access the Java classes.
Sure, there are lots of ways, but you said about the limited resource...
IMHO define your own lightweight RPC-like protocol and use sockets on TCP/IP to communicate. Actually in this case there's no need to use full advantages of RPC etc... You need only to define API for this particular case and implement it on both sides. In this case you can serialize your packets to quite small. You can even assign a kind of GUIDs to your remote methods and use them to save the traffic and speed-up your intercommunication.
The advantage of sockets usage is that your solution will be pretty scalable.
You could try the PHP/Java integration.
Also, if the communication is one-way (something like "sendmail for IM"), you could write out the PHP requests to a file and monitor that in your Java app.
I was also faced with this problem recently. The Resin solution above is actually a complete re-write of PHP in Java along the lines of JRuby, Jython and Rhino. It is called Quercus. But I'm guessing for you as it was for me, tossing out your Apache/PHP setup isn't really an option.
And there are more problems with Quercus besides: the free version is GPL, which is tricky if you're developing commercial software (though not as tricky as Resin would like you to believe (but IANAL)) and on top of that the free version doesn't support compiling to byte code, so its basically an interpreter written in Java.
What I decided on in the end was to just exchange simple messages over HTTP. I used PHP's json_encode()/json_decode() and Java's json-lib to encode the messages in JSON (simple, text-based, good match for data model).
Another interesting and light-weight option would be to have Java generate PHP code and then use PHP include() directive to fetch that over HTTP and execute it. I haven't tried this though.
If its the actual HTTP calls you're concerned about (for performance), neither of these solutions will help there. All I can say is that I haven't had problems with the PHP and Java on the same LAN. My feeling is that it won't be a problem for the vast majority of applications as long as you keep your RPC calls fairly course-grained (which you really should do anyway).
Sorry, this is a bit of a quick answer but: i heard the Resin app server has support for integrating java and PHP.
They claim they can smash php and java together: http://www.caucho.com/resin-3.0/quercus/
I've used resin for serving J2ee applications, but not for its PHP support.
I'd be interested to hear of such adventures.
Why not use web service?
Make a Java layer and put a ws access(Axis, SpringWS, etc...) and the Php access the Java layer using one ws client.
I think it's simple and useful.
I've come across this page which introduces a means to link the two layers. However, it still requires a middle layer (TCP/IP). Moreover, other services may exploit the Java service as well because it accepts all incoming connections.
http://www.devx.com/Java/Article/20509
[Researching...]
we have this scenario:
A server which contains needed data and client component which these data wants.
On the server are stored 2 types of data:
- some information - just a couple of strings basically
- binary data
We have a problem with getting binary data. Both sides are written in Java 5 so we have couple of ways....
Web Service is not the best solution because of speed, memory etc...
So, What would you prefer?
I would like to miss low level socket connection if possible...
thanks in advance
Vitek
I think the only way to do LARGE amounts of data is going to be with raw socket access.
You will hit the Out of Memory issues on large files with most other methods.
Socket handling is really pretty straight forward in Java, and it will let you stream the data without loading the entire file into memory (which is what happens behind the scenes without your own buffering).
Using this strategy I managed to build a system that allowed for the transfer of arbitrarily large files (I was using a 7+ GB DVD image to test the system) without hitting memory issues.
Take a look at the W3C standard MTOM to transfer binary data as part of a SOAP service. It is efficient in that it sends as a binary and can also send as buffered chunks. It will also interop with other clients or providers:
How to do MTOM Interop
Server Side - Sending Attachments with SOAP
You might want to have a look at protobuf, this is the library that google uses to exchange data. Its very efficient and extensible. On a sidenote, Never underestimate the bandwidth of a station wagon full of 1TB harddisks!
I've tried converting the binary data to Base64 and then sending it over via SOAP calls and it's worked for me. I don't know if that counts as a web service, but if it does, then you're pretty much stuck with sockets.
Some options:
You could use RMI which will hide the socket level stuff for you, and perhaps gzip the data...but if the connection fails it won't resume for you. Probably will encounter memory issues too.
just HTTP the data with a binary mime type (again perhaps configuring gzip on the webserver). similar problem on resume.
spawn something like wget (I think this can do resume)
if the client already has the data (a previous version of it), rsync would copy only the changes
What about the old, affordable and robust FTP? You can for example easily embed an FTP server in your server-side components and then code a FTP client. FTP was born exactly for that (File Transfer Protocol, isn't it?), while SOAP with attachments was not designed with that stuff in mind and can perform very badly.
For example you could have a look at:
http://mina.apache.org/ftpserver/
But there are other implementations out there, Apache Mina is just the first one I can recall.
Good luck & regards
Is sneakernet an option? :P
RMI is well known for its ease-of-use and its memory leaks. Be warned. Depending on just how much data we're talking about, sneakernet and sockets are both good options.
Consider GridFTP as your transport layer. See also this question.