I am tasked with writing a client-side data download system (on Linux) that uses FTP or HTTP to download terabyte-sized data from external partners to our local site. Our company's network admin tells me that I cannot exceed a certain bandwidth. What is the best way for me to implement such a system? Do existing libraries exist?
I am open to writing my own FTP and HTTP clients (in either C or Java on Linux) but would prefer to stay out of the kernel. I know that I can limit the rate at which my FTP/HTTP client calls a socket read(), but what happens if the server-side calls write() faster than my limit?
You could build another layer on top of an InputStream: In the read method, you can count the bytes so far. If the number of bytes/second exceed a certain limit, let the download thread sleep for a while. TCP's flow control does the rest.
I know Apache JMeter simulates slow connections. You could maybe take look at the code.
If you know the network path delay you could just set your TCP receive buffer size to the desired bandwidth-delay product. That will throttle the sender all right. But the resulting value may be too small for your platform, so it may adjust it upwards. Check the value after you set it.
Does your netadmin know that TCP automatically shares bandwidth fairly?
Are you open to off the shelf GUI or command line products? Filezillia provides this.
There also is a linux command line client called lftp. A settable parameter is net:limit-total-rate which will limit the rate of transfer. Since this client supports multiple transfers at one time, it also has a parameter net:limit-rate.
To keep it simple, if you are on linux you just could use wget instead of re-inventing the wheel? Take a look at the --limit-rate switch.
But back on topic :) This answer could get you started: How can I implement a download rate limited in Java?
Related
I have 2 Java applications. First I may edit as much as I wish, but I will compile it to machine code later on. Second one I am not able to edit, but I may write an addon for it. I need to make that addon be able to talk with first application. Generally simply send strings to each other. Input and Output streams of a process is not an option for me. I am thinking of using a tcp socket client/server or a file which will act as a buffer. But both ways seem a liitle bit ugly to me, could anyone propose me a better idea?
It depends on what kind of data you wish to transfer.
If it is only Strings, then:
if number of process = 2 and if you are sure of it, then stdin &8 stdout is the best way forward. You can create a Process using ProcessBuilder and then get the streams to communicate. The other process can just to System.out to transfer message. This is preferred to Socket, because you dont have to handle graceful closing of socket etc. (In case it fails and the port is not un-binded successfully, it can be a big trouble)
if number of process > 2 and less than say 10, you can probably use Sockets and communicate through Socket. This should work well, though extra effort goes in gracefully managing sockets.
if number of process is Large, then JMS should be used. It does a lot of things which you dont need to handle. Too big a task if the number of processes are less.
So in your case, process is the best way forward.
If the data you wish to transfer, can even be Objects. RMI can be used given the number of processes are less. If more, use JMS again.
Edit: Now for all the above, there is a lot of dirty work involved. For a change, if you are looking at something new & exciting, I would advice akka. It is a actor based model which communicate with each other using Messages.
The beauty is, the actors can be on same JVM or another (very little config) and akka takes care the rest for you. I haven't seen a more cleaner way than doing this :)
What about to use JMS ?
You can use according to your needs, either the Publish/Sunbscribe or Point-to-Point Models.
Another approach is having DB table to store your data, one process can insert and other process can read it when ever required. When you are using JMS, there is likeliness of loosing data, But storing in db would be failsafe and future proof.
I have a system where in I get requests via HTTP call to my PHP code(producer). This code adds the request parameters to a table in MYSQL(queue). This is then taken and processed by a java program(consumer). In my first implementation both producer and consumer was in PHP(with MYSQL queue). Then as load increased this proved inefficient and so I made the consumer java. Now I think polling MYSQL table for the queue from my java app is getting inefficient(vey high cpu usage for MYSQL process). Is there a better way to implement this queue (sharing memory between PHP code and Java app or something)?
Yes, you've got many options. The first is obviously to convert this into a client-server service, and pass either text or binary messages between them. You might want to look into webservices if you're a masochist, or a simpler REST service, or CORBA / COM+ and other for binary serializations. And then there's various queues, like MQseries, RabbitMQ, etc. Sometimes the middle-man is fast enough and efficient enough, or a direct call would suffice.
The next is a more direct link if your platforms are within the same server or cluster, something like JavaBridge and others (do a search for "java php bridge", and several will crop up. There's even a PHP interpreter written in Java for the JVM which gives you full compatibility between the two which might do the trick for you.
Is there a reliable, cross-platform way to do IPC (between two JVMs running on the same host) in Java (J2SE) that doesn't rely on the network stack?
To be more specific, I have a server application that I'd like to provide a small "monitoring" GUI app for. The monitor app would simply talk to the server process and display simple status information. The server app has a web interface for most of its interaction, but sometimes things go wrong (port conflict, user forgot password) that require a local control app.
In the past I've done this by having the server listen on 127.0.01 on a specific port and the client communicates that way. However, this isn't as reliable as I'd like. Certain things can make this not work (Windows's network stack can be bizarre with VPN adapters, MediaSense, laptops lid closing/power saving modes). You can imagine the user's confusion when the tool they use to diagnose the server doesn't even think the server is running.
Named Pipes seem plausible, but Java doesn't seem to have an API for them unless I'm mistaken. Ideas? Third party libraries that support this? My performance requirements are obviously extremely lax in case that helps.
One of my specialties is really low-tech solutions. Especially if your performance requirements aren't critical:
The low-low tech alternative to named pipes is named FILES. Think yourself up a protocol where one app writes a file and another reads it. If need be, you can do semaphoring between them.
Remember that a rename is pretty much an atomic operation, so you could calmly write a file in some process and then make it magically appear in its entirety by renaming/moving it from somewhere that wasn't previously visible.
You can poll for data by checking for appearance of a file (in a loop with a SLEEP in it), and you can signal completion by deleting the file.
An added benefit is that you can debug your app using the DIR command :)
Depending on how much data you need to pass between the server and the diagnostic tool you could:
go low-tech and have a background thread check a file in the file system; fetch commands from it; write ouput into a second to be picked up by the diagnostic tool.
build a component that manages an input/output queue in shared memory connecting to it via JNI.
Consider JMX. I do not know if any of the Windows JVM's allow JMX over shared memory.
Does Windows even have named pipes? I was going to suggest it. You'd just have to use an exec() to create it.
Map a read_write byte buffer into memory from a FileChannel. Write status information into the byte buffer, then call force() to get it written out. On the monitor side, open up the same file and map it into memory too. Poll it periodically to find out the status.
I am designing a application where many clients connect to a central server. This server keeps these connections, sending keep-alives every half-hour. The server has a embedded HTTP server, which provides a interface to the client connections (ex. http://server/isClientConnected?id=id). I was wondering what is the best way to go about this. My current implementation is in Java, and I just have a Map with ID's as the key, but a thread is started for each connection, and I don't know if this is really the best way to do this. Any pointers would be appreciated.
Thanks,
Isaac Waller
Use the java.nio package, as described on this page: Building Highly Scalable Servers with Java NIO. Also read this page very carefully: Architecture of a Highly Scalable NIO-Based Server.
Personally I'd not bother with the NIO internals and use a framework like Apache MINA or xSocket. NIO is complicated and easy to get wrong in very obscure ways. If you want it to "just work", then use a framework.
With a single thread per connection you can usually scale up to about 10,000 connections on a single machine. For a Windows 32 machine, you probably will hit a limit around 1,000 connections.
To avoid this, you can either change the design of your program, or you can scale out (horizontal). You have to weight the cost of development with the cost of hardware.
The single thread per user, with a single continuous connection is usually the easiest programming model. I would stick with this model until you reach the limits of your current hardware. At that point, I would decide to either change the code, or add more hardware.
If the clients will be connected for long periods of time, allocating a thread per client can be problematic. Each thread on the server requires a certain amount of resources (memory for the stack, for example).
You could use Jetty Continuations to handle the client request with fewer threads by using asynchronous servlets.
Read more about the the Reactor pattern. There is an implementation for that in Java (it uses channels instead of thread for client).
It is easy to implement and very efficient.
I know that such type of questions exist in SF but they are very specific, I need a generic suggestion. I need a feature for uploading user files which could be of size more that 1 GB. This feature will be an add-on to the existing file-upload feature present in the application which caters to smaller files. Now, here are some of the options
Use HTTP and Java applet. Send the files in chunks and join them at the server. But how to throttle the n/w.
Use HTTP and Flex application. Is it better than an applet wrt browser compatibility & any other environment issues?
Use FTP or rather SFTP rather than HTTP as a protocol for faster upload process
Please suggest.
Moreover, I've to make sure that this upload process don't hamper the task of other users or in other words don't eat up other user's b/w. Any mechanisms which can be done at n/w level to throttle such processes?
Ultimately customer wanted to have FTP as an option. But I think the answer with handling files programmatically is also cool.
Use whatever client side language you want (a Java App, Flex, etc.), and push to the server with HTTP PUT (no Flex) or POST. In the server side Java code, regulate the flow of bytes in your input stream loop. A crude, simple, sample snippet that limits bandwidth to no faster than an average <= 10KB/second:
InputStream is = request.getInputStream();
OutputStream os = new FileOutputStream(new File("myfile.bin"));
int bytesRead = 0;
byte[] payload = new byte[10240];
while (bytesRead >= 0) {
bytesRead = is.read(payload);
if (bytesRead > 0)
os.write(payload, 0, bytesRead);
Thread.currentThread().sleep(1000);
}
(With more complexity one could more accurately regulate the single stream bandwidth, but it gets complex when considering socket buffers and such. "Good enough" is usually good enough.)
My application does something similar to the above--we regulate both up (POST and PUT) and (GET) down stream bandwidth. We accept files in the 100s of MB every day and have tested up to 2GB. (Beyond 2GB there is the pesky Java int primitive issues to deal with.) Our clients are both Flex and curl. It works for me, it can work for you.
While FTP is great and all, you can avoid many (but not all) firewall issues by using HTTP.
If you want to reduce bandwidth you may want to send the data compressed (unless its compressed already) This may save 2-3 times the data volume depending on what you are sending.
For an example of good practice for uploading large files, and the various ways of tackling it, have a look at flickr.com (you may have to sign up to get to the uploader page)
They provide various options, including HTTP form upload, a java desktop client, or some kind of javascript-driven gadget that I can't quite figure out. They don't seem to use flash anywhere.
For sending files to a server, unless you have to use HTTP, FTP is the way to go. Throttling, I am not completely sure of, at least not programmatically.
Personally, it seems like limitations of the upload speed would be better accomplished on the server side though.