Is there an efficient way to limit the bandwidth of a certain java process?
I am familiar with solutions like trickle to limit bandwidth of a certain process on run time
sudo trickle -s -d 1024 /path/to/app.sh
But when dealing with java processes it makes it more of a challenge because the application initiates a JVM or in some cases a WRAPPER service that initiates a JVM - that means that solutions like 'trickle' will not work.
I can try and limit (using trickle) the whole java process (by wrapping / messing up with /usr/bin/java s.link) - UGLY.
Does anyone know of a better solution for limiting the bandwidth of a java process (JVM)?
Thanks!
Unfortunately I don't think trickle can do it.
I have similar issue and I solved it via throttling bandwidth on a particular port. For example you application opens port 34567 to communicate, then you can apply firewall setting and throttle it down.
On a mac I am using "ipfw", example:
sudo ipfw pipe 1 config bw 5KByte/s
sudo ipfw add 2 pipe 1 src-port 6666
On linux I am using "tc", examples & source: http://www.cyberciti.biz/faq/linux-traffic-shaping-using-tc-to-control-http-traffic/
As a final solution, you can create bash script that monitors processes and picks ones you need and throws port throttling on it.
The question is not really clear.
Do you have control of the Java code? Otherwise, are you the System Administator?
If you are using a Java code you could use the Socket paradigm and then limit each socket connection by using the following method: setPerformancePreferences(int connectionTime, int latency, int bandwidth).
In the other case, the bandwidth limitation capability depends by the OS and the way the Java applications are executed.
Related
I would like to have an automated test for the following scenario:
User logs in and performs some "lengthy" operation. For example tries to upload a file.
In the middle of that lengthy operation, the connection to the server is lost for a period of time, and then restored.
Application does whatever it's supposed to do in this case. For example shows a message to the user asking if they want to retry.
I want steps 1. and 3. to be done by Selenium. Question is: how can I do the step 2. as part of the automated test?
Details on the tools and environment:
Selenium is on Java with Junit
Test must run on Linux and Windows
It will also run on 3 browsers: Firefox, Chrome, IE 11
Couple of solutions I was thinking about:
Call some script that manipulates a connection on the machine.
Manipulate proxy settings of the browser (proxy points to fake address, thus connection is broken).
Use a real proxy, which could be controlled from the code (e.g. by some commands), so it can behave as "working" or "broken" connection.
None of these solutions is ideal for various reasons.
So: did anyone try to solve a similar problem? Would like to hear your solution or alternative ideas, that you think may work. Thanks.
Option 1
Stubbing
You'll have to identify what would be the exception thrown and what will be the component that throws it in a real case scenario. You can do that easily, simulate the scenario in your machine and when the exception is thrown, the stack-trace will tell you exactly what component thrown it.
Then you'll have to extend the component that throws the exception and inject it in the proper place and, ultimately, create an API to trigger that exception.
If you think you need a framework to automate this kind of tests, have a look to to Fitnesse.
Option 2
Simulation
Simulating a real network problem, would be overly complicated and the benefits are not worth the effort in this case (imo).
Anyway... Linux has an excellent built in network emulation layer called netem that allows any kind of seamless interaction with the network traffic. This kernel module is controlled by the tc command line interface.
It's not very easy to get it right when you want to apply those condition to a single port because you'll have to:
Netfilter rule to mark the packets that we want to limit
Traffic Control policy
Filter to bind the packets to the policy
A simple example would be if you want to have 10% packet loss on the whole eth0, in which case you'll use:
tc qdisc change dev eth0 root netem loss 10%
Then you'll have to wrap this functionality in some java controllable way.
If I failed on discouraging you, here is a good guide: TrafficControl
You can execute the following commands on windows on cmd to disconnect and reconnect network.
> ipconfig /release
> ipconfig /renew
Using this you can use the Java Runtime class to execute the command.
Runtime.getRuntime().exec("ipconfig /release");
I have tested this on windows and it works.
The Linux equivalent of the cmd commands are as follows
> sudo ifconfig eth0 up
> sudo ifconfig eth0 down
Note that eth0 here is the name of my Ethernet connection. You can list the names of all the connections using
> ifconfig -a
You can look at the following thread to execute bash through Java - How to execute bash command with sudo privileges in Java?
I am tasked with writing a client-side data download system (on Linux) that uses FTP or HTTP to download terabyte-sized data from external partners to our local site. Our company's network admin tells me that I cannot exceed a certain bandwidth. What is the best way for me to implement such a system? Do existing libraries exist?
I am open to writing my own FTP and HTTP clients (in either C or Java on Linux) but would prefer to stay out of the kernel. I know that I can limit the rate at which my FTP/HTTP client calls a socket read(), but what happens if the server-side calls write() faster than my limit?
You could build another layer on top of an InputStream: In the read method, you can count the bytes so far. If the number of bytes/second exceed a certain limit, let the download thread sleep for a while. TCP's flow control does the rest.
I know Apache JMeter simulates slow connections. You could maybe take look at the code.
If you know the network path delay you could just set your TCP receive buffer size to the desired bandwidth-delay product. That will throttle the sender all right. But the resulting value may be too small for your platform, so it may adjust it upwards. Check the value after you set it.
Does your netadmin know that TCP automatically shares bandwidth fairly?
Are you open to off the shelf GUI or command line products? Filezillia provides this.
There also is a linux command line client called lftp. A settable parameter is net:limit-total-rate which will limit the rate of transfer. Since this client supports multiple transfers at one time, it also has a parameter net:limit-rate.
To keep it simple, if you are on linux you just could use wget instead of re-inventing the wheel? Take a look at the --limit-rate switch.
But back on topic :) This answer could get you started: How can I implement a download rate limited in Java?
I am starting too many processes in Java using:
Runtime.getRuntime().exec("java -ja myJar.jar")
When it reaches around 350 processes, I get an IO exception:
Cannot run program "java": java.io.IOException: error=24, Too many open files
Exception in creating process
at java.lang.ProcessBuilder.start(ProcessBuilder.java:475)
at java.lang.Runtime.exec(Runtime.java:610)
at java.lang.Runtime.exec(Runtime.java:448)
at java.lang.Runtime.exec(Runtime.java:345)
In each process I am using one database connection.
I am running ubuntu 32 bit OS. But when I run:
ulimit -u
I can see that process limit is unlimited. What could be the problem?
All systems have their limits - sounds like you've hit your system's limit.
In linux, creating new processes consumes lots of inodes (like windows handles), which is a lot like a file handle. The only way around it is to allocate more via kernel settings (I don't know how offhand).
Have you considered starting lots of java Threads instead? They would consume a lot less system resources.
The problem is that you have too many files open, not too many processes in operation. To check the file limit do:
ulimit -n
It will commonly be 1024.
Check http://www.puschitz.com/TuningLinuxForOracle.shtml and search for ulimit for good instructions on changing this limit.
Running 350 JVM instances is not the normal way. Can you redesign and run 350 "main threads" within the same JVM. That is how servlet container works. All web applications are running in the same JVM.
PD. ulimit man pages says to see max. open files is ulimit -n.
So basically the problem is described in the title.
The server works in the following way:
Listens to a new connection
Once connection is requested - adds the request to the Q,
Continues listening to a new connection
Separate process takes care of a Q and spawns a new thread to deal with the clients' requests.
The server code is similar to this tutorial (everything is in try / catch, unfortunately I cant show the source-code - company policy)
It seems to work very well, until the number of clients exceeds ~ 50, Then it just hangs with no exceptions / warnings / etc. There is a cpu thread limit of 32k, no limits on the number of open files / open sockets / etc. OS = CentOS 5.5 (same seems to happen in ubuntu tho). The server logs data to MySQL using ODBC. Separate stress tests of both showed that I can have up to 32k java processes (limited by /proc/sys/kernel/threads-max ) and MySQL can perform up to 20k simple operations / second, so Im assuming the problem is with the sockets.
So the question really is:
What is the limiting factor in socket connections and how can I make it bigger?
OR am I looking in the wrong place?
The chances are that you have induced a deadlock somewhere in the code. The key indicator here is if by 'hang' you mean the CPU usage of the server drops to nothing and no futher activity is seen in the server.
When the server hangs run jdk tool: jstack against it's process. This should show you what is waiting on what lock. Also in the tool kit is jvisualvm and if on a unix box a simple kill -3 pid will do a thread dump to stderr.
With out the code or at least a reproducable sample I'm afraid I can't help much more. One thing you might want to look at is using jetty as your embedded server instead of a hand roled one, they have already been through the deadlock/threading pain so you don't have to.
Don´t know if this will help you and if your are using it, but try to run your socket server with java switch "-server",this will select the Java HotSpot Server VM.The -server turns on the optimizing JIT along with a few other "server-class" settings. Generally you get the best performance out of this setting. The default VM is -client.
Also check your other params, so your socket server don´t run with minimal resources
Have a nice day
I have a third party component which tries to send too many UDP messages to too many separate addresses in a certain situation. This is a burst which happens when the software is started and the situation is temporary. I'm actually not sure is it the plain amount of the messages or the fact that each of them go to a separate IP address.
Anyway, changing the underlying protocol or the problematic component is not an option, so I'm looking for a workaround. The StackTrace looks like this:
java.io.IOException: No buffer space available
at java.net.PlainDatagramSocketImpl.send(Native Method)
at java.net.DatagramSocket.send(DatagramSocket.java:612)
This issue occurs (at least) with Java versions 1.6.0_13 and 1.6.0_10 and Linux versions Ubuntu 9.04 and RHEL 4.6.
Are there any Java system properties or Linux configuration tweaks which might help?
I've finally determined what the issue is. The Java IOException is misleading since it is "No buffer space available" but the root issue is that the local ARP table has been filled. On Linux, the default ARP table lookup is 1024 (files /proc/sys/net/ipv4/neigh/default/gc_thresh1, /proc/sys/net/ipv4/neigh/default/gc_thresh2, /proc/sys/net/ipv4/neigh/default/gc_thresh3).
What was happening in my case (and I assume your case), is that your Java code is sending out UDP packets from an IP address that is in the same subnet as your destination addresses. When this is the case, the Linux machine will perform an ARP lookup to translate the IP address into the hardware MAC address. Since you are blasting out packets to many different IPs the local ARP table fills up quickly, hits 1024, and that is when the Java exception is thrown.
The solution is simple, either increase the limit by editing the files I mentioned earlier, or move your server into a different subnet than your destination addresses, which then causes the Linux box to no longer perform neighbor ARP lookups (instead will be handled by a router on the network).
When sending lots of messages, especially over gigabit ethernet in Linux, the stock parameters for your kernel are usually not optimal. You can increase the Linux kernel buffer size for networking through:
echo 1048576 > /proc/sys/net/core/wmem_max
echo 1048576 > /proc/sys/net/core/wmem_default
echo 1048576 > /proc/sys/net/core/rmem_max
echo 1048576 > /proc/sys/net/core/rmem_default
As root.
Or use sysctl
sysctl -w net.core.rmem_max=8388608
There are tons of network options
See Linux Network Tuning by IBM
and More tuning information
Might be a bit complicated but as I know, Java uses the SPI1 pattern for the network sub-library. This allows you to change the implementation used for various network operations. If you use OpenJDK then you could gain some hints how and what to wrap with your implementation. Then, in your implementation you slow down the I/O with some sleeps for example.
Or, just for fun, you could override the default DatagramSocket with your modified implementation. Have the same package name for it and - as I know - it will take precedence over the default JRE class. At least this method worked for me on some buggy 3rd party library.
Edit:
1Service Provider Interface is a method to separate client and service code within an API. This separation allows different client and different provider implementations. Can be recognized from the name ending in Impl usually, just like in your stack trace java.net.PlainDatagramSocketImpl is the provider implementation where the DatagramSocket is the client side API.
You commented that you don't want to slow down the communication the entire way. There are several hacks to avoid it, for example measure the time in your code and slow the communication within the first 1-2 minutes starting at your first incoming method call. Then you can skip the sleep.
Another option would be to identify the misbehaving class in the library, JAD it and fix it. Then replace the original class file in the library.
I'm also currently seeing this problem as well with both Debian & RHEL. At this point I believe I've isolated it down to the NIC and/or the NIC driver. What hardware configuration do you have this also exhibits this problem? This seems to only occur on new Dell PowerEdge servers that we recently acquired that have Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet NICs.
I too can confirm that it is the rapid generation of outbound UDP packets to many different IP addresses in a short window. I've attempted to write a simple Java application that can reproduce it (since ours is occurring with snmp4j).
EDIT
Look at my answer here: Java IOException: No buffer space available while sending UDP packets on Linux
I have got this error when i tried to run coherence cluster in two local JVM using the WIFI connection to database..
If i run it using the ethernet - it runs well.