I have several instances of a distributed application running on the localhost; every instance communicate with others through certain ports, all instances together make an ensemble.
(I'm actually talking about ZooKeeper, running on Linux)
Now I want to write unit tests to emulate ensemble partitioning.
E.g. I have 5 instances, and I want to split them into two groups of 3 and 2 so that an instance from one group couldn't communicate to an instance from another group. It would emulate real-world situation when 3 machines are in one datacenter, 2 machines are in another one, and datacenters become partitioned.
The problem is essentially to make a socket work selectively: speak to one socket, but don't speak to another. One solution that comes to mind is to abstract communication layer and inject testing rules into it (in the form of "if I'm an instance from one group I'm not not allowed to speak to an instance from another group -- close socket or ignore data or whatever").
But maybe there exist some tools for this, maybe some testing framework?
In general, how do you test such cases in your distributed apps?
P.S. Though question is tagged with "java" (since ZooKeeper is written in Java), it'd be great to hear solutions for any other language, or language-independent ones -- maybe some linux guru-tricks.
Maybe this answer will save a few hours of googling for someone.
Linux has a very good firewall utility, iptables. It allows you to block communication between processes, like this:
iptables -A INPUT -p tcp --sport <source port> --dport <dest port> -j DROP
While not a full-blown unit testing framework by any measure, this helps a bit with manual testing in my case.
I would be tempted to say that this is integration testing rather than unit testing (or that it will be really hard to do proper unit testing of such thing).
The few times I needed to test such things, I've used virtualisation (e.g. VMWare) to test the system. You can test reboot, shutdown, etc. of nodes all from one physical machine running several images.
In the past I have disconnected network cables. You can always disable a network interface via the OS.
try blockade. https://github.com/dcm-oss/blockade
It's a docker based tool to emulate the behaviour you want. Runs on linux only, but there is a Vagrant file for setup if you want to run it from another OS.
Related
I am using Spring Boot mail and ActiveMQ to build an email system. I followed this example project. Because our application QPS is small one server is enough to handle the requests. In the example project ActiveMQ, sender, and receiver are all on the same server. Is this a good practice for small application? Or I should put ActiveMQ, sender, and receiver on three separate machines?
It's depends...
The size of the application is irrelevant. It depends more on your requirements for availability, scalability and data safety.
If you have everything on the same machine you have a single point of risk. If the machine crash you lost everything on that machine. But this setup is the most
simple one (also for maintenance) and the change that the server will crash is low. Modern machines are able to handle a big load.
If you have a really high load and/or a requirement for guaranteed delivery you should use multiple systems with producers that sends messages to an ActiveMQ cluster (also distributed over multiple machines). The consumers, also on more than one machine. Use also load balancers to connect/interface to the machines.
You can also have a setup in the middle of both example setups (simple and
complex).
If you are able to reproduce all the messages (email messages in your example), and the load is not so high, I will advise you to put it simple all on the same machine.
The short answer is it depends. The longn answer is measure it. The use of small application criteria is flawed. You can have both on the same server if your server have all the resources required by your application and message queue broker, and not impacting the performance of end user.
I would suggest run your performance tests to test your criteria then decide your target environment setup.
The simplest setup is everything on the same box. If this one box has enough CPU and disk space, why not ? One (performance) advantage is that nothing needs to go over the network.
If you are concerned about fault-tolerance, replicate that whole setup on a second machine.
I would like to have an automated test for the following scenario:
User logs in and performs some "lengthy" operation. For example tries to upload a file.
In the middle of that lengthy operation, the connection to the server is lost for a period of time, and then restored.
Application does whatever it's supposed to do in this case. For example shows a message to the user asking if they want to retry.
I want steps 1. and 3. to be done by Selenium. Question is: how can I do the step 2. as part of the automated test?
Details on the tools and environment:
Selenium is on Java with Junit
Test must run on Linux and Windows
It will also run on 3 browsers: Firefox, Chrome, IE 11
Couple of solutions I was thinking about:
Call some script that manipulates a connection on the machine.
Manipulate proxy settings of the browser (proxy points to fake address, thus connection is broken).
Use a real proxy, which could be controlled from the code (e.g. by some commands), so it can behave as "working" or "broken" connection.
None of these solutions is ideal for various reasons.
So: did anyone try to solve a similar problem? Would like to hear your solution or alternative ideas, that you think may work. Thanks.
Option 1
Stubbing
You'll have to identify what would be the exception thrown and what will be the component that throws it in a real case scenario. You can do that easily, simulate the scenario in your machine and when the exception is thrown, the stack-trace will tell you exactly what component thrown it.
Then you'll have to extend the component that throws the exception and inject it in the proper place and, ultimately, create an API to trigger that exception.
If you think you need a framework to automate this kind of tests, have a look to to Fitnesse.
Option 2
Simulation
Simulating a real network problem, would be overly complicated and the benefits are not worth the effort in this case (imo).
Anyway... Linux has an excellent built in network emulation layer called netem that allows any kind of seamless interaction with the network traffic. This kernel module is controlled by the tc command line interface.
It's not very easy to get it right when you want to apply those condition to a single port because you'll have to:
Netfilter rule to mark the packets that we want to limit
Traffic Control policy
Filter to bind the packets to the policy
A simple example would be if you want to have 10% packet loss on the whole eth0, in which case you'll use:
tc qdisc change dev eth0 root netem loss 10%
Then you'll have to wrap this functionality in some java controllable way.
If I failed on discouraging you, here is a good guide: TrafficControl
You can execute the following commands on windows on cmd to disconnect and reconnect network.
> ipconfig /release
> ipconfig /renew
Using this you can use the Java Runtime class to execute the command.
Runtime.getRuntime().exec("ipconfig /release");
I have tested this on windows and it works.
The Linux equivalent of the cmd commands are as follows
> sudo ifconfig eth0 up
> sudo ifconfig eth0 down
Note that eth0 here is the name of my Ethernet connection. You can list the names of all the connections using
> ifconfig -a
You can look at the following thread to execute bash through Java - How to execute bash command with sudo privileges in Java?
I need to transfer data from one process to another.
I'm quite familiar with the subject when the two processes were originated from C code - not once I used files, signals and pipes in C-code to accomplish it, but never have I tried to do it between two processes where one was originated from Java code, and the other from C code.
Since all the above methods require the (Linux) native API, and the JVM is on the way, I have decided to use a socket to communicate between these two processes, and I have a couple of questions:
How common is it to use socket to communicate between two processes on the same machine?
Does the fact that there is no designated "Server" and "Client" can set any obstacles (implementation-wise)?
The reason I'm asking is that everywhere I read online, there is always one process defined as 'server', and one as 'client'. This is not the case in my situation. Plus, I never tried to use socket for this purpose.
How common is it to use socket to communicate between two processes on the same machine?
It's quite common for a certain class of interaction patterns: when two independently-launched programs need a bidirectional communication channel. You cannot easily use pipes for that ("independently-launched" interferes). You can use FIFOs, but you need two, someone needs to set them up, and there are other quirks.
Does the fact that there is no designated "Server" and "Client" can set any obstacles (implementation-wise)?
The distinction between "client" vs. "server" is first and foremost about roles in establishing communication: the server sets up the communication interface and waits for one or more clients to open a connection to it. Being a "server" does not necessarily imply supporting multiple clients (neither concurrently nor serially), nor does it necessarily imply anything about communication passing over the socket connection after it is established. If you use sockets then you do have client and server, but if you have no other way to designate which process should have which role then you can choose arbitrarily.
One trick with sockets in Java is that although the Java standard library supports them, it supports only network sockets, not UNIX-domain sockets. The latter are more commonly used in UNIX and Linux applications where communication is inherently restricted to processes running on the same machine, but network sockets listening to (only) the loopback interface can nevertheless serve that purpose, too.
On modern systems, local TCP connections are just as fast as UNIX-domain sockets, so using them is not a problem.
Connecting 2 processes together in a language and platform agnostic way can easily be achieved with a Socket. It's supported by all languages and platforms, and can easily be replaced with another method if wanted.
From your explanation I gathered that the Java process would be the server. Sockets are quite risk-free, since they don't require special permissions (for ports over 1024 at least) or any other special handling.
Just pay attention when designing the (application level) protocol your processes will be communicating through.
You might want to use the Java Native Interface. It might be exactly what you want. - Based on your aproach to use sockets on both programs.
You might want to look into shared memory on linux.
BUT: Using sockets is not a bad thing here in general, but i doubt it is common practice*.
*I lack proof, that this is not common practice tho.
I want to create Java Network servers which share one IP address. Something like the Piranha cluster:
Is there any solution similar to this?
P.S They have to work as a cluster. If one server is down the second one should handle the traffic.
Well the obvious solution would be to try to build your Java servers behind the Piranha layer; i.e. implement the application services on "real server 1", "real server 2", etcetera in Java
I'm pretty sure that you can't implement a Piranha-like solution in (pure) Java. The IP level load balancing is implemented in the network stack in the OS kernel (I think) of the "director". That rules out (pure) Java for two reasons:
It is impractical to put Java code inside the kernel.
To do it in user space in Java would entail using native code to read and write raw network packets. That is not possible in pure Java.
Besides, the chances are that you'd get better network throughput if the director layer was not implemented in Java, pure or otherwise.
Of course, there are other ways to do load balancing as well ...
Just create your standalone tcp/ip servers to listen on different ports (and ofcourse the IP address would be same as this is your requirement)
I have a Java application, and I need it to be high available.
I was thinking of FastMPJ, like running multiple instances on different PCs. Every minute the app will check if master instance is running, and if not, the other will run instead of it.
I'd like to ask if it is a good solution, or if there is any better.
A more general solution is to use a load-balancing system, that is: you have N instances of the application running with the same privileges (if possible on different hardware), then a redundant load-balancer in front selects one of those based on the actual load for each request/task.
The benefit of this solution is obviously, that hardware is actually used and doesn't sit somewhere idle, waiting on the 0.01% case to jump in. Then the instance is actually tested all the time, and errors will be reported when they happen (like faulty hardware), and you prevent a: "Oh... the backup isn't even working". And on top of that you balance the load between machines adaptively.
In one of my project while implementing a exchange we used Apache Qpid for high availability and my experiense was quite satisfaotory. It scales very well too. I have been running application up to 32 node clusters. Please find further details here and let me know in case u need any further infromation:
http://qpid.apache.org/releases/qpid-0.18/books/AMQP-Messaging-Broker-Java-Book/html/High-Availability.html
Hope it helps:)
One often forgets that there must also be high availability from the application to database as well. It is my experience that the data access layer is where most of the application bottlenecks occur. So make sure you have a good application aware DB load balancer. Oracle has a solid solution but is for Oracle databases only. PostGres has an open source version. Heimdall Data is a commercial solution.