I have multiple vms running all of the modules in the project. Each request created by the user has to be processed by all modules, but needs to be done only once. So if VM1 picks up a request then module1 can process the request partially, next VM1 or VM2 or any other VM in cluster can pick up and process for module2. And so on.
Since each VM is of limited capacity i would like to use a load balancer for allocating work among individual VM's.
Are there load balancers(open source for java) available which can solve this or do i need to implement it using several load balancing algos(round robin,weighted etc) for solving my requirement?
Edit 1:
Each module is a java class which is independent in itself but needs previous modules to be done before its started.Each Vm is listening to a message bus. As and when a message appears in the bus any of the vm can pick up this and start working on it.
You can try HAProxy (TCP/HTTP loadbalancer ) which is open source, feature rich and quite widely used. Apart from good documentation you can find lots of information available.
Depending on the exact semantics of the problem you're trying to parallelize, you might get good results by chunking your problem into "work packets" of some size and keeping them in a central queue. Then, just have each VM poll a packet from said queue as soon as it finished the previous packet. This is called self-scheduling.
Related
I am using Spring Boot mail and ActiveMQ to build an email system. I followed this example project. Because our application QPS is small one server is enough to handle the requests. In the example project ActiveMQ, sender, and receiver are all on the same server. Is this a good practice for small application? Or I should put ActiveMQ, sender, and receiver on three separate machines?
It's depends...
The size of the application is irrelevant. It depends more on your requirements for availability, scalability and data safety.
If you have everything on the same machine you have a single point of risk. If the machine crash you lost everything on that machine. But this setup is the most
simple one (also for maintenance) and the change that the server will crash is low. Modern machines are able to handle a big load.
If you have a really high load and/or a requirement for guaranteed delivery you should use multiple systems with producers that sends messages to an ActiveMQ cluster (also distributed over multiple machines). The consumers, also on more than one machine. Use also load balancers to connect/interface to the machines.
You can also have a setup in the middle of both example setups (simple and
complex).
If you are able to reproduce all the messages (email messages in your example), and the load is not so high, I will advise you to put it simple all on the same machine.
The short answer is it depends. The longn answer is measure it. The use of small application criteria is flawed. You can have both on the same server if your server have all the resources required by your application and message queue broker, and not impacting the performance of end user.
I would suggest run your performance tests to test your criteria then decide your target environment setup.
The simplest setup is everything on the same box. If this one box has enough CPU and disk space, why not ? One (performance) advantage is that nothing needs to go over the network.
If you are concerned about fault-tolerance, replicate that whole setup on a second machine.
On a distributed environment I have many nodes running the same code.
I have a process that handles an event and I want to make sure it is only handled once.
What are the recomended solutions for this requirement?
Here are my options from the top of my head:
Using a message broker (RabbitMQ, ActiveMQ, Kafka).
for instance, creating only one queue for this message.
But I don't like adding another component to the stack.
I dont really like adding a message brokers to the stack just for process sync.
use quartz.
http://www.quartz-scheduler.org/documentation/quartz-2.x/configuration/ConfigJDBCJobStoreClustering .
But I need a shared DB.
What are my other options?
I am working as a developer on a batch processing solution, how it works is that we split a big file and process it across jvms. So we have 4 processor jvms which take a chunk of file and process it and 1 gateway jvm job of gateway jvm is to split the file into no. of processor jvms i.e. 4 and send a rest request which is consumed by processor jvms, rest request has all the details the file location it has to pick the file from and some other details
Now if i want to add another processor jvm without any downtime is there any way we can do it. Currently we are maintaining the urls for 4 jvms in a property file is there any better way to do it ? which provided me the ability to add more jvms without restarting any component
You can consider setting up a load balancer and putting your JVM(s) behind it. The load balancer would be responsible for distributing the incoming requests to the JVMs.
This way you can scale up or scale down your JVM depending on the work load. Also, if one of the JVMs are not working, other part of your system need not care about it anymore.
Not sure what is your use case and tech stack you are following. But it seems that you need to have distributed system with auto-scaling and dynamic provisioning capabilities. Have you considered Hadoop or Spark clusters or Akka?
If you can not use any of it, then solution is to maintain list of JVMs in some datastore (lets say in a table); its dynamic data meaning one can add/remove/update JVMs. Then you need a resource manager who can decide whether to spin up a new JVM based on load or any other conditional logic. This resource manager needs to monitor entire system. Also, whenever you create a task or chunk or slice data then distribute it using message queues such as ApacheMQ, ActiveMQ. You can also consider Kafka for complex use cases. Now a days, application servers such as websphere (Libery profile), weblogic also provide auto-scaling capability. So, if you are already using any of such application server then you can think of making use of that capability. I hope this helps.
I can use the Java MQ Api to put and get messages.
I can also disable gets and put on a queue.
During a migration project, we'll have an App running in parallell. Old and New. Old and New will have their own separate queues. I regulary have messages from a client going to Old. Occasionally want the msgs to flow to New instead.
wondering if MQ supports a gate/switch concept. where via API I can point a queue to go only to New, or only to Old, for a short time.
Trying to avoid going to message based routing via WMB since I dont have to do that today. THe parallel mode is only for a few months.
You do not mention the version of MQ or whether there are message affinities or dependence on preserving the MQMD.MsgID. These are critical in devising a solution to this problem. I'll try to describe enough options so that at least one will be viable whatever version you are at.
Pub/Sub
The easiest thing to do is to have the messages arrive on an alias over a topic. Any message that arrives is published immediately on that topic. Then it is a simple matter to generate administrative subscriptions to direct messages to the queues on which the apps needing the messages are listening. This is entirely a configuration change and requires no external components, processes or code. It is available from v7.1 of MQ and higher, which is to say any of the currently supported versions of MQ.
The down side is that IBM MQ will change the MQMD.MsgID from the time the message is received on the topic to the time it is published on the application's input queue. This breaks the app's ability to use the MQMD.MsgID of the incoming message as a correlation ID when replying. If the requesting app pre-loads the correlation ID or doesn't rely on a correlation ID, this is not an issue.
Aliasing
But for apps where this is an issue, it gets a bit harder. You can alias over a queue and have inbound messages land on the alias. When you need to switch from one queue to another, you change the alias. There are a couple issues with this. The first is that it is never possible to deliver the message stream to more than one of the applications. In a parallel processing test it is often desirable to do exactly that and then compare summary or detail reports.
The second problem is more operational in nature. It isn't possible to change the alias while it is open. If the messages arrive over a RCVR, RQSTR or `CLUSRCVR channel, no problem. Stop the channe, switch the alias and restart the channel. In a series of MQSC script commands this can be done faster than it can be typed. However, if the applications putting the messages are connected in bindings mode or via client directly to the alias, they must all be stopped in order to change the alias.
That said, aliasing works on all versions of MQ out of the box.
Physical copy
One solution that's been around for quite some time is to use the Q program (SupportPac MA01) to direct the messages. In this scenario, the queue on which messages land is a local queue. The Q program is either triggered or set to constantly listen on the queue. When a message arrives, Q then copies it to one or both of the destination queues.
Switching the behavior if Q is triggered involves pre-defining 2 or 3 processes where each defines a different behavior - move new messages to QUEUEA, to QUEUEB or to both. Changing the queue's PROCESS attribute to point to a different process results in an instantaneous change of the behavior.
Alternatively, if Q is configured to listen on the queue forever then changing the behavior involves use of three different scripts to execute it where one causes messages to be copied to QUEUEA, another to QUEUEB and another to both queues. Changing the behavior involves killing the script and starting a different one.
The Q program works with all versions of MQ, regardless of whether it is triggered or scripted.
Downsides to this approach include the obvious - more moving parts. You have to trigger the queue or else make a transactional program act like a daemon. Not hard but if you are betting the business on it then perhaps some monitoring is in order to make sure the input queue doesn't start building.
Recommendation
Of all these methods, I really like the Pub/Sub version. It is extremely reliable, has the least moving parts, and if anything breaks it's under IBM support. When you need to change something, you can do that with minimal impact to the running applications. If at all possible, use that.
I want to find or develop an application that can run as a daemon, notify the administrator by email or sms when the Java applications running on a host get any exceptions or errors. I know JVMTI can achieve part of my goal, but it will impact performance of the monitored applications(I don't know how much will it be, it will be acceptable if it's slight), besides it seems to be a troublesom job to develop a JVMTI agent and I'm not sure what would happen if several applications running at the same time using the same agent. Is there any better solutions? Thanks in advance.
One way would be to use a logging system like log4j that publishes all errors occuring on system A to a logging server on system B from which you can monitor the errors occured. This isn't a completely generic solutation however, since only exceptions propagated to log4j (or any other logging system) would be handled - but it may be a good start.
The best solution is to have the Java application send its errors via email/sms. The problem is that programs will generate exceptions and handle correctly in normal operation. You only want particular exception.
Failing this you could write a log reader, which reads the logs of the application. This is tricky to get right, but it can be done.
An application can generate 1000+ exception per days and still be behaving normally because the application knows how to handle these exceptions. e.g. every time a socket connection is closed an exception can be thrown.
IMO, the best approach is to deploy an external monitoring system. This can:
monitor multiple applications
monitor infrastructure services
monitor network availability and machine accessibility,
monitor resources such as processor and file system usage.
Applications can be monitored in a variety of ways, including:
by processing log events,
by watching for application restarts,
by "pinging" the application's web apis to check service liveness, and
by using the application's JMX interfaces.
This information can be filtered and prioritized in an intelligent fashion, and critical events can be reported by whatever means is most appropriate.
You don't want individual applications sending emails, because they don't have sufficient information to do a decent job. Furthermore, putting the reporting logic into individual applications is likely to lead to inconsistent implementation, poor configurability, and so on.
There is a nearby alternative to JVMTI : JPDA. This infrastructure allows you to create a remote "debugger" (yes, that's what you're planning to do) using Java code, and connect it to the VM using either local or remote connection.
There will be, like for JVMTI, an overhead to program execution. However, as the Trace.java example shows, it's quite simple to both implement and connect to target VM.
Finally, notice if you want to instrument code run by application server (JBoss, Glassfish, Tomcat, you name it) there are various other means available.
I follow the pattern where every exception gets logged to a table.
Then an RSS feed selects from that table.
I subscribe to the RSS feed in MS Outlook at work and also on my Android phone with a program called NewsRob. NewsRob let me set my phone to alert me when there is something new.
I blog about how to do this HERE. It is in .net, but you get the idea.
As a related step I found a way to notify myself when something DIDN'T happen. That blog is HERE.
There are loads of applications out there that do what you are looking for in a way that does not impact performance. Have you had a look at Kibana/ElasticSearch, or Splunk or Logscape for enterprise solutions ( they both also have free versions).
I'm going to echo what has already been said and highlight what java already provides and what you can do with an external monitoring system. Java already provides:
log4j - log ERRORS, WARNINGS, FATAL and Exceptions to a file
JMX - Create custom application metrics and you also have access to java.lang/* which will give you heap memory usage , garbage collection, thread counters etc.
JVM gc logging - you can log all your garbage collection events to a file and watch for any long Full GC collections.
An external monitoring system will allow you to set alerts triggered off different operational scenarios. You will also get visualisation of your system performance through charts. I've used Logscape's java app in the past to monitor 30 java processes spread out over3 hosts.