I need some advice regarding potential hosting solutions - there is an incredible amount of choice and confusing options out there.
Basically I have a Java application that contains an embedded ActiveMQ message broker. The job of this application is to:
1) Process Messages (JMS) recieved on the broker from 10-15 sources
2) Publish Messages (JMS) to a different JMS broker on another server (in our office).
So, I am looking for something that will not cost the earth (this is only for testing purposes) but could offer decent RAM and processing speed options so that we can really test the limits of the application (we need to see what the bottleneck is, whether it is Active MQ or the processing APP).
Also, the outgoing bandwidth costs will need to be a consideration. Again, the volumes will be sporadic and sometimes signficant (depending on the intensity of testing periods).
Any recommendations would be appreciated.
Thanks
Maybe Heroku is what you are searching for. There is a monthly free tier and you can run any Java application. There is no ActiveMQ available, but via add-ons you can use RabbitMQ, which also supports the JMS. You can increase performance of you application at any time - have a look at Herokus pricing.
Related
I am using Spring Boot mail and ActiveMQ to build an email system. I followed this example project. Because our application QPS is small one server is enough to handle the requests. In the example project ActiveMQ, sender, and receiver are all on the same server. Is this a good practice for small application? Or I should put ActiveMQ, sender, and receiver on three separate machines?
It's depends...
The size of the application is irrelevant. It depends more on your requirements for availability, scalability and data safety.
If you have everything on the same machine you have a single point of risk. If the machine crash you lost everything on that machine. But this setup is the most
simple one (also for maintenance) and the change that the server will crash is low. Modern machines are able to handle a big load.
If you have a really high load and/or a requirement for guaranteed delivery you should use multiple systems with producers that sends messages to an ActiveMQ cluster (also distributed over multiple machines). The consumers, also on more than one machine. Use also load balancers to connect/interface to the machines.
You can also have a setup in the middle of both example setups (simple and
complex).
If you are able to reproduce all the messages (email messages in your example), and the load is not so high, I will advise you to put it simple all on the same machine.
The short answer is it depends. The longn answer is measure it. The use of small application criteria is flawed. You can have both on the same server if your server have all the resources required by your application and message queue broker, and not impacting the performance of end user.
I would suggest run your performance tests to test your criteria then decide your target environment setup.
The simplest setup is everything on the same box. If this one box has enough CPU and disk space, why not ? One (performance) advantage is that nothing needs to go over the network.
If you are concerned about fault-tolerance, replicate that whole setup on a second machine.
I have a scenario with these particular demands:
Production ready & stable.
Point to point connection, with the producer behind a firewall and a consumer in the cloud. It might be possible to split the traffic between a couple of producers\consumers, but all the traffic still has to traverse a single WAN connection which will probably be the bottleneck.
High throughput - something along the order of 300 Mb/sec (may be up to 1Gb!). Message sizes vary from ~1KB to possibly several MBs.
Guaranteed delivery a must - every message has to arrive at the consumer eventually, so we need to start saving messages to disk in the event of a momentary network outage or risk running out of memory.
Message order is not important, messages are timestamped and can be re-arranged at the consumer.
Highly preferable but not as important - should run on both linux & windows (JVM seems the obvious choice)
I've been looking at so many MQs lately, and I don't have any hands-on experience with any.
Thought it will be a better idea to ask someone with experience.
We're considering mostly Kafka, but I'm not sure it's the best for our use case, seems to be tailored to distributed deployment & mutliple topics\consumers\producers. Also, definitely not production ready on windows.
What about Apache ActiveMQ or Apollo\Artemis? RabbitMQ seems not to be a good fit for our performance requirements. Or maybe there's some Java library that has the features we need without a middleman broker?
Any help making sense of this kludge would be greatly appreciated.
If anyone comes across this, we went with Kafka in the end. Its performance is impressive and so far it's very stable on linux. No attempt yet to run it on windows in production deployments.
UPDATE 12/3/2017:
Works fine and very stable on Linux, but on Windows this is not usable in production. Old data never gets deleted due to leaky file handles, the relevant Jira is being ignored since 2013: https://issues.apache.org/jira/browse/KAFKA-1194
Hello I am a student just learning to use Netty and MySQL.
I am building a server for my android and iOS application. I built my server based on using Netty 4.0.6 example HttpUploadServer.
The server's primary task is to send/recieve and save images and audio files(about 1mb in total). About 10,000 requests will be sent daily.
One of my advisor said that two things should be the most thought about when developing a server.
Scaling up and out
High availability
However, (as I am just learning server programming) I have no idea how to do them. The only thing I can think to increase scalability and availability is something like Amazon's Elastic Load Balancer.
I know this is a very broad question, but please give me a headway.
How can I increase scalability and availablity using Java(Espcially Netty)?
Scaling up can be achieved trough many techniques
Having multiple instances: aka Elastic Load Balancers
Sharding: server 1 handles requests for users A-M server 2 handles requests for users N-Z
Add caching: Are you servicing the same request multiple times? Throw some memory at the problem at keep serving the same answer
Simplify your workload!
The really important question you need to answer is what is limiting your ability to server N+1 clients. Are you running out of sockets, memory, cpu time, db transactions?
Like any profiling problem work out what your dominant problem is and solve it.
I am looking for a simple persistent buffer as temporary storage for JSON messages in a Java application. Memory usage should be relatively constant and not depend on the number of messages in the buffer. It would nice to be able to replay messages from a point in the past. Deletion of old messages should be efficient. Needs to be able to handle 1m messages/h.
Currently my application uses a local RabbitMQ broker which shovels messages to a remote RabbitMQ broker. When the remote broker is down or not accepting messages the local RabbitMQ broker's memory usage rises with the queue length and eventually it stops accepting messages. I want to swap this out for a local disk based buffer and a thread copying messages to the remote RabbitMQ broker.
Anyone have any ideas? I have looked at Kafka but it seems like overkill for my use-case. MongoDB is a possibility but I am worried about its memory usage.
Memory usage is always an issue in any system.I am using MongoDB for production and when I compare with similar solutions (CouchDB,CouchBase,redis.io), MongoDB is really good in memory management and easiness of implementation. But I should admit , I never had a chance to test Riak more in detail.
I am storing 5.000.000 user records with 4 index fields and all user session behind a rest/web service api which uses a messaging service behind.
My messaging service uses another db instance on the same server.
My user records have at least 20 fields and session records have just 5 fields.
My ubuntu servers never used more than 10 GB rams even with heavy loading processes.
Hope this helps to figure out.
ps: all depend on data model and how you implement your infrastructure.
Regards,
EDIT:
I think this is a good slideshow about using MongoDB for messaging.
and a nice article about MongoDB and messaging.
You can use the test code and see the results are ok for your solution.
Please don't forget to share your results if you test.
I am currently investigating what Java compatible solutions exist to address my requirements as follows:
Timer based / Schedulable tasks to batch process
Distributed, and by that providing the ability to scale horizontally
Resilience, no SPFs please
The nature of these tasks (heavy XML generation, and the delivery to web based receiving nodes) means running them on a single server using something like Quartz isn't viable.
I have heard of technologies like Hadoop and JavaSpaces which have addressed the scaling and resilience end of the problem effectively. Not knowing whether these are quite suited to my requirements, its hard to know what other technologies might fit well.
I was wondering really what people in this space felt were options available, and how each plays its strengths, or suits certain problems better than others.
NB: Its worth noting that schedule-ability is perhaps a hangover from how we do things presently. Yes there are tasks which ought to go at certain times. It has also been used to throttle throughput at times when no mandate for set times exists.
Asynchronous always brings JMS to mind for me. Send the request message to a queue; a MessageListener is plucked out of the pool to handle it.
This can scale, because the queue and listener can be on a remote server. The size of the listener thread pool can be configured. You can have different listeners for different tasks.
UPDATE: You can avoid having a single point of failure by clustering and load balancing.
You can get JMS without cost using ActiveMQ (open source), JBOSS (open source version available), or any Java EE app server, so budget isn't a consideration.
And no lock-in, because you're using JMS, besides the fact that you're using Java.
I'd recommend doing it with Spring message driven POJOs. The community edition is open source, of course.
If that doesn't do it for you, have a look at Spring Batch and Spring Integration. Both of those might be useful, and the community editions are open source.
Have you looked into GridGain? I am pretty sure it won't solve the scheduling problem, but you can scale it and it happens like "magic", the code to be executed is sent to a node and it is executed in there. It works fine when you don't have a database connection to be sent (or anything that is not serializable).