Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Questions asking us to recommend or find a book, tool, software library, tutorial or other off-site resource are off-topic for Stack Overflow as they tend to attract opinionated answers and spam. Instead, describe the problem and what has been done so far to solve it.
Closed 8 years ago.
Improve this question
I have an application that makes heavy use of JMS (OpenMQ at the moment). We need durable persistence on some of the messages in flight, and that needs to survive the database blowing up, so we've moved the broker's storage to JDBC as it means that we can replicate and back up a database cluster and know that both the relational and message stores are at the same point in time when restoring from a backup.
We're finding that OpenMQ's JDBC backed storage method is very slow (tens of messages/s). Are there any brokers out there that perform well using JDBC? We're ideally after thousands of messages/s, but can probably tolerate 100s/s.
Backing a messaging server with JDBC is never going to be as fast using an optimized message store. Most of the time, a broker simply writes messages to disk but do not read. A traditional database is often more optimized towards read performance and complex queries than maximum write performance.
ActiveMQ with replicated LevelDB seems to be a decent fit for your requirements. It will automatically replicate the message store to "slave" brokers while still allow high throughput.
Another option would be to dig into the database part rather than the broker part. You will probably notice similar patterns with other brokers as well, if you hook them up to JDBC. Is there a way to make your database work more efficient with write performance? SSD disks? Settings? etc. As I said, by default, most databases are tweaked towards read performance and performance of complex queries.
Related
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Questions asking us to recommend or find a book, tool, software library, tutorial or other off-site resource are off-topic for Stack Overflow as they tend to attract opinionated answers and spam. Instead, describe the problem and what has been done so far to solve it.
Closed 8 years ago.
Improve this question
I am building an centralized inventory system using java(swing and servlets). I have local data stored in desktop application which needs to be in sync with an server.
Requirement is:-
1) When local stores are online ,data must be synchronized with server in real time.
2) Local store will have access to their data even if they go offline, but when they go online all local changes must be reflected on server.
I got across microsoft sync framework which is exactly what I need. Is there something equivalent in java?
There are a number of Open Source Replication and Synchronization Tools Written in Java:
Of those listed the following might handle your "offline" requirement:
SymmetricDS – SymmetricDS is a web based, database independent, data synchronization system. Beyond replicating tables between relational databases the software incorporates functionality to handle a large number of databases, manage low-bandwidth connections and work within periods of network outage. The system supports two way replication, guaranteed delivery and multiple active schemas.
I'm sure the others are worth a quick look as well.
See also Strategy for Offline/Online data synchronization for some great discussion on things to think about/watch out for.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Questions asking us to recommend or find a tool, library or favorite off-site resource are off-topic for Stack Overflow as they tend to attract opinionated answers and spam. Instead, describe the problem and what has been done so far to solve it.
Closed 8 years ago.
Improve this question
My use case is to develop a telecom billing in Java. Knowing that i must calculate 60 000 bills per day, i need to distribute the load of the calculation on several(three or four) servers.
That is called Clustering ( correct me if it's wrong).
My Solution is to develop a middleware that will distribute for each server a list of client who should be charged, and then each server will calculate the bill and generate a PDF file.
Could you give me some more ideas, for example which Java class do i need, or which methods shall i use.
Thank you
if you use jms (maybe as part of using j2ee?) you could simply use a jms queue for this - have several consumers (mdbs in j2ee) on every node, and send the list of clients to process to the queue.
the queue will guarantee that every message (==client) will be handled by one and only one listener, and since each node will have a limited number of listeners you get work distribution this way.
try apache hadoop.
http://hadoop.apache.org/
It will be perfect for this kind of divide and conquer tasks.
Spring batch will be a good fit for that: http://projects.spring.io/spring-batch/
It has support for automatic restart of jobs and partitioning ( distribution over many machines ).
There is an entire chaptr in the docs dedicated to scaling/distribution: http://docs.spring.io/spring-batch/2.2.x/reference/html/scalability.html
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Questions asking us to recommend or find a tool, library or favorite off-site resource are off-topic for Stack Overflow as they tend to attract opinionated answers and spam. Instead, describe the problem and what has been done so far to solve it.
Closed 9 years ago.
Improve this question
I'm currently using NewRelic (http://newrelic.com/) for monitoring my java applications. Point is I would like to monitor Zookeeper, Hornetq and other java-based application that run without server or application container.
Looks like NewRelic does not support this things..
Could you please tell me if there is a good universal monitor tool that I can use for this purpose ? (rather than JMX / Jconsole...)
You might be interested in Ganglia: http://ganglia.sourceforge.net/
For Zookeeper: https://github.com/andreisavu/zookeeper-monitoring/tree/master/ganglia
Ganglia is a scalable distributed monitoring system for high-performance computing systems such as clusters and Grids. It is based on a hierarchical design targeted at federations of clusters. It leverages widely used technologies such as XML for data representation, XDR for compact, portable data transport, and RRDtool for data storage and visualization. It uses carefully engineered data structures and algorithms to achieve very low per-node overheads and high concurrency. The implementation is robust, has been ported to an extensive set of operating systems and processor architectures, and is currently in use on thousands of clusters around the world. It has been used to link clusters across university campuses and around the world and can scale to handle clusters with 2000 nodes.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Questions asking us to recommend or find a tool, library or favorite off-site resource are off-topic for Stack Overflow as they tend to attract opinionated answers and spam. Instead, describe the problem and what has been done so far to solve it.
Closed 9 years ago.
Improve this question
what is a better nosql database for creating a system to record advertisement data for about 50 to 200 millions insert per day, the aggregation of the data will be used to show the pattern of how users engage with the ads. I really like MongoDB but it seems that major industry players are picking Riak for the job. It seems that Mongo had to flush some caveats in last 2 releases and the current version seems to be pretty good for the job, any idea?
It seems mongodb with hadoop (http://docs.mongodb.org/ecosystem/tutorial/getting-started-with-hadoop/ ) fits your data requirements. You can store data in mongodb and run aggregation jobs (map/reduce) on hadoop cluster.
I'd use Java Berkeley DB from Oracle.
Very powerful and easy to use Open source but not free.
This is not a type of question that could be answered as Product1 or Product2. It is just too small amount of information given. There is no info about the environment, where the system will run, what type of information will be inserted, how are you going to aggregate it.
The best way is to try:
write a test using Product1,
write a the same thing with a Product2
start inserting the data which looks as close as possible to the
data you are assuming to get in real environment
make measurements of speed and whatever factors you need
and only based on that you will be able to determine what suits you
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Questions asking us to recommend or find a tool, library or favorite off-site resource are off-topic for Stack Overflow as they tend to attract opinionated answers and spam. Instead, describe the problem and what has been done so far to solve it.
Closed 9 years ago.
Improve this question
I think tow months ago. I found a google's open source project that can store key value pairs with high performance. But i forget the name. Could anybody tell me? or you can have some other suggestions for me? I have been using BerkerlyDB, but I found BerkerlyDb is not fast enough for my program. However, berkerylyDB is convenient to use as it appears as a java lib jar, which can be integraed with my program seamlessly. My program is also written in Java.
Two strong competitors in the DHT (Distributed Hash Table) 'market':
Cassandra (created by Facebook, in use by Digg and Twitter)
HBase
Here is a presentation about Cassandra. On slide 20 you'll see some speed benchmarks- 0.12 ms / write
(You can search around for the whole presentation, including Eric Evans talking)
Nobody mentions leveldb and yet this post is at the top when searching for "good key value store". Leveldb in my experience is simply awesome. It's so fast I couldn't believe it.
I've been trying quite a few databases for a task I was doing. I tried:
windows azure table storage (expensive, value size max 1 Mb and each property size is max 64 Kb)
redis (awesome if you have as much ram as you please)
mongodb (awesome as long as there is enough ram, breaks after that point)
sql server (expensive, needs maintenance, such as rebuilding indexes and eventually still not fast enough)
sqlite (free, but not as simple to use as leveldb and not fast)
leveldb. If you can model your job as to reading large consecutive chunks of data through an iterator then you'll get great speed. Writing is also pretty fast. Combine it with ssd disk and you'll love it.
Bigtable?
Redis
http://code.google.com/p/redis/
Maybe you should describe what features you need. If it doesn't need to be distributed (does it?) then I would try using the H2 Database. For those who think "it can't be fast because it's using SQL" please note that when using prepared statement, SQL parsing is only done once. Disclaimer: I'm the main author of H2.
Many answer seem to automatically assume need for distribution; but that seems odd if question refers to BDB.
With that in mind, beyond Redis and H2 (which are both good), there is also Tokyo Cabinet to consider, which seems to offer benefits over BDB. And one more newer possibility is Krati.
I think you saw Guava or Google collections.