Im currently developing a Java Server Application which has a lot of tables with some data that is modified frequently. All of the data has to get often fully retrieved (to display it to the user or to index it). As MySQL queries seem to be expensive (and problematic because I need the data async) I came up with the idea of literally loading in the entire table as a local cache. That way I can index the data always from the local cache and I only need to send changes to the mysql. Fortunately the data is not that big so I dont have to worry about OutOfMemory Exceptions.
What is more efficient, repeatingly sending queries to MySQL or load all data in local cache to do operations?
Thanks for reading,
Regards, Tech!
Related
I have an existing database in a file. I want to load the database in memory; because I'm doing a lot queries and the database isn't very large (<50MB) to fasten those queries. Is there any way to do this?
50 MB easily fits in the OS file cache; you do not need to do anything.
If the file locking results in a noticeable overhead (which is unlikely), consider using the exclusive locking mode.
You could create a RAM drive and have the database use these files instead of your HDD/SSD hosted files. If you have insane performance requirements your could go for a in memory database as well.
Before you do for any in memory solutions: what is "a lot of queries" an what is the expected response time per query? Chances are that the database program isn't the performance bottleneck, but slow application code or inefficient queries / lack of indexes / ... .
I think SQLite does not support concurrent access to the database, which would waste a lot of performance. If write occur rather infrequently, you could boost your performance by keeping copies of the database and have different threads read different SQLite instances (never tried that).
Either of the solutions suggested by CL and Ray will not perform as well as a true in-memory database due to the simple fact of the file system overhead (irrespective of whether the data is cached and/or in a RAM drive; those measure will help, but you can't beat getting the file system out of the way, entirely).
SQLite allows multiple concurrent readers, but any write transaction will block readers until it is complete.
SQLite only allows a single process to use an in-memory database, though that process can have multiple threads.
You can't load (open) a persistent SQLite database as an in-memory database (at least, the last time I looked into it). You'll have to create a second in-memory database and read from the persistent database to load the in-memory database. But if the database is only 50 MB, that shouldn't be an issue. There are 3rd party tools that will then let you save that in-memory SQLite database and subsequently reload it.
I am programming a server application (chat server side) with java, which receive requests and send to a target.
I have several tables in the database, and when the server
application starts, I programmed it to copy all the content of the
database to Map tables (in the ram) in order to speed up the pull
push data while the application running.
Dose this correct way? Or you suggest me to pull data from the database directly when I need a detail. and remove the Map<>
tables from the ram!?
I suffer from memory leak.
Does dealing with the database slows the application?
Whether caching all or some of the data in memory makes sense depends on your use case. It can improve performance but it adds complexity which might not be needed.
You can load millions or even billions of records into a JVM, but much more then this you need an off heap storage such as a data store for this purpose or a database. Using off heap memory, you can have trillions of records in a JVM but this is rarely needed.
We are using MongoDB as an intermediate storage for an application that allows the user uploading downloading video files.
We are using GridFS API from a Java applications , as is very convenient for the case (We found it more appropriate, faster and reliable than storing the files in a table in a relation database).
Once the videos have been processed from the DB (and stored into physical files) we can remove them, but we have the problem that the new space is not just reallocated and is instead "used" without any util data. We have tried to repair the database, as suggested in posts like Auto compact the deleted space in mongodb? but this had the database down for few days! , which is not ideal as it needs to be running 24/7. (We have come accoss this recently when the DB was without free space).
I am not very knowledgeable in this topic and so would like to get opinions for a solution you know/use that would be efficient: Allow the storage in blocks and be easy to reallocate memory once the blocks/chunks are not needed.
Some options are:
1) Have two Mongo DB: from time to time export and import data(all
except the tables used by GridFS that contain the videos) from one
db to the other. First db can be dropped and space defragmented
again. This seems a bit complex and not good if it needs to be done
frequently as we don't have much total space for the DB.
2) Store them in a relational database (for a table without relation
and these special characteristics does not seem ideal but works
if other solutions don't)
3) ...
If it serves, the application is deployed in a J2EE infrastructure.
Thanks.
I am developing a simple client server application in Android, although I have a doubt. I will save the data from app at server, and the app will need access them. I would like to know what's the best approach to manage data; Using two databases, one in the server and another in the smartphone (It will need synchronization), or only one at server (All requests will have to go to server) ?
Thanks in advance.
Well, frankly speaking, it depends on the kind of data you are handling.
If your data is totally dynamic and varies at short intervals of time, you should always be querying the server.
On the other hand, if your data is relatively static and remains same over longer periods of time, it would be wise to cache the data in your local database, instead of querying it again and again from the server.
Another thing you have to keep in mind is the size of the data. If the data is really very large and you are storing it in your local database, then you need to clear the old data after a certain interval to make sure that your app is not eating up your memory.
It depends on your needs. Manage very large databases in the app is not the appropriate way (because the app will be very large after a while). You should store all of your data on the server and use SQLite in the app for cache only, but do not cache everything, just the data that don't change often.
In an application I'm working on, I need a write-behind data log. That is, the application accumulates data in memory, and can hold all the data in memory. It must, however, persist, tolerate reasonable faults, and allow for backup.
Obviously, I could write to a SQL database; Derby springs to mind for easy embedding. I'm not tremendously fond of the dealing with a SQL API (JDBC however lipsticked) and I don't need any queries, indices, or other decoration. The records go out, and on restart, I need to read them all back.
Are there any other suitable alternatives?
Try using a just a simple log file.
As data comes in, store in memory and write (append) to a file. write() followed by fsync() will guarantee (on most systems, read your system and filesystem docs carefully) that the data is written to persistent storage (disc). These are the same mechanisms any database engine would use to get data in persistent storage.
On restart, reload the log. Occasionally, trim the front of the log file so data usage doesn't grow infinitely. Or, model the log file as a circular buffer the same size as what you can hold in memory.
Have you looked at (now Oracle) Berkeley DB for Java? The "Direct Persistence Layer" is actually quite simple to use. Docs here for DPL.
Has different options for backups comes with a few utilities. Runs embedded.
(Licensing: a form of the BSD License I beleive.)