In a search application, I need to keep track of the files and their locations. Currently am using a database table for this, but since I have to connect to the db every time I need to retrieve such data, this is obviously not efficient. Is there a method I can load the table to memory and use it? I won't need to modify it while it's in the memory.
Thank You!
If all you want to do is retrieve one table into memory you can do this with a single SELECT statement. You can build a collection like a Map from the ResultSet. After that get the information you want from the Map.
You could populate any of the several Java databases out there that have an in-memory mode, like HSQLDB, Derby, or H2. You might also look at SQLite, which isn't specifically Java but has various Java connectors as described in this Q&A here on StackOverflow.
But you don't have to connect to a DB each time you need to query it, you can use a connection pool to manage a set of connections you can reuse. Since usually the main delay is establishing a connection, this can lead to quite lot per-query overhead.
You could also use one of caching products like Ehcache, Memcache, Coherence and many others. I have some knowledge in using Ehache. Configure Hibernate to cache a particular query or entity object or a POJO. All subsequent searches with same criteria will be fetched from cache.
I believe similar features are provided by other products as well.
Your sentence "I won't need to modify it while it's in the memory." does not reflect the title of your question, where you apparently want to modify an commit back your data after using it.
If you simply want to speedup your app, why don't you store your data in some kind of variable? Depending on your development tool, it could be some kind of session variable.
Related
I thought about this solution: get data from web service, insert into table and then join with other table, but it will affect perfomance and, also, after this I must delete all that data.
Are there other ways to do this?
You don't return a record set from a web service. HTTP knows nothing about your database or result sets.
HTTP requests and responses are strings. You'll have to parse out the data, turn it into queries, and manipulate it.
Performance depends a great deal on things like having proper indexes on columns in WHERE clauses, the nature of the queries, and a lot of details that you don't provide here.
This sounds like a classic case of "client versus server". Why don't you write a stored procedure that does all that work on the database server? You are describing a lot of work to bring a chunk of data to the middle tier, manipulate it, put it back, and then delete it? I'd figure out how to have the database do it if I could.
no, you don't need save anything into database, there's a number of ways to convert XML to table without saving it into database
for example in Oracle database you can use XMLTable/XMLType/XQuery/dbms_xml
to convert xml result from webservice into table and then use it in your queries
for example:
if you use Oracle 12c you can use JSON_QUERY: Oracle 12ะก JSON
XMLTable: oracle-xmltable-tutorial
this week discussion about converting xml into table data
It is common to think about applications having a three-tier structure: user interface, "business logic"/middleware, and backend data management. The idea of pulling records from a web service and (temporarily) inserting them into a table in your SQL database has some advantages, as the "join" you wish to perform can be quickly implemented in SQL.
Oracle (as other SQL DBMS) features temporary tables which are optimized for just such tasks.
However this might not be the best approach given your concerns about performance. It's a guess that your "middleware" layer is written in Java, given the tags placed on the Question, and the lack of any explicit description suggests you may be attempting a two-tier design, where user interface programs connect directly with the backend data management resources.
Given your apparent investment in Oracle products, you might find it worthwhile to incorporate Oracle Middleware elements in your design. In particular Oracle Fusion Middleware promises to enable "data integration" between web services and databases.
I needed to implement a utility server that tracks few custom variables that will be sent from any other server. To track the variables, a key value collection, either JDK defined or custom needs to be used.
Here are few considerations -
Keeping all the variables in memory of the server all the time is memory intensive.
This server needs to be a very lightweight server and I do not want heavy database operations.
Is there a pre-defined streaming collection which can serialize the data after a threshold memory and retrieve it on need basis?
I hope I am clear in defining the problem statement.
Please suggest if any other better approach.
this thing looks very promising, but is in development stage...
JDBM3
Edit Current version of the file backed collections: MapDB.
Database
What you've described sounds exactly like you should use a database (i.e. indexed key/value store, too big for memory but want performance benefits of in-memory caching where possible).
I'd recommend a lightweight embedded database such as H2 - it's small, fast and should suit your purposes very well.
Have you thought of using an on the shelf nosql queue value store? Redis for example?
If you want it java only you have the option of using a lib like ehcache, it would have the functionalities you need.
I have an upcoming project where the core of it will be storing a mapping between two integers. ( 1234 in column A maps to 4567 in column B). There are roughly 1000 mappings. A lookup on the mappings will be done every time a user hits a certain url on the site.
It seems like inserting it into our relational database is overkill. The overhead of selecting it out on every hit seems high. On the other hand, storing it an XML file and loading that flat file from disk every time theres a hit, also seems less than optimal.
So my question is this: what is the ideal data structure and method to persist this mapping?
The system architecture is tomcat + apache + mysql. The code will be running in tomcat.
EDIT:
Mappings are static, I won't need to change them. Seems like the XML file in a hashmap is the way to go.
I would use a properties file or an XML file, load it into memory (as a HashMap<Integer, Integer>) on startup and then just serve from the hashmap.
If you need to change the mapping at execution time, you could either write it back immediately or potentially just write changes incrementally (and update the in-memory map), with a process to unify the original file and the changes on startup. This doesn't work terribly well if you need to scale to multiple servers, of course - at that point you need to work out what sort of consistency you need etc. A database is probably the simplest way of proceeding, but it depends on the actual requirements.
I agree a relational database seems a bit of overkill. You may want to look at a NoSQL database. MongoDB is my personal favourite, but there are plenty out there. Do a search on NoSQL databases.
A NoSQL database will allow you to store this mapping as a simple document, with extremely faster searching and updating of the data. Obviously it's another technology in your stack though, so that's something for you to consider.
You could try using an in-memory database like H2 or HSQLDB. The memory footprint will likely be larger than with in-memory hashmap and file, but on the upside you can use SQL for querying and updating and don't need to worry about concurrent access.
I had completed my project Address Book in Java core, in which my data is stored in database (MySql).
I am facing a problem that when i run my program on other computer than tere is the requirement of creating the hole data base again.
So please tell me any alternative for storing my data without using any database software like mysql, sql etc.
You can use an in-memory database such as HSQLDB, Derby (a.k.a JavaDB), H2, ..
All of those can run without any additional software installation and can be made to act like just another library.
I would suggest using an embeddable, lightweight database such as SQLite. Check it out.
From the features page (under the section Suggested Uses For SQLite):
Application File Format. Rather than
using fopen() to write XML or some
proprietary format into disk files
used by your application, use an
SQLite database instead. You'll avoid
having to write and troubleshoot a
parser, your data will be more easily
accessible and cross-platform, and
your updates will be transactional.
The whole point of StackOverflow was so that you would not have to email around questions/answers :)
You could store data in a filesystem, memory (use serialisation etc) which are simple alternatives to DB. You can even use HSQLDB which can be run completely in memory
If you data is not so big, you may use simple txt file and store everything in it. Then load it in memory. But this will lead to changing the way you modify/query data.
Database software like mysql, sql etc provides an abstraction in terms of implementation effort. If you wish to avoid using the same, you can think of having your own database like XML or flat files. XML is still a better choice as XML parsers or handlers are available. Putting your data in your customised database/flat files will not be manageable in the long run.
Why don't you explore sqlite? It is file based, means you don't need to install it separately and still you have the standard SQL to retrieve or interact with the data? I think, sqlite will be a better choice.
Just use a prevayler (.org). Faster and simpler than using a database.
I assume from your question that you want some form of persistent storage to the local file system of the machine your application runs on. In addition to that, you need to decide on how the data in your application is to be used, and the volume of it. Do you need a database? Are you going to be searching the data different fields? Do you need a query language? Is the data small enough to fit in to a simple data structure in memory? How resilient does it need to be? The answers to these types of questions will help lead to the correct choice of storage. It could be that all you need is a simple CSV file, XML or similar. There are a host of lightweight databases such as SQLite, Berkelely DB, JavaDB etc - but whether or not you need the power of a database is up to your requirements.
A store that I'm using a lot these days is Neo4j. It's a graph database and is not only easy to use but also is completely in Java and is embedded. I much prefer it to a SQL alternative.
In addition of the others answers about embedded databases I was working on a objects database that directly serialize java objects without the need for ORM. Its name is Sofof and I use it in my projects. It has many features which are described in its website page.
I need to create project in which there are two databases local and remote. Remote database needs to be synchronized daily with local database reflecting changes made in local database.
I am using JAVA. Database is ORACLE. I have JAVA/JPA code that does CRUD operations on local database.
How to synchronize changes to remote database.
I would not do this in Java, but look for native Oracle database synchronization mechanisms/tools. This will
be quicker to implement
be more robust
have faster replication events
be more 'correct'
Please look at some synchronization products. SQL Anywhere from Sybase where I work is one such product. You may be able to get a developer/evaluation copy that you can use to explore your options. I am sure Oracle has something similar too.
The basic idea is to be able to track the changes that have happened in the central database. This is typically done by keeping a timestamp for each row. During the synchronization, the remote database provides the last sync time and the server sends to it all rows that have changed since then. Note that the rows that have been deleted in the central database will need some special handling to ensure they get deleted from the remote database.
A true two-way synchronization is lot more complex. You need to also upload the changes from remote database to central and also some conflict resolution strategies have to be implemented for the cases when the same row has been changed in both the remote and central database in incompatible way in the two.
The general problem is too complex to be explained in a respone here but I hope I have been able to provide some useful pointers.
The problem is that what you are asking can range from moderately difficult (for a simple, not very robust system) to a very complex product that could keep a small team busy for a year depending on requirements.
That's why the other answers said "Find another way" (basically)
If you have to do this for a class assignment or something, it's possible but it probably won't be quick, robust or easy.
You need server software on each side, a way to translate unknown tables to data that can be transferred over the wire (along with enough meta-data to re-create it on the other side) and you'll probably want to track database changes (perhaps with a flag or timestamp) so that you don't have to send each record over every time.
It's a hard enough problem that we can't really help much. If I HAD to do that for a customer, I'd quote him at least a man year of work to get it even moderately reliable.
Good Luck
Oracle has a sophistication replication functionality to synchronise databases. Find out more..
From your comments it appears you're using the Oracle Lite: this supports replication, which is covered in the Lite documentation.
Never worked with it, but http://symmetricds.codehaus.org/ might be of use