Is there a way to log all database tables (and maybe other objects) that the application running within Tomcat reads from/write to? All of the database access is performed either via Hibernate (wired through Spring) or Sql2o.
We are in process of migrating a legacy database, and it would be nice to automatically separate tables that are actually used from the ones that are pure legacy (and we've got lots of those!).
Thanks!
You can create new Aspect since you are already using Spring, and monitor all your read/write services, so every time any of it is called, you can log it.
It should be quite easy, define new aspect (f.e. MyAspect), define before/after or round actions (which will log your table names somewhere), and just put the annotation on your services (#MyAspect).
Another way is to create triggers on your tables (not sure if that is an option), but I'd preffer an aspect... It's much cleaner approach.
Related
Background : I don't want to hardcode properties in a java constant file, coz every time I want to change to one of the property_value, I have to build and deploy the entire code again. I don't even want to keep them in application.yaml/application.properties file, coz my properties are huge in numbers(100s). So, I have decided to maintain properties in a properties table in oracle DB.
I can think of two approaches :
Read the property value as and when required by firing a sql query.
Load all/part of the properties at the time of starting application and have global point of access by caching them.
As I need few of the properties in the beginning itself, I want to go with approach number 2.
I wanted to go for singleton bean, but this requires me to know all of the keys(property_names) in the beginning itself, and makes the singleton look ugly having 100s of member variables. Here I was planning to fire query by using pre-construct function of spring bean. The main problem here is the bean requires to be changed every time I add/delete properties from the properties table.
Another approach I could think of was to go with a Map<String,String> as both the property_name and property_value columns of my table are of VARCHAR type. But the question is how can I get global point of access to this map ?
Any better approaches much appreciated!
Many thanks in advance
If you deploy your app on web server you can take advantage of it. Many of web server supports changing configuration on the fly (using web console) such as JBoss EAP.
The other idea is to build your own properties lib. Create a function which read the data. I'd prefer using NoSQL database than RDBMS because of lightweight, great performance and scale very well. I wouldn't use caching unless it persisted somewhere and good on availability.
I've heard about Spring Cloud Config to externalize configuration but i never deep dive into it.
Some of us uses Apache Commons Configuration that support properties reload. Take a look at this thread.
I have an application with a huge code base which uses an Oracle database. I want to develop an hibernate app which can interact with incoming and outgoing request from the above said application without any dependencies of database.
Like if I want to change the database to mysql or postgresql it would not have any problem. Is this practical? Can it be done? Asking for help.
As to practicality, very seldom does an app ever change databases. While the idea sounds great it isn't often done and generally the benefits you can get from using built in database features sometimes outweighs the work of keeping it database independent.
As to it being done, it certainly can between SQL databases. To go from SQL to noSQL is a bit more tricky as they are in the process of supporting them in JPA. If interested in that take a look at Hibernate OGM. If you want to truly keep it so you can easily switch databases you need to stick to the JPA standard. See this on generating JPA compliant entities from the database. So long as you use ONLY JPA you can easily switch between the databases that provide a JPA implementation. Then you just include the correct implementation set the dialect and you are switched.
If you have access to change the current application it will probably be easier to just update each of the actions that contain the hard coded queries with your JPA code. If you have unit testing that would make this process much easier as well.
If you want to write something new, but not change the front end, you would need to handle whatever actions your forms on the front end are submitting. Making sure to make them available at the same path and with the same HTTP methods (GET, POST, PUT, etc.), that take the same parameters, and returning the same structure as what your actions due today.
Both approaches would allow you to go action by action replacing them. With writing something new though, replacing them one at a time is a little more difficult if both the new app and old app aren't in the same domain OR if authentication/authorization is involved.
Good luck and best wishes!
I am developing java application based on spring framework.
It
Connects to a MySQL database
Gets data from MySQLTable1 in POJOs
Manipulates (update,delete) it in memory
Inserts into a Netezza database table
The above 4 processes are done for each client (A,B,C) every hour.
I am using a spring JDBC template to get the data like this:
SELECT COL1,COL2,COL3 FROM MySQLTable1 WHERE CLIENTID='A' AND COL4='CONDITION'
and read each record into a POJO before I write it to a Netezza table.
There are going to be multiple instance of this application running every hour through a scheduler.
So Client A and Client B can be running concurrently but the SELECT will be unique,
I mean data for:
SELECT COL1,COL2,COL3 FROM MySQLTable1 WHERE CLIENTID='A' AND COL4='CONDITION'
will be different from
SELECT COL1,COL2,COL3 FROM MySQLTable1 WHERE CLIENTID='B' AND COL4='CONDITION'
But remember all of these are stored in memory as POJOs.
My questions are :
Is there a risk of data contamination?
Is there a need to implement database transaction using spring data transaction manager?
Does my application really need to use something like Spring Batch to deal with this?
I appreciate your thoughts and feedback.
I know this is a perfect scenario for using an ETL tool but that is out of scope.
Is there a risk of data contamination?
It depend on what you are doing with your data but I don't see how you can have data contamination if every instance is independant, you just have to make sure that every instances that run concurrently are not working on the same data (Client ID).
Is there a need to implement database transaction using spring data transaction manager?
You will probably need a transaction for insertion into the Netezza table. You certainly want your data to have a consistent state in the result table. If an error occur in the middle of the process, you'll probably want to rollback everything that was inserted before it failed. Regarding the transaction manager, you don't especially need the Spring transaction manager, but since you are using Spring it might be a good option.
Does my application really need to use something like Spring Batch to deal with this?
Does it really need it, probably not, but Spring Batch was made for those kind of application, so it might help you to structure your application (Spring Batch provides reusable functions that are essential in processing large volumes of records, including logging/tracing, transaction management, job processing statistics, job restart, skip, and resource management). Everything can be made without the framework and it might be overkill to use it if you have a really small application. But at the end, if you need those features, you'll probably want to use it...
Spring Batch is ETL, so using it would be a good fit for this use case and also a good alternative to a commercial ETL tool.
Is there a risk of data contamination? Client A and B read separate data, so they can never interfere with each other by reading or writing the same data by accident. The risk would be if two clients with the same ID are created, but that is not the case.
Is there a need to implement database transaction using spring data transaction manager?
There is no mandatory need to do that, although programatic transaction management has many pitfalls and is best avoided. Spring Batch would manage transactions for you, as well as other aspects such as paging.
Does my application really need to use something like Spring Batch to deal with this? There is no mandatory need to do this, although it would help a lot, especially in the paging aspect. How will you handle queries that return thousands of rows? Without a framework this needs to be handled manually.
Im currently working my way towards JPA 2.0 and I start of liking how easy it is to maintain persistent data.
What I'm currently trying to accomplish is using JPA in a basic desktop application. The application should allow me to open embedded databases which are on my file system. I chose H2 databases for now, but I can really live switching to JavaDB or anything else.
What Im trying to accomplish is, that one can open the database file without previously define a persistence-unit in the persistence.xml file.
I can easily define a unit and persist objects, but it needs to be configured first.
I want to write some sort of database browser which allows opening without preconfiguration and recompiling.
http://www.objectdb.com/java/jpa/start/connection
I saw that ObjectDB allows access for this type of PersistenceFactory creation, but I was not able to transfer this example to other databases.
Am I totally wrong with the way I approach this probblem? Is JPA not designed with on-the-fly database access?
Thank you for your help,
Johannes
Not part of the JPA standard. Some implementations may offer their own API to do it. For example with DataNucleus if you go to this page http://www.datanucleus.org/products/accessplatform_3_0/jpa/persistence_unit.html at the end you can create dynamic persistence-units (and hence EMFs), and that implementation obviously allows persistence to the widest range of datastores you'll get anywhere
You can pass a Map of properties to createEntityManagerFactory() call that defines the database connection info, etc. The property names are the same as in the persistence.xml. I assume most JPA providers support this, EclipseLink does.
You will still need to define the set of classes for the database and map them.
If you do not have any classes either, than you could look into EclipseLink's dynamic support,
http://wiki.eclipse.org/EclipseLink/Examples/JPA/Dynamic
If you want to make a database browser accessing different databases, you can't use a PU/Entity Manager (imo).
You'll need a dialogue asking a user for the IP/Port of the database, the username/password, the database name to access, and the type of database.
Then all you need to do is create a socket, send requests over the socket, and parse the response into a view.
Since both the request and the response are database specific, the user has to select the proper database driver.
Our design has one jvm that is a jboss/webapp (read/write) that is used to maintain the data via hibernate (using jpa) to the db. The model has 10-15 persistent classes with 3-5 levels of depth in the relationships.
We then have a separate jvm that is the server using this data. As it is running continuously we just have one long db session (read only).
There is currently no intra-jvm cache involved - so we manually signal one jvm from the other.
Now when the webapp changes some data, it signals the server to reload the changed data. What we have found is that we need to tell hibernate to purge the data and then reload it. Just doing a fetch/merge with the db does not do the job - mainly in respect of the objects several layers down the hierarchy.
Any thoughts on whether there is anything fundamentally wrong with this design or if anyone is doing this and has had better luck with working with hibernate on the reloads.
Thanks,
Chris
A Hibernate session loads all data it reads from the DB into what they call the first-level cache. Once a row is loaded from the DB, any subsequent fetches for a row with the same PK will return the data from this cache. Furthermore, Hibernate gaurentees reference equality for objects with the same PK in a single Session.
From what I understand, your read-only server application never closes its Hibernate session. So when the DB gets updated by the read-write application, the Session on read-only server is unaware of the change. Effectively, your read-only application is loading an in-memory copy of the database and using that copy, which gets stale in due course.
The simplest and best course of action I can suggest is to close and open Sessions as needed. This sidesteps the whole problem. Hibernate Sessions are intended to be a window for a short-lived interaction with the DB. I agree that there is a performance gain by not reloading the object-graph again and again; but you need to measure it and convince yourself that it is worth the pains.
Another option is to close and reopen the Session periodically. This ensures that the read-only application works with data not older than a given time interval. But there definitely is a window where the read-only application works with stale data (although the design guarantees that it gets the up-to-date data eventually). This might be permissible in many applications - you need to evaluate your situation.
The third option is to use a second level cache implementation, and use short-lived Sessions. There are various caching packages that work with Hibernate with relative merits and demerits.
Chris, I'm a little confused about your circumstances. If I understand correctly, you have a both a web app (read/write) a standalone application (read-only?) using Hibernate to access a shared database. The changes you make with the web app aren't visible to the standalone app. Is that right?
If so, have you considered using a different second-level cache implementation? I'm wondering if you might be able to use a clustered cache that is shared by both the web application and the standalone application. I believe that SwarmCache, which is integrated with Hibernate, will allow this, but I haven't tried it myself.
In general, though, you should know that the contents of a given cache will never be aware of activity by another application (that's why I suggest having both apps share a cache). Good luck!
From my point of view, you should change your underline Hibernate cache to that one, which supports clustered mode. It could be a JBoss Cache or a Swarm Cache. The first one has a better support of data synchronization (replication and invalidation) and also supports JTA.
Then you will able to configure cache synchronization between webapp and server. Also look at isolation level if you will use JBoss Cache. I believe you should use READ_COMMITTED mode if you want to get new data on a server from the same session.
The most used practice is to have a Container-Managed Entity Manager so that two or more applications in the same container (ie Glassfish, Tomcat, Websphere) can share the same caches.
But if you don't use an Application container, because you use Play! for instance, then I would build some webservices in the primary Application to read/write consistently in the cache.
I think using stale data is an open door for disaster. Just like Singletons become Multitons, read-only applications are often a write sometimes.
Belt and braces :)