Are there any recommendations, best practices or good articles on providing integration hooks ?
Let's say I'm developing a web based ordering system. Eventually I'd like my client to be able to write some code, packaged it into a jar, dump it into the classpath, and it would change the way the software behaves.
For example, if an order comes in, the code
1. may send an email or sms
2. may write some additional data into the database
3. may change data in the database, or decide that the order should not be saved into the database (cancel the data save)
Point 3 is quite dangerous since it interferes too much with data integrity, but if we want integration to be that flexible, is it doable ?
Options so far
1. provide hooks for specific actions, e.g. if this and that occurs, call this method, client will write implementation for that method, this is too rigid though
2. mechanism similar to servlet filters, there is code before the actual action is executed and code after, not quite sure how this could be designed though
We're using Struts2 if that matters.
This integration must be able to detect a "state change", not just the "end state" after the core action executes.
For example if an order changes state from In Progress to Paid, then it will do something, but if it changes from Draft to Paid, it should not do anything.The core action in this case would be loading the order object from the database, changing the state to Paid, and saving it again (or doing an sql update).
Many options, including:
Workflow tool
AOP
Messaging
DB-layer hooks
The easiest (for me at the time) was a message-based approach. I did a sort-of ad-hoc thing using Struts 2 interceptors, but a cleaner approach would use Spring and/or JMS.
As long as the relevant information is contained in the message, it's pretty much completely open-ended. Having a system accessible via services/etc. means the messages can tap back in to the main app in ways you haven't anticipated.
If you want this to work without system restarts, another option would be to implement handlers in a dynamic language (e.g., Groovy). Functionality can be stored in a DB. Using a Spring factory makes this pretty fun and reduces some of the complexity of a message-based approach.
One issue with a synchronous approach, however, is if a handler deadlocks or takes a long time; it can impact that thread at the least, or the system as a whole under some circumstances.
Related
To understand if the spring events fits the task im working on I need to understand how they work, where are they stored?
as I can guess they are stored in a spring application context and disappears if the application crashes, is my guess correct?
Spring events are intended to use when calling methods directly would create too much coupling. If you need to track events for auditing or replay purposes you have to save the events yourself. Based on your comments, there are many ways to achieve this, based on the topology and purpose of the application (list not complete):
Model entities that represent the events and store them in a repository
Incorporate a message broker such as Kafka that support message persistence
Install an in-memory cache such as Hazelcast
Use a cloud message service such as AWS SQS
Lastly, please make sure that you carefully evaluate which options suits your needs best. Options 2 to 4 all introduce heavy complexity and distributed applications can bring sorrow and misery to your life. Go for the simplest option if you can and only resort the other options if absolutely necessary.
Our application uses java / sql server.
We have ETL jobs (around 35 for different upstreams) using sprint batch. Some of the code is in java and some in database. We want to track lifecycle of a job from database. E.g. when a job started, when a particular component got called, when a method / stored procedure got called and how much time that took. The purpose is to do health check which component is taking more time and in case some stored procedure takes lots of time in production we should be able to query database. Moreover, we also want to store intermediate calculations for audit and debug purpose.
This time tracking and intermediate calculations would be stored besides normal application logging.
Current solution we have implemented is normalized tables in database (e.g. Job, Task, status, etc) for which we have stored procedure wrapper and then have java classes as well to call those stored procedures.
We are not redesigning our application, so wanted to check what is the best approach to track such information. AOP? but I believe that usually gets called for before and after what about the intermediate calculations we want to store?
Our current approach is working, but it is cluttering code as method is doing logging & auditing, instead of just concentrating on the main logic.
A free and open-source tool you should consider is Jamon, it is a comprehensive monitoring framework that provides a lots of useful features:
JAMon allows developers to track their applications performance and
behavior using predefined modules. There are modules that
automatically monitor : SQL, HTTP page requests, Spring beans, method
invocations, Log4j, and Exceptions. Other modules are often easy to
build. JAMon keeps track of the following metrics for any of the items
it tracks in the modules: hits, total, average, min, max and
concurrency (average, max, current/active) to name a few.
Now about storing calculation, I would suggest to break your methods in smaller sub-methods and then use AOP or any other tool to capture the returned value and perform whatever operation you want on these data.
In addition, if you need to have more details on the database layer I would recommend log4jdbc, which will give you nice audit and metrics around jdbc calls. For example you'll be able to get the execution time, the in and out parameters of called procedures, parameters provided to any statements.
You can even extends this tool to provide custom behavior (audit only some procedures, do something specific with collected data.
Aspects are a very good way to isolate the timing code in one place.
Stored procedures seem unnecessary to me. A simple SQL INSERT ought to do the trick. It's fine if you're using the stored proc as an interface to hide the schema from users, but I doubt that this table will evolve much.
Logging, timing, and auditing are the "hello world" of aspect oriented programming.
How should I design an application comprised of numerous (but identical) independent processes that need to communicate data to an enterprise application and be monitored and accessible by a web interface?
Here's a more concrete example in Java:
The independent processes are multiple instances of a standalone J2SE application that receives on initialization data about a "user" entity and then starts doing stuff regarding this "user" (this is an infinite process and so any batch sort of design would be wrong here and also similarly, the starting time of these processes is irrelevant)
The enterprise application is a set of J2EE beans and web-services that implement business logic, DB access etc.. and that are (for example) hosted on GlassFish.
The web front is a set of JSPs (perhaps also on GlassFish) that work with the beans.
Now ideally, I want a way for the processes in (1) to be able to invoke methods from the beans in (2), but also for the beans in (2) to be able and update the processes (1) about things.
So these are the required flows of executions, assuming there are 10 independent process of (1) running for 10 different users (consider a "user" something easily identifiable by, say, a number):
Something happens in one of the processes of (1) and they invoke a method from the enterprise application (2) with some data.
One of the real, human, users (which was already identified by the web app) clicks something on a web-page of (3), this invokes a method in (2), and then some "magical" entity (which I have no idea how to name) finds the independent process from (1) that is responsible for this particular user and updates the process with some new data.
My best approach so far is to expose these J2SE apps by JMX and go from there, but I have one thing I don't understand - who or what should be holding a key-pair list of the sort "the process at URI X is responsible for user Y" and then directing the calls accordingly.
BTW, please feel free to give any advice outside of the Java platform (!), as long as it is a platform that can be scaled easily.
EDIT:
Also, is there a way to "host" such independent processes on some app-server? Something that will re-spawn processes if they fail, allow for deployment and monitoring of such processes on remote machines etc.?
There has been some time since I have used Java Message Service in the past so I am afraid I am not up-to-date with the technical details, but from your description it seems like it would suit your case, to handle communication between the adminstration GUI and the client processes.
There are various options (I believe you are interested for asynchronous communication) so you should take a look on the latest developments to examine yourself if it fits your case or not.
Regarding the data size that the server would exchange with the processes I believe this is a different topic and I must say that the answer depends. Would it be better to send all data in the message? Or would the message be just a notification so the client to be notified and then connect to some enterprise bean to check some new state? I would prefer the latter case but this is something you should decide based on your requirements. I wouldn't blindly exclude the first option unless I had some apparent evidence that this wouldn't work.
Regarding the scaling I don't think it can be much worse then the scaling of your rest of your beans. As much the server is concerned they processes are all clients that need to be served.
Please take the above advice with a grain of salt: I don't know specifics of your problem/design. I am speaking more about in a general way.
I hope that helps
I'm developing an MVC spring web app, and I would like to store the actions of my users (what they click on, etc.) in a database for offline analysis. Let's say an action is a tuple (long userId, long actionId, Date timestamp). I'm not specifically interested in the actions of my users, but I take this as an example.
I expect a lot of actions by a lot of (different) users par minutes (seconds). Hence the processing time is crucial.
In my current implementation, I've defined a datasource with a connection pool to store the actions in a database. I call a service from the request method of a controller, and this service calls a DAO which saves the action into the database.
This implementation is not efficient because it waits that the call from the controller and all the way down to the database is done to return the response to the user. Therefore I was thinking of wrapping this "action saving" into a thread, so that the response to the user is faster. The thread does not need to be finished to get the reponse.
I've no experience in these massive, concurrent and time-critical applications. So any feedback/comments would be very helpful.
Now my questions are:
How would you design such system?
would you implement a service and then wrap it into a thread called at every action?
What should I use?
I checked spring Batch, and this JobLauncher, but I'm not sure if it is the right thing for me.
What happen when there are concurrent accesses at the controller, the service, the DAO and the datasource level?
In more general terms, what are the best practices for designing such applications?
Thank you for your help!
Take a singleton object # apps level and update it with every user action.
This singleton object should have a Hashmap as generic, which should get refreshed periodically say after it reached a threshhold level of 10000 counts and save it to DB, as a spring batch.
Also, periodically, refresh it / clean it upto the last no.# of the records everytime it processed. We can also do a re-initialization of the singleton instance , weekly/ monthly. Remember, this might lead to an issue of updating the same in case, your apps is deployed into multiple JVM. So, you need to implement the clone not supported exception in singleton.
Here's what I did for that :
Used aspectJ to mark all the actions of the user I wanted to collect.
Then I sent this to log4j with an asynchronous dbAppender...
This lets you turn it on or off with log4j logging level.
works perfectly.
If you are interested in the actions your users take, you should be able to figure that out from the HTTP requests they send, so you might be better off logging the incoming requests in an Apache webserver that forwards to your application server. Putting a cluster of web servers in front of application servers is a typical practice (they're good for serving static content) and they are usually logging requests anyway. That way the logging will be fast, your application will not have to deal with it, and the biggest work will be writing a script to slurp the logs into a database where you can do analysis.
Typically it is considered bad form to spawn your own threads in a Java EE application.
A better approach would be to write to a local queue via JMS and then have a separate component, e.g., a message driven bean (pretty easy with EJB or Spring) which persists it to the database.
Another approach would be to just write to a log file and then have a process read the log file and write to the database once a day or whenever.
The things to consider are: -
How up-to-date do you need the information to be?
How critical is the information, can you lose some?
How reliable does the order need to be?
All of these will factor into how many threads you have processing your queue/log file, whether you need a persistent JMS queue and whether you should have the processing occur on a remote system to your main container.
Hope this answers your questions.
I've been scratching my head around developing a simple plugin based architecture on top of Spring, for one of my current apps. No matter how much separation one could achieve using patterns like MVC, one always reaches a point where coupling is inevitable.
Thus, I started weighing options. At first I thought that filters are a good one. Every plugin I'd make would be a filter, which then I will simply insert into the filter map. Of course, this will create a bit of overhead when enumerating and checking all the filters, but at least , controllers won't have to care what has happened to the data before it reached them, or what happens afterwards, they will just care to fetch the models (through DAO or whatnot) and return them.
The problem with this is that not all of my app requests are HTTP-based. Some are based on emails, others are internally scheduled (timed), so Filters won't help much, unless I try to adapt every type of incoming request to HTTPRequest, which would be too much.
Another one I thought about was annotation based AOP, where I annotate every method, where the plugin would intercept methods based on certain conventions. My problem with is that first I am not so experienced with AOP in general, and second, simply writing all those conventions already suggests a bit of coupling
By far the option that mostly appeals to my way of thinking is using Spring-based events. Every type of request handler within my app (web controller, email handler, etc) will be a sort of an event dispatcher, which will dispatch Spring events on every major action. On the other hand, plugins will simply listen for when a particular event happens, and do some logic. This will allow me to utilize point #1 as well, as some of those plugins could be filters as well, i.e when they receive a notification that a certain controller action is done, they may just decide to do nothing, and rather wait for when they get called by the filter chain. I see this as a somewhat nice approach. Of course here comes the overhead again, of dispatching events, plus the fact that every involved class will eb coupled with Spring forever, but I see this as a necessary evil.
My main concern regarding Spring events is performance, both in terms of latency, and memory footprint.
I am still not an expert, so a bunch of feedback here would be of tremendous help. Are spring events the best for this type of architecture, or there is another solution that I've missed? I am aware that there might even be some third-party solutions out there already, so I'd be glad if someone could point out one or two tried and proven ones.
Thanks.
The concept of a plugin can be achieved with the Spring bean factory. If you create a common interface you can define multiple beans that implement it and inject them where needed. Or you can use a factorybean to deliver the right plugin for the job.
Your idea of using events is called an 'Event Driven Architecture'. This goes a lot further than just plugins because it not only decouples from the implementation but also offers the possibility to decouple from which instance is used (multiple handlers), which location (multiple machines) and the time at which the request is handled (asynchronous handling). The tradeoff is an increased overall complexity, a reduced component-level complexity and the need for a messaging infrastructure. Often JMS is used, but if you just want a single-node setup both Spring and Mule offer simple in-memory modes as well.
To help you further you should expand a bit on the requirements you are trying to meet and the architectural improvements you want. So far you have mentioned that you want to use plugins and described some possible solutions, but you have not really described what you are trying to achieve.