i use the latest Acitiviti 5.22.0 engine (to be more concrete i use Alfresco Process Services 1.6.3) and i have implemented a Spring bean that gets executed every 10 minutes to generate a JSON representation of all my processes (process name, startDate, endDate, current taskName(s) and assignee(s)), to send them to an audit server. The problem is, that i only need to send all changed processes since the last run.
I do not want to send the JSON as soon as a process changes but to do a batch update of my audit system every 10 minutes.
To accomplish this, i've tried different approaches. My latest one:
Create a event listener bean that listens to all PROCESS_STARTED, PROCESS_COMPLETED, PROCESS_CANCELLED, TASK_COMPLETED, ...
Every time the event is triggered, store a process variable "_dirty" and set it to true
Every 10 minutes (wenn my JSON-bean is executed) query for all processes with the "_dirty" variable set to true
After sending the JSON to the audit system, set all "_dirty" process variables to false.
The problem with this approach: I am not able to update the "_dirty" variable after a process is ended. At least i don't know how.
My second approach would be to to store the processInstanceId on every event into a "global" property, but i don't know how to store this "global" property into database in case the server restarts. Is there a way to persist a property or an Entity into DB without creating an extra table, DAO, etc.?
Any ideas on how to solve this task? All tips are very much appreciated!
AFAIK, There's no such option
But you look at this. and see if it can be helpful in your case.
https://www.activiti.org/userguide/#_database_tables
As Linus suggested: This is not possible, so I needed some completely different approach.
I am creating an Ad-Hoc task now and store my properties as a local task variable. The Ad-Hoc task is owned by a system account and not assigned to anybody. This way I can make sure, no one of my real users tries to "complete" the task. Also I've written some code to generate the task if needed, so in case i want to clean it, it is created automatically the next time i want to store data.
Creating an Ad-Hoc task is quite easy by using org.activiti.engine.TaskService autowiring into my class.
Task task = taskService.newTask();
task.setDelegationState(DelegationState.PENDING);
task.setName("Some name goes here");
task.setTenantId("your tenant id (if any)");
task.setOwner("your system accounts ID");
task.setCategory("i use a special category to later query for the task");
taskService.saveTask(task);
After saving the task to the database, I can use the taskService to store and retrieve variables like this:
taskService.setVariableLocal(task.getId(), "variableKey", "variableValue");
Or query for the task like this:
Task task = taskService.createTaskQuery().taskDelegationState(DelegationState.PENDING).taskCategory("your special category").singleResult();
Not a very nice solution (I recommend having the task cached in a bean or something, so you don't need to query it all the time or even cache its values or something), but it works.
Related
I would like to retrieve data from my in-memory H2 database via rest endpoint using Spring and Java8. I have 2 endpoints: one to retrieve data, and second one to add data to database.
How can I achieve something like it is described below in easiest way? I am not sure what solution can be better I thought about JMS Queue or CompletableFuture (if it is possible). It should work for few users, they will call to retrieve data saved under their id number.
Scenario:
User calls rest-endpoint to retrieve data.
If data is present in database then it is retrieved and returned to user.
If data is not present in database then connection is hold for 60 seconds and if during that time something will appear in database (added via endpoint to add new data) then data will be returned.
If data is not present in database and new data won’t appear in 60 seconds then endpoint returns no content.
There were multiple ways of doing that and if requirements is clear then i suggest below two approaches.
Approach 1:
Find and retrieve if data available without waiting.
If data not available set resource id and retrieveTime in header and respond to consumer.
Based resource id you can be ready with data if available.
In this way you can sure about your endpoint service time always consistent and ideally it shouldn't be more than 3 seconds.
Approach 2.
if data not available then out sleep in 60 seconds (not in database connection scope) and then again try with same thread.
Don't need any queue or ansyc process here.
Here your loosing resources and service time would take more.
Apart from other approaches, if your systems using eventing then use eventing approach when there is record persistent then send event to consumer (all database has the feature to send event to source system today).
I have a scenario where I check for a specific value in the Database every 10 seconds or so. And, if the value is YES, then I execute a bunch of shell scripts from a Java application.
Now, the value in database is only updated to YES once in a while depending on the user submitting a job on a web page. Therefore, running a while loop to check for this value in database seems to be a very bad design and I would like to implement a much cleaner approach using listeners (Observer design pattern).
How would such an implementation look like? Any examples I can follow to do this?
Yes there is much better job. So there is something called binlog reader in mysql. Thats how master and slave sync is done in mysql cluster database.
So either you write your own logic over https://github.com/shyiko/mysql-binlog-connector-java which gets all the chane event on table
or use https://github.com/zendesk/maxwell to read events from particular table and whenver any change in value is there check if it matches your condition and excute the script or java application on basis of that instead of running it as a cron.
The general idea is to use DB triggers, register DB listener from Java side and be notified from DB side when some event has happened.
Pls review proposed solutions
How to implement a db listener in Java
I'm currently developing an application in Java that connects to a MySQL database using JDBC, and displays records in jTable. The application is going to be run by more than one user at a time and I'm trying to implement a way to see if the table has been modified. EG if user one modifies a column such as stock level, and then user two tries to access the same record tries to change it based on level before user one interacts.
At the moment I'm storing the checksum of the table that's being displayed as a variable and when a user tries to modify a record it will do a check whether the stored checksum is the same as the one generated before the edit.
As I'm new to this I'm not sure if this a correct way to do it or not; as I have no experience in this matter.
Calculating the checksum of an entire table seems like a very heavy-handed solution and definitely something that wouldn't scale in the long term. There are multiple ways of handling this but the core theme is to do as little work as possible to ensure that you can scale as the number of users increase. Imagine implementing the checksum based solution on table with million rows continuously updated by hundreds of users!
One of the solutions (which requires minimal re-work) would be to "check" the stock name against which the value is updated. In the background, you'll fire across a query to the table to see if the data for "that particular stock" has been updated after the table was populated. If yes, you can warn the user or mark the updated cell as dirty to indicate that that value has changed. The problem here is that the query won't be fired off till the user tries to save the updated value. Or you could poll the database to avoid that but again hardly an efficient solution.
As a more robust solution, I would recommend using a database which implements native "push notifications" to all the connected clients. Redis is a NoSQL database which comes to mind for this.
Another tried and tested technique would be to forgo direct database connection and use a middleware layer like a messaging queue (e.g. RabbitMQ). Message queues enable design of systems which communicate using message. So for e.g. every update the stock value in the JTable would be sent across as a message to an "update database queue". Once the update is done, a message would be sent across to a "update notification queue" to which all clients would be connected. This will enable all of them to know that the value of a given stock has been updated and act accordingly. The advantage to this solution is that you get to keep your existing stack (Java, MySQL) and can implement notifications without polling the DB and killing it.
Checksum is a way to see if data has changed.
Anyway I would suggest you store a column "last_update_date", this column is supposed to be always updated at every update of the record.
So you juste have to store this date (precision date time) and do the check with that.
You can also add a column version number : a simple counter incremented by 1 at each update.
Note:
You can add a trigger on update for updating last_update_date, it should be 100% reliable, maybe you don't need a trigger if you control all updates.
When using in network communication:
A checksum is a count of the number of bits in a transmission unit
that is included with the unit so that the receiver can check to see
whether the same number of bits arrived. If the counts match, it's
assumed that the complete transmission was received.
So it can be translated to check 2 objects are different, your approach is correct.
Is there an efficient way to create a copy of table structure+data in HBase, in the same cluster? Obviously the destination table would have a different name. What I've found so far:
The CopyTable job, which has been described as a tool for copying data between different HBase clusters. I think it would support intra-cluster operation, but have no knowledge on whether it has been designed to handle that scenario efficiently.
Use the export+import jobs. Doing that sounds like a hack but since I'm new to HBase maybe that might be a real solution?
Some of you might be asking why I'm trying to do this. My scenario is that I have millions of objects I need access to, in a "snapshot" state if you will. There is a batch process that runs daily which updates many of these objects. If any step in that batch process fails, I need to be able to "roll back" to the original state. Not only that, during the batch process I need to be able to serve requests to the original state.
Therefore the current flow is that I duplicate the original table to a working copy, continue to serve requests using the original table while I update the working copy. If the batch process completes successfully I notify all my services to use the new table, otherwise I just discard the new table.
This has worked fine using BDB but I'm in a whole new world of really large data now so I might be taking the wrong approach. If anyone has any suggestions of patterns I should be using instead, they are more than welcome. :-)
All data in HBase has a certain timestamp. You can do reads (Gets and Scans) with a parameter indicating that you want to the latest version of the data as of a given timestamp. One thing you could do would be to is to do your reads to server your requests using this parameter pointing to a time before the batch process begins. Once the batch completes, bump your read timestamp up to the current state.
A couple things to be careful of, if you take this approach:
HBase tables are configured to store the most recent N versions of a given cell. If you overwrite the data in the cell with N newer values, then you will lose the older value during the next compaction. (You can also configure them to with a TTL to expire cells, but that doesn't quite sound like it matches your case).
Similarly, if you delete the data as part of your process, then you won't be able to read it after the next compaction.
So, if you don't issue deletes as part of your batch process, and you don't write more versions of the same data that already exists in your table than you've configured it to save, you can keep serving old requests out of the same table that you're updating. This effectively gives you a snapshot.
There are two different processes developed in Java running independently,
If any of the process modifyies the table, can i get any intimation? As the table is modified. My objective is i want a object always in sync with a table in database, if any modification happens on table i want to modify the object.
If table is modified can i get any intimation regarding this ? Do Database provide any facility like this?
We use SQL Server and have certain triggers that fire when a table is modified and call an external binary. The binary we call sends a Tib rendezvous message to notify other applications that the table has been updated.
However, I'm not a huge fan of this solution - Much better to control writing to your table through one "custodian" process and have other applications delegate to that. To enforce this you could change permissions on your table so that only your custodian process can write to the database.
The other advantage of this approach is being able to provide a caching layer within your custodian process to cater for common access patterns. Granted that a DBMS performs caching anyway, but by offering it at the application layer you will have more control / visibility over it.
No, database doesn't provide these services. You have to query it periodically to check for modification. Or use some JMS solution to send notifications from one app to another.
You could add a timestamp column (last_modified) to the tables and check it periodically for updates or sequence numbers (which are incremented on updates similiar in concept to optimistic locking).
You could use jboss cache which provides update mechanisms.
One way, you can do this is: Just enclose your database statement in a method which should return 'true' when successfully accomplished. Maintain the scope of the flag in your code so that whenever you want to check whether the table has been modified or not. Why not you try like this???
If you're willing to take the hack approach, and your database stores tables as files (eg, mySQL), you could always have something that can check the modification time of the files on disk, and look to see if it's changed.
Of course, databases like Oracle where tables are assigned to tablespaces, and tablespaces are what have storage on disk it won't work.
(yes, I know this is a bad approach, that's why I said it's a hack -- but we don't know all of the requirements, and if he needs something quick, without re-writing the whole application, this would technically work for some databases)