I have a requirement to fetch the records from the Dataverse in which some changes have been done in specif columns values. For example, let's say we have a table named employee in which we have a field called position which can be changed over time from intern, software developer, development lead, etc. If we have 10 records currently and if the position of one of the employees gets changed, I need only that one employee record. I have gone through Retrieve and detect changes to table definitions but I believe it is related to changes in the schema and not related changes in the data. I am using the Spring Boot with Java 11 and to work with Dataverse I am using the Olingo library and also may use the Web APIs if required. Is there a way to detect the changes in the data as described above?
EDIT
To add more details we will have a scheduled job that triggers at X minutes which needs to fetch the employee data for which position has changed from the last fetch time of X minutes. As we can see in the image below, all 3 records are being updated in that X minutes internal and last modified time has been updated for all. I need to fetch the records highlighted in green for which position attribute has changed. For a record with Id 2, I don't need to fetch it as the position is the same.
Solution 1: Custom changes table
If you may and can extend your current Dataverse environment
Create a new table called Employee Change. Add a column of type Lookup named Employee and link it to your Employee table
Modify Main Form and add Employee column to the form
Create a workflow process which would fire on field change. Inside the workflow process you create an Employee Change record and set lookup column value to the changed record
You can now query Employee Change table for changed records. You would need to expand the lookup column to get required columns from Employee table.
Example Web API query:
GET [Organization URI]/api/data/v9.1/employee?$select=createdon, employeeid
&$expand=employeeid($select=employeeid,fullname,column2,column3) HTTP/1.1
Accept: application/json
OData-MaxVersion: 4.0
OData-Version: 4.0
More info on expanding lookup columns can be found here
Solution 2: Auditing
Use built-in auditing feature
Make sure auditing is enabled. Details can be found in docs
Enable auditing on required column in Employee table
Query audit records for changes in Employee table. You have to pay attention only to changes to specific attributes of interest
You will get a list of records changed and then you have to query once again to retrieve columns of the records
Solution 3: Push instead of pull
It might make more sense to push changes from Dataverse to your API instead of constantly querying for changes.
You could use Microsoft Power Automate to create a simple flow which would call your API / platform when change is detected in Dataverse
A good start could be exploring the following Power Automate template: When a record is updated in Microsoft Dataverse, send an email. You could then replace "send email" steps with querying other APIs
Related
I am building an application that does some processing after looking up a database (oracle).
Currently, I have configured the application with Spring Integration and it polls data in a periodic fashion regardless of whether any data is updated or inserted.
The problem here is, I cannot add or use any column to distinguish between old and new records. Also, for no insert or update in table as well, poller polls data from database and feeds the data into message channel.
For that, I want to switch to database change notification and I need to register the query something like
SELECT * FROM EMPLOYEE WHERE STATUS='ACTIVE'
now this active status is true for old and new entries and I want to eliminate the old entries from my list. So that, only after a new insert or an existing update, I want to get data which are added newly or updated recently.
Well, that is really very sad that you can't modify the data model in the database. I'd really suggest to try to insist to change the table for your convenience. For example might really be just one more column LAST_MODIFIED, so could to filter the old records and only poll those which date is very fresh.
There is also possibility in Oracle like trigger, so you can perform some action on INSERT/UPDATE and modify some other table for your interest.
Otherwise you don't have choice unless use one more extra persistence service to track loaded records. For example MetadataStore based on Redis or MongoDB: https://docs.spring.io/spring-integration/docs/4.3.12.RELEASE/reference/html/system-management-chapter.html#metadata-store
I have designed a Web based project.
I am using Mysql database. I will perform all persistence logic in java using hibernate. All the client side actions will be done in javascript.
Here my problem is,
If two users are trying to update same record simultaneously at different places.
Initially User-1 updates by giving full information related to a single object and called save information method.
At the other end User-2 updates same record by giving partial information and called save information method.
If User-1 information is saved first User-2 information will overwrite first given information. Hence some information might loss which user-1 given but he doesn't know some thing is loosed.
Please give some suggestions to overcome this problem.
I recommend you use the Optimistic Lock. Basically this technique is to have a field in the table to tell Hibernate which your version and thus whether an object with smaller version try to overwrite the data in a larger version hibernate will throw an exception. This versioning field is usually a numeric field that hibernate increases with every update or date field. The flow is something like:
1 - The record is inserted into the base. At this point the "version" field is set to zero.
2 - The X user query the record with version 0.
3 - The Y user query the record with version 0.
4 - The Y user updates the registry information. At that moment the hibernate automatically increments the version of record for 1.
5 - The X user updates the information on the version 0 and try to save. At that moment the hibernate finds that the record is already in version 1 that is greater than the version that the user X is using, in that it throws an exception stating the problem and not allowing overwriting the most current information.
To implement this strategy simply create a numeric field in your table and then apply #Version:
#Version
#Column(name = "version")
private Integer version;
What you need to consider, is a locking strategy for your data. Using Hibernate, by default you have no locking (a.k.a. Ostrich locking or "last save wins"). Roughly, the other two options are optimistic locking and pessimistic locking.
Optimistic locking means that you do not prevent users editing data concurrently, but you will inform a user if his edit failed because the data was saved from elsewhere after it was loaded from the DB.
Pessimistic locking means that you prevent multiple users for editing the data concurrently. This is a bit more complicated form of locking and is usually neither practical nor required.
More info on implementing a locking strategy can be found from Hibernate documentation. Which strategy you should choose depends a lot on your application and whether many users are expected to frequently edit the same information.
Before user 2 updates the DB, you can check if the information in the DB (e.g. the row) is the same as it was when user 2 reached the update/edit page. e.g. you could do a SELECT on the row when the user reaches the page and again after the user has made the changes (i.e. before the row is updated) and compare these before you update the DB.
If the row is the same, there was no changes. If the row is different, someone else had edited it.
I have a web project that uses a database to store data that is used to generate tasks that would be processed for remote machines to alter that records and store new data. My problem here is that I have to store all that changes on each table but I don't need all these information. For example, a table A could have 5 fields but I only need 2 for historical purposes. Another table B could have 3 and I would have to add another one (date for example). Also, I don't need changes during daily task generation, only the most recent one.
Which is the best way to maintain a change history? Someone told me that a good idea is having two tables, the A (B) table and another one called A_history (B_history) with the needed fields. This is actually what I'm doing, using triggers to insert into history tables but I don't feel comfortable with this approach. My project uses Spring (Spring-data, Hibernate and JPA) and if I change the DB (currently MySQL) I'd have to migrate triggers. Is there a good way to manage history records? Tables could be generated with Hibernate/JPA annotations.
If I maintain the two tables approach, can I add a method to the repository to fetch rows from current table and history table at once?
For this pourpose there is a special Hibernate Envers project. See official documentation here. Just configure it, annotate necessary properties with #Audited annotation and that's all. No need for DB triggers.
One pitfall: if you want to have a record for each delete operation then you need to use Session.delete(entity) way instead of HQL "delete ...".
EDIT. Also take a look into native auditing support of spring data jpa.
I am not a database expert. What I have seen them do boils down to a few ways of approach.
1) They add a trigger to the transactional table that copies inserts and updates to a history table but not deletes. This means any queries that need to include history can be done from the history table since all the current info is there too.
a) They can tag each entry in the history table with time and date and
keep track of all the states of the original records.
b) They can only
keep track of the current state of the original record and then it
settles when the original is deleted.
2) They have a periodic task that goes around and copies data marked as deletable into the history table. It then deletes the data from the transactional table. Any queries in the transactional table have to make sure to ignore the deletable rows. Any queries that need history have to search both tables and merge the results.
3) If the volume of data isn't too large, they just leave everything in one table and mark some entries as historical. Queries have to ignore historical rows. Queries that include history are easy. This may slow down database access as the table grows to include many unused rows but that can sometimes be ameliorated by clever use of indexes.
I have a facebook 'like' application - a virtual white board for multiple 'teams' who share a 'wall' common to that project. There are about 9-12 entities for which I capture the data. I'm trying to have the user's homepage display the update of activities that have happened since the past login - like how facebook posts notifications:
"[USER] has done [some activity] on [some entity] - 20 minutes ago"
where [...] are clickable links and the activities are primarily (rather only) CRUD.
I'll have to persist these updates. I'm using MySQL as the backend DB and thought of having an update table per project that could store the activities. But it seems there needs to be one trigger per table and that would just be redundant. More so It's difficult to nail down the tabular schema for that update table since there are many different entities.
The constraint is to use MySQL but I'm open to other options of "how" to achieve this functionality.
Any ideas?
PS: Using jQuery + REST + Restlet + Glassfish + MySQL + Java
It doesn't have to be handled at the database level. You can have a transaction logging service that you call in each operation. Each transaction gets a (unique, sequential) key.
Store the key of the last item the person saw, and show any updates where the key is higher, the update the last key seen.
A periodic routine can go through the user accounts and see what is the lowest seen transaction log key across all users (i.e. what is the newest log entry that all users have already seen) and delete/archive any entries with a key <= that one.
I have an existing application that I am working w/ and the customer has defined the table structure they would like for an audit log. It has the following columns:
storeNo
timeChanged
user
tableChanged
fieldChanged
BeforeValue
AfterValue
Usually I just have simple audit columns on each table that provide a userChanged, and timeChanged value. The application that will be writing to these tables is a java application, and the calls are made via jdbc, on an oracle database. The question I have is what is the best way to get the before/after values. I hate to compare objects to see what changes were made to populate this table, this is not going to be efficient. If several columns change in one update, then this new table will have several entries. Or is there a way to do this in oracle? What have others done in the past to track not only changes but changed values?
This traditionally what oracle triggers are for. Each insert or update triggers a stored procedure which has access to the "before and after" data, which you can do with as you please, such as logging the old values to an audit table. It's transparent to the application.
http://asktom.oracle.com/pls/asktom/f?p=100:11:0::::P11_QUESTION_ID:59412348055
If you use Oracle 10g or later, you can use built in auditing functions. You paid good money for the license, might as well use it.
Read more at http://www.oracle.com/technology/pub/articles/10gdba/week10_10gdba.html
"the customer has defined the table structure they would like for an audit log"
Dread words.
Here is how you would implement such a thing:
create or replace trigger emp_bur before insert on emp for each row
begin
if :new.ename = :old.ename then
insert_audit_record('EMP', 'ENAME', :old.ename, :new.ename);
end if;
if :new.sal = :old.sal then
insert_audit_record('EMP', 'SAL', :old.sal, :new.sal);
end if;
if :new.deptno = :old.deptno then
insert_audit_record('EMP', 'DEPTNO', :old.deptno, :new.deptno);
end if;
end;
/
As you can see, it involves a lot of repetition, but that is easy enough to handle, with a code generator built over the data dictionary. But there are more serious problems with this approach.
It has a sizeable overhead: an
single update which touches ten
field will generate ten insert
statements.
The BeforeValue and AfterValue
columns become problematic when we
have to handle different datatypes -
even dates and timestamps become
interesting, let alone CLOBs.
It is hard to reconstruct the state
of a record at a point in time. We
need to start with the earliest
version of the record and apply the
subsequent changes incrementally.
It is not immediately obvious how
this approach would handle INSERT
and DELETE statements.
Now, none of those objections are a problem if the customer's underlying requirement is to monitor changes to a handful of sensitive columns: EMPLOYEES.SALARY, CREDIT_CARDS.LIMIT, etc. But if the requirement is to monitor changes to every table, a "whole record" approach is better: just insert a single audit record for each row affected by the DML.
I'll ditto on triggers.
If you have to do it at the application level, I don't see how it would be possible without going through these steps:
start a transaction
SELECT FOR UPDATE of the record to be changed
for each field to be changed, pick up the old value from the record and the new value from the program logic
for each field to be changed, write an audit record
update the record
end the transaction
If there's a lot of this, I think I would be creating an update-record function to do the compares, either at a generic level or a separate function for each table.