Scan a database table periodically for new rows - java

I have a Java application that reads rows from table A and writes those rows to table B.
Now, my other requirement is to check table A periodically (every x minutes) for newly inserted rows and move them to table B. If it helps, table A has created_at and updated_at TIMESTAMP fields.
Is there any way I can check for newly inserted rows in a certain interval, based on the TIMESTAMP value, add those new rows to a List and re-use the Java method that I already have, to write to table B?
I'm a Java/MySQL noob and would highly appreciate any advice/suggestion that'd help me get started. I'm using MariaDB database.

Method 1: One way you can do this by adding an additional column indicating status as either active or processed. Once the row is picked from Table A and sent to Table B you can update the status of the column as Processed. This can be done using simple update SQL query. You can keep checking the table A after every X min for any record with status as open with a simple select query like select * from order_book where status = ' OPEN'.
Method 2: You can use a trigger that as soon as the record is inserted in Table A send the record to Table B. Benefit of this method is that it doesn't required you to keep a Connection opened to read the table. This will be done by the DBMS directly
CREATE TRIGGER trigger_UpdateTableB ON TableA
FOR INSERT AS
BEGIN
INSERT INTO
TableB
(
primary_column,
column_tableA
)
SELECT
primary_column,
column_tableB,
FROM
INSERTED
END
Method 3: You can also achieve a similar thing easily using Apache Camel.
<route>
<!--To set up a route that generates an event every 60 seconds -->
<from uri="timer://foo?fixedRate=true&period=60000" />
<to uri="sqlComponent:{{sql.selectTableA}}" />
<to uri="sqlComponent:{{sql.updateStatus}}" />
<to uri="sqlComponent:{{sql.InsertTableB}}" />
</route>
Method 4: You can also create a Scheduled Job in java using ScheduledExecutorService. You can configure the scheduled job to run maybe once in an hour (according to the use case) and pick up new records from table A and insert them into table B.
Appendix -
https://mariadb.com/kb/en/library/trigger-overview/ To read more about triggers in MariaDB
http://camel.apache.org/getting-started.html To know more about Apache Camel
https://mkyong.com/java/java-scheduledexecutorservice-examples/

Related

create with params don't insert new Rows to the database table

im trying to use the createwithparm method programmatically in adf
to insert new record in the database but it doesnt works
i have db table with 2 generated values with before insert triggers
and i will pass 2 values
and this is my code
OperationBinding operation = ADFUtils.findOperation("CreateWithParams");
Object result = operation.execute();
and from the edit action binding I've referenced the 2 values i want to pass
{pageFlowScope.userBean.investorNumber}
{pageFlowScope.userBean.tempCode}
but nothing is inserted in the database
and there is nothing in the log
Given that you said "nothing is inserted into the database", I have to ask: Do you understand how ADF BC(EO, VO, AM) works? When you submit a page, for example with createwithparam, it updates the EO and VOs in the ADF BC middle tier model, in memory. Nothing is written to the database. You must issue a COMMIT through the enclosing Application Module to get the data written to the db.
This might help.

Update Query using Temporary Table is taking a lot of time in Postgres

I have a Main Table named REF_SERVICE_OFFERING where-in I already have 3 Million+ data. Now, I wanted to update the 3M records in Java based on some specific condition.
We decided to create a temporary table (where-in the records within will be used to Update the Main Table) and using the below query to Update the Main Table. The temporary table will hold more than 200k records :
UPDATE REF_SERVICE_OFFERING SET
PART_PRICE_BILL_TYPE= TEMP.PART_PRICE_BILL_TYPE,
part_price_unit_type=TEMP.part_price_unit_type,
part_price_allowed_units=TEMP.part_price_allowed_units,
part_price_discount=TEMP.part_price_discount,
part_price_source_id=TEMP.part_price_source_id
FROM REF_SERVICE_OFFERING RSO JOIN ref_offer_temp1 TEMP
ON TEMP.RECORD_NUM = RSO.RECORD_NUM
AND TEMP.SO_NAME = RSO.SO_NAME
AND TEMP.SERVICE_CASE_TYPE = RSO.SERVICE_CASE_TYPE
AND TEMP.WORK_ORDER_TYPE = RSO.WORK_ORDER_TYPE
WHERE (RSO.PART_PRICE_BILL_TYPE IS NOT NULL OR TRIM(RSO.PART_PRICE_BILL_TYPE) NOT LIKE '')
AND (RSO.PART_PRICE_EXCP_SOURCE_ID IS NOT NULL OR TRIM(RSO.PART_PRICE_EXCP_SOURCE_ID) NOT LIKE '')
Our database is Postgres 9.6. But this update is taking a lot of time and never edning. We also tried dumping only 10k records in the temporary table which will be used to update 4L records.
We tried doing EXPLAIN command and couldnt figure out the reason why.
Any help would be really appreciated.

Comparing 2 tables in different databases and do a delta

I have a standalone Java application with embedded H2 Database(D1) and table T1. I have another mysql database(D2) with table T2 hosted on a server. My Requirement is to pull the full data from D2.T2 and push it into D1.T1 . T1 and T2 both have same table definition. First pull is not a problem. Starting from 2nd pull I only need to pull the rows which got updated in D2.T2 and update that in D1.T2 . How can this be achieved? I have to find the delta from the last time i pulled and then update those rows.
If the structures on both tables are the same you can try the following SQL statement:
INSERT INTO your_table
(your_column, ...)
VALUES
(your_value, ...)
ON DUBLICATE KEY UPDATE
your_column = your_value, ...
Your have to get the entries from your source database and execute the above statement for each one on your target database.

Before insert trigger in mysql giving issue when insert is fired from Java code

DELIMITER $$
CREATE TRIGGER `classdost`.`tr_xyz_media`
BEFORE INSERT ON classdost.xyz_media
FOR EACH ROW
BEGIN
DECLARE t_id INT(20);
IF NEW.id < 5000000 THEN
INSERT INTO xyz_media_temp (insert_date) VALUES(CURDATE());
SELECT MAX(xyz_media_temp.id) INTO t_id FROM xyz_media_temp;
SET New.id = t_id;
END IF;
END$$
DELIMITER ;
I have this trigger in MySQL this is running fine when I fire any Insert query in PHPMyadmin but when same Insert is fired from java code it is not executing. I tried changing the code made all request as Ajax with async: false and then I found out that that the earlier requests are still running in background and my later request is giving exception as no ID is returned from database yet.
What can be done to avoid this issue?
If you say requests are still running in background, then it is possible that your table is locked (MyISAM tables), and you cannot modify it.
See if table is locked using this query -
SHOW OPEN TABLES FROM database_name;
This query will show you open tables, whicj cannot be edited (INSERT, UPDATE or etc.). Then check you Java application, find code that locks tables.

How can I prevent duplicated insert in oracle database with Java?

Let me give the example first. It is a log table.
User A subscribe the service A = OK
User A unsubscribe the service A = OK
User A subscribe the service A again = OK
User A subscribe the service A again = Not OK, because you can't subscribe same service at the same time.
Sometimes the client goes crazy and send 5 subscribe requests at the same time ( 4 tomcat servers behind), if I do nothing in this situation then 5 same records will be inserted.
As you can see, I can't use unique constraint here.
I guess perhaps I can use some single thread block in Oracle, but not sure..
I tried "merge" , but I guess it is used in specific records instead of last record.
begin single thread
select the last record
if the last record is the same then don't insert.
if the last record is not the same then insert.
end single thread
Is it possible and how to achieve ?
Perhaps you need to check for the user id, and service type. if same user trying to subscribe same service before the previous subcribed service is performed, then alert the user.
or maybe you want to limit the user to subscribe in only some given duration, say: user can only subscribe same service in each 1 day
You can update the record if the record already exist, for example:
Make a query to check if the record with particular user and service is exist:
SELECT * FROM table WHERE userid = userid AND serviceid=serviceid
If the query return any result, means its exist. then do update:
UPDATE table SET column1='value', column2='value2' ... WHERE userid = userid AND serviceid = serviceid
else, if no result returned, means the user haven't subscribe the service. then insert record:
INSERT INTO table(column1, column2, ...) values ('value1', 'value2', ...)
I think you could solve this problem with constraint. When user subscribes it inserts a row when it unsubscribes it deletes it. A row must be unique for same user and same service.
If you do not want to delete rows add ACTIVE column to this table and make constraint on USER + SERVICE + ACTIVE.
I do not fully understand your problem, but it seems you need to implement mutual exclusion somewhere. Have you tried with a SELECT ... FOR UPDATE?
http://www.techonthenet.com/oracle/cursors/for_update.php
I tried "MERGE" and subquery to solve this case.
By the way, this problem is only happened when subscribe. First, I get the status(subscribe or unsubscribe) from the last record of a user and the service. If the last status in table is 'subscribe', means this subscribed request might be the duplicated one.
MERGE INTO subscr_log M
USING
(SELECT status
FROM subscr_log
WHERE rid=
(SELECT MAX(rid)
FROM monthly_subscr_log
WHERE SCRID ='123456'
AND service_item='CHANNEL1'
)
) C
ON (C.status ='SUB' ) -- try to see the last record is subscribe or not
WHEN MATCHED THEN
UPDATE SET M.REASON='N/A' WHERE M.STATUS='XXXXXXX' --do impossible sql
WHEN NOT MATCHED THEN
INSERT VALUES (9999,8888,'x','x','x','x','x','x','x','x','x',sysdate,'x','x','x','x');

Categories

Resources