i have an application that receive Id from Web service and store it into database if received Id for any reason like connection failed could not store in database i want to know what id was.
is there any way to do it or is there any sample code?
i want use java language.
Use triggers. If you use triggers than you can revoke the transactions in sql.
Do you want a way to make sure that you don't lose the ids even if there is failure in the connection?
If so, why don't you save the last Id you receive in your code that way, you can check if there are more IDs after the last one in case of an outage, Or in your code you can make sure that the web service is reachable, and if it isn't store the IDs till the connection is restored.
Related
I am developing a dictionary application and using many external sources to collect the data.
This data is collected from those sources only for the first time, after that i persist it to my db and fetch it from their.
The problem i am facing is, some words like set, cut, put etc have 100's of meanings and many examples as well. It takes around 10 seconds to Persist all this data to mysql. I am using mybatis to persist data. And because of this, the response time is getting screwed up. Without this database persist, i get response in 400-500ms, if i show data directly after fetching from sources.
I am trying to find a way to persist the data in background. I am using MVC pattern so dao layer is separate.
Is it a good idea to use threading in the dao layer as a solution? Or should I use some messaging tool like Kafka to send a message to persist the given word in background? What else can I do?
Note: I prefer MySQL as the db right now, will probably use redis for caching later on.
My global answer on question + further comments:
Do not bulk insert with Mybatis foreach. Instead you shall execute the statement in a java iteration over the list of object to store, using ExecutorType Reuse or Batch(Read the documentation).
For transactions, in main mybatis-config xml, configure the environment:
transactionManager type JDBC to manage the transaction in the code session = sessionFactory.openSession(); session.commit(); session.rollback();
transactionManager type MANAGED to let the container manage.
Furthermore, you can let the web app send the response, while a new thread takes its time to store the data.
I need to get the only data which is updated, I am getting complete data which is affecting the performance. So whenever a client hit the API I want to send the only changes (updated data) from DB which is related to client.
There is a scenario just like facebook. If user goes offline at 12 o'clock for 1 hour i.e he will be get online at 1 o'clock. Now I need to send the notifications to him that which activities are done within offline time period.
I am using timestamp right now but Is their any other better option ?
I don't want to use timestamp or flag for last session ending time.
Thanks in advance
Try to use Versioning concept there u can get latest record .. or try to get latest record from database of that client.
There must be a column which might be an identification for updates, like last_modified_date or something. Apply the same in where clause and execute the query.
I am not sure but i know in mongo db oplog can be use to monitored changes in database , i guess like that mysql log can be use to do such monitoring if you can access it with your code. may be i am still not sure you can give it a try
I am trying to sync multiple databases whose items have GUID for IDs, meaning that one item has the same ID on all databases.
My question is:
If i modify or create on item on 1 database, and want to synchronize this change to the other database should i:
1.) Check if the item is new or just modified, if its new then use the save() function, if its modified then use the update() function
or
2.)Do not check if its new or modified and just use the saveOrUpdate() function?
After seeing your use case in the comments, I think the best approach is to track (on both the client and server) when the last updated/last synced time was. In the event that the last sync time is null, or comes before the last updated time, you know that the data needs to be synced.
Now, on to the heart of your question: how to sync it. The client need not know the state of a server when it sends an object to you. In fact, it shouldn't. Consider the case where the client posts an object, your server receives it and process it, but the connection dies before your client receives the response. This is a very valid scenario and will result in a mis-match of data. As a result, any way that you try to determine whether or not the server has received an object (from the client) is likely to end up in a bad state.
The best solution is really to create an idempotent endpoint on the server (an upsert method, or saveOrUpdate as you referred to it in your question) which is able to determine what to do with the object. The server can query it's database by primary key to determine if it has the object or not. If it does, it can update, if not, it can insert.
Understandably, performance is important as well as the data. But, stick with primary keys in the database and that one additional select query you add should be extremely minimal (sub-10ms). If you really want to squeeze some more performance out, you could always use memcache or redis as a caching layer to determine if you have a certain GUID in your database. This way, you only have to hit memory (not your database) to determine if an object exists or not. The overhead of that would be measured only in the latency between your web server and cache server (since a memory read is incredibly cheap).
tl;dr
Upsert (or saveOrUpdate) is the way to go. Try not to track the state of one machine on another.
i am going to call a function which will retrieve some data value from database. but before that i am sending those data. i am just checking whether those data properly inserted or not with this function call. but inserting data taking some time to insert into the database but my function calling starts before it actually inserts the data into the database. As because of that it is finding that no data is inserted in the database. Can any one tell how do i resolve this issue. How to synchronize this. whether i should get the proper result after the proper insertion into the database. i cant use here runnable interface or thread class. the think that i have to do is to call the data access function after certain time so that data gets enough time to get inserted into the database. please help me out.
Don't know what language you are using, but maybe the function has a parameter which causes it to wait until the query finishes before returning? Something mentioning the word "synchronous?"
Use the Database Driver
I'm not familiar with JDBC so I'm not sure which tools are/aren't available to you, but it seems like you're doing more work than you need to.
Typically the database driver will inform you whether the query was executed successfully, so you should not need to have your application query the data afterward to verify that the data is there. Instead, ask the driver for errors to see whether there was a problem with the query.
If you're inserting a large about of data and your database supports it, you may want to use a transaction to perform your insert. This will pass all of the data into the db, attempt the insert, and warn you of any problems.
If there are problems with the transaction, you can roll back, and the database state will be the same as when you started (obviously you will need to handle the errors to save your data). If there are no problems, you can finish committing the transaction, and rest assured that the database state matches the application state.
Alternatives
If for some reason the above methods won't work, you can try to resolve the race condition using an event pattern. In simple terms, you want to raise an event when the data is done inserting to alert the validator that it can start reading data. The validator will listen for that event and trigger when it hears it.
There are two different processes developed in Java running independently,
If any of the process modifyies the table, can i get any intimation? As the table is modified. My objective is i want a object always in sync with a table in database, if any modification happens on table i want to modify the object.
If table is modified can i get any intimation regarding this ? Do Database provide any facility like this?
We use SQL Server and have certain triggers that fire when a table is modified and call an external binary. The binary we call sends a Tib rendezvous message to notify other applications that the table has been updated.
However, I'm not a huge fan of this solution - Much better to control writing to your table through one "custodian" process and have other applications delegate to that. To enforce this you could change permissions on your table so that only your custodian process can write to the database.
The other advantage of this approach is being able to provide a caching layer within your custodian process to cater for common access patterns. Granted that a DBMS performs caching anyway, but by offering it at the application layer you will have more control / visibility over it.
No, database doesn't provide these services. You have to query it periodically to check for modification. Or use some JMS solution to send notifications from one app to another.
You could add a timestamp column (last_modified) to the tables and check it periodically for updates or sequence numbers (which are incremented on updates similiar in concept to optimistic locking).
You could use jboss cache which provides update mechanisms.
One way, you can do this is: Just enclose your database statement in a method which should return 'true' when successfully accomplished. Maintain the scope of the flag in your code so that whenever you want to check whether the table has been modified or not. Why not you try like this???
If you're willing to take the hack approach, and your database stores tables as files (eg, mySQL), you could always have something that can check the modification time of the files on disk, and look to see if it's changed.
Of course, databases like Oracle where tables are assigned to tablespaces, and tablespaces are what have storage on disk it won't work.
(yes, I know this is a bad approach, that's why I said it's a hack -- but we don't know all of the requirements, and if he needs something quick, without re-writing the whole application, this would technically work for some databases)