Develop Reference

How to get only updated data from mysql

I need to get the only data which is updated, I am getting complete data which is affecting the performance. So whenever a client hit the API I want to send the only changes (updated data) from DB which is related to client.
There is a scenario just like facebook. If user goes offline at 12 o'clock for 1 hour i.e he will be get online at 1 o'clock. Now I need to send the notifications to him that which activities are done within offline time period.
I am using timestamp right now but Is their any other better option ?
I don't want to use timestamp or flag for last session ending time.
Thanks in advance

Try to use Versioning concept there u can get latest record .. or try to get latest record from database of that client.

There must be a column which might be an identification for updates, like last_modified_date or something. Apply the same in where clause and execute the query.

I am not sure but i know in mongo db oplog can be use to monitored changes in database , i guess like that mysql log can be use to do such monitoring if you can access it with your code. may be i am still not sure you can give it a try

Is checksum a good way to see if table has been modified in MySQL?

I'm currently developing an application in Java that connects to a MySQL database using JDBC, and displays records in jTable. The application is going to be run by more than one user at a time and I'm trying to implement a way to see if the table has been modified. EG if user one modifies a column such as stock level, and then user two tries to access the same record tries to change it based on level before user one interacts.
At the moment I'm storing the checksum of the table that's being displayed as a variable and when a user tries to modify a record it will do a check whether the stored checksum is the same as the one generated before the edit.
As I'm new to this I'm not sure if this a correct way to do it or not; as I have no experience in this matter.

Calculating the checksum of an entire table seems like a very heavy-handed solution and definitely something that wouldn't scale in the long term. There are multiple ways of handling this but the core theme is to do as little work as possible to ensure that you can scale as the number of users increase. Imagine implementing the checksum based solution on table with million rows continuously updated by hundreds of users!
One of the solutions (which requires minimal re-work) would be to "check" the stock name against which the value is updated. In the background, you'll fire across a query to the table to see if the data for "that particular stock" has been updated after the table was populated. If yes, you can warn the user or mark the updated cell as dirty to indicate that that value has changed. The problem here is that the query won't be fired off till the user tries to save the updated value. Or you could poll the database to avoid that but again hardly an efficient solution.
As a more robust solution, I would recommend using a database which implements native "push notifications" to all the connected clients. Redis is a NoSQL database which comes to mind for this.
Another tried and tested technique would be to forgo direct database connection and use a middleware layer like a messaging queue (e.g. RabbitMQ). Message queues enable design of systems which communicate using message. So for e.g. every update the stock value in the JTable would be sent across as a message to an "update database queue". Once the update is done, a message would be sent across to a "update notification queue" to which all clients would be connected. This will enable all of them to know that the value of a given stock has been updated and act accordingly. The advantage to this solution is that you get to keep your existing stack (Java, MySQL) and can implement notifications without polling the DB and killing it.

Checksum is a way to see if data has changed.
Anyway I would suggest you store a column "last_update_date", this column is supposed to be always updated at every update of the record.
So you juste have to store this date (precision date time) and do the check with that.
You can also add a column version number : a simple counter incremented by 1 at each update.
Note:
You can add a trigger on update for updating last_update_date, it should be 100% reliable, maybe you don't need a trigger if you control all updates.

When using in network communication:
A checksum is a count of the number of bits in a transmission unit
that is included with the unit so that the receiver can check to see
whether the same number of bits arrived. If the counts match, it's
assumed that the complete transmission was received.
So it can be translated to check 2 objects are different, your approach is correct.

Hibernate bulk operations migrate databases

I wrote a small executable jar using Spring & Spring Data JPA to migrate data from a database, converting objects from original database (throught several tables) to valid objects for the new database and then insert the new objects in new database.
Problem is : I process a large amount of data (200 000) and doing my insert one by one is really time consuming (1hr, all the time is spent for the INSERT operations which happen after validating/transforming incoming data, it is not spent for the retrieval from original database nor validation/conversion).
I already had suggestions about it :
[Edit because i didn't explain it well] As I am doing a
extract-validate-transform-insert, do my insert (which are valid
because they are verified first) X objects by X objects (instead of
inserting it one by one). That is the suggestion from the frist
answer : tried it but that not so efficient, stil time consuming.
Instead of saving directly in database, save the insert into a .sql file and then import the file directly in database. But how to translate myDao.save() to a final SQL output and then write it to a file.
Use Talend : know as probably the best way, but too long to re-do everything. I'd like to find a way using java and refactor my jar.
Other ideas ?
Note : one important point is that if one valisation fails I want to continue to process other data, I only log an error.
Thanks

You should pause and think for a minute: what could cause an error when inserting your data into the database? Short of "your database is hosed", there are two posibilities:
There is a bug in your code;
The data coming in is bad.
If you have a bug in your code, you would be better of if all your data load is reverted. You will get another chance to transfer data after you fix your code.
If the data coming in is bad, or is suspected bad, you should add a step for validating your data. So, your process workflow might look like this: Extract --> Validate --> Transform --> Load. If the incoming data is invalid, write it into the log or load into a separate table for erroneous data.
You should keep all your process run in the same transaction, using the same Hibernate session. Keeping all 200K reords in memory would be pushing it. I suggest using batching (see http://docs.jboss.org/hibernate/orm/3.3/reference/en-US/html/batch.html). In two words, after a predetermined number of records, say 1000, flush and clear your Hibernate session.

function call problem

i am going to call a function which will retrieve some data value from database. but before that i am sending those data. i am just checking whether those data properly inserted or not with this function call. but inserting data taking some time to insert into the database but my function calling starts before it actually inserts the data into the database. As because of that it is finding that no data is inserted in the database. Can any one tell how do i resolve this issue. How to synchronize this. whether i should get the proper result after the proper insertion into the database. i cant use here runnable interface or thread class. the think that i have to do is to call the data access function after certain time so that data gets enough time to get inserted into the database. please help me out.

Don't know what language you are using, but maybe the function has a parameter which causes it to wait until the query finishes before returning? Something mentioning the word "synchronous?"

Use the Database Driver
I'm not familiar with JDBC so I'm not sure which tools are/aren't available to you, but it seems like you're doing more work than you need to.
Typically the database driver will inform you whether the query was executed successfully, so you should not need to have your application query the data afterward to verify that the data is there. Instead, ask the driver for errors to see whether there was a problem with the query.
If you're inserting a large about of data and your database supports it, you may want to use a transaction to perform your insert. This will pass all of the data into the db, attempt the insert, and warn you of any problems.
If there are problems with the transaction, you can roll back, and the database state will be the same as when you started (obviously you will need to handle the errors to save your data). If there are no problems, you can finish committing the transaction, and rest assured that the database state matches the application state.
Alternatives
If for some reason the above methods won't work, you can try to resolve the race condition using an event pattern. In simple terms, you want to raise an event when the data is done inserting to alert the validator that it can start reading data. The validator will listen for that event and trigger when it hears it.

Java MySQL Programming

I am accessing a MySQL table that has over 1 million or more Records. I am using My SQL query browser which is unable to grab all the records and it break the connection in the middle.
Now I have to write a Java Program which access that particular table without being broken in the middle as this table will be modified and accessed frequently.
Can you experts suggest me how should do I go over this problem
either I create an Index on the table and how do I create index

There are different reasons why a MySQL connection might break during a query. Can you give the exact error message you receive?
A simplified explanation on how to add an index to the table for a simple query
Look at the field(s) in the WHERE
clause of the query
Add an index on the field(s) using
ALTER TABLE ADD INDEX
Use EXPLAIN on the query and check
if the query is actually using the
index.
IF you want more specific help, Post the SHOW CREATE TABLE and the EXPLAIN of your query.

MySQL query browser limits the number of records to be displayed for performance reasons, because it is an interactive program and nobody like to wait for half an hour before the program crashes with an out-of-memory error. You can change these limits in the settings.
Your Java program will face similar problems.
When using large datasets it is important to plan how you are going to access that dataset and create the necessary indexes.
It would be useful to edit the question to show the structure of the data. Generqlly it looks like this :
CREATE INDEX idx_customer_name ON customer (name);
Here are more details
If you just want to dump the data to work on the data using Excel you can try this on the commandline
mysqldump -u [username] -p -t -T/path/to/directory [database] --fields-enclosed-by=\" --fields-terminated-by=,
In my experience this is a very painful exercise as Excel really is not made to deal with this amount of rows, and the dump format usually is slightly, but infuriatingly incompatible.
Your best bet is to invest an hour of your time to go through a SQL tutorial like sql fundamentals and play with MySQL query browser to get a feel of what you can do with SQL. I guarantee your investment paid itself back by tomorrow.

I am not very well used to MySQL programming, but generally indexes are used to arrange the values of one or more columns in a database table in specific order.
SYNTAX
CREATE INDEX IndexName ON tableName (column);
Just go through this tutorial for more information,
http://dev.mysql.com/doc/refman/5.0/en/create-index.html