I am writing a Java program that needs to continue checking if the auto increment value of a given database table has changed. Currently, the program does this by querying the database in an infinite loop on a separate thread.
public class StackOverflow implements Runnable {
#Override
public void run()
{
while(true)
{
// Assume that 'currentMessageID' has already been declared as type integer
// and that 'getLatestMessageID()' queries the database.
if(currentMessageID < queryHandler.getLatestMessageID())
{
int latestMessageID = queryHandler.getLatestMessageID();
for(int x = ajaxChat.currentMessageID + 1; x <= latestMessageID; x++)
{
// Do something when the auto increment value is greater than the last
// known auto increment value.
}
}
}
}
}
While this works just fine, it puts a significant strain on the database server since
SELECT `auto_increment` FROM INFORMATION_SCHEMA.TABLES WHERE table_name = 'SOrocks'
is being called over and over. Is there any way that I could watch for the auto increment value to change without hammering the database server with the same query over and over again?
Unluckyly if you can work only from the application side, a thread that polls the db is the only solution which comes to my mind BUT, if you can also change the DB side, you can always make a trigger on the database to call your java method (example with Oracle).
Related
I am working on Spring+Hibernate based web application.
In this application I have to do calculation on 50000 records available into the database.
Current logic :-
Loop through 0 to 50000 (All 50000 records are independent to each other)
Select ith element
Do calculation on ith element (Drop CALCULATION_TEMP table if exist, create new table CALCULATION_TEMP and insert calculation in CALCULATION_TEMP table)
Do some calculation on step 3 table and get the result
Put step 4 result into Results table
Currently these all calculation taking around 38 hours to complete with single thread.
Now we want to run this system by multiple threads.
For testing purpose I have taken 50 records.
Using Single thread it is taking around 30 sec.
Using two threads :-
Half records performing by first thread and rest of the records by second thread.
Now I am using two TEMP tables for both threads. (TEMP1 and TEMP2)
It is taking 225 sec.
Rough Code :-
for (int i = 0; i < recordsSize; i++) {
final int j = i;
String recordId = list.get(i);
// Method call : Code for creating CALCULATION_TEMP table
// CALCULATION_TEMP table will contain dynamic number of column. It is depends on the record data (50 to 70 columns)
// return flag value
boolean flag = xyzMethod(....);
if (flag) {
// All calculation done in this method
// Around 600 - 700 rows will be created into CALCULATION_TEMP table on the basis of calculation logic
Object fileMapColumnData[] = /* Method call */;
// Insert result for one record into RESULT table for unique recordId (this result is calculated in CALCULATION_TEMP table)
insertIntoResultTable(....);
// Drop CALCULATION_TEMP table
} else {
LOGGER.error("Unable to calculate ... because of some wrong data");
loggerDTO.getCustomLogger().severe("Unable to calculate ... because of some wrong data");
}
if (i % 100 == 0) {
calculationDao.flushAndClear();
}
// Thread for showing process completion status in percentage
Thread t = new Thread() {
#Override
public void run() {
getPercentageDone((float) recordsSize, (float) (j + 1));
}
};
t.start();
}
Please suggest, How I can improve the performance.
Creating/Droping temporary tables take a lots of time by 50000 times. Can you do calculations without temporary table ? It can improve peformance.
There is no hard and fast rule for performance improvements. If you are having constraints and business decisions, we need to know the calculations and how the threads are managed, connections are managed and the number of loops involved etc. There are so many things to consider. Start with checking the pain point areas, the time taken for it and improve each method first.
I am developing desktop app in Java 7. I have here a situation. At the method below
private synchronized void decryptMessage
(CopyOnWriteArrayList<Integer> possibleKeys, ArrayList<Integer> cipherDigits)
{
// apply opposite shift algorithm:
ArrayList<Integer> textDigits = shiftCipher(possibleKeys, cipherDigits);
// count CHI squared statistics:
double chi = countCHIstatistics(textDigits);
if(chi < edgeCHI) // if the value of IOC is greater or equal than that
{
System.err.println(chi + " " + possibleKeys + " +");
key = possibleKeys; // store most suitable key
edgeCHI = chi;
}
}
I count the value called 'chi' and based on that if 'chi' is less than 'edgeCHI' value I save the key at instance variable. That method is invoked by some threads, so I enforce synchronization.
When all the threads complete the program continues to execute by passing control to a method which controls the sequence of operations. Then this line has been executed at that method:
System.err.println(edgeCHI+" "+key+" -");
It prints correct value of 'chi', as has been printed the last value of 'chi' at decryptMessage method, but the value of key is different. The 'decryptMessage' method has been invoked by threads which generate key values.
I store the key value as global variable
private volatile CopyOnWriteArrayList<Integer> key = null; // stores the most suitable key for decryption.
Why do I have two different key values? The values itself are not important. The matter is that the value of key printed at the last call at 'decryptMessage' method (when chi < edgeCHI) must match the one printed at the method which controls the flow of operations.
This is how you create threads:
for(int y = 0; y < mostOccuringL.length; y++){// iterate through the five most frequent letters
for(int i = (y + 1); i < mostOccuringL.length; i++ ){//perform letter combinations
int [] combinations = new int[2];
combinations[0] = y;
combinations [1] = i;
new KeyMembers(""+y+":"+i ,combinations, keywords, intKeyIndex, cipherDigits).t.join();
}
}
Within run method you invoke decryptMesssage method in order to identify most feasible decryption key.
I have been trying to figure out what is the prob for two days, but I don't get it.
Suggestions?
Relying on syserr (or sysout) printing to determine an order of execution is dangerous - especially in multi-threaded environments. There is absolutely no guarantuee when the printing actually occurs or if the printed messages are in order. Maybe what you see as "last" printed message of one of the threads wasn't the "last" thread modifying the key field. You cannot say that by looking only at sterr output.
What you could do is use a synchronized setter for the key field, that increases an associated access counter whenever the field is modified and print the new value along with the modification count. This way you can avoid the problems of syserr printing and reliably determine what the last set value was. e.g. :
private long keyModCount = 0;
private synchronized long update(CopyOnWriteArrayList<Integer> possibilities, double dgeChi) {
this.keys = possibilites;
this.edgeChi = edgeChi; // how is edgeChi declared? Also volatile?
this.keyModCount++;
return this.keyModCount;
}
And inside decryptMessage:
if(chi < edgeCHI) // if the value of IOC is greater or equal than that
{
long sequence = update(possibleKeys, chi);
System.err.println("["+ sequence +"]"+ chi + " " + possibleKeys + " +");
}
To provide an answer we would need to see more of the (simplified if necessary) code that controls the thread execution.
Solution has been found. I just changed CopyOnWriteArrayList data type into ArrayList at the point where field variable gets correct key. It works as expected now.
I am trying to validate some text files. In the front end i am using JTextarea, The below method is called on every time the user enter 'Enter' key. If the file is too big,say 5000 lines and if the user enters many times 'Enter' key then, i am getting unexpected results, like even if the line is valid, it shows it as invalid.
Is there any thing to do with sleep, should i have to increase the sleep time or something else has to be done? Any ideas will be helpful
private TreeSet validate(int curLine, TreeSet errorSet) {
int increment = 0;
int nextLine = 0;
if (curLine == lines.length || errorSet.size() != 0) {
return errorSet;
} else {
String line = lines[curLine];
//validation starts. After validation, line is incremented as per the requirements
increment = 1 //As per requirement. Depends on validation results of the line
if (increment > 0) {
try{
Thread.currentThread().sleep(100);
}catch(Exception ex){
System.out.println(ex);
}
nextLine = (curLine + increment);
validate(nextLine, errorSet);
}
}
return errorSet;
}
I wouldnt look at making the sleep time any longer/shorter. Instead, I would consider doing a better time at marshalling the trigger to validate. Is there any reason to allow the input of a validation request while one is in progress? if not, i would look at blocking the call to validate while a current process is still not complete.
If you think that multiple validations should be able to occur in tandem, I would then look to the creation of a thread pool for these actions. Testing my determine how many threads can concurrently run, and therefore determine the size of your threadpool. At this point, system memory may also play an important point, so you may want to look at those statistics while testing as well.
My problem is this: I am trying to process about 1.5 million rows of data in Spring via JDBCTemplate coming from MySQL. With such a large number of rows, I am using the RowCallbackHandler class as suggested here
The code is actually working, but's SLOW... The thing is that no matter what I set the fetch size to, I seem to get approximately 350 records per fetch, with a 2 to 3 second delay between fetches (from observing my logs). I tried commenting out the store command and confirmed that behavior stayed the same, so the problem is not with the writes.
There are 6 columns, only 1 that is a varchar, and that one is only 25 characters long, so I can't see throughput being the issue.
Ideally I'd like to get more like 30000-50000 rows at a time. Is there a way to do that?
Here is my code:
protected void runCallback(String query, Map params, int fetchSize, RowCallbackHandler rch)
throws DatabaseException {
int oldFetchSize = getJdbcTemplate().getFetchSize();
if (fetchSize > 0) {
getJdbcTemplate().setFetchSize(fetchSize);
}
try {
getJdbcTemplate().query(getSql(query), rch);
}
catch (DataAccessException ex) {
logger.error(ExceptionUtils.getStackTrace(ex));
throw new DatabaseException( ex.getMessage() );
}
getJdbcTemplate().setFetchSize(oldFetchSize);
}
and the handler:
public class SaveUserFolderStatesCallback implements RowCallbackHandler {
#Override
public void processRow(ResultSet rs) throws SQLException {
//Save each row sequentially.
//Do NOT call ResultSet.next() !!!!
Calendar asOf = Calendar.getInstance();
log.info("AS OF DATE: " + asOf.getTime());
Long x = (Long) rs.getLong("x");
Long xx = (Long) rs.getLong("xx");
String xxx = (String) rs.getString("xxx");
BigDecimal xxxx = (BigDecimal)rs.getBigDecimal("xxxx");
Double xxxx = (budgetAmountBD == null) ? 0.0 : budgetAmountBD.doubleValue();
BigDecimal xxxxx = (BigDecimal)rs.getBigDecimal("xxxxx");
Double xxxxx = (actualAmountBD == null) ? 0.0 : actualAmountBD.doubleValue();
dbstore(x, xx, xxx, xxxx, xxxxx, asOf);
}
}
And what is your query? Try to create an indexex for fields you are searching/sorting. That will help.
Second strategy: in memory cache implementation. Or using of hibernate plus 2nd level cache.
Both this technics can significantly speed up your query execution.
Few Questions
How long does it takes if you query the DB directly. Another issue could be ASYNC_NETWORK_IO delay between application and DB hosts.
did you check it without using Spring
The answer actually is to do setFetchSize(Integer.MIN_VALUE) while this totally violates the stated contract of Statement.setFetchSize, the mysql java connector uses this value to stream the resultset. This results in tremendous performance improvement.
Another part of the fix is that I also needed to create my own subclass of (Spring) JdbcTemplate that would accomodate the negative fetch size... Actually, I took the code example here, where I first found the idea of setting fetchSize(Integer.MIN_VALUE)
http://javasplitter.blogspot.com/2009/10/pimp-ma-jdbc-resultset.html
Thank you both for your help!
I need to implement global object collecting statistics for web server. I have Statistics singleton, which has method addSample(long sample), which subsequently call updateMax. This has to be obviously thread-safe. I have this method for updating maximum of whole Statistics:
AtomicLong max;
private void updateMax(long sample) {
while (true) {
long curMax = max.get();
if (curMax < sample) {
boolean result = max.compareAndSet(curMax, sample);
if (result) break;
} else {
break;
}
}
}
Is this implementation correct? I am using java.util.concurrent, because I believe it would be faster than simple synchronized. Is there some other / better way to implement this?
As of Java 8, LongAccumulator has been introduced.
It is advised as
This class is usually preferable to AtomicLong when multiple threads
update a common value that is used for purposes such as collecting
statistics, not for fine-grained synchronization control. Under low
update contention, the two classes have similar characteristics. But
under high contention, expected throughput of this class is
significantly higher, at the expense of higher space consumption.
You can use it as follows:
LongAccumulator maxId = new LongAccumulator(Long::max, 0); //replace 0 with desired initial value
maxId.accumulate(newValue); //from each thread
I think it's correct, but I'd probably rewrite it a little for clarity, and definitely add comments:
private void updateMax(long sample) {
while (true) {
long curMax = max.get();
if (curMax >= sample) {
// Current max is higher, so whatever other threads are
// doing, our current sample can't change max.
break;
}
// Try updating the max value, but only if it's equal to the
// one we've just seen. We don't want to overwrite a potentially
// higher value which has been set since our "get" call.
boolean setSuccessful = max.compareAndSet(curMax, sample);
if (setSuccessful) {
// We managed to update the max value; no other threads
// got in there first. We're definitely done.
break;
}
// Another thread updated the max value between our get and
// compareAndSet calls. Our sample can still be higher than the
// new value though - go round and try again.
}
}
EDIT: Usually I'd at least try the synchronized version first, and only go for this sort of lock-free code when I'd found that it was causing a problem.
With Java 8 you can take advantage of functional interfaces and a simple lamda expression to solve this with one line and no looping:
private void updateMax(long sample) {
max.updateAndGet(curMax -> (sample > curMax) ? sample : curMax);
}
The solution uses the updateAndGet(LongUnaryOperator) method. The current value is contained in curMax and using the conditional operator a simple test is performed replacing the current max value with the sample value if the sample value is greater than the current max value.
as if you didn't have your pick of answers, here's mine:
// while the update appears bigger than the atomic, try to update the atomic.
private void max(AtomicDouble atomicDouble, double update) {
double expect = atomicDouble.get();
while (update > expect) {
atomicDouble.weakCompareAndSet(expect, update);
expect = atomicDouble.get();
}
}
it's more or less the same as the accepted answer, but doesn't use break or while(true) which I personally don't like.
EDIT: just discovered DoubleAccumulator in java 8. the documentation even says this is for summary statistics problems like yours:
DoubleAccumulator max = new DoubleAccumulator(Double::max, Double.NEGATIVE_INFINITY);
parallelStream.forEach(max::accumulate);
max.get();
I believe what you did is correct, but this is a simpler version that I also think is correct.
private void updateMax(long sample){
//this takes care of the case where between the comparison and update steps, another thread updates the max
//For example:
//if the max value is set to a higher max value than the current value in between the comparison and update step
//sample will be the higher value from the other thread
//this means that the sample will now be higher than the current highest (as we just set it to the value passed into this function)
//on the next iteration of the while loop, we will update max to match the true max value
//we will then fail the while loop check, and be done with trying to update.
while(sample > max.get()){
sample = max.getAndSet(sample);
}
}