My little application uploads table to a database .
Connection ....
...
...
PreparedStatement ps ...
ps.executeUpdate() ; // this takes a while so I want to create a progress bar
How can I track progress of ps.executeUpdate() ? Can anybody post a java code to do that ? I need to create a progress bar
(My database is Teradata)
You can leverage the Performance Monitor and Performance Control (PMPC) Open APIs using SQL to monitor the progress of a session as it executes a SQL Statement in Teradata 13 and higher.
Get Completed Steps, Host, RunVProcNo for all sessions running as user:
SELECT HostID
, RunVProcNo
, ReqStepsCompletedCnt
FROM TABLE(MonitorSession(-1,'{username}', 0)) AS T2;
You can always obtain the current session for a given connection by running:
SELECT SESSION;
With the Session Number in hand you can isolate the number of records returned by the previous MonitorSession() call.
Get Total Number of Steps for SQL being executed:
SELECT MAX(StepNum) AS TotalNumSteps
FROM TABLE(MonitorSQLSteps({Host},{Session},{RunVProcNo)) AS T1;
With those two statements you should be able to track how something is progressing to update a progress bar in the application. `
There are several possibilities to display "something is happening". But the problem is, you have no progress indicator or listener on a database statement.
"A direct tracking of progress in the database management system is not possible." Regarding this I was corrected by the other answerer. So there are possibilities to track the progress but these are very database specific. So this is also a way to go if you have only TeraData in mind.
So here are some other possibilities to produce some kind of process feedback:
Slice your update in multiple steps (n) (if it is possible). You know how many steps you have to go, that would be your 100 percent. Every step should increase your progressbar by 100/n percent.
This splitting of your SQL is not always a way to go. Therefore you could display a work in progress indicator (an indetermined one).
You could guess the time to process and build a progressbar on that value, like windows does on file copy.
In every case your database processing should happen in a worker thread (different from EDT), but for sure you are aware of that.
Related
We are tuning up a server where clients can create queries and send them to Oracle. This server can create a pool and have connections on standby. The number of standby connections is something we can control and that is what we are trying to tune up. So, while we are tuning up this minimal number of connection on standby we were checking the V$SESSION table to see the connections standing by and when they were active. At that moment of being active is when we started to see this number of "connections" grow up to 70 or 80 at a time while the query was executing. My guess is that these are not connections per se. Looks like the places where it is reading the data from?? I am not sure and that is why I'd like to know. What are those? They only show when the query executes. Here is the query I am using to check in oracle what my connections are doing:
select TO_CHAR(s.prev_exec_start, 'DD-MON-YYYY HH24:MI:SS') as "LAST_RAN", s.*
from V$SESSION s
where username = 'MY_USER_NAME';
It would help if you could show some of the output from your query, but it is very possible that what you are seeing are parallel execution threads for some of the queries. These will spawn automatically to handle the parallel processing requirements and generally have process names like 'P000', 'P001', etc.
The level of parallelism for a query can be defined as part of the related object properties, or within query hints. Short making changes at the system level to parallel processing behavior (see Documentation), or disabling parallel processing entirely for these sessions - which might completely kill query performance - there isn't much you can do.
A logon trigger to disable parallel execution might look like this:
CREATE OR REPLACE TRIGGER APP_SCHEMA_LOGON_TRG
AFTER LOGON ON APP_USER.SCHEMA
BEGIN
execute immediate 'ALTER SESSION DISABLE PARALLEL QUERY';
END;
/
I am fetching records (of large data set, around 1 Million records)from MariaDB in batches of size 500 (by using 'limit').
For each fetch iteration I am opening and closing the connection.
In my peer review I was advised to fetch the result set once and batch process by iterating on the result set itself, i.e. without closing the connection.
Is the second method right way of doing it ?
Edit : After I fetch records in batches of size 500 I am updating a field for each record and putting it on a messaging queue.
Yes, the second method is the right way to do it. Here are some reasons:
There is overhead to running the query multiple times.
The underlying data might change in the tables you are using, and the separate batches might be inconsistent.
You are depending on the ordering of the results and the sorting might have duplicates.
Your program starts
Connect to database
do some SQL (selects/inserts/whatever)
do some more SQL (selects/inserts/whatever)
do some more SQL (selects/inserts/whatever)
...
Disconnect from database
Your program ends
That is, keep the connection open as long as needed during the program. (Even if you don't explicitly disconnect, the termination of your program will do the termination. This is important to note when doing a web site -- each 'page' is essentially a separate 'program'; the db connection cannot be held between pages.)
You have another implied question... "Should I grab a batch of rows at once, then process them in the client?" The answer is "It depends".
If the processing can be done in SQL, it is probably much more efficient to do it there. Example: summing up some numbers.
If you fetch some rows from one table, then for each of those rows, fetch row(s) from another table... It will be much more efficient to use an SQL JOIN.
"Batching" may not be relevant. The client interface is probably
Fetch rows from a table (possibly all rows, even if millions)
Look at each row in turn.
Please provide the specifics of what you will be doing with the million rows so we can discuss more specifically.
Polling loop:
If you check for new things to do only once a minute, do reconnect each time.
Given that, it does not make sense to hang onto a resultset between pools.
Opening and closing a database connection is quite time consuming. Keeping the connection open will save a lot of time.
I have a J2EE server, currently running only one thread (the problem arises even within one single request) to save its internal model of data to MySQL/INNODB-tables.
Basic idea is to read data from flat files, do a lot of calculation and then write the result to MySQL. Read another set of flat files for the next day and repeat with step 1. As only a minor part of the rows change, I use a recordset of already written rows, compare to the current result in memory and then update/insert it correspondingly (no delete, just setting a deletedFlag).
Problem: Despite a purely sequential process I get lock timeout errors (#1204) and Innodump show record locks (though I do not know how to figure the details). To complicate things under my windows machine everything works, while the production system (where I can't install innotop) has some record locks.
To the critical code:
Read data and calculate (works)
Get Connection from Tomcat Pool and set to autocommit=false
Use Statement to issue "LOCK TABLES order WRITE"
Open Recordset (Updateable) on table order
For each row in Recordset --> if difference, update from in-memory-object
For objects not yet in the database --> Insert data
Commit Connection, Close Connection
The Steps 5/6 have an Commitcounter so that every 500 changes the rows are committed (to avoid having 50.000 rows uncommitted). In the first run (so w/o any locks) this takes max. 30sec / table.
As stated above right now I avoid any other interaction with the database, but it in future other processes (user requests) might read data or even write some fields. I would not mind for those processes to read either old/new data and to wait for a couple of minutes to save changes to the db (that is for a lock).
I would be happy to any recommendation to do better than that.
Summary: Complex code calculates in-memory objects which are to be synchronized with database. This sync currently seems to lock itself despite the fact that it sequentially locks, changes unlocks the tables without any exceptions thrown. But for some reason row locks seem to remain.
Kind regards
Additional information:
Mysql: show processlist lists no active connections (all asleep or alternatively waiting for table locks on table order) while "show engine INNODB" reports a number of row locks (unfortuantely I can't understand which transaction is meant as output is quite cryptic).
Solved: I wrongly declared a ResultSet as updateable. The ResultSet was closed only on a "finalize()" method via Garbage Collector which was not fast enough - before I reopended the ResultSet and tried therefore to aquire a lock on an already locked table.
Yet it was odd, that innotop showed another query of mine to hang on a completely different table. Though as it works for me, I do not care about oddities:-)
I want to delete duplicate record in my database table, and I do it in java by using this query
String sql = "DELETE e1 FROM tweet_after_preprocessing e1, tweet_after_preprocessing e2 WHERE e1.tweet = e2.tweet AND e1.tweet_after_preprocessing_id > e2.tweet_after_preprocessing_id"
The problem is when there are so many records in my database table, the process will take so long, and make my program look not curently running.
and I want to use progress bar to show progress of the executing, how can I do that?. I don't now the maximum and the minimum value, so how can i accessing the progress bar?.
You can create an indeterminate progress bar by setting the property indeterminate to true: JProgressBar.html#setIndeterminate().
Also it is wise to not execute long lasting work in the EDT but use a different thread for this.
The problem is that you're doing all your work on the EDT, which is blocking your GUI. You need to do the loading on another Thread, so your GUI can still update, display, and respond to user input.
Once you have the work on another thread, then from that thread, you can post updates to the EDT using the SwingUtilities.invokeLater() method.
You could also look into the SwingWorker class, which handles some of that for you.
Recommended reading: http://docs.oracle.com/javase/tutorial/uiswing/concurrency/
Excellent question.
Jenkins uses the time of the last build to guess how long the current build will take to run. So for example if the last time a build ran it took 10 minutes, and it is 5 minutes into a current build it will show that it is 50% complete.
You can do something similar. Maybe query the DB first to see how many items need to be deleted and have a table of how long it will take for the amount of items being deleted and how many items are in the table.
My application does queries to database (non-queries too).
I show to the user indeterminate progressbar while transaction.
The problem is that I don't know when to close the progressbar, because I have no indication or signal object of query completion.
If your queries are taking so long that you need a progress bar, I'd recommend taking a hard look at your database to see how you can speed them up.
Since a database operation blocks, you know it's done when you return.
The progress bar suggests that you need some kind of polling mechanism to check a metric that indicates how much you have to do and how many have been done. Since you don't give any details, I can only guess what those might be.
But an AJAX call in your JSP to poll and update the progress bar is most likely what you need.