Develop Reference

Getting Error while running "Alter table <TableName> not logged initially" through Java

There are millions of records in table which I need to delete but I need to log off the transaction logs for that I use Alter Table not logged initially but it is throwing error and makes the table inaccessible. There is no partition in table but table contain Index and Sequences. Also autocommit is off.
Error : DB2 SQL Error: SQLCODE=-911, SQLSTATE=40001, SQLERRMC=68, DRIVER=3.65.77
Getting the above error only when running through java, not getting any error if running from client.
Need to know in which all cases and scenarios the query can fail or what need make ensure before running this query. How to handle this scenario in code.

When asking for help with Db2, always put into your question the Db2-server version and platform (Z/OS, i-Series, Linux/Unix/Windows) because the answers can depend on those facts.
The sqlcode -911 with sqlerrmc 68 is a lock-timeout. This is not a deadlock. Your job
might not be the only job that is concurrently accessing the table. Monitoring functions and administrative views let you see which locks exist at any moment in time (e.g. SNAPLOCK and SNAP_GET_LOCK table function and many others).
Refer to the Db2 Knowledge Centre for details of the suggestions below, to educate yourself.
Putting the table into not-logged-initially for your transaction is high risk, especially if you are a novice, because if your transaction fails then you can lose the entire table.
If you persist with that approach, take precautions and rehearse point in time recovery in case your actions cause damage. Verify your backups and recovery steps carefully.
With autocommit disabled, one can lock a table in exclusive mode, but this can cause a service-outage on production if the target table is a hot table. One can also force off applications that are holding locks if you have the relevant rights.
If there are any other runnning jobs (i.e. not your own code) accessing the table while you try to alter that table when the -911 is almost inevitable. Your approach may be unwise.
Bulk delete can be achieved by other means, it depends on what you wish to trade-off.
This is a frequently asked question. It's not RDBMS specific either.
Consider doing more research, as this is a widely discussed topic.
Alternative approaches for bulk delete include these:
batching the logged deletes, commit once per batch, adjustable batch size
( to ensure you avoid a -964 transaction-log-full situation).
This requires programming a loop, and you should condsider 'set current timeout not wait'
along with retrying automatically later any failed batches (e.g batches that failed
due to locks). This approach yields a
slow and gradual removal of rows, but increases concurrency. You are trading
a long slow execution for minimal impact on other running jobs.
Create an identical shadow table, into which you insert only the rows that you
wish to keep. Then use truncate table ... immediate on the target table
(this is an unlogged action)
and finally restore the preserved-rows from the shadow-table into the target-table.
A less safe variation of this is to export only the rows you want to keep and then
import-replace
depending on Db2-licence and frequency of purge, migrating the data (or some of the data) into a range partitioned table, and using detach may be the better long term solution
Refer to the on-line Db2 Knowledge Center for details of the above suggestions.

Counting Number Of Specific Record In Database

I have a application which needs to aware of latest number of some records from a table from database, the solution should be applicable without changing the database code or add triggers or functions to it ,so I need a database vendor independent solution.
My program written in java but database could be (SQLite,MySQL,PostgreSQL or MSSQL),for now I'm doing Like that:
In a separate thread that is set as a daemon my application sends a simple command through JDBC to database to be aware of latest number of the records with condition:
while(true){
SELECT COUNT(*) FROM Mytable WHERE exited='1'
}
and this sort of coding causes DATABASE To lock,slows down the whole system and generates huge DB Logs which finally brings down the whole thing!
how can i do it in a right way to always have latest number of certain records or only counting when the number changed?

A SELECT statement should not -- by itself -- have the behavior that you are describing. For instance, nothing is logged with a SELECT. Now, it is possible that concurrent insert/update/delete statements are going on, and that these cause problems because the SELECT locks the table.
Two general things you can do:
Be sure that the comparison is of the same type. So, if exited is a number, do not use single quotes (mixing of types can confuse some databases).
Create an index on (exited). In basically all databases, this is a single command: create index idx_mytable_exited on mytable(exited).
If locking and concurrent transactions are an issue, then you will need to do more database specific things, to avoid that problem.

As others have said, make sure that exited is indexed.
Also, you can set the transaction isolation on your query to do a "dirty read"; this indicates to the database server that you do not need to wait for other processes' transactions to commit, and instead you wish to read the current value of exited on rows that are being updated by those other processes.
SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED is the standard syntax for using "dirty read".

Reset/clear MySQL user variables

I have bunch of MySQL queries that use temporary tables to split complex/expensive queries into small pieces.
create temporary table product_stats (
product_id int
,count_vendors int
,count_categories int
,...
);
-- Populate initial values.
insert into product_stats(product_id) select product_id from product;
-- Incrementally collect stats info.
update product_stats ... join vendor ... set count_vendors = count(vendor_id);
update product_stats ... join category... set count_categories = count(category_id);
....
-- Consume the resulting temporary table.
select * from product_stats;
The problem is that, as I use connection pool, these tables are not cleared even if I close the java.sql.Connection.
I can manually remove them (drop temporary table x;) one by one before executing the needed queries, but that may take place for mistakes.
Is there a way (JDBC/MySQL , API/configuration) to reset all the temporary tables created within the current session without closing the database connection (as you know, I'm not reffering to java.sql.Connection.close()), so that I can still use the advantages that provides connection pool?
Edited:
It seems that only from MySQL version 5.7.3 they started imlpementing the "reset connection" feature. (Release note: https://dev.mysql.com/doc/relnotes/mysql/5.7/en/news-5-7-3.html) However, I will not use it for the moment because version 5.7 is still on a development release.

Q: Is there a way (JDBC/MySQL , API/configuration) to reset all the temporary tables created within the current session without closing the database connection.
A: No. There's no "reset" available. You can issue DROP TEMPORARY TABLE foo statements within the session, but you have to provide the name of the temporary table you want to drop.
The normative pattern is for the process that created the temporary table to drop it, before the connection is returned to the pool. (I typically handle this in the finally block.)
If we are expecting other processes may leave temporary tables in the session (and to be defensive, that's what we expect), we typically do a DROP TEMPORARY TABLE IF EXISTS foo before we attempt to create a temporary table.
EDIT
The answer above is correct for MySQL up through version 5.6.
#mr.Kame (OP) points out the new mysql_reset_connection function (introduced in MySQL 5.7.3).
Reference: 22.8.7.60 mysql_reset_connection() http://dev.mysql.com/doc/refman/5.7/en/mysql-reset-connection.html
Looks like this new function achieves nearly the same result as we'd get by disconnecting from and reconnecting to MySQL, but with less overhead.
(Now I'm wondering if MariaDB has introduced a similar feature.)

H2 Database - Creating Indexes

I'm using the H2 database - running in embedded mode - and when my app starts up I load the H2 database with data from a mySQL database. I'm using linked tables to point to the mySQL tables.
My issue is that I'm trying to speed up the time that H2 takes to create the indexes on the tables, particularly for larger tables (5Million+).
Does anyone know if it is safe to run the CREATE INDEX commands in a separate thread while I load the next table's data into H2?
For example:
Thread 1: Loads table 1 -> tells Thread 2 to start creating indexes and then Thread 1 loads table 2, etc.
I can't use the MVCC mode when loading the tables because later on I need to use the MULTI_THREADED mode when I do my selects. When I try using the MULTI_THREADED mode I got locking errors even though I was loading data into discrete tables.
Many thanks!

What might work (but I'm not sure if it's faster) is to create the tables and indexes first, and then load the tables in parallel. This should avoid locking problems in the system table.

I would also like to add the method rst.findColumn("columnName") to find the indexes AFTER getting the result set of the table. rst is a ResultSet object. This is what I have used.

Another way to dramatically improve H2 loading and especially indexing performance is to set the initial memory close to what the expected memory requirement is. As one example, this one change allowed an app with about a 1.5GB requirement to startup in 47 seconds instead of failing after 15 - 20 minutes. Prior to this, we were seeing GC Overhead limit exceeded and JVMTI errors.
Add the following to your VM arguments (as an example):
-Xms2g
-Xmx4g

Tips on Speeding up JDBC writes?

I am writing a program that does a lot of writes to a Postgres database. In a typical scenario I would be writing say 100,000 rows to a table that's well normalized (three foreign integer keys, the combination of which is the primary key and the index of the table). I am using PreparedStatements and executeBatch(), yet I can only manage to push in say 100k rows in about 70 seconds on my laptop, when the embedded database we're replacing (which has the same foreign key constraints and indices) does it in 10.
I am new at JDBC and I don't expect it to beat a custom embedded DB, but I was hoping it to be only 2-3x slower, not 7x. Anything obvious that I maybe missing? does the order of the writes matter? (i.e. say if it's not the order of the index?). Things to look at to squeeze out a bit more speed?

This is an issue that I have had to deal with often on my current project. For our application, insert speed is a critical bottleneck. However, we have discovered for the vast majority of database users, the select speed as their chief bottleneck so you will find that there are more resources dealing with that issue.
So here are a few solutions that we have come up with:
First, all solutions involve using the postgres COPY command. Using COPY to import data into postgres is by far the quickest method available. However, the JDBC driver by default does not currently support COPY accross the network socket. So, if you want to use it you will need to do one of two workarounds:
A JDBC driver patched to support COPY, such as this one.
If the data you are inserting and the database are on the same physical machine, you can write the data out to a file on the filesystem and then use the COPY command to import the data in bulk.
Other options for increasing speed are using JNI to hit the postgres api so you can talk over the unix socket, removing indexes and the pg_bulkload project. However, in the end if you don't implement COPY you will always find performance disappointing.

Check if your connection is set to autoCommit. If autoCommit is true, then if you have 100 items in the batch when you call executeBatch, it will issue 100 individual commits. That can be a lot slower than calling executingBatch() followed by a single explicit commit().
I would avoid the temptation to drop indexes or foreign keys during the insert. It puts the table in an unusable state while your load is running, since nobody can query the table while the indexes are gone. Plus, it seems harmless enough, but what do you do when you try to re-enable the constraint and it fails because something you didn't expect to happen has happened? An RDBMS has integrity constraints for a reason, and disabling them even "for a little while" is dangerous.

You can obviously try to change the size of your batch to find the best size for your configuration, but I doubt that you will gain a factor 3.
You could also try to tune your database structure. You might have better performances when using a single field as a primary key than using a composed PK. Depending on the level of integrity you need, you might save quite some time by deactivating integrity checks on your DB.
You might also change the database you are using. MySQL is supposed to be pretty good for high speed simple inserts ... and I know there is a fork of MySQL around that tries to cut functionalities to get very high performances on highly concurrent access.
Good luck !

try disabling indexes, and reenabling them after the insert. also, wrap the whole process in a transaction