I am looking similar way to jdbc driver in Java, to perform a batch of updates in PHP.
In jdbc there is an API of PreparedStatement.executeBatch(), which executes the whole statements in one round trip to the DB.
Does PHP PDO has similar API, and if not, does starting transaction, doing the updates and then commit will do the same effect of executing all updates in one round trip to the DB or each update will round trip to the DB and immediately executing the statement (although not visible to others, since it is in transaction)?
There is no such thing like "batch update" in Mysql. There are SQL queries only.
As long as you can do your updates in one query, your updates will be done in one round-trip. Otherwise there would be many. No matter what API being used.
Speaking of single SQL queries, there are 2 possible ways
CASE statement in WHERE
A neat trick with INSERT(!) query with ON DUPLICATE UPDATE statement. Which will actually update your data.
PHP PDO doesn't have batch execution of queries.
Running many inserts and updates in a transaction usually improves greatly the execution speed. If you're making batch jobs on a database you should run queries in bulks within a transaction.
Related
My Scenario is i have big query with lot of joins and lot of decode/case calls in select and i am passing one param to where condition from java and i see for 150000 rows java fetch is very slow but query is running faster in SQL developer client interface.
i thought of creating or replacing a view which takes one parameter and call that view from java.
Did not find resource to know how to pass prams to create or replace view statement from java ?
Any one suggest other approach that fetches rows quickly ?
Using oracle 12c and driver is jdbc7 and jdk8
First (and easiest):
Set the JDBC fetch size to a high number in your statement. There is a setFetchSize(int) method on Statement, PreparedStatement, CallableStatement, and ResultSet objects.
This defaults to something small like 10 rows. Set that to a reasonably high number, such as 500 or more.
This is a setting that will definitely slow down a query that pulls back hundreds of thousands of records.
Second:
Verify that the query is indeed running fast in SQL Developer, to the last row.
You can export to a file or try wrapping the query in a PL/SQL statement that will loop through all records.
If you wish, you can use AUTOTRACE in SQL*Plus to your advantage:
SET TIMING ON
SET AUTOTRACE TRACEONLY
<your query>
This will run the query to the end, pulling all records over the network but not displaying them.
The goal here is to prove that your SQL statement is indeed returning all records as quickly as needed.
If not, then you have a standard tuning exercise. Get it running to completion quickly in SQL Developer first.
I am working with mysql in Java.
Basically, I have multiple queries that each create a table in the database, along with a single ALTER statement that adjusts the auto-increment initial value for one of my attributes. I am executing those queries as a transaction--namely either all are committed to the database or none are. But to do so I have create a separate Statement for each query - 8 in total - and execute each. After, I commit all the results. And then I have to close each Statement.
But this seems inefficient. To many Statements. So I wonder whether batch methods would work. My concern is that batch methods execute all the queries simultaneously, and since I have Referential Integrity Constraints and the ALTER query there is a dependancy between the tables - and thus the order in which they are created matters. Is this not correct ? Am I misunderstanding how batch statements work ?
If my logic above is correct, then should I possibly group a few queries together (that are not related) and use batch methods to execute them. This will then reduce the number of Statements I have.
I don't think you can batch DDL (i.e. create, drop, alter). Also, it's not a great idea, performance wise, to require dynamic DDL.
You can batch DML statements using Statement.addBatch(String) (i.e. select, insert, update and delete statements) and then call Statement.executeBatch().
I am trying to create multiple tables (upto 20) via java.sql prepared statement batch execute. Most of tables are related to eachother. But there is some confusion in my mind.
1) set connection auto commit true or false?
2) Is there any special pattern for BatchExecute.? like up down. I want to parent table create query must execute first.
3) If error ouccurs all the batch is rollback?
The behavior of batch execution with auto commit on is implementation defined, some drivers may not even support that. So if you want to use batch execution, set auto commit to false.
That said, some databases implicitly commit each DDL statement; this might interfere with correct working of batched execution. I would advise to take the safe route and not use batched execution for DDL, but to use a normal Statement and execute(String) for executing DDL.
Actually using batch execution in this case does not make much sense. Batch execution gives you a (big) performance improvement when inserting or updating thousands of rows at once.
You just need to have all your statements within a transaction:
call Connection.setAutoCommit(false)
execute your create-table statements with Statement.executeUpdate
call Connection.commit()
You need to order the create-table statements yourself based on the foreign-keys between them.
As Mark pointed out, the DB you are using might commit each create-table right away and ignore the transaction. Not all DBs support transactional creation of tables. You will need to test this or do some more research regarding this aspect.
I've been looking around trying to determine some Hibernate behavior that I'm unsure about. In a scenario where Hibernate batching is properly set up, will it only ever use multiple insert statements when a batch is sent? Is it not possible to use a DB independent multi-insert statement?
I guess I'm trying to determine if I actually have the batching set up correctly. I see the multiple insert statements but then I also see the line "Executing batch size: 25."
There's a lot of code I could post but I'm trying to keep this general. So, my questions are:
1) What can you read in the logs to be certain that batching is being used?
2) Is it possible to make Hibernate use a multi-row insert versus multiple insert statements?
Hibernate uses multiple insert statements (one per entity to insert), but sends them to the database in batch mode (using Statement.addBatch() and Statement.executeBatch()). This is the reason you're seeing multiple insert statements in the log, but also "Executing batch size: 25".
The use of batched statements greatly reduces the number of roundtrips to the database, and I would be surprised if it were less efficient than executing a single statement with multiple inserts. Moreover, it also allows mixing updates and inserts, for example, in a single database call.
I'm pretty sure it's not possible to make Hibernate use multi-row inserts, but I'm also pretty sure it would be useless.
I know that this is an old question but i had the same problem that i thought that hibernate batching means that hibernate would combine multiple inserts into one statement which it doesn't seem to do.
After some testing i found this answer that a batch of multiple inserts is just as good as a multi-row insert. I did a test inserting 1000 rows one time using hibernate batch and one time without. Both tests took about 20s so there was no performace gain in using hibernate batch.
To be sure i tried using the rewriteBatchedStatements option from the MySQL Connector/J which actually combines multiple inserts into one statement. It reduced the time to insert 1000 records down to 3s.
So after all hibernate batch seems to be useless and a real multi-row insert to be much better. Am i doing something wrong or what causes my test results?
The Oracle bulk insert collect an array of entyty and pass in a single block to the db associating to it a unic ciclic insert/update/delete.
Is unic way to speed network throughput .
Oracle suggest to do it calling a stored procedure from hibernate passing it an array of datas.
http://biemond.blogspot.it/2012/03/oracle-bulk-insert-or-select-from-java.html?m=1
Is not only a software problem but infrastructural!
Problem is network data flow optimization and TCP stack fragmentation.
Mysql have function.
You have to do something like what is described in this article.
Normal transfer on network the correct volume of data is the solution
You have also to verify network mtu and Oracle sdu/tdu utilization respect data transferred between application and database
I am trying to create a program that updates 2 different tables using sql commands. The only thing I am worried about is that if the program updates one of the tables and then loses connection or whatever and does NOT update the other table there could be an issue. Is there a way I could either
A. Update them at the exact same time
or
B. Revert the first update if the second one fails.
Yes use a SQL transaction. Here is the tutorial:JDBC Transactions
Depending on the database, I'd suggest using a stored procedure or function based on the operations involved. They're supported by:
MySQL
Oracle
SQL Server
PostgreSQL
These encapsulate a database transaction (atomic in nature -- it either happens, or it doesn't at all), without the extra weight of sending the queries over the line to the database... Because they already exist on the database, the queries are parameterized (safe from SQL injection attacks) which means less data is sent -- only the parameter values.
Most SQL servers support transactions, that is, queueing up a set of actions and then having them happen atomically. To do this, you wrap your queries as such:
START TRANSACTION;
*do stuff*
COMMIT;
You can consult your server's documentation for more information about what additional features it supports. For example, here is a more detailed discussion of transactions in MySQL.