I want to insert some 4K rows in the MySql db. I don't want to fire 4k 'insert' queries. Is there any way by which I can fire only one insert query to store those 4k rows in the db.
I searched on internet and everywhere I found that the users are doing bulk insert into the db from a file.
In my case, I have the data in the memory and I don't want to first write that data to a file to do bulk insert. If I do that then I will add delay in the program.
You could write a single insert query that would do several inserts in a single call to the database :
insert into your_table (field1, field2, field3)
values
(value1_1, value1_2, value1_3),
(value2_1, value2_2, value2_3),
(value3_1, value3_2, value3_3)
Here, with the example I've given, this single query would have inserted three rows in the table.
MySQL's LOAD DATA command might be useful to you: http://dev.mysql.com/doc/refman/5.5/en/load-data.html
With reference to Pascal's suggestion unless your command exceeds max_allowed_packet then you should be able to execute this query. In many cases it works best to create few smaller inserts with say 1000 rows in each.
You can execute your statements in batch, some code example can be found here.
Also, setAutoCommit(false), and call conn.commit(); after executeBatch() to minimise the number of commits.
Insert bulk more than 7000000 record in 1 minutes in database(superfast query with calculation)
mysqli_query($cons, '
LOAD DATA LOCAL INFILE "'.$file.'"
INTO TABLE tablename
FIELDS TERMINATED by \',\'
LINES TERMINATED BY \'\n\'
IGNORE 1 LINES
(isbn10,isbn13,price,discount,free_stock,report,report_date)
SET RRP = IF(discount = 0.00,price-price * 45/100,IF(discount = 0.01,price,IF(discount != 0.00,price-price * discount/100,#RRP))),
RRP_nl = RRP * 1.44 + 8,
RRP_bl = RRP * 1.44 + 8,
ID = NULL
')or die(mysqli_error());
$affected = (int) (mysqli_affected_rows($cons))-1;
$log->lwrite('Inventory.CSV to database:'. $affected.' record inserted successfully.');
RRP and RRP_nl and RRP_bl is not in csv but we are calculated that and after insert that.
In mySql you can use load data infile
LOAD DATA INFILE 'C:\MyTextFile'
INTO TABLE myDatabase.MyTable
FIELDS TERMINATED BY ','
Related
I have a Main Table named REF_SERVICE_OFFERING where-in I already have 3 Million+ data. Now, I wanted to update the 3M records in Java based on some specific condition.
We decided to create a temporary table (where-in the records within will be used to Update the Main Table) and using the below query to Update the Main Table. The temporary table will hold more than 200k records :
UPDATE REF_SERVICE_OFFERING SET
PART_PRICE_BILL_TYPE= TEMP.PART_PRICE_BILL_TYPE,
part_price_unit_type=TEMP.part_price_unit_type,
part_price_allowed_units=TEMP.part_price_allowed_units,
part_price_discount=TEMP.part_price_discount,
part_price_source_id=TEMP.part_price_source_id
FROM REF_SERVICE_OFFERING RSO JOIN ref_offer_temp1 TEMP
ON TEMP.RECORD_NUM = RSO.RECORD_NUM
AND TEMP.SO_NAME = RSO.SO_NAME
AND TEMP.SERVICE_CASE_TYPE = RSO.SERVICE_CASE_TYPE
AND TEMP.WORK_ORDER_TYPE = RSO.WORK_ORDER_TYPE
WHERE (RSO.PART_PRICE_BILL_TYPE IS NOT NULL OR TRIM(RSO.PART_PRICE_BILL_TYPE) NOT LIKE '')
AND (RSO.PART_PRICE_EXCP_SOURCE_ID IS NOT NULL OR TRIM(RSO.PART_PRICE_EXCP_SOURCE_ID) NOT LIKE '')
Our database is Postgres 9.6. But this update is taking a lot of time and never edning. We also tried dumping only 10k records in the temporary table which will be used to update 4L records.
We tried doing EXPLAIN command and couldnt figure out the reason why.
Any help would be really appreciated.
I'm trying to move a large number of records from one MySQL instance two another inside RDS. They are on different VPCs and different AWS accounts, so I can't create a data pipeline that would do the copy for me.
I've written a quick java program that connects to both the import database and the export database and does the following:
query the import database for the highest in table.primary_key with SELECT MAX(primary_key) FROM table
get a result set from the export table with SELECT * FROM table WHERE(primary_key > max_from_import) LIMIT 1000000
create a PreparedStatement object from the import connection and set the queryString to INSERT INTO table (col1....coln) VALUES (?....n?)
iterate over the result set and set the prepared statement columns to the ones from the result cursor (with some minor manipulations to the data), call execute on the PreparedStatement object, clear its' parameters, then move to the next result.
With this method I'm able to see around 100000 records being imported an hour, but I know that from this question that a way to optimize inserts is not to create a new query each time, but to append more data with each insert. i.e.
INSERT INTO table (col1...coln) VALUES (val1...valn), (val1...valn)....(val1...valn);
Does the jdbc driver know to do this, or is there some sort of optimization I can make on my end to improve insert run time?
UPDATE:
Both answers recommended using the add and execute batch, as well as removing auto commit. Removing auto commit saw a slight improvement (10%), doing the batch yielded a run time of less than 50% of the individual inserts.
You need to use batch insert. Internally, Connector/J (MySQL JDBC driver) can rewrite batch inserts into multi values insert statements.
(Note that this is the default Connector/J behavior. You can add
the option useServerPrepStmts=true to the JDBC url to enable server side prepared statements)
The code looks like the following:
try(PreparedStatement stmt = connection.prepareStatement(sql)) {
for(value : valueList) {
stmt.clearParameters();
stmt.setParameter(1, value);
stmt.addBatch();
}
stmt.executeBatch();
}
The code above will generate a multi value insert:
INSERT tablename(field) VALUES(value1), (value2), (value3) ...
First create a JDBC connection to Destination database and make its auto commit property to false.
After that in a loop do the following
Read N(for example 1000) number of rows from Source database and write that to destination database.
After some inserts commit destination database connection.
Sample code to get more idea is given below
Connection sourceCon = getSourceDbConnction();
Connection destCon = getDestinationDbConnction();
destCon.setAutoCommit(false);
int i=0;
String query;
while((query=getInsertQuery()!=null)
{
statement.executeUpdate(query);
i++;
if(i%10 == 0)
{
destCon.commit();
i=0;
}
}
destCon.commit();
The getInsertQuery function should give string in INSERT INTO table (col1...coln) VALUES (val1...valn), (val1...valn)....(val1...valn); format.
Also it should return null, if all tables are processed.
If you are using Prepared Statements, you can use addBatch and executeBatch functions. Inside loop add values using addBatch function. After some inserts call executeBatch.
I need to insert data from excel to database which looks:
Id Name Phone Joining Date Subject
1 A 11111 14-Mar-2001 Cse
2 B 22222 25-Dec-2016 IT
3 C 33333 12-Dec-2011 ECE
If I have to perform batch insert in a single table then I am able to do it using spring jdbctemplate(batchUpdate(...)).
But I want it to insert data in multiple tables e.g. 1st 3 columns in Table1, next 2 in Table2, next n in table3 like this way.
For reading data I am using POI API and after extracting data m keeping it in List of Map object which looks:
allObj=[{0=1.0, 1=A, 2=11111.0, 3=2001-3-14 0:0:0, 4=Cse}, {0=2.0, 1=B, 2=22222.0, 3=2016-12-25 0:0:0, 4=IT}, {0=3.0, 1=C, 2=33333.0, 3=2011-12-12 0:0:0, 4=ECE}]
How to perform this tasks? not asking full solution but a hint. Thanks
If coding is required then inform I am not posting it as it is lengthy and common.
EDITED:
Few didn't understand the Question!
I think u know batch update. I am using JdbcTemplate of spring.
suppose I have table T1 as:
Id|Name|Phone|Joining Date| Subject in Database(using MYSQL)
Now, I have an excel file with the corresponding values.I can read it and batch insert it into database by JdbcTemplate in that table.
But Now I have two table as T1: Id|Name|Phone
and T2: Joining Date| Subject
I have the same excel file.
NOW my question comes into the frame.
How to insert the values in two tables? If you get the question kindly remove your -ve vote.
LOAD DATA LOCAL INFILE
'C:\\temp\\file.csv'
INTO TABLE table_name
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\r\n'
IGNORE 1 LINES
(#col1,#col2)
set
column1 = #col1,
column2 = #col2;
Above query for table1, run same for other tables by changing column_names accordingly.
I have a java program that reads a csv file line by line and does MERGE in the table using the Merge statement provided by DB2.
For example, If i have a 1000 lines in CSV it would run the executeUpdate 1000 times.
My challenge is that i run this script every hour so there is a possibility that new entries would come into the csv file or values would get updated or rows might be deleted (sometimes the row will come again in future)
I think i am doing the insert if not exist and update if exist part correct but i am not sure how do i do the delete part for rows that were inserted in the past but no longer in the latest csv file ?
Also what would be an ideal situation here to avoid doing 1000 insert statement do i use prepared statement ? Can someone give me an example ?
String sql_merge = " MERGE INTO IM4BMX.IEMCOMPUTER_DETAILS AS A USING (VALUES ('"+ComputerID+"', '"+Environment+"')) AS A_TMP ( ComputerID , Environment) ON A.ComputerID = A_TMP.ComputerID WHEN MATCHED THEN UPDATE SET ComputerID = '"+ComputerID+"', Environment = '"+Environment+"' WHEN NOT MATCHED THEN INSERT ( ComputerID , Environment ) VALUES (A_TMP.ComputerID, A_TMP.Environment ) ELSE IGNORE ";
stmt.executeUpdate(sql_merge);
Try adding a when not matched clause to your query.
Ex. WHEN NOT MATCHED
THEN DELETE
If you CSV file contains data that does not match what is in the DB then it should delete that row.
Look at this link for further information.
Merge Statement Explained
I have a table with columns RECID,regionID, areaID ,UserID ,AppNo, NextAppNo(Accepts null)
and data is as follows (NextApp is not null in the first row only):
1,102,Ar-1,xyz,1,1
2,102,Ar-1,qwe,2
3,102,Ar-1,fgh,3
4,102,Ar-1,yuyi,4
1) Add a record
Now I want to add another record:
First I get the max number of APPID where areaID = Ar-1 and regionID =102 (Java)
Then, if max appNo is 0, add nextappNo, otherwise do nothing
Then insert into db
Is it possible to do this in one query?
2) Delete a record
In another query i want to delete the first record:
First I select data where areaID = 'Ar-1' and regionID =102
Then in a loop (Java) I check if Recid is the same id I want to delete
From there onward I update the the column approver and also, if the record which we are going to delete has approverNO = 1, then also update the next approver to one of the next record
Then delete the record from table.
You can use sql procedure to do it. It will be easier when you execute that single procedure with any type of operation such as insert,delete,update, etc.,