I investigate one problem on the database (Aurora MySQL 5.7.mysql_aurora.2.10.0).
In my AWS service, we change the database structure automatically through the java code, namely through spring (schema.sql + spring.sql.init.mode=always). Examples: Create table if exist, alter table, etc.
And after some time, I get a DB "waiting for table metadata lock" error (photo below), which leads to the fall of the kubernetes pod (above limit CPU usage).
There is the same service working with GCP (Big Query) doing the same auto-configuration through the same piece of code and there is no such problem.
Schema.sql file
CREATE TABLE IF NOT EXISTS job_detail (
job_id varchar(50) NOT NULL,
created_by varchar(100) DEFAULT NULL,
dataflow_id varchar(50) DEFAULT NULL,
periodic_tasks longtext,
error_details longtext DEFAULT NULL,
end_date datetime DEFAULT NULL,
executed_date datetime DEFAULT NULL,
failed_records longtext,
total_records int(11) DEFAULT 0,
created_date datetime DEFAULT NULL,
updated_by varchar(100) DEFAULT NULL,
updated_date datetime DEFAULT NULL,
job_status varchar(100) DEFAULT NULL,
job_summary longtext,
job_name varchar(100) DEFAULT NULL,
project_id varchar(50) NOT NULL,
PRIMARY KEY (job_id)
);
ALTER TABLE job_detail CHANGE COLUMN created_by created_by VARCHAR(255) NULL DEFAULT NULL;
ALTER TABLE job_detail CHANGE COLUMN updated_by updated_by VARCHAR(255) NULL DEFAULT NULL;
Photo of queries logs
Related
I'm using JDBC MySqlDataSource class to handle a database for my app. I'm trying to make a method that formats the database for app's use - creates tables and constraints. Basically it's a script generated from phpMyAdmin export option. I've made a class called DatabaseTemplate with a static String containing the script and then I use it in my formatDatabase method.
public void formatDatabase() {
try {
sql.executeUpdate(DatabaseTemplate.GetScript());
} catch (SQLException e) {
System.out.println("Connection not found.");
e.printStackTrace();
}
}
It makes an SQL exception "Syntax not correct". My script string looks like this:
CREATE TABLE `arenas` (
`arena_id` int(11) NOT NULL,
`name` varchar(50) COLLATE utf16_polish_ci NOT NULL,
`location` varchar(50) COLLATE utf16_polish_ci NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf16 COLLATE=utf16_polish_ci;
CREATE TABLE `contestants` (
`contestant_id` int(11) NOT NULL,
`name` varchar(50) COLLATE utf16_polish_ci NOT NULL,
`surname` varchar(50) COLLATE utf16_polish_ci NOT NULL,
`nickname` varchar(50) COLLATE utf16_polish_ci NOT NULL,
`score` int(11) NOT NULL,
`language` varchar(50) COLLATE utf16_polish_ci NOT NULL,
`contact_info` text COLLATE utf16_polish_ci NOT NULL,
`additional_info` text COLLATE utf16_polish_ci,
`team_id` int(11) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf16 COLLATE=utf16_polish_ci;
CREATE TABLE `matches` (
`match_id` int(11) NOT NULL,
`sideA` int(11) NOT NULL,
`sideB` int(11) NOT NULL,
`sideA_score` int(11) DEFAULT NULL,
`sideB_score` int(11) DEFAULT NULL,
`time` datetime NOT NULL,
`tournament` int(11) NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf16 COLLATE=utf16_polish_ci;
CREATE TABLE `system_users` (
`sys_usr_id` int(11) NOT NULL,
`login` varchar(20) COLLATE utf16_polish_ci NOT NULL,
`pw_hash` int(32) NOT NULL,
`permissions` varchar(5) COLLATE utf16_polish_ci NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf16 COLLATE=utf16_polish_ci;
CREATE TABLE `teams` (
`team_id` int(11) NOT NULL,
`name` varchar(50) COLLATE utf16_polish_ci NOT NULL,
`where_from` varchar(50) COLLATE utf16_polish_ci NOT NULL,
`leader_id` int(11) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf16 COLLATE=utf16_polish_ci;
CREATE TABLE `tournaments` (
`tournament_id` int(11) NOT NULL,
`name` varchar(50) COLLATE utf16_polish_ci NOT NULL,
`type` enum('solo','team') COLLATE utf16_polish_ci NOT NULL,
`arena_id` int(11) NOT NULL,
`operator` int(11) NOT NULL,
`additional_info` text COLLATE utf16_polish_ci
) ENGINE=InnoDB DEFAULT CHARSET=utf16 COLLATE=utf16_polish_ci;
ALTER TABLE `arenas`
ADD PRIMARY KEY (`arena_id`);
ALTER TABLE `contestants`
ADD PRIMARY KEY (`contestant_id`),
ADD KEY `team_id` (`team_id`);
ALTER TABLE `matches`
ADD PRIMARY KEY (`match_id`),
ADD KEY `sideA` (`sideA`),
ADD KEY `sideB` (`sideB`),
ADD KEY `tournament` (`tournament`);
ALTER TABLE `system_users`
ADD PRIMARY KEY (`sys_usr_id`);
ALTER TABLE `teams`
ADD PRIMARY KEY (`team_id`),
ADD KEY `leader_id` (`leader_id`);
ALTER TABLE `tournaments`
ADD PRIMARY KEY (`tournament_id`),
ADD KEY `arena_id` (`arena_id`),
ADD KEY `operator` (`operator`);
ALTER TABLE `contestants`
MODIFY `contestant_id` int(11) NOT NULL AUTO_INCREMENT;
ALTER TABLE `contestants`
ADD CONSTRAINT `contestants_ibfk_1` FOREIGN KEY (`team_id`) REFERENCES `teams` (`team_id`);
ALTER TABLE `matches`
ADD CONSTRAINT `matches_ibfk_1` FOREIGN KEY (`sideA`) REFERENCES `teams` (`team_id`),
ADD CONSTRAINT `matches_ibfk_2` FOREIGN KEY (`sideB`) REFERENCES `teams` (`team_id`),
ADD CONSTRAINT `matches_ibfk_3` FOREIGN KEY (`tournament`) REFERENCES `tournaments` (`tournament_id`);
ALTER TABLE `teams`
ADD CONSTRAINT `teams_ibfk_1` FOREIGN KEY (`leader_id`) REFERENCES `contestants` (`contestant_id`);
ALTER TABLE `tournaments`
ADD CONSTRAINT `tournaments_ibfk_1` FOREIGN KEY (`arena_id`) REFERENCES `arenas` (`arena_id`),
ADD CONSTRAINT `tournaments_ibfk_2` FOREIGN KEY (`operator`) REFERENCES `system_users` (`sys_usr_id`);
I'm guessing it's because of how String in java uses \n. I tried removing it, changing it to \r\n, it's still the same exception. How do I format this script so that it executes properly?
It might be good to read the Java documentation on issuing SQL queries as it seems you are using the wrong method in your code.
If you are curious about the difference between DDL (Data Definition Language) and DML (Data Manipulation Language), there are valuable articles on them on Wikipedia.
You might find it easier to place the sql in a dedicated file setup.sql and then load the file when needed and execute the sql from the file. If your code contains more than one SQL statement, you could store each in a separate file.
That way your SQL query will:
actually be readable to a human and
there will be no issues with multi-line strings in Java.
I'm using H2 to test my database code. To do that, I start by reading in a set of SQL files and using RunScript.execute to execute the SQL.
My real database is MySQL, and I have tested the SQL scripts to make sure they work and they do. The problem is decidedly inside of H2. Here's the set of scripts:
CREATE TABLE stores (
id integer AUTO_INCREMENT PRIMARY KEY,
name varchar(255) NOT NULL,
UNIQUE(name),
created_at timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
updated_at timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
deleted_at timestamp
);
CREATE TABLE regions (
id integer AUTO_INCREMENT PRIMARY KEY,
name varchar(255) NOT NULL,
UNIQUE(name),
created_at timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
updated_at timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
deleted_at timestamp
);
CREATE TABLE products (
id integer AUTO_INCREMENT PRIMARY KEY,
store_id integer NOT NULL,
name varchar(255) NOT NULL,
region_id integer NOT NULL,
created_at timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
updated_at timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
deleted_at timestamp,
UNIQUE(store_id, name),
FOREIGN KEY store_fk(store_id) REFERENCES stores(id),
FOREIGN KEY region_fk(region_id) REFERENCES regions(id)
);
Running these 3 scripts to create the tables using MySQL Workbench works fine. Running them from JUnit via the follow code does not:
Reader storeTableSql = new InputStreamReader(
DatabaseConnectorTests.class.getResourceAsStream(
"/migrations/V1__create_stores_table.sql"));
Reader regionTableSql = new InputStreamReader(
DatabaseConnectorTests.class.getResourceAsStream(
"/migrations/V2__create_regions_table.sql"));
Reader productTableSql = new InputStreamReader(
DatabaseConnectorTests.class.getResourceAsStream(
"/migrations/V3__create_products_table.sql"));
this.dc = DatabaseConnector.getConnection();
RunScript.execute(dc, storeTableSql);
RunScript.execute(dc, regionTableSql);
RunScript.execute(dc, productTableSql);
I get the following error:
org.h2.jdbc.JdbcSQLException: Syntax error in SQL statement "CREATE TABLE PRODUCTS (
ID INTEGER AUTO_INCREMENT PRIMARY KEY,
STORE_ID INTEGER NOT NULL,
NAME VARCHAR(255) NOT NULL,
REGION_ID INTEGER NOT NULL,
CREATED_AT TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
UPDATED_AT TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
DELETED_AT TIMESTAMP,
UNIQUE(STORE_ID, NAME),
FOREIGN KEY STORE_FK[*](STORE_ID) REFERENCES STORES(ID),
FOREIGN KEY REGION_FK(REGION_ID) REFERENCES REGIONS(ID)
) "; expected "("; SQL statement:
CREATE TABLE products (
id integer AUTO_INCREMENT PRIMARY KEY,
store_id integer NOT NULL,
name varchar(255) NOT NULL,
region_id integer NOT NULL,
created_at timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
updated_at timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
deleted_at timestamp,
UNIQUE(store_id, name),
FOREIGN KEY store_fk(store_id) REFERENCES stores(id),
FOREIGN KEY region_fk(region_id) REFERENCES regions(id)
) [42001-196]
This code works in MySQL, and I am running H2 in MySQL compatibility mode via the following connection string: jdbc:h2:file:~/database;MODE=MySQL. I'm not sure what else I can do here. This error seems very strange to me, as I have no idea why it would think I need ANOTHER (.
Does anyone know how to fix this? Or have a better suggestion on how to seed my test database? I cannot use hibernate for this task due to a restriction on the system I am running on so I'm forced to do a lot manually.
Thank you!
From this post, but with a little more explanation.
Foreign key constraints in H2 do not allowed named constraints for whatever reason. The solution to this problem is to simply remove the name:
CREATE TABLE products (
id integer AUTO_INCREMENT PRIMARY KEY,
store_id integer NOT NULL,
name varchar(255) NOT NULL,
region_id integer NOT NULL,
created_at timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
updated_at timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
deleted_at timestamp,
UNIQUE(store_id, name),
FOREIGN KEY (store_id) REFERENCES stores(id),
FOREIGN KEY (region_id) REFERENCES regions(id)
);
The error message is so unclear and misleading. Hopefully this helps someone.
All answers to the issue "Incorrect string value" point to using utf8mb4. However, I'm still using that I still get that error.
Running
SELECT CCSA.character_set_name, collation_name FROM information_schema.`TABLES` T,
information_schema.`COLLATION_CHARACTER_SET_APPLICABILITY` CCSA
WHERE CCSA.collation_name = T.table_collation
AND T.table_schema = "test"
AND T.table_name = "products";
produces
character_set_name collation_name
utf8mb4 utf8mb4_unicode_ci
And running
SELECT character_set_name, collation_name FROM information_schema.`COLUMNS`
WHERE table_schema = "test"
AND table_name = "products"
AND column_name = "description";
produces the same result. Then, when I try to update the column description:
UPDATE test.products SET description='Esperienza molto negativa... 😁😁😁😁SCONSIGLIATISSIMO' WHERE id='50'
I get the abovementioned error
1366: Incorrect string value: '\xF0\x9F\x98\x81\xF0\x9F...' for column 'description' at row 1
I've tried this update both through a jdbc connection jdbc:mysql://localhost:3306/test?useUnicode=yes;characterEncoding=utf8; and directly through MySQLWorkbench. I don't know what else to try of look for. What am I missing?
EDIT
Output of SHOW CREATE TABLE
CREATE TABLE `products` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`manufacturerId_fk` int(11) NOT NULL,
`type_fk` varchar(20) COLLATE utf8mb4_unicode_ci NOT NULL DEFAULT 'TEST',
`ownerId_fk` int(11) DEFAULT NULL,
`rating_fk` int(11) NOT NULL,
`description` varchar(2000) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`authorName` varchar(75) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`authorUrl` varchar(100) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`created` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
`updated` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
PRIMARY KEY (`id`),
UNIQUE KEY `manufacturer_authorUrl_UNIQUE` (`manufacturerId_fk`,`authorUrl`),
KEY `fk_products_manufacturers_idx` (`manufacturerId_fk`),
KEY `fk_products_users_idx` (`ownerId_fk`),
KEY `fk_products_validRatings_idx` (`rating_fk`),
KEY `fk_products_productTypes` (`type_fk`),
CONSTRAINT `fk_products_productTypes` FOREIGN KEY (`type_fk`) REFERENCES `productTypes` (`type`) ON DELETE NO ACTION ON UPDATE NO ACTION,
CONSTRAINT `fk_products_manufacturers` FOREIGN KEY (`manufacturerId_fk`) REFERENCES `manufacturers` (`id`) ON DELETE NO ACTION ON UPDATE NO ACTION,
CONSTRAINT `fk_products_validRatings` FOREIGN KEY (`rating_fk`) REFERENCES `validRatings` (`value`) ON DELETE NO ACTION ON UPDATE NO ACTION
) ENGINE=InnoDB AUTO_INCREMENT=418 DEFAULT CHARSET=utf8mb4
A similar question occurred to me with a text field.
(MySQL 1366 error incorrect string value)
I confirmed the database, table, column character set were all utf8mb4.
The problem was solved by changing the field type to longtext.
Hope this could help someone.
Caused by: java.sql.SQLException: Invalid default value for
'updated_at' Query:
CREATE TABLE IF NOT EXISTS saved_query (id int(11)
NOT NULL AUTO_INCREMENT,name varchar(255) NOT NULL,description
varchar(255) DEFAULT NULL,query longtext,params_json
longtext,created_at timestamp DEFAULT CURRENT_TIMESTAMP,updated_at
=timestamp NOT NULL, PRIMARY KEY (id)) Parameters: []
Pasting the query for readability:
CREATE TABLE IF NOT EXISTS saved_query
(
id INT(11) NOT NULL auto_increment,
name VARCHAR(255) NOT NULL,
description VARCHAR(255) DEFAULT NULL,
query LONGTEXT,
params_json LONGTEXT,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP NOT NULL,
PRIMARY KEY (id)
)
Mysql version is 5.7.13. The query above works in versions 5.6 and 5.5. The following query works in 5.7 but not in <5.7 versions:
CREATE TABLE IF NOT EXISTS saved_query
(
id INT(11) NOT NULL auto_increment,
name VARCHAR(255) NOT NULL,
description VARCHAR(255) DEFAULT NULL,
query LONGTEXT,
params_json LONGTEXT,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (id)
)
The error in 5.6 is:
Caused by: java.sql.SQLException: Incorrect table definition; there
can be only one TIMESTAMP column with CURRENT_TIMESTAMP in DEFAULT or
ON UPDATE clause Query:
CREATE TABLE IF NOT EXISTS saved_query (id
int(11) NOT NULL AUTO_INCREMENT,name varchar(255) NOT NULL,description
varchar(255) DEFAULT NULL,query longtext,params_json
longtext,created_at timestamp DEFAULT CURRENT_TIMESTAMP,updated_at
timestamp DEFAULT CURRENT_TIMESTAMP, PRIMARY KEY (id)) Parameters: []
What should be the query that works in all these versions?
The problem is because of sql_modes. You have to remove the sql_mode to make it work. Remove these SQL modes
NO_ZERO_IN_DATE
NO_ZERO_DATE
or
before you run any query, paste this code in the first line of your code:
SET SQL_MODE = "NO_AUTO_VALUE_ON_ZERO";
SET time_zone = "+00:00";
or
before you run any query, paste this code in the first line:
SET sql_mode = '';
TIMESTAMP can not be NULL, and mysql add DEFAULT CURRENT_TIMESTAMP for column. But may be only one column width DEFAULT CURRENT_TIMESTAMP. Just use '0000-00-00 00:00:00'.
CREATE TABLE IF NOT EXISTS saved_query
(
id INT(11) NOT NULL auto_increment,
name VARCHAR(255) NOT NULL,
description VARCHAR(255) DEFAULT NULL,
query LONGTEXT,
params_json LONGTEXT,
created_at TIMESTAMP DEFAULT '0000-00-00 00:00:00',
updated_at TIMESTAMP DEFAULT '0000-00-00 00:00:00',
PRIMARY KEY (id)
)
I am working on a grails project. I have following query that I am trying to execute
String CHECK_FOR_HIGH_TRADE_VOLUME_QUERY = "Update LocationTrade lt set lt.hasVeryHighVolume=true where lt.locationIndices=? AND lt.trade.volume>20000";
...
LocationTrade.executeUpdate(CHECK_FOR_HIGH_TRADE_VOLUME_QUERY, [indices]);
The relationship between LocationTrade and Trade is unidirectional many-to-one. So, LocationTrade has a reference to Trade but Trade class does not have reference to the List of LocationTrade.
On execution, I get the following exception.
org.springframework.dao.InvalidDataAccessResourceUsageException: could not execute update query; SQL [update location_trade cross join set has_very_high_volume=1 where location_indices_id=? and volume>20000]; nested exception is org.hibernate.exception.SQLGrammarException: could not execute update query
and
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'set has_very_high_volume=1 where location_indices_id=997 and volume>20000' at line 1
It seems that generated query is wrong. There should have been a join with the Trade table, but that is missing. I am unable to identify the error that I made here. Can some of you help me?
Creation Script for the two tables (I have stripped some of the uninteresting columns)
CREATE TABLE `location_trade` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`version` bigint(20) NOT NULL,
`auto_status` varchar(255) DEFAULT NULL,
`exclusion_reason_description` varchar(255) DEFAULT NULL,
`exclusion_reason_id` bigint(20) DEFAULT NULL,
`exclusion_reason_title` varchar(255) DEFAULT NULL,
`location_indices_id` bigint(20) DEFAULT NULL,
`manual_status` varchar(255) DEFAULT NULL,
`trade_id` bigint(20) DEFAULT NULL,
`absolute_price` decimal(19,6) DEFAULT NULL,
`flag` varchar(255) DEFAULT NULL,
`auto_exclusion_reason` varchar(255) DEFAULT NULL,
`date_created` datetime DEFAULT NULL,
`exclusion_reason_text` varchar(255) DEFAULT NULL,
`last_updated` datetime DEFAULT NULL,
`has_very_high_volume` bit(1) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `FK858985A90CAA966` (`location_indices_id`),
KEY `FK858985AB5FA6A69` (`trade_id`),
CONSTRAINT `FK858985A90CAA966` FOREIGN KEY (`location_indices_id`) REFERENCES `location_indices` (`id`),
CONSTRAINT `FK858985AB5FA6A69` FOREIGN KEY (`trade_id`) REFERENCES `trade` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=25405 DEFAULT CHARSET=latin1;
CREATE TABLE `trade` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`version` bigint(20) NOT NULL,
`comments` varchar(1020) DEFAULT NULL,
`end_date` datetime DEFAULT NULL,
`price` decimal(19,6) DEFAULT NULL,
`price_type` varchar(255) DEFAULT NULL,
`source_id` bigint(20) DEFAULT NULL,
`start_date` datetime DEFAULT NULL,
`trade_date` datetime DEFAULT NULL,
`trade_name` varchar(255) DEFAULT NULL,
`volume` decimal(19,6) DEFAULT NULL,
`volume_units` varchar(255) DEFAULT NULL,
`date_created` datetime DEFAULT NULL,
`last_updated` datetime DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `FK697F1642D085935` (`source_id`),
CONSTRAINT `FK697F1642D085935` FOREIGN KEY (`source_id`) REFERENCES `job_source` (`id`),
) ENGINE=InnoDB AUTO_INCREMENT=26567 DEFAULT CHARSET=latin1;
Thanks
The Hibernate documentation says:
No join, either implicit or explicit, can be specified in a bulk HQL query. Sub-queries can be used in the where-clause, where the subqueries themselves may contain joins.
lt.trade.volume is an implicit inner join between LocationTrade and Trade, so the query is invalid. You'll have to rewrite it to something like the following:
update LocationTrade lt set lt.hasVeryHighVolume=true where lt.locationIndices=?
and lt.id in (
select lt2.id from LocationTrade lt2 where lt2.trade.volume > 20000)
Or you'll have to use a SQL query instead.