Develop Reference

Fix liquibase and Orient DB compatibility

I upgraded liquibase version in my Spring Boot App from 3.10.2 to 4.17.2 and I'm getting error
Suppressed: com.orientechnologies.orient.core.sql.OCommandSQLParsingException: Error parsing query: CREATE TABLE DATABASECHANGELOGLOCK (ID INT NOT NULL, LOCKED BOOLEAN NOT NULL, LOCKGRANTED datetime, LOCKEDBY VARCHAR(255), CONSTRAINT PK_DATABASECHANGELOGLOCK PRIMARY KEY (ID)) Encountered " <CREATE> "CREATE "" at line 1, column 1.Was expecting one of:
<LET> ...
<EXPLAIN> ...
DB name="mdp"
Error Code="1"
at com.orientechnologies.orient.core.sql.parser.OStatementCache.throwParsingException(OStatementCache.java:155)
at com.orientechnologies.orient.core.sql.parser.OStatementCache.parse(OStatementCache.java:147)
at com.orientechnologies.orient.core.sql.parser.OStatementCache.get(OStatementCache.java:86)
at com.orientechnologies.orient.core.sql.parser.OStatementCache.get(OStatementCache.java:65)
at com.orientechnologies.orient.core.sql.OSQLEngine.parse(OSQLEngine.java:80)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentEmbedded.command(ODatabaseDocumentEmbedded.java:650)
at com.orientechnologies.orient.server.OConnectionBinaryExecutor.executeQuery(OConnectionBinaryExecutor.java:1489)
at com.orientechnologies.orient.client.remote.message.OQueryRequest.execute(OQueryRequest.java:143)
at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.sessionRequest(ONetworkProtocolBinary.java:355)
at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.execute(ONetworkProtocolBinary.java:239)
at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:67)
As I can see liquibase can not create his own tables. I started getting this error after version update.
Can somebody tell me what can be a reason and how can I fix it. I can provide any information about my app whatever you need. Thanks!

Spring boot testing with liquibase fails

I have been trying for quite some time to figure out a solution for my problem, to no avail.
Anyway, i have a bunch of integration tests (in a nonstandard directory testRegression parallel to the standard test directory).
These integration tests use an h2 in memory database. In production as well as for testing i am using liquibase to simulate the schema evolution.
My properties (in application-testRegession.properties) look as follows:
spring.liquibase.enabled=true
spring.liquibase.user=sa
spring.liquibase.password=
spring.liquibase.change-log=classpath:/liquibase/changelog-master.xml
spring.datasource.url=jdbc:p6spy:h2:mem:testdb;MODE=PostgreSQL;DB_CLOSE_DELAY=-1;DATABASE_TO_UPPER=FALSE;INIT=CREATE SCHEMA IF NOT EXISTS nmc\\;CREATE SCHEMA IF NOT EXISTS mkt\\;CREATE SCHEMA IF NOT EXISTS cdb\\;CREATE SCHEMA IF NOT EXISTS pg_temp
spring.datasource.driver-class-name=com.p6spy.engine.spy.P6SpyDriver
spring.datasource.username=sa
spring.datasource.password=
The error i consistenly keep getting is:
2020-07-21 15:57:34.173 INFO [liquibase.lockservice.StandardLockService] [Test worker:13]: Successfully acquired change log lock
2020-07-21 15:57:34.303 INFO [liquibase.changelog.StandardChangeLogHistoryService] [Test worker:13]: Creating database history table with name: PUBLIC.DATABASECHANGELOG
2020-07-21 15:57:34.305 INFO [liquibase.executor.jvm.JdbcExecutor] [Test worker:13]: CREATE TABLE PUBLIC.DATABASECHANGELOG (ID VARCHAR(255) NOT NULL, AUTHOR VARCHAR(255) NOT NULL, FILENAME VARCHAR(255) NOT NULL, DATEEXECUTED TIMESTAMP NOT NULL, ORDEREXECUTED INT NOT NULL, EXECTYPE VARCHAR(10) NOT NULL, MD5SUM VARCHAR(35), DESCRIPTION VARCHAR(255), COMMENTS VARCHAR(255), TAG VARCHAR(255), LIQUIBASE VARCHAR(20), CONTEXTS VARCHAR(255), LABELS VARCHAR(255), DEPLOYMENT_ID VARCHAR(10))
2020-07-21 15:57:34.307 INFO [liquibase.lockservice.StandardLockService] [Test worker:13]: Successfully released change log lock
2020-07-21 15:57:34.309 WARN [org.springframework.boot.web.servlet.context.AnnotationConfigServletWebServerApplicationContext] [Test worker:13]: Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liquibase' defined in class path resource [org/springframework/boot/autoconfigure/liquibase/LiquibaseAutoConfiguration$LiquibaseConfiguration.class]: Invocation of init method failed; nested exception is liquibase.exception.DatabaseException: Table "DATABASECHANGELOG" already exists; SQL statement:
CREATE TABLE PUBLIC.DATABASECHANGELOG (ID VARCHAR(255) NOT NULL, AUTHOR VARCHAR(255) NOT NULL, FILENAME VARCHAR(255) NOT NULL, DATEEXECUTED TIMESTAMP NOT NULL, ORDEREXECUTED INT NOT NULL, EXECTYPE VARCHAR(10) NOT NULL, MD5SUM VARCHAR(35), DESCRIPTION VARCHAR(255), COMMENTS VARCHAR(255), TAG VARCHAR(255), LIQUIBASE VARCHAR(20), CONTEXTS VARCHAR(255), LABELS VARCHAR(255), DEPLOYMENT_ID VARCHAR(10)) [42101-197] [Failed SQL: (42101) CREATE TABLE PUBLIC.DATABASECHANGELOG (ID VARCHAR(255) NOT NULL, AUTHOR VARCHAR(255) NOT NULL, FILENAME VARCHAR(255) NOT NULL, DATEEXECUTED TIMESTAMP NOT NULL, ORDEREXECUTED INT NOT NULL, EXECTYPE VARCHAR(10) NOT NULL, MD5SUM VARCHAR(35), DESCRIPTION VARCHAR(255), COMMENTS VARCHAR(255), TAG VARCHAR(255), LIQUIBASE VARCHAR(20), CONTEXTS VARCHAR(255), LABELS VARCHAR(255), DEPLOYMENT_ID VARCHAR(10))]
2020-07-21 15:57:34.309 INFO [com.zaxxer.hikari.HikariDataSource] [Test worker:13]: HikariPool-3 - Shutdown initiated...
2020-07-21 15:57:34.324 INFO [com.zaxxer.hikari.HikariDataSource] [Test worker:13]: HikariPool-3 - Shutdown completed.
2020-07-21 15:57:34.326 INFO [org.apache.catalina.core.StandardService] [Test worker:13]: Stopping service [Tomcat]
2020-07-21 15:57:34.342 INFO [org.springframework.boot.autoconfigure.logging.ConditionEvaluationReportLoggingListener] [Test worker:13]:
Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2020-07-21 15:57:34.345 ERROR [org.springframework.boot.SpringApplication] [Test worker:13]: Application run failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liquibase' defined in class path resource [org/springframework/boot/autoconfigure/liquibase/LiquibaseAutoConfiguration$LiquibaseConfiguration.class]: Invocation of init method failed; nested exception is liquibase.exception.DatabaseException: Table "DATABASECHANGELOG" already exists; SQL statement:
So how can i get around this issue? My basic understanding is that each test class creates its own ApplicationContext. For that it creates and loads a liquibase bean into it.
However, this problem occurs only for 2 out of 42 tests.
I would really like to get to the bottom of this and understand whats going on.
Can anyone shed light on my problem?
ADDITIONALLY
The test all run fine individually, but when run as a group they fail.
UPDATE 1
The relevant properties are as follows:
spring.main.allow-bean-definition-overriding=true
spring.datasource.url=jdbc:p6spy:h2:mem:testdb;MODE=PostgreSQL;DB_CLOSE_DELAY=-1;DATABASE_TO_UPPER=FALSE;INIT=CREATE SCHEMA IF NOT EXISTS nmc\\;CREATE SCHEMA IF NOT EXISTS mkt\\;CREATE SCHEMA IF NOT EXISTS cdb\\;CREATE SCHEMA IF NOT EXISTS pg_temp
spring.datasource.driver-class-name=com.p6spy.engine.spy.P6SpyDriver
spring.datasource.username=sa
spring.datasource.password=
spring.datasource.hikari.connectionTimeout=10000
spring.datasource.hikari.idleTimeout=60000
spring.datasource.hikari.maxLifetime=180000
spring.datasource.hikari.maximumPoolSize=50
spring.h2.console.enabled=true
spring.h2.console.path=/h2-console
My configuration is:
#Configuration
#ComponentScan(
basePackages = {
"com.aareal.nmc"
},
excludeFilters = {
#ComponentScan.Filter(type = FilterType.ASSIGNABLE_TYPE, value = CommandLineRunner.class)
})
#EnableTransactionManagement
#Profile("testRegression")
#SpringBootApplication(exclude = SecurityAutoConfiguration.class)
#EnableConfigurationProperties(LiquibaseProperties.class)
public class RegressionTestConfig {
My two tests are annotated as:
#RunWith(SpringRunner.class)
#SpringBootTest(
classes = {
RegressionTestConfig.class
},
//webEnvironment = SpringBootTest.WebEnvironment.DEFINED_PORT)
webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
Thanks

I had the same issue, and it seems to have been caused by case sensitive checking of the database table name. That is, the table was created as 'DATABASECHANGELOG', but Liquibase was checking for the existence of 'databasechangelog'.
The fix (at least for an H2 database) is to specify case insensitive identifiers in the database URL. For example:
jdbc:h2:mem:~/mydb;CASE_INSENSITIVE_IDENTIFIERS=TRUE
Explanation: The Spring test process fires up one or more instances of the Spring container to run tests on. If it thinks the config is exactly the same for two tests it will re-use an instance, otherwise it will start a new one. The instances are shared to avoid needing to start a whole new Springboot application for each test. But, the problem is that the instances may share some resources, such as databases and network ports. Therefore errors might occur from trying to start multiple instances at the same time. In this case, the test suite is starting two instances using the same database, but the second one is trying to re-run the whole Liquibase setup because the case sensitive issue means it doesn't see the table has already been created.

For my special case (that is for internal testing only, not production) what i have the following:
src
|-- main
|-- test
|-- testRegression
Workaround
Decide on the version of liquibase to use (i chose 4.0.0, which is the most recent at this point)
Create a file "src/testRegression/java/liquibase/changelog/StandardChangeLogHistoryService.java"
Open the original liquibase file "StandardChangeLogHistoryService.java" (mine is in ~//.gradle/caches/modules-2/files-2.1/org.liquibase/liquibase-core/4.0.0/23a5317eb5005b4765cd85e6f3a2cc4bb55c0daa/liquibase-core-4.0.0-sources.jar which i copied and unzipped)
and copy its contents 1:1 into the newly created file in 2.
Add a catch block by changing code (around line 396) from
if (SqlGeneratorFactory.getInstance().supports(sql, database)) {
executor.execute(sql);
getDatabase().commit();
} else {
to
if (SqlGeneratorFactory.getInstance().supports(sql, database)) {
try {
executor.execute(sql);
getDatabase().commit();
} catch (DatabaseException excptn) {
Scope.getCurrentScope()
.getLog(getClass())
.warning(
"Table '"
+ getDatabase()
.escapeTableName(
getLiquibaseCatalogName(),
getLiquibaseSchemaName(),
getDatabaseChangeLogTableName())
+ "' already exists.");
}
} else {
This is simply a workaround since there could be legitimate reasons for the ChangeLogTable table creation to fail. However, it already existing should not be cause for a major failure in my opinion.
My current view is that this is something which ought to be addressed/fixed in the official liquibase code base.
The following post(s) were helpful:
https://github.com/liquibase/liquibase-cache/issues/1

H2 Database - "Unknown data type" with empty data type

I'm getting a very weird error from H2 where it tells me that it doesn't know a data type but doesn't tell me which one it is.
That's the error message:
Unknown data type: ; SQL statement:
CREATE TABLE bans (id INT NOT NULL AUTO_INCREMENT, player_id INT NOT NULL, operator_id INT NOT NULL, end DATETIME NULL, reason VARCHAR(1024) NOT NULL, PRIMARY KEY (id), INDEX (player_id), INDEX (end), FOREIGN KEY (player_id) REFERENCES players(id), FOREIGN KEY (operator_id) REFERENCES players(id)) [50004-200]
and this is the SQL query in plain:
CREATE TABLE bans (id INT NOT NULL AUTO_INCREMENT, player_id INT NOT NULL, operator_id INT NOT NULL, end DATETIME NULL, reason VARCHAR(1024) NOT NULL, PRIMARY KEY (id), INDEX (player_id), INDEX (end), FOREIGN KEY (player_id) REFERENCES players(id), FOREIGN KEY (operator_id) REFERENCES players(id))
I neither understand what H2 is trying to tell me, nor do I see what's wrong with that query. I tried playing around with the spacing but to no avail.
Edit 1:
I'm opening the database connection with this JDBC string:
jdbc:h2:%s;AUTO_SERVER=%s;DATABASE_TO_UPPER=FALSE
(And using String.format to set the two appropriate values. First one naturally being the base file name and the second either TRUE or FALSE (either work).)
I am not changing any other settings or enabling any other modes. The only thing I do is create 2 tables before and inserting a bit of data into them.

You cannot use non-standard INDEX(player_id) and INDEX(end) in H2 without a MySQL compatibility mode. (Actually database indexes aren't covered by the Standard.)
Either use a MySQL compatibility mode, or use a separate CREATE INDEX command such as
CREATE INDEX ON bans(end);
Index on player_id column is not needed, because non-unique index will be created automatically by H2 for constraint FOREIGN KEY (player_id) REFERENCES players(id).
Please also note that behavior of DATABASE_TO_UPPER=FALSE was changed since 1.4.198. With this setting all column names are case-sensitive. You may want to use DATABASE_TO_LOWER=TRUE instead.

Another thing to watch out for are forbidden characters in column names.
e.g. if you use a column name such as #Column(name="my-column") then you'll actually see the exact same error message.
Unknown data type: ; SQL statement:
When you're really stuck, connect to the console webpage of the database, and try to execute the query there. (There's usually a query in the exception message).
If you run an embedded H2 database, then you can enable the console webpage by setting the following properties:
spring.datasource.url=jdbc:h2:mem:testdb
spring.datasource.driverClassName=org.h2.Driver
spring.datasource.username=sa
spring.datasource.password=password
spring.jpa.database-platform=org.hibernate.dialect.H2Dialect
# view database on http://localhost:8080/h2-console
spring.h2.console.enabled=true

Sometimes you get this exception when you have the same function in H2 and in your dev DB, but the functionality is not the same:
For example CONVERT function in H2 and Orcacle.
In this case you can override the H2 function when you put this in your H2 URL: BUILTIN_ALIAS_OVERRIDE=1;
Then you have to create an alias in your test resources sql:
CREATE
ALIAS CONVERT FOR "com.example.config.H2Functions.convert";
Then create this class with the emethod in java:
public class H2Functions {
public static String convert(String param1, String param2) {
return StringUtils.stripAccents(param1);
}}
If you are using Junit5 you have to implement org.junit.jupiter.api.extension.BeforeEachCallback;
#Override
public void beforeEach(ExtensionContext context) throws Exception {
DataSource dataSource = getDataSource(context);
backupDatabase(dataSource);
}
private void backupDatabase(DataSource dataSource) {
try {
Connection connection = dataSource.getConnection();
Statement statement = connection.createStatement();
RunScript.execute(connection, IOUtils.getReader(H2Extension.class.getResourceAsStream("/h2_functions.sql")));
connection.close();
} catch (SQLException e) {
throw new RuntimeException(e);
}
}
And put these annotations on your test class:
#SpringBootTest
#ExtendWith({SpringExtension.class, H2Extension.class})

H2 SQL Syntax Error `expected "., COMMENT, ("` using Flyway w/ Spring Boot Test

I am using Spring Boot 2 with Data JPA and Flyway on a Postgres database. Everything works fine in production. Now I'm trying to write tests that will run on an H2 Embedded Database for testing purposes. But, the tests encounter a SQL syntax error. But, I don't understand what's wrong with the syntax:
CREATE TABLE mysite_user (
id int8 NOT NULL,
thirdparty_user_id varchar(255) NULL,
email varchar(255) NULL,
first_name varchar(255) NULL,
PRIMARY KEY (id)
);
CREATE INDEX mysite_user_thirdparty_user_id_idx ON mysite_user USING btree (thirdparty_user_id) ;
This is the error:
Migration V1__Initial.sql failed
--------------------------------
SQL State : 42001
Error Code : 42001
Message : Syntax error in SQL statement "CREATE INDEX MYSITE_USER_THIRDPARTY_USER_ID_IDX ON MYSITE_USER USING[*] BTREE (THIRDPARTY_USER_ID) "; expected "., COMMENT, ("; SQL statement:
CREATE INDEX mysite_user_thirdparty_user_id_idx ON mysite_user USING btree (thirdparty_user_id) [42001-199]
Location : db/migration/V1__Initial.sql (/Users/me/Development/mysite-website/target/classes/db/migration/V1__Initial.sql)
Line : 20
Statement : CREATE INDEX mysite_user_thirdparty_user_id_idx ON mysite_user USING btree (thirdparty_user_id)
What am I doing wrong?

Arguably, what you're doing wrong is using different databases between prod and test. As you've seen, your tests end up having to be coded around the differences between the databases, as well as potentially overlooking subtle edge cases which work in one db and not the other.
If you want to go down this route it is possible - see Best way for "database specific" sql scripts with Flyway

H2 in memory database is getting confused between column name, and the value I want to use

Context : Trying to test a spring boot application, trying to initialize data via the data.sql file in the src/test/resources/ directory
I have the following in memory db :
spring.datasource.url=jdbc:h2:mem:db;DB_CLOSE_DELAY=-1;DATABASE_TO_UPPER=false;
I also have the following SQL table :
CREATE TABLE `roles` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`name` varchar(60) DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `UK_nb4h0p6txrmfc0xbrd1kglp9t` (`name`)
) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=utf8;
and here is my insert statement (in data.sql)
INSERT INTO roles(name) VALUES ("ROLE_USER");
When I run my spring unit tests, I get this error :
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactory' defined in class path resource [org/springframework/boot/autoconfigure/orm/jpa/HibernateJpaConfiguration.class]: Initialization of bean failed; nested exception is org.springframework.jdbc.datasource.init.ScriptStatementFailedException: Failed to execute SQL script statement #2 of URL [file:/Users/me/IdeaProjects/proj/out/test/resources/data.sql]: INSERT INTO roles(name) VALUES ("ROLE_USER"); nested exception is org.h2.jdbc.JdbcSQLSyntaxErrorException: Column "ROLE_USER" not found; SQL statement:
INSERT INTO roles(name) VALUES ("ROLE_USER") [42122-198]
To me, everything looks correct... I'm using a simple insert statement, on a table which definitely exists... and I am inserting a value which corresponds well with the column definition. But for some reason, it throws that error saying Column "ROLE_USER" does not exist, and I'm not sure why.

Ah! I found the solution - turns out that the SQL statement needed single quotes for the values... so the correct way would be :
INSERT INTO roles(name) VALUES ('ROLE_USER');
instead of :
INSERT INTO roles(name) VALUES ("ROLE_USER");