I am using flyway version 2.3, I have an sql patch which inserts a varchar into a table having character sequence that Flyway treats as placeholders. I want to flyway to ignore placeholders and run the script as is.
The script file is
insert into test_data (value) values ("${Email}");
And the Java code is
package foobar;
import com.googlecode.flyway.core.Flyway;
public class App
{
public static void main( String[] args )
{
// Create the Flyway instance
Flyway flyway = new Flyway();
// Point it to the database
flyway.setDataSource("jdbc:mysql://localhost:3306/flywaytest", "alpha", "beta");
// Start the migration
flyway.migrate();
}
}
This can be done by splitting $ and { in the expression:
insert into test_data (value) values ('$' || '{Email}')
You can change the value of the placeholder suffix or prefix to a different value and you should be OK.
try this properties:
final var flyway = Flyway.configure()
.dataSource(DataSourceProvider.getInstanceDataSource())
.locations("path")
.outOfOrder(true)
.validateOnMigrate(false)
.placeholderReplacement(false)
.load();
In my MySQL migration script this worked:
I just escaped the first { characters, like this:
'...<p>\nProgram name: $\{programName}<br />\nStart of studies: $\{startOfStudies}<br />\n($\{semesterNote})\n</p>...'
This way flyway didn't recognize them as placeholders, and the string finally stored doesn't contain the escape character.
...<p>
Program name: ${programName}<br />
Start of studies: ${startOfStudies}<br />
(${semesterNote})
</p>...
I had exactly the same problem, but the accepted answer didn't fit my requirements. So I solved the problem in another way and post this answer hoping that it'll be useful to other people coming here from Google search.
If you cannot change the placeholder suffix and prefix, you can trick Flyway into believing there are no placeholders by using an expression. E.g.:
INSERT INTO test_data(value) VALUES (REPLACE("#{Email}", "#{", "${"));
This is useful if you've already used placeholders in lots of previous migrations. (If you just change placeholder suffix and prefix, you'll have to change them in previous migration scripts, too. But then the migration script checksums won't match, Flyway will rightfully complain, and you'll have to change checksums in the schema_version table by calling Flyway#repair() or manually altering the table.)
Just add a property to your bootstrap.properties (or whatever you use)
flyway.placeholder-replacement = false
In 2021, the simple answer is to set the placeholderReplacement boolean to false:
flyway -placeholderReplacement="false"
The configuration parameter placeholderReplacement determines whether placeholders should be replaced.
Reference: https://flywaydb.org/documentation/configuration/parameters/placeholderReplacement
Related
Can I generate Liquibase changelog from a DB based on table name prefix.
Example:
If I have a DB schema and it has following tables:
abc
abcd
abcdef
xyz
I just want to generate ChangeLog for tables starting with "abc". So changelog for tables
abc,
abcd,
abcdef
Can someone help me if there's a way to do this?
It's possible with maven or liquibase command line if you're using liquibase version > 3.3.2.
Take a look at the release notes
Liquibase 3.3.2 is officially released. It is primarily a bugfix
release, but has one major new feature: object
diffChangeLog/generateChangeLog object filtering.
includeObjects/excludeObjects logic
You can now set an includeObjects or excludeObjects paramter on the
command line or Ant. For maven, the parameteres are diffExcludeObjects
and diffIncludeObjects. The format for these parameters are:
An object name (actually a regexp) will match any object whose name matches the regexp.
A type:name syntax that matches the regexp name for objects of the given type
If you want multiple expressions, comma separate them
The type:name logic will be applied to the tables containing columns, indexes, etc.
NOTE: name comparison is case sensitive. If you want insensitive
logic, use the (?i) regexp flag.
Example Filters:
“table_name” will match a table called “table_name” but not “other_table” or “TABLE_NAME”
“(i?)table_name” will match a table called “table_name” and “TABLE_NAME”
“table_name” will match all columns in the table table_name
“table:table_name” will match a table called table_name but not a column named table_name
“table:table_name, column:*._lock” will match a table called table_name and all columns that end with “_lock”
So try using excludeObjects or includeObjects parameters with generateChangeLog command
UPDATE
I've used liquibase command line, and this command does the trick (for mysql database):
liquibase
--changeLogFile=change.xml
--username=username
--password=password
--driver=com.mysql.cj.jdbc.Driver
--url=jdbc:mysql://localhost:3306/mydatabase
--classpath=mysql-connector-java-8.0.18.jar
--includeObjects="table:abc.*"
generateChangeLog
This works for me on Windows 10:
liquibase.properties:
changeLogFile=dbchangelog.xml
classpath=C:/Program\ Files/liquibase/lib/mysql-connector-java-8.0.20.jar
driver=com.mysql.cj.jdbc.Driver
url=jdbc:mysql://localhost:3306/liquibase?serverTimezone=UTC
username=root
password=password
schemas=liquibase
includeSchema=true
includeTablespace=true
includeObjects=table:persons
C:\Users\username\Desktop>liquibase generateChangeLog
Liquibase Community 4.0.0 by Datical
Starting Liquibase at 11:34:35 (version 4.0.0 #19 built at 2020-07-13 19:45+0000)
Liquibase command 'generateChangeLog' was executed successfully.
You can download mysql-connector here, find the generateChangeLog documentation here and more information on includeObjects here.
I am currently using spring-data-jpa version 1.9.4.
I have a MySql table with columns project(integer), summary(varchar), and description(varchar).
I have a regex that I would like to use to search the summary and/or description field meaning that if it finds it in summary does not need to apply regex to description.
The repository method I am attempting to use is:
List<Issue> findByProjectAndSummaryOrDescriptionRegex(long project, String regex)
The error I am receiving is:
java.lang.IllegalArgumentException: Unsupported keyword REGEX (1):
[MatchesRegex, Matches, Regex]
It is difficult in my company environment to update/upgrade versions, so if the issue is NOT my syntax but rather the then if someone knows which version now supports 'Regex' for query derivation or where I could find that specific information I would be grateful. I have looked at the Changelog and it appears that 1.9.4 should support but it appears not.
Thanks for your help!
JD
EDIT 1: I am aware of the #Query annotation but have been asked by my lead to only use that as a last resort if I cannot find the correct version which supports keyword REGEX [MatchesRegex, Matches, Regex]
I would recommend using native query (with #Query annotation) if the Spring data syntax does not work, e.g.:
#Query(nativeQuery=true, value="SELECT * FROM table WHERE project = ?1 AND (summary regexp ?2 OR description regexp ?2)")
List<Issue> findByProjectAndSummaryOrDescription(long project, String regex);
Update
If native query is not an option then (a) could you try it with single column and see if that works and (b) could you try by appending regex to both the columns, e.g.:
List<Issue> findByProjectAndDescriptionRegex(long project, String regex);
List<Issue> findByProjectAndSummaryRegexOrDescriptionRegex(long project, String regex, String regex);
In a followup, I discovered by doing some digging that the authoratative list will reside in the org.springframework.data.jpa.repository.query.JpaQueryCreator class. So for future folks that want to know which keywords from the 'Documented' list are ACTUALLY implemented, look inside JpaQueryCreator and you will the keywords supported as case arguments inside a switch!
Hope this helps!
PS - as suspected, REGEX was not supported in my version
try tu use #Query with param nativeQuery = true inside You can use database regexp_like function :
#Query(value = "select t.* from TABLE_NAME t where regexp_like(t.column, ?1)", nativeQuery = true)
Documentation :
https://www.techonthenet.com/oracle/regexp_like.php
In an application with a custom database migrator which we want to replace with Flyway.
These migrations are split into some categories like "account" for user management and "catalog" for the product catalog.
Files are named $category.migration.$version.sql. Here, $category is one of the above categories and $version is an integer version starting from 0.
e.g. account.migration.23.sql
Although one could argue that each category should be a separate database, in fact it isn't and a major refactoring would be required to change that.
Also I could use one schema per category, but again this would require rewriting all SQL queries.
So I did the following:
Move $category.migration.$version.sql to /sql/$category/V$version__$category.sql (e.g. account.migration.1.sql becomes /sql/account/V1_account.sql)
Use a metadata table per category
set the baseline version to zero
In code that would be
String[] _categories = new String[] { "catalog", "account" };
for (String _category : _categories) {
Flyway _flyway = new Flyway();
_flyway.setDataSource(databaseUrl.getUrl(), databaseUrl.getUser(), databaseUrl.getPassword());
_flyway.setBaselineVersion(MigrationVersion.fromVersion("0"));
_flyway.setLocations("classpath:/sql/" + applicationName);
_flyway.setTarget(MigrationVersion.fromVersion(_version + ""));
_flyway.setTable(category + "_schema_version");
_flyway.setBaselineOnMigrate(true); // (1)
_flyway.migrate();
}
So there would be the metadata tables catalog_schema_version and account_schema_version.
Now the issue is as follows:
Starting with an empty database I would like to apply all pre-existing migrations per category, as done above.
If I remove _flyway.setBaselineOnMigrate(true); (1), then the catalog migration (the first one) succeeds, but it would complain for account that the schema public is not empty.
Likewise setting _flyway.setBaselineOnMigrate(true); causes the following behavior:
The migration of "catalog" succeeds but V0_account.sql is ignored and Flyway starts with V1_account.sql, maybe because it somehow still thinks the database was already baselined?
Does anyone have a a suggestion for resolving the problem?
Your easiest solution is to keep the schema_version tables in another schema each. I've answered a very similar question here.
Regarding your observation on baseline, those are expected traits. The migration of account starts at v1 because with the combination of baseline=0, baselineOnMigrate=true and a non empty target schema (because catalog has populated it) Flyway has determined this is a pre-existing database that is equal to the baseline - thus start at v1.
What collation is available to H2 Database that does not ignore spaces but at the same time recognizes characters with umlauts and without as the same?
For example, it should treat "Ilkka Seppälä" and "Ilkka Seppala" as the same. It also needs to treat "MSaifAsif" and "M Saif Asif" as different (because of the spaces)
I found the answer to my question. To get my desired outcome to work, I had to do two things:
add icu4j as a dependency to the project which made H2 use the ICU4J collator.
testCompile 'com.ibm.icu:icu4j:55.1'
This is mentioned in the documentation H2 DB Reference - SET COLLATION . (It does not explain though the difference between the default collator and ICU4J's.
Add SET COLLATION ENGLISH STRENGTH PRIMARY to the JDBC url:
jdbc:h2:mem:test;MODE=MySQL;INIT=CREATE SCHEMA IF NOT EXISTS "public"\;SET COLLATION ENGLISH STRENGTH PRIMARY
A snippet of my unit test which works after adding ICU4J:
#Test
public void testUnicode() throws Exception {
Author authorWithUnicode = new Author();
authorWithUnicode.setName("Ilkka Seppälä");
authorRepository.save(authorWithUnicode);
Author authorWithSpaces = new Author();
authorWithSpaces.setName("M Saif Asif");
authorRepository.save(authorWithSpaces);
assertThat(authorRepository.findByName("Ilkka Seppälä").get()).isNotNull();
assertThat(authorRepository.findByName("Ilkka Seppala").get()).isNotNull();
assertThat(authorRepository.findByName("M Saif Asif").get()).isNotNull();
assertThat(authorRepository.findByName("MSaifAsif")).isEqualTo(Optional.empty());
}
Previously, without ICU4J, if H2 was initialized with SET COLLATION ENGLISH STRENGTH PRIMARY, the 4th assert would fail because it would treat the String with spaces as the same with the one without spaces. Without SET COLLATION, the second assert would fail because it would treat the name with letter "a" with umlaut as different from the one without.
I am trying to use Jooq to do an INSERT into a PostgreSQL database. The query fails if the String includes a backslash character with SQL state code: 42601 which means SYNTAX ERROR.
Jooq: 3.4.4
postgresql driver: 8.4-702.jdbc4
PostgreSQL: "PostgreSQL
8.4.20 on x86_64-redhat-linux-gnu, compiled by GCC gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-4), 64-bit"
JDK 1.8.0_25
Spring Tool Suite 3.6.0.RELEASE
Database:
CREATE TABLE datahub.test (
body TEXT NOT NULL
);
Jooq code generated using maven:
jooq-codegen-maven version 3.4.4
generator.name: org.jooq.util.DefaultGenerator
generator.database.name: org.jooq.util.postgres.PostgresDatabase
Unit test
#RunWith(SpringJUnit4ClassRunner.class)
#ContextConfiguration(locations = {"/spring-config.xml"})
public class BatchExceptionJooqTest {
private static Logger log = LogManager.getLogger(BatchExceptionJooqTest.class);
#Autowired
private DSLContext db;
#Test
public void runBasicJooqTest(){
try{
final List<InsertQuery<TestRecord>> batchUpdate = Lists.newLinkedList();
InsertQuery<TestRecord> insertQuery = db.insertQuery(TEST);
insertQuery.addValue(TEST.BODY, "It's a bit more complicated than just doing copy and paste... :\\");
batchUpdate.add(insertQuery);
db.batch(batchUpdate).execute();
}catch(Exception e){
log.error(e);
}
}
}
Problem
The test fails with an exception:
2014-12-26 17:11:16,490 [main] ERROR BatchExceptionJooqTest:36 :runBasicJooqTest - org.jooq.exception.DataAccessException: SQL [null]; Batch entry 0 insert into "datahub"."test" ("body") values ('It''s a bit more complicated than just doing copy and paste... :\') was aborted. Call getNextException to see the cause.
The test passes, if instead of String: "It's a bit more complicated than just doing copy and paste... :\\" I use String: "It's a bit more complicated than just doing copy and paste... :\\\\". This seems a bit inconsistent when compared to what is happening to the the single quote during the operation. It is correctly doubled so as to get through the SQL parser. Not so with the backslash.
I read somewhere that escaping a backslash with another backslash is not part of the SQL standard and Postgre has changed its default behavior lately. However I am not clear on the meaning of the manual p 4.1.2.2 - it seems to indicate that double backslashes should work and there is not really any reason for jooq not to do it.
So.. could someone please explain if the described situation in Jooq:
Is desired behavior and there is no workaround besides doubling all incoming backslashes my application is processing?
Is desired behavior but there is a configuration change I can do to make Jooq process the backslashes in a similar manner to the single quotes?
Is it a bug?
What am I doing incorrectly?
Thank you
You are using PostgreSQL 8.x. In that version, the system defaulted to accepting backslash escaped string literals even without the preceding E.
To avoid this, you should set the server configuration variable standard_conforming_strings to ON.
It is, of course, strongly recommended that you migrate to a version of PostgreSQL higher than 8.x, as the 8.x versions have reached end-of-life and are no longer supported.
jOOQ 3.5 has introduced org.jooq.conf.Settings.backslashEscaping (https://github.com/jOOQ/jOOQ/issues/3000). This was mainly introduced for MySQL, which still today defaults to non-standards compliant string literal escaping using backslashes.
Note that this setting affects only inlined bind values, so it will not escape backslashes when binding values to a PreparedStatement.
I agree with RealSkeptic's answer, which suggests you change the database behaviour or upgrade to a newer PostgreSQL version.