Develop Reference

Flyway partial migration of legacy application

In an application with a custom database migrator which we want to replace with Flyway.
These migrations are split into some categories like "account" for user management and "catalog" for the product catalog.
Files are named $category.migration.$version.sql. Here, $category is one of the above categories and $version is an integer version starting from 0.
e.g. account.migration.23.sql
Although one could argue that each category should be a separate database, in fact it isn't and a major refactoring would be required to change that.
Also I could use one schema per category, but again this would require rewriting all SQL queries.
So I did the following:
Move $category.migration.$version.sql to /sql/$category/V$version__$category.sql (e.g. account.migration.1.sql becomes /sql/account/V1_account.sql)
Use a metadata table per category
set the baseline version to zero
In code that would be
String[] _categories = new String[] { "catalog", "account" };
for (String _category : _categories) {
Flyway _flyway = new Flyway();
_flyway.setDataSource(databaseUrl.getUrl(), databaseUrl.getUser(), databaseUrl.getPassword());
_flyway.setBaselineVersion(MigrationVersion.fromVersion("0"));
_flyway.setLocations("classpath:/sql/" + applicationName);
_flyway.setTarget(MigrationVersion.fromVersion(_version + ""));
_flyway.setTable(category + "_schema_version");
_flyway.setBaselineOnMigrate(true); // (1)
_flyway.migrate();
}
So there would be the metadata tables catalog_schema_version and account_schema_version.
Now the issue is as follows:
Starting with an empty database I would like to apply all pre-existing migrations per category, as done above.
If I remove _flyway.setBaselineOnMigrate(true); (1), then the catalog migration (the first one) succeeds, but it would complain for account that the schema public is not empty.
Likewise setting _flyway.setBaselineOnMigrate(true); causes the following behavior:
The migration of "catalog" succeeds but V0_account.sql is ignored and Flyway starts with V1_account.sql, maybe because it somehow still thinks the database was already baselined?
Does anyone have a a suggestion for resolving the problem?

Your easiest solution is to keep the schema_version tables in another schema each. I've answered a very similar question here.
Regarding your observation on baseline, those are expected traits. The migration of account starts at v1 because with the combination of baseline=0, baselineOnMigrate=true and a non empty target schema (because catalog has populated it) Flyway has determined this is a pre-existing database that is equal to the baseline - thus start at v1.

How to restrict ‘ this special character in textarea using regular expression in java servlet [duplicate]

I have the following string value: "walmart obama 👽💔"
I am using MySQL and Java.
I am getting the following exception: `java.sql.SQLException: Incorrect string value: '\xF0\x9F\x91\xBD\xF0\x9F...'
Here is the variable I am trying to insert into:
var1 varchar(255) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL`
My Java code that is trying to insert "walmart obama 👽💔" is a preparedStatement. So I am using the setString() method.
It looks like the problem is the encoding of the values 👽💔. How can I fix this? Previously I was using Derby SQL and the values 👽💔 just ended up being two sqaures (I think this is the representation of the null character)
All help is greatly appreciated!

What you have is EXTRATERRESTRIAL ALIEN (U+1F47D) and BROKEN HEART (U+1F494) which
are not in the basic multilingual plane. They cannot be even represented in java as one char, "👽💔".length() == 4. They are definitely not null characters and one will see squares if you are not using fonts that support them.
MySQL's utf8 only supports basic multilingual plane, and you need to use utf8mb4 instead:
For a supplementary character, utf8 cannot store the character at all,
while utf8mb4 requires four bytes to store it. Since utf8 cannot store
the character at all, you do not have any supplementary characters in
utf8 columns and you need not worry about converting characters or
losing data when upgrading utf8 data from older versions of MySQL.
So to support these characters, your MySQL needs to be 5.5+ and you need to use utf8mb4 everywhere. Connection encoding needs to be utf8mb4, character set needs to be utf8mb4 and collaction needs to be utf8mb4. For java it's still just "utf-8", but MySQL needs a distinction.
I don't know what driver you are using but a driver agnostic way to set connection charset is to send the query:
SET NAMES 'utf8mb4'
Right after making the connection.
See also this for Connector/J:
14.14: How can I use 4-byte UTF8, utf8mb4 with Connector/J?
To use 4-byte UTF8 with Connector/J configure the MySQL server with
character_set_server=utf8mb4. Connector/J will then use that setting
as long as characterEncoding has not been set in the connection
string. This is equivalent to autodetection of the character set.
Adjust your columns and database as well:
var1 varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NOT NULL
Again, your MySQL version needs to be relatively up-to-date for utf8mb4 support.

Weirdly, I found that REMOVING &characterEncoding=UTF-8 from the JDBC url did the trick for me with similar issues.
Based on my properties,
jdbc_url=jdbc:mysql://localhost:3306/dbName?useUnicode=true
I think this supports what #Esailija has said above, i.e. my MySQL, which is indeed 5.5, is figuring out its own favorite flavor of UTF-8 encoding.
(Note, I'm also specifying the InputStream I'm reading from as UTF-8 in the java code, which probably doesn't hurt)...

All in all, to save symbols that require 4 bytes you need to update characher-set and collation for utf8mb4:
database table/column:
alter table <some_table> convert to character set utf8mb4 collate utf8mb4_unicode_ci
database server connection (see)
On my development enviromnt for #2 I prefer to set parameters on command line when starting the server:
mysqld --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
btw, pay attention to Connector/J behavior with SET NAMES 'utf8mb4':
Do not issue the query set names with Connector/J, as the driver will not detect that the character set has changed, and will continue to use the character set detected during the initial connection setup.
And avoid setting characterEncoding parameter in connection url as it will override configured server encoding:
To override the automatically detected encoding on the client side, use the characterEncoding property in the URL used to connect to the server.

How I solved my problem.
I had
?useUnicode=true&characterEncoding=UTF-8
In my hibernate jdbc connection url and I changed the string datatype to longtext in database, which was varchar before.

Append the line useUnicode=true&characterEncoding=UTF-8 to your jdbc url.
In your case the data is not being send using UTF-8 encoding.

I faced the same issue and solved it by setting the Collation to utf8_general_ci for each column.

I guess MySQL doesn't believe this to be valid UTF8 text. I tried an insert on a test table with the same column definition (mysql client connection was also UTF8) and although it did the insert, the data I retrieved with the MySQL CLI client as well as JDBC didn't retrieve the values correctly. To be sure UTF8 did work correctly, I inserted an "ö" instead of an "o" for obama:
johan#maiden:~$ mysql -vvv test < insert.sql
--------------
insert into utf8_test values(_utf8 "walmart öbama 👽💔")
--------------
Query OK, 1 row affected, 1 warning (0.12 sec)
johan#maiden:~$ file insert.sql
insert.sql: UTF-8 Unicode text
Small java application to test with:
package test.sql;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
public class Test
{
public static void main(String[] args)
{
System.out.println("test string=" + "walmart öbama 👽💔");
String url = "jdbc:mysql://hostname/test?useUnicode=true&characterEncoding=UTF-8";
try
{
Class.forName("com.mysql.jdbc.Driver").newInstance();
Connection c = DriverManager.getConnection(url, "username", "password");
PreparedStatement p = c.prepareStatement("select * from utf8_test");
p.execute();
ResultSet rs = p.getResultSet();
while (!rs.isLast())
{
rs.next();
String retrieved = rs.getString(1);
System.out.println("retrieved=\"" + retrieved + "\"");
}
}
catch (Exception e)
{
e.printStackTrace();
}
}
}
Output:
johan#appel:~/workspaces/java/javatest/bin$ java test.sql.Test
test string=walmart öbama 👽💔
retrieved="walmart öbama "
Also, I've tried the same insert with the JDBC connection and it threw the same exception you are getting.
I believe this to be a MySQL bug. Maybe there's a bug report about such a situation already..

I had kind of the same problem and after going carefully against all charsets and finding that they were all right, I realized that the bugged property I had in my class was annotated as #Column instead of #JoinColumn (javax.presistence; hibernate) and it was breaking everything up.

execute
show VARIABLES like "%char%”;
find character-set-server if is not utf8mb4.
set it in your my.cnf, like
vim /etc/my.cnf
add one line
character_set_server = utf8mb4
at last restart mysql

This setting useOldUTF8Behavior=true worked fine for me. It gave no incorrect string errors but it converted special characters like à into multiple characters and saved in the database.
To avoid such situations, I removed this property from the JDBC parameter and instead converted the datatype of my column to BLOB. This worked perfect.

Besides,data type can use blob install of varchar or text.

UTF-8 won't persist on Hibernate + MySQL

I'm trying to save some values in MySQL database by using Hibernate, but most Lithuanian characters won't get saved, including ąĄ čČ ęĘ ėĖ įĮ ųŲ ūŪ(they are saved as ?), however, šŠ žŽ do get saved.
If I do inserts manually, then those values are properly saved, so the problem is most likely in Hibernate configuration.
What I have tried so far:
hibernate.charset=UTF-8
hibernate.character_encoding=UTF-8
hibernate.use_unicode=true
---------
properties.put(PROPERTY_NAME_HIBERNATE_USE_UNICODE,
env.getRequiredProperty(PROPERTY_NAME_HIBERNATE_USE_UNICODE));
properties.put(PROPERTY_NAME_HIBERNATE_CHARSET,
env.getRequiredProperty(PROPERTY_NAME_HIBERNATE_CHARSET));
properties
.put(PROPERTY_NAME_HIBERNATE_CHARACTER_ENCODING,
env.getRequiredProperty(PROPERTY_NAME_HIBERNATE_CHARACTER_ENCODING));
---------
private void registerCharachterEncodingFilter(ServletContext aContext) {
CharacterEncodingFilter cef = new CharacterEncodingFilter();
cef.setForceEncoding(true);
cef.setEncoding("UTF-8");
aContext.addFilter("charachterEncodingFilter", cef)
.addMappingForUrlPatterns(null, true, "/*");
}
As described here
I tried adding ?useUnicode=true&characterEncoding=utf-8 to db connection url.
As described here
I ensured that my db is set to UTF-8 charset. phpmyadmin > information_schema > schemata
def db_name utf8 utf8_lithuanian_ci NULL
This is how I save into db:
//Controller
buildingService.addBuildings(schema.getBuildings());
List<Building> buildings = buildingService.getBuildings();
System.out.println("-----------");
for (Building b : schema.getBuildings()) {
System.out.println(b.toString());
}
System.out.println("-----------");
for (Building b : buildings) {
System.out.println(b.toString());
}
System.out.println("-----------");
//Service:
#Override
public void addBuildings(List<Building> buildings) {
for (Building b : buildings) {
getCurrentSession().saveOrUpdate(b);
}
}
First set of println contains all Lithuanian characters, while second replaces most with ?
EDIT: Added details
insert into buildings values (11,'ąĄčČęĘ', 'asda');
select short, hex(short) from buildings;
//Šalt. was inserted via hibernate
//letters are properly displayed:
ąĄčČęĘ | C485C484C48DC48CC499C498
MIF Šalt. | 4D494620C5A0616C742E
select address, hex(address) from buildings;
Šaltini? <...> | C5A0616C74696E693F20672E2031412C2056696C6E697573
//should contain "ų"
--------
show create table buildings;
buildings | CREATE TABLE `buildings` (
`id` int(11) NOT NULL,
`short` varchar(255) COLLATE utf8_lithuanian_ci DEFAULT NULL,
`address` varchar(255) COLLATE utf8_lithuanian_ci DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_lithuanian_ci
EDIT:
I did not find a proper solution, so I came up with a workaround. I ended up escaping/unescaping characters, storing them like this: \uXXXX.

Let's verify that they were stored correctly... Please do SELECT col, HEX(col) ... to fetch some cell with Lithuanian characters. A correctly stored ą will show C485. The others should show various hex values of C4xx or C5xx. 3F is ?.
But, more importantly, 4 characters do show. Š should be C5A0 if properly stored as utf8. However, I suspect, you will see 8A, implying that the column in the table is really declared as CHARACTER SET latin1. (The 4 characters show up in the first column of my charset blog ).
Do SHOW CREATE TABLE to see how the column is defined. If it says latin1, then the problem is with the table definition, and you probably ought to start over.

You have to ensure that every component taking part in data entry uses UTF-8 encoding explicitly.
If you enter the values via browser, make sure that the
page displaying the results with the following header
Content-Type: text/html; charset=utf-8.
The input form is defined as follows
<form action="submit" accept-charset="UTF-8">...</form>.
If you are creating String objects from byte array, make sure you
explicitly state the Charset in the constructor.
If your entry happens from a text file, that file has to be UTF-8
encoded.
If it is hardcoded directly in your code, then the source has to be
UTF-8 encoded.

The fact that your DB holds correct UTF-8 (two or more bytes for a special letter) is reassuring.
If you get one single ? for a special letter, it was attempted to do a UTF-8 conversion to some encoding that does not contain those letters. And that seems to be the case. The letters that are converted correctly are in the ISO-8859-1 or Windows-1252 range. The others are not.
Now ISO-88591-1 aka Latin-1 is the default HTTP encoding, default in java EE server. You might like to do before writing:
response.setCharacterEncoding("UTF-8");
Now one problem with System.out.println is that it uses the system default encoding. Logging to a file with a logger is more interesting. Or debugging and inspecting the String and its char array.
That the schema does seemingly work, may be that the schema Strings stem immediately from a Java source, and the editor encoding and javac compiler encoding differ. This can be checked by u-escaping the string literals in java: "\u0105" instead of "ą".
Make a unit test that writes and reads from the database.

Backslash in Jooq/PostgreSQL INSERT clause

I am trying to use Jooq to do an INSERT into a PostgreSQL database. The query fails if the String includes a backslash character with SQL state code: 42601 which means SYNTAX ERROR.
Jooq: 3.4.4
postgresql driver: 8.4-702.jdbc4
PostgreSQL: "PostgreSQL
8.4.20 on x86_64-redhat-linux-gnu, compiled by GCC gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-4), 64-bit"
JDK 1.8.0_25
Spring Tool Suite 3.6.0.RELEASE
Database:
CREATE TABLE datahub.test (
body TEXT NOT NULL
);
Jooq code generated using maven:
jooq-codegen-maven version 3.4.4
generator.name: org.jooq.util.DefaultGenerator
generator.database.name: org.jooq.util.postgres.PostgresDatabase
Unit test
#RunWith(SpringJUnit4ClassRunner.class)
#ContextConfiguration(locations = {"/spring-config.xml"})
public class BatchExceptionJooqTest {
private static Logger log = LogManager.getLogger(BatchExceptionJooqTest.class);
#Autowired
private DSLContext db;
#Test
public void runBasicJooqTest(){
try{
final List<InsertQuery<TestRecord>> batchUpdate = Lists.newLinkedList();
InsertQuery<TestRecord> insertQuery = db.insertQuery(TEST);
insertQuery.addValue(TEST.BODY, "It's a bit more complicated than just doing copy and paste... :\\");
batchUpdate.add(insertQuery);
db.batch(batchUpdate).execute();
}catch(Exception e){
log.error(e);
}
}
}
Problem
The test fails with an exception:
2014-12-26 17:11:16,490 [main] ERROR BatchExceptionJooqTest:36 :runBasicJooqTest - org.jooq.exception.DataAccessException: SQL [null]; Batch entry 0 insert into "datahub"."test" ("body") values ('It''s a bit more complicated than just doing copy and paste... :\') was aborted. Call getNextException to see the cause.
The test passes, if instead of String: "It's a bit more complicated than just doing copy and paste... :\\" I use String: "It's a bit more complicated than just doing copy and paste... :\\\\". This seems a bit inconsistent when compared to what is happening to the the single quote during the operation. It is correctly doubled so as to get through the SQL parser. Not so with the backslash.
I read somewhere that escaping a backslash with another backslash is not part of the SQL standard and Postgre has changed its default behavior lately. However I am not clear on the meaning of the manual p 4.1.2.2 - it seems to indicate that double backslashes should work and there is not really any reason for jooq not to do it.
So.. could someone please explain if the described situation in Jooq:
Is desired behavior and there is no workaround besides doubling all incoming backslashes my application is processing?
Is desired behavior but there is a configuration change I can do to make Jooq process the backslashes in a similar manner to the single quotes?
Is it a bug?
What am I doing incorrectly?
Thank you

You are using PostgreSQL 8.x. In that version, the system defaulted to accepting backslash escaped string literals even without the preceding E.
To avoid this, you should set the server configuration variable standard_conforming_strings to ON.
It is, of course, strongly recommended that you migrate to a version of PostgreSQL higher than 8.x, as the 8.x versions have reached end-of-life and are no longer supported.

jOOQ 3.5 has introduced org.jooq.conf.Settings.backslashEscaping (https://github.com/jOOQ/jOOQ/issues/3000). This was mainly introduced for MySQL, which still today defaults to non-standards compliant string literal escaping using backslashes.
Note that this setting affects only inlined bind values, so it will not escape backslashes when binding values to a PreparedStatement.
I agree with RealSkeptic's answer, which suggests you change the database behaviour or upgrade to a newer PostgreSQL version.

How to ignore placeholder expressions for Flyway?

I am using flyway version 2.3, I have an sql patch which inserts a varchar into a table having character sequence that Flyway treats as placeholders. I want to flyway to ignore placeholders and run the script as is.
The script file is
insert into test_data (value) values ("${Email}");
And the Java code is
package foobar;
import com.googlecode.flyway.core.Flyway;
public class App
{
public static void main( String[] args )
{
// Create the Flyway instance
Flyway flyway = new Flyway();
// Point it to the database
flyway.setDataSource("jdbc:mysql://localhost:3306/flywaytest", "alpha", "beta");
// Start the migration
flyway.migrate();
}
}

This can be done by splitting $ and { in the expression:
insert into test_data (value) values ('$' || '{Email}')

You can change the value of the placeholder suffix or prefix to a different value and you should be OK.

try this properties:
final var flyway = Flyway.configure()
.dataSource(DataSourceProvider.getInstanceDataSource())
.locations("path")
.outOfOrder(true)
.validateOnMigrate(false)
.placeholderReplacement(false)
.load();

In my MySQL migration script this worked:
I just escaped the first { characters, like this:
'...<p>\nProgram name: $\{programName}<br />\nStart of studies: $\{startOfStudies}<br />\n($\{semesterNote})\n</p>...'
This way flyway didn't recognize them as placeholders, and the string finally stored doesn't contain the escape character.
...<p>
Program name: ${programName}<br />
Start of studies: ${startOfStudies}<br />
(${semesterNote})
</p>...

I had exactly the same problem, but the accepted answer didn't fit my requirements. So I solved the problem in another way and post this answer hoping that it'll be useful to other people coming here from Google search.
If you cannot change the placeholder suffix and prefix, you can trick Flyway into believing there are no placeholders by using an expression. E.g.:
INSERT INTO test_data(value) VALUES (REPLACE("#{Email}", "#{", "${"));
This is useful if you've already used placeholders in lots of previous migrations. (If you just change placeholder suffix and prefix, you'll have to change them in previous migration scripts, too. But then the migration script checksums won't match, Flyway will rightfully complain, and you'll have to change checksums in the schema_version table by calling Flyway#repair() or manually altering the table.)

Just add a property to your bootstrap.properties (or whatever you use)
flyway.placeholder-replacement = false

In 2021, the simple answer is to set the placeholderReplacement boolean to false:
flyway -placeholderReplacement="false"
The configuration parameter placeholderReplacement determines whether placeholders should be replaced.
Reference: https://flywaydb.org/documentation/configuration/parameters/placeholderReplacement