Develop Reference

Insert/update to sql_ascii encoding postgreSQL

Here is a postgreSQL with server encoding SQL_ASCII. When I get data, I must use function convert_to(column1, 'SQL_ASCII') in select, and then use new String(value1, 'GBK') in java to get the right value.
But, when I send data by insert/update, the value in DB always error. Anyone can tell me how to send SQL including Chinese or other character by Java?
Apache DBCP config:
driverClassName=org.postgresql.Driver
url=jdbc:postgresql://127.0.0.1:5432/fxk_db_sql_ascii
username=test
password=test
initialSize=10
maxTotal=10
maxIdle=10
minIdle=5
maxWaitMillis=1000
removeAbandonedOnMaintenance=true
removeAbandonedOnBorrow=true
removeAbandonedTimeout=1
connectionProperties=useUnicode=true;characterEncoding=SQL_ASCII;allowEncodingChanges=true
SQL query in java:
String sql = "select user_id, first_name as first_name, convert_to(first_name, 'sql_ascii') as first_name1, last_name as last_name, convert_to(last_name, 'sql_ascii') as last_name1 from public.tbl_users";
ResultSet rs = stmt.executeQuery(sql);
List<Map<String, Object>> list = new ArrayList<Map<String, Object>>();
ResultSetMetaData md = rs.getMetaData();
int columnCount = md.getColumnCount();
while (rs.next()) {
Map<String, Object> rowData = new HashMap<String, Object>();
for (int i = 1; i <= columnCount; i++) {
rowData.put(md.getColumnName(i), rs.getObject(i)==null?"":new String(rs.getBytes(i),"GBK"));
}
list.add(rowData);
}
rs.close();
But how should I do while insert/update?

Avoid SQL_ASCII
You should be using a server encoding of UTF8 rather than SQL_ASCII.
The documentation is quite clear about this matter, and even includes a warning to not do what you are doing. To quote (emphasis mine):
The SQL_ASCII setting behaves considerably differently from the other settings. When the server character set is SQL_ASCII, the server interprets byte values 0-127 according to the ASCII standard, while byte values 128-255 are taken as uninterpreted characters. No encoding conversion will be done when the setting is SQL_ASCII. Thus, this setting is not so much a declaration that a specific encoding is in use, as a declaration of ignorance about the encoding. In most cases, if you are working with any non-ASCII data, it is unwise to use the SQL_ASCII setting because PostgreSQL will be unable to help you by converting or validating non-ASCII characters.
Use UTF8
Use an encoding of UTF8, meaning UTF-8. This can handle the characters for any language including Chinese.
And the UTF8 encoding allows Postgres to make use of the new support for International Components for Unicode (ICU) in Postgres 10 and later.
Java also uses Unicode encoding. Just let your JDBC driver handle the marshaling of text between Java and the database.

How to restrict ‘ this special character in textarea using regular expression in java servlet [duplicate]

I have the following string value: "walmart obama 👽💔"
I am using MySQL and Java.
I am getting the following exception: `java.sql.SQLException: Incorrect string value: '\xF0\x9F\x91\xBD\xF0\x9F...'
Here is the variable I am trying to insert into:
var1 varchar(255) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL`
My Java code that is trying to insert "walmart obama 👽💔" is a preparedStatement. So I am using the setString() method.
It looks like the problem is the encoding of the values 👽💔. How can I fix this? Previously I was using Derby SQL and the values 👽💔 just ended up being two sqaures (I think this is the representation of the null character)
All help is greatly appreciated!

What you have is EXTRATERRESTRIAL ALIEN (U+1F47D) and BROKEN HEART (U+1F494) which
are not in the basic multilingual plane. They cannot be even represented in java as one char, "👽💔".length() == 4. They are definitely not null characters and one will see squares if you are not using fonts that support them.
MySQL's utf8 only supports basic multilingual plane, and you need to use utf8mb4 instead:
For a supplementary character, utf8 cannot store the character at all,
while utf8mb4 requires four bytes to store it. Since utf8 cannot store
the character at all, you do not have any supplementary characters in
utf8 columns and you need not worry about converting characters or
losing data when upgrading utf8 data from older versions of MySQL.
So to support these characters, your MySQL needs to be 5.5+ and you need to use utf8mb4 everywhere. Connection encoding needs to be utf8mb4, character set needs to be utf8mb4 and collaction needs to be utf8mb4. For java it's still just "utf-8", but MySQL needs a distinction.
I don't know what driver you are using but a driver agnostic way to set connection charset is to send the query:
SET NAMES 'utf8mb4'
Right after making the connection.
See also this for Connector/J:
14.14: How can I use 4-byte UTF8, utf8mb4 with Connector/J?
To use 4-byte UTF8 with Connector/J configure the MySQL server with
character_set_server=utf8mb4. Connector/J will then use that setting
as long as characterEncoding has not been set in the connection
string. This is equivalent to autodetection of the character set.
Adjust your columns and database as well:
var1 varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NOT NULL
Again, your MySQL version needs to be relatively up-to-date for utf8mb4 support.

Weirdly, I found that REMOVING &characterEncoding=UTF-8 from the JDBC url did the trick for me with similar issues.
Based on my properties,
jdbc_url=jdbc:mysql://localhost:3306/dbName?useUnicode=true
I think this supports what #Esailija has said above, i.e. my MySQL, which is indeed 5.5, is figuring out its own favorite flavor of UTF-8 encoding.
(Note, I'm also specifying the InputStream I'm reading from as UTF-8 in the java code, which probably doesn't hurt)...

All in all, to save symbols that require 4 bytes you need to update characher-set and collation for utf8mb4:
database table/column:
alter table <some_table> convert to character set utf8mb4 collate utf8mb4_unicode_ci
database server connection (see)
On my development enviromnt for #2 I prefer to set parameters on command line when starting the server:
mysqld --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
btw, pay attention to Connector/J behavior with SET NAMES 'utf8mb4':
Do not issue the query set names with Connector/J, as the driver will not detect that the character set has changed, and will continue to use the character set detected during the initial connection setup.
And avoid setting characterEncoding parameter in connection url as it will override configured server encoding:
To override the automatically detected encoding on the client side, use the characterEncoding property in the URL used to connect to the server.

How I solved my problem.
I had
?useUnicode=true&characterEncoding=UTF-8
In my hibernate jdbc connection url and I changed the string datatype to longtext in database, which was varchar before.

Append the line useUnicode=true&characterEncoding=UTF-8 to your jdbc url.
In your case the data is not being send using UTF-8 encoding.

I faced the same issue and solved it by setting the Collation to utf8_general_ci for each column.

I guess MySQL doesn't believe this to be valid UTF8 text. I tried an insert on a test table with the same column definition (mysql client connection was also UTF8) and although it did the insert, the data I retrieved with the MySQL CLI client as well as JDBC didn't retrieve the values correctly. To be sure UTF8 did work correctly, I inserted an "ö" instead of an "o" for obama:
johan#maiden:~$ mysql -vvv test < insert.sql
--------------
insert into utf8_test values(_utf8 "walmart öbama 👽💔")
--------------
Query OK, 1 row affected, 1 warning (0.12 sec)
johan#maiden:~$ file insert.sql
insert.sql: UTF-8 Unicode text
Small java application to test with:
package test.sql;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
public class Test
{
public static void main(String[] args)
{
System.out.println("test string=" + "walmart öbama 👽💔");
String url = "jdbc:mysql://hostname/test?useUnicode=true&characterEncoding=UTF-8";
try
{
Class.forName("com.mysql.jdbc.Driver").newInstance();
Connection c = DriverManager.getConnection(url, "username", "password");
PreparedStatement p = c.prepareStatement("select * from utf8_test");
p.execute();
ResultSet rs = p.getResultSet();
while (!rs.isLast())
{
rs.next();
String retrieved = rs.getString(1);
System.out.println("retrieved=\"" + retrieved + "\"");
}
}
catch (Exception e)
{
e.printStackTrace();
}
}
}
Output:
johan#appel:~/workspaces/java/javatest/bin$ java test.sql.Test
test string=walmart öbama 👽💔
retrieved="walmart öbama "
Also, I've tried the same insert with the JDBC connection and it threw the same exception you are getting.
I believe this to be a MySQL bug. Maybe there's a bug report about such a situation already..

I had kind of the same problem and after going carefully against all charsets and finding that they were all right, I realized that the bugged property I had in my class was annotated as #Column instead of #JoinColumn (javax.presistence; hibernate) and it was breaking everything up.

execute
show VARIABLES like "%char%”;
find character-set-server if is not utf8mb4.
set it in your my.cnf, like
vim /etc/my.cnf
add one line
character_set_server = utf8mb4
at last restart mysql

This setting useOldUTF8Behavior=true worked fine for me. It gave no incorrect string errors but it converted special characters like à into multiple characters and saved in the database.
To avoid such situations, I removed this property from the JDBC parameter and instead converted the datatype of my column to BLOB. This worked perfect.

Besides,data type can use blob install of varchar or text.

ResultSet's getString method returns ? (question marks) when the string contains multibyte characters

The database is oracle 11gR2 and the charset of the database is AL32UTF8 and the column I'm trying to query is NVARCHAR2.
ResultSet rs = statement.executeQuery(query);
while (oRS.next())
{
String s = oRS.getString(1);
System.out.println(s);
}
All the non-English multi byte characters are displayed as question marks.
I tried replacing the getString with getBytes and tried converting the byte array to String with "UTF-8" charset, no luck.
My intention was not to print on the console, but I am embedding this list of strings to a HTML file (a list control ). Even there it displays the strings with question marks. The HTML page was encoded with ISO-8859-1

I changed the html page's encoding to UTF-8 and it started working. So the problem was not with the JDBC.

Special chars in JAVA

สวัสดี Mr.Java Sp'e c'i'a'l'' '
I tried to parse the String using below code but I could't make
simply it shows the wrong value.
String s = "สวัสดี Mr.Java Sp'e c'i'a'l'' '"";
s = s.replaceAll("'", "'");
//s = s.replaceAll("'", "''");
StringEscapeUtils.escapeHtml(s);
I am trying to get from JSP and save in SQL Server DB and show using JSP and update.
But some times in JSP it shows the converted &apos in jsp as it is instead of Special
Chars.
Very Simple is Here I have shown this String(สวัสดี Mr.Java Sp'e c'i'a'l'' ') in StackOverflow they
save in their DB and Shows and allows me to update this is what I
wanted.

OK. So lets look at what your code does:
// line 1
String s = "สวัสดี Mr.Java Sp'e c'i'a'l'' '";
We have a String with various international characters in it ... and some "'" characters.
// line 2
s = s.replaceAll("'", "'");
Assuming that those are really "'" characters characters, we will replace all instances of "'" with an XML / HTML character entity giving us:
"สวัสดี Mr.Java Sp'e c'i'a'l'' '"
And so ...
// line 3
s = StringEscapeUtils.escapeHtml(s);
This replaces any active HTML / XML characters with character references. This includes the ampersand characters "&" that you previously inserted. The result is this:
"&#xxxx;&#xxxx;&#xxxx;&#xxxx; Mr.Java Sp'e
c'i'a'l'' '"
(The &#xxxx; numeric character references encode those Thai (?) characters.)
When you embed that in an HTML document and display it, you will see "สวัสดี Mr.Java Sp'e c'i'a'l'' '"
See what has happened? You have HTML escaped your HTML escaped apostrophies!!
So what do you really need to do?
There is no need replace apostrophes with '. Apostrophes are legal in HTML text.
There should be no need to add HTML escapes so that you can store text in a database:
Any modern database will allow you to store Unicode strings without any special encoding.
If you are trying to prevent the database's SQL parser getting confused by quotes in the text you are storing, you are doing it the wrong way. The right way to do this is to use a PreparedStatement, add parameter placeholders to the query, and use the PreparedStatement.setXxx methods to provide the parameter values. The execute (or whatever) will take care of any SQL escaping that needs to be done.

Handling character encoding from Java to PHP to MySQL

In Java I pass a String to PHP.
In PHP I take that String and do a search for it with a MySQL query.
Here is php code:
$query = $database->escape_value(trim($_POST['query']));
$result = mysqli_query($dbconnection, Data::getSearchQuery($query));
while ($row = mysqli_fetch_assoc($result)) {
$output[] = $row;
}
print(json_encode($output));
mysqli_close($dbconnection);
public static function getSearchQuery($item_query) {
$query = "
SELECT i.item, i.item_id, c.category, c.cat_id
FROM items as i
LEFT JOIN master_cat AS c
ON (c.cat_id = i.cat_id)
WHERE i.item LIKE '%{$item_query}%'
ORDER BY i.item ASC;";
return $query;
}
This always works if I use regular characters on my U.S. keyboard. But the moment I start using irregular characters, the search turns empty.
I can verify that MySQL stores the data AS THE USER ENTERS IT. So if they typed Beyoncè, that is how database stores it.
But when I search for Beyoncè (or whatever) in the above code, it returns empty.
How should I handle the char. encoding here?

Three points to think of:
1) The $item_query variable could come in wrong encoding.
2) >>I can verify that MySQL stores the data AS THE USER ENTERS IT
This can get tricky. If one writes an iso8859-1 encoded string to an utf-8 database, the string is obviously stored incorrectly. If that string is read with a client (i. e. phpmyadmin or mysql command line tool) configured to iso8859-1, the string is correctly returned - although its representation in the database is clearly wrong.
3) The MySql settings:
Have your set utf-8 for the connection itself? What about charsets and collations for the database/the table?
https://dev.mysql.com/doc/refman/5.5/en/charset-syntax.html
UPDATE:
I assume you want everything to be UTF-8. Kind of quick hack to test:
Beyoncé has 7 characters (see MySQL CHAR_LENGTH function)
in UTF-8, it occupies 8 bytes (see MySQL LENGTH function). The eight bytes are, represented in a one-byte-per-character encoding like windows-1252, something like Beyoncé.
This leads to the following diagnostic tests ...
The PHP-issued SQL command
"SELECT CHAR_LENGTH($item_query), LENGTH($item_query);"
should then return a result of (7, 8) to show us that the $item_query variable is probably correctly encoded and the database likes UTF-8. (7, 7) would mean $item_query wasn't UTF-8, and (8, 8) would mean the database doesn't want to deal with UTF-8 yet. If the latter is the case, then perhaps issue a SET NAMES 'UTF8'; before the query.
Similarly, the PHP-issued SQL command
SELECT CHAR_LENGTH('Beyoncé'), LENGTH('Beyoncé');
should return the result (7, 8) to show us that your PHP editor is configured to edit UTF-8 php files.
Repeat the previous step with phpmyadmin (or any SQL client) to be sure that this client uses UTF-8, too.
No table was involved yet! The SQL command
SELECT CHAR_LENGTH(somecolumn), LENGTH(somecolumn) FROM sometable;
(with sometable having UTF-8 character encoding and somecolumn containing some diacritical characters) should tell you if UTF-8 was used when storing values to the table.
If all previous tests passed, test again with LIKE. Even 'Beyoncé' LIKE 'Beyonce' should work then. For more information, google MySQL collation.