Related
In Liferay we are looking for articles that satisfy certain conditions by using the following code:
Hits hits = indexSearcherHelper.search(searchContext, query);
Search query which we use is defined as:
BooleanFilter filter = new BooleanFilter();
filter.addRequiredTerm(Field.GROUP_ID, globalSiteId);
filter.addRequiredTerm(Field.STATUS, WorkflowConstants.STATUS_APPROVED);
filter.addRequiredTerm("ddmStructureKey", "TEST");
filter.addRequiredTerm("head", true);
MatchAllQuery query = new MatchAllQuery();
query.setPreBooleanFilter(filter);
and this search finds multiple hits.
Then we attempt to get the article like this:
JournalArticleResource journalArticleResource = journalArticleResourceLocalService.getArticleResource(GetterUtil.getLong(hits.toList().get(0).get(Field.ENTRY_CLASS_PK)));
JournalArticle article = journalArticleLocalService.getArticle(journalArticleResource.getGroupId(), journalArticleResource.getArticleId());
However, this produces following error:
No JournalArticleResource exists with the primary key 809477.
In 95% of cases, this code works as expected. But in some cases (on some environments), it appears that index search found results which are not valid. Why does this happen?
Can it be that index has some stale records that are from old, already deleted articles? Do we need to reindex the database?
UPDATE 1: I have observed a very strange behaviour of the index search:
The following code:
for (int counter = 0; counter < 10; counter++)
{
System.out.println(counter);
System.out.println(indexSearcherHelper.search(searchContext, query).toList().size());
}
produces this result:
0
0
1
4
2
7
3
0
4
4
5
7
6
0
7
4
8
7
9
0
There is only 1 result in reality that needs to be found. On all other environments this code keeps finding just one result in all 10 searches, since we added only 1 article.
In this case, however, it keeps finding no results, 4 results, 7 results and keeps repeating the same pattern.
What is going on here? Is database corrupted? Is it Liferay bug? How can the same search return different number of results?
(By the way, last year we did a live database migration from one server to another, that is, migration of the database while Liferay was up and running [not too good idea] to reduce the production downtime, so I am afraid that we might be experiencing the database corruption here.)
UPDATE 2: as requested in the comments, here is the version of the Liferay that we are using and an example of the search with values of some fields modified since this is a production example from closed source application.
Version:
Liferay Community Edition Portal 7.0.4 GA5 (Wilberforce / Build 7004 / October 23, 2017)
System.out.println(hits.toList().get(0));
{
ddmTemplateKey=[673861],
entryClassPK=[809477],
ddm__keyword__673858__LActive_hr_HR=[true],
publishDate=[20211116063000],
ddm__keyword__673858__SActive_hr_HR=[false],
ddm__keyword__673858__GNA_en_US_String_sortable=[ne],
ddm__text__673858__OList_hr_HR_String_sortable=[32554651079],
classNameId=[0],
ddm__keyword__673858__SActive_en_US_String_sortable=[false],
ddm__keyword__673858__O_hr_HR_String_sortable=[opis pop upa],
modified_sortable=[1637050218921],
title_hr_HR=[Test ss n],
ddm__keyword__673858__O_en_US=[Opis pop upa],
version=[2.4],
ddm__keyword__673858__B_en_US=[grey],
ddm__keyword__673858__SActive_hr_HR_String_sortable=[false],
ddm__keyword__673858__OAll_en_US_String_sortable=[false],
status=[0],
ddm__keyword__673858__GPA_en_US=[OK],
publishDate_sortable=[1637044200000],
content_hr_HR=[OK 32554651079 NE true Opis pop upa all true Test pop najnoviji Utorak grey false all false /ervices],
ddm__keyword__673858__TR_en_US=[all],
ddm__keyword__673858__B_hr_HR=[grey],
uid=[com.liferay.journal.model.JournalArticle_PORTLET_811280],
localized_title_en_US_sortable=[test ss n],
layoutUuid=[],
ddm__text__673858__OList_en_US=[32554651079],
ddm__keyword__673858__GNA_hr_HR=[NE],
ddm__keyword__673858__TR_en_US_String_sortable=[all],
ddm__keyword__673858__GNA_hr_HR_String_sortable=[ne],
createDate=[20211115132217],
ddm__keyword__673858__OAll_hr_HR_String_sortable=[false],
displayDate_sortable=[1637044200000],
ddm__keyword__673858__O_en_US_String_sortable=[opis pop upa],
entryClassName=[com.liferay.journal.model.JournalArticle],
ddm__keyword__673858__N_en_US=[Test pop najnoviji Utorak],
ddm__keyword__673858__S_hr_HR_String_sortable=[all],
userId=[30588],
localized_title_en_US=[test ss n],
ddm__keyword__673858__N_hr_HR_String_sortable=[test pop najnoviji utorak],
ddm__keyword__673858__OListActive_hr_HR=[true],
ddm__keyword__673858__GPA_hr_HR_String_sortable [ok],
treePath=[, 673853],
ddm__keyword__673858__B_en_US_String_sortable=[grey],
ddm__keyword__673858__S_hr_HR=[all], groupId=[20152],
ddm__keyword__673858__B_hr_HR_String_sortable=[grey],
createDate_sortable=[1636982537964],
classPK=[0],
ddm__keyword__673858__S_en_US_String_sortable=[all],
ddm__keyword__673858__GPA_hr_HR=[OK],
scopeGroupId=[20152],
articleId_String_sortable=[809475],
ddm__keyword__673858__OAll_hr_HR=[false],
modified=[20211116081018],
ddm__keyword__673858__LActive_hr_HR_String_sortable=[true],
ddm__keyword__673858__L_hr_HR=[/ervices],
localized_title_hr_HR_sortable=[test ss n],
ddm__keyword__673858__L_en_US=[/ervices],
visible=[true],
ddmStructureKey=[TEST],
ddm__keyword__673858__OAll_en_US=[false],
defaultLanguageId=[hr_HR],
ddm__keyword__673858__L_hr_HR_String_sortable=[/ervices],
viewCount_sortable=[0],
folderId=[673853],
classTypeId=[673858],
ddm__text__673858__OList_hr_HR=[32554651079],
ddm__keyword__673858__TR_hr_HR_String_sortable=[all],
companyId=[20116],
rootEntryClassPK=[809477],
ddm__keyword__673858__LA_en_US_String_sortable=[true],
displayDate=[20211116063000],
ddm__keyword__673858__OListActive_hr_HR_String_sortable=[true],
ddm__keyword__673858__SActive_en_US=[false],
ddm__keyword__673858__OListActive_en_US=[true],
ddm__keyword__673858__LActive_en_US=[true],
content=[OK 32554651079 NE true Opis pop upa all true Test pop najnoviji Utorak grey false all false /ervices],
head=[true],
ddm__keyword__673858__GPA_en_US_String_sortable=[ok],
ddm__keyword__673858__OListActive_en_US_String_sortable=[true],
ratings=[0.0],
expirationDate_sortable=[9223372036854775807],
viewCount=[0],
ddm__text__673858__OList_en_US_String_sortable=[32554651079],
localized_title_hr_HR=[test ss n],
expirationDate=[99950812133000],
ddm__keyword__673858__N_en_US_String_sortable=[test pop najnoviji utorak],
roleId=[20123, 20124, 20126],
ddm__keyword__673858__S_en_US=[all],
articleId=[809475],
ddm__keyword__673858__N_hr_HR=[Test pop najnoviji Utorak],
userName=[tuser%40admin -],
localized_title=[test ss n],
stagingGroup=[false],
headListable=[true],
ddm__keyword__673858__L_en_US_String_sortable=[/ervices],
ddm__keyword__673858__O_hr_HR=[Opis pop upa],
ddm__keyword__673858__TR_hr_HR=[all],
ddm__keyword__673858__GNA_en_US=[NE]
}
You might be using the wrong service, try using the journalArticleLocalService.
The id of the journal article resource is the id of the journal article plus 1, so if you have more than one article, in most cases it wont produce the error, but will return the wrong article.
Perhaps you are hitting some inconsistencies in your Elasticsearch index: JournalArticles that don't exist in the Database but they exist in the Elasticsearch.
You can double check this and correct it using my Liferay Index Checker, see https://github.com/jorgediaz-lr/index-checker#readme
Once you have installed it, you have to:
Check the "Display orphan index entries" option
Click on "Check Index"
If you have any orphan results, you can remove them clicking on the "Remove orphans" button.
Ok, so I am a little new to MySQL querys and I am wondering if this would be the correct syntax to get if a value is false in a row.
"SELECT `status`
FROM `friends`
WHERE `requester`= '"+requester+"'
AND `requested` = '"+requested+"'
AND `status`= 'false'";
If that is wrong, I'm trying to fetch the requests where you are the requester and the status is false through a java method.
first thing i would do is check to be sure the variables contain what you want them to. maybe even put in the data directly to the query at first without variables. get it to what you want it to be.
System.out.println("the requester is: ");
System.out.println(requester);
System.out.println("the requested is: ");
System.out.println(requested);
"SELECT 0 FROM friends WHERE requester= '"+requester+"' AND requested = '"+requested+"' AND status= 'false'"
generally 0 is false and 1 is true in programming languages so you can just pull out a 0 when its false.
you may want to also do a like comparison on the requester and requested. you could do it on the status column if you want but that shouldn't be necessary
"SELECT 0 FROM friends WHERE requester LIKE '%"+requester+"%' AND requested LIKE '%"+requested+"%' AND status= 'false'"
Your query want work only if there is some white space char in your value else your query is fine to check try this use like
SELECT status FROM friends WHERE requester= '"+requester+"' AND requested = '"+requested+"' AND status like '%false';
try this man
status is false
in your case, false is a string, not a boolean
I have run into an interesting problem which I'm pretty sure is the fault of HashMap. Consider the following debug code (AMap is a HashMap, key is a value passed to this method)
System.out.println("getBValues - Given: " + key);
System.out.println("getBValues - Contains Key: " + AMap.containsKey(key));
System.out.println("getBValues - Value: " + AMap.get(key));
for(Map.Entry<A,HashSet<B>> entry : AMap.entrySet()) {
System.out.println("getBValues(key) - Equal: " + (key.equals(entry.getKey())));
System.out.println("getBValues(key) - HashCode Equal: "+(key.hashCode() == entry.getKey().hashCode()));
System.out.println("getBValues(key) - Key: " + entry.getKey());
System.out.println("getBValues(key) - Value: " + entry.getValue());
}
Now in this Map I insert a single key (Channel) and value. Later I try and get the value back with get() and run this debug code which in my case gives this output:
getBValues - Given: Channel(...)
getBValues - Contains Key: false <--- Doesnt contain key?!
getBValues - Value: null <--- Null (bad)
getBValues(key) - Equal: true <--- Given key and AMap key is equal
getBValues(key) - HashCode Equal: true
getBValues(key) - Key: Channel(Same...)
getBValues(key) - Value: [] <--- Not null (This is the expected result)
As you can see, fetching the key from the HashMap directly doesn't work but looping through I get the exact same key, meaning its there it just can't be found with get(). My question is what would cause this? How can get() not find a key that exists?
I would provide an some example code of this but I can't seem to reproduce this independently.
Any suggestions on what might be causing this?
I'll bet you didn't override equals and hashCode properly in your key Channel class. That would explain it.
Joshua Bloch tells you how to do it correctly in his "Effective Java" Chapter 3.
http://java.sun.com/developer/Books/effectivejava/Chapter3.pdf
From what I can see, we still haven't ruled out if it has to do with immutability.
If you do:
aMap.put(key, value);
key.setFieldIncludedInHashCodeAndEquals(25);
then you would get the result from above.
To rule this out, either show us more of your code or, in the for loop in your example above, add
System.out.println(aMap.get(entry.getKey()));
Also, use a debugger. That way, you can see if your object is in the correct bucket.
Messy, complicated question, but here goes. I'm working on an integration project with Google Checkout, and there is a Google Checkout GWT service that returns the currency conversion rates used by the Checkout web interface to convert USD into local currencies. This endpoint is hosted at https://market.android.com/publish/gwt/, and staring at Firebug I see this going to the server:
7|0|6|https://market.android.com/publish/gwt/|FCCA4108CB89BFC2FEC78BA7363D4AF6|com.google.wireless.android.vending.developer.
shared.MerchantService|getCurrencyExchangeRates|com.google.common.money.CurrencyCode/112449834|java.util.ArrayList/4159755760
|1|2|3|4|2|5|6|5|235|6|13|5|18|5|81|5|53|5|72|5|102|5|121|5|177|5|175|5|205|5|204|5|55|5|86|-1|
and this being returned
//OK[235,3,'D0JA',2,86,3,'CXXg',2,55,3,'DW2A',2,204,3,'X9NA',2,205,3,'EuvA',2,175,3,'VIig',2,177,3,'E2Dw',2,121,3,'E4ziA',2,1
02,3,'do$Q',2,72,3,'T82w',2,53,3,'Ds0Q',2,81,3,'Cq5g',2,18,3,'Dlfg',2,13,1,["com.google.common.collect.RegularImmutableList/4
40499227","com.google.common.money.SimpleMoney/627983206","com.google.common.money.CurrencyCode/112449834"],0,7]
Forgive the odd formatting: can't quite get the code block to format right.
Wandering the web for hours on end I was able to determine that the RegularImmutableList class is in the Guava libraries (at http://code.google.com/p/guava-libraries/). What I'm looking for is:
I can't find the com.google.common.money.SimpleMoney or com.google.common.money.CurrencyCode classes anywhere: anyone seen them?
The GWT wire format appears to be an odd JSON string. I see various references to Google Groups messages talking about descriptions of the wire format, but can't find the underlying messages or any coherent reference that would let me reverse this: anyone have a handle on a handy reference? If I can at least understand WHAT the encoding is I might be able to get away without the class files from question 1 above.
I started wandering through the Android Market api library at http://code.google.com/p/android-market-api/, figuring they have to have done SOME of the Android Market communication integration, and they appear to have done so using protobufs. Is there any decent reference for the GWT/protobufs communication bits?
The underlying reason for this craziness is that I need to be able to take regular exchange rate values from Google Checkout so when I'm importing sales transactions in foreign currencies I can do the conversion at the prevailing rate at the time of the transaction. The current Checkout reporting formats do NOT provide this, so most folks end up using alternative sources of exchange rates that don't match what Google uses. It is clearly a shortcoming on the part of Google Checkout's integration interface, but if we got started on shortcomings of Google Checkout's interface we'd be here all week. My intention is to poll the Checkout interface for newly fulfilled orders and then request the appropriate exchange rate table so I can figure out in near real-time what the incoming payments are. I've got the polling bit down pat but can't quite get past the exchange rate bit.
While trying to create a script to bulk upload in-app products for my application (CSV upload constantly failed with obscure error messages), I have managed to understand the GWT AJAX protocol.
It's actually pretty simple, except it requires you to know structure of all used classes. Or guess it, as is the case with internal classes used by Google. :)
I'll use examples from the question to explain the protocol in detail.
Request format
7|0|6|https://market.android.com/publish/gwt/|FCCA4108CB89BFC2FEC78BA7363D4AF6|com.google.wireless.android.vending.developer.shared.MerchantService|getCurrencyExchangeRates|com.google.common.money.CurrencyCode/112449834|java.util.ArrayList/4159755760|1|2|3|4|2|5|6|5|235|6|13|5|18|5|81|5|53|5|72|5|102|5|121|5|177|5|175|5|205|5|204|5|55|5|86|-1|
The request is pipe-delimited list of tokens with the following meaning:
7 - protocol version
0 - flags. 1 is FLAG_ELIDE_TYPE_NAMES, 2 is FLAG_RPC_TOKEN_INCLUDED
6 - string token count
6 string tokens:
https://market.android.com/publish/gwt/
FCCA4108CB89BFC2FEC78BA7363D4AF6
com.google.wireless.android.vending.developer.shared.MerchantService
getCurrencyExchangeRates
com.google.common.money.CurrencyCode/112449834
java.util.ArrayList/4159755760
The actual encoded request, which references strings from the list above using 1-based indices:
1 - https://market.android.com/publish/gwt/ - base URL
2 - FCCA4108CB89BFC2FEC78BA7363D4AF6 - some hash, which is references as serializationPolicyStrongName in GWT sources.
3 - com.google.wireless.android.vending.developer.shared.MerchantService - service name
4 - getCurrencyExchangeRates - method name
2 - parameter count. Parameter types follow:
5 - com.google.common.money.CurrencyCode/112449834
6 - java.util.ArrayList/4159755760
Serialized parameters. Each object is represented either by its classname and list of serialized fields or by negative integer back-reference to previously encountered object. In our case we have two objects:
5 - com.google.common.money.CurrencyCode/112449834, which only has one integer field: 235
6 - java.util.ArrayList/4159755760, which has one integer length field 13, followed by 13 serialized list items. Note that 12 of them are CurrencyCode objects serialized just as the above one, and the last one is a backreference (-1) to the very first object we encountered while (de-)serializing this request, i.e. CurrencyCode(235)
Response format
//OK[235,3,'D0JA',2,86,3,'CXXg',2,55,3,'DW2A',2,204,3,'X9NA',2,205,3,'EuvA',2,175,3,'VIig',2,177,3,'E2Dw',2,121,3,'E4ziA',2,102,3,'do$Q',2,72,3,'T82w',2,53,3,'Ds0Q',2,81,3,'Cq5g',2,18,3,'Dlfg',2,13,1,["com.google.common.collect.RegularImmutableList/440499227","com.google.common.money.SimpleMoney/627983206","com.google.common.money.CurrencyCode/112449834"],0,7]
The response is very similar in format to the request except it's JS-formatted array (though not JSON, as it uses invalid single quotes), and it's in reverse order.
The field meaning is as follows:
7 - protocol version
0 - flags, same as for request
Array of string tokens:
com.google.common.collect.RegularImmutableList/440499227
com.google.common.money.SimpleMoney/627983206
com.google.common.money.CurrencyCode/112449834
And then goes one serialized object of type 1 - com.google.common.collect.RegularImmutableList/440499227 with one integer length field 13, followed by 13 serialized objects of class 2 - com.google.common.money.SimpleMoney/627983206. Each SimpleMoney object has two fields, for example:
'Dlfg' - long integer field encoded as base64 number. This particular one is 940000
3, 18 - CurrencyCode object with integer field 18
What you are looking at is GWT-RPC serialization format. Unfortunatelly it is not publicly documented. Fortunatelly GWT is open-source so you could look at the source to see how it is produced.
Note: This format might change between GWT versions (I known it did in 2.2). This is most likelly also a reason why Google does not document it - if they did they'd need to keep it backward compatible.
Class names that you see are Java classes that Google Checkout uses internally. When GWT is compiled to JS the names get mangled so you don't see them any more.
As noted this is GWT-RPC.
What you are trying to do is reverse-engineer Google internal APIs. I wouldn't do that because, a. It might change without notice, breaking your app and, b. I'm sure Goog wouldn't like it and it probably violates the service agreement (have you read it?).
I have some code made in VB that may be useful for you to realize how to parse GWT Serialized strings. "Datos" contains the string you received.
aAux = Split(Datos, ",[")
aAux(1) = Replace(aAux(1), "],0,7]", "")
aAux(0) = Replace(aAux(0), "//OK[", "")
aAux(0) = Replace(aAux(0), "'", "")
aDescripcion = Split(aAux(1), """,""")
aValor = Split(aAux(0), ",")
InvertirArray aValor
For X = 0 To UBound(aValor)
If Not IsNumeric(aValor(X)) Then
Exit For
End If
If adescripcion(Int(aValor(X))-1) = "gov.senasa.embalajemadera.shared.domain.Pais/3238585366" Then
For Y = X + 1 To UBound(aValor)
If Int(aValor(Y)) = "" Then '- Do what you want
end if
If adescripcion(Int(aValor(Y))) = "java.lang.Integer/3438268394" Then
'- Do what you want
Next Y
End If
Next X
Of course you have to adapt it to your needs and you will have to play a little bit with the arrays...
InvertirArray:
Public Sub InvertirArray(ByRef Arr() As String)
'- el array va tiene que empezar en 0
Dim X As Long
Dim Hasta As Long
Dim Tmp As String
If UBound(Arr) Mod 2 = 0 Then
'- Es impar
Hasta = UBound(Arr) + 1
Else
Hasta = UBound(Arr)
End If
For X = LBound(Arr) To UBound(Arr) \ 2
Tmp = Arr(X)
Arr(X) = Arr(UBound(Arr) - X)
Arr(UBound(Arr) - X) = Tmp
Next X
end sub
And of course you need to decode and encode Long Numbers and dates. So:
Public Function EncodeDateGwt(Numero As Double, Optional isDate As Boolean = False) As String
Dim s As String
Dim a As Double
Dim i As Integer
Dim u As Integer
Dim Base As String
Numero = IIf(isDate, Numero * 1000, Numero)
Base = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789_$"
Do While Val(Numero) <> 0
a = Numero
i = 0
Do While a >= 64
i = i + 1
a = a / 64
Loop
If i <> u - 1 And u <> 0 Then EncodeDateGwt = EncodeDateGwt & String(u - i - 1, Left(Base, 1))
a = Int(a)
EncodeDateGwt = EncodeDateGwt + Mid(Base, a + 1, 1)
Numero = Numero - a * (64 ^ i)
u = i
Loop
EncodeDateGwt = EncodeDateGwt & String(i, Left(Base, 1))
End Function
Public Function DecodeDateGwt(Texto As String, Optional isDate As Boolean = False) As Long
Dim Base As String
Dim a As Integer
Base = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789_$"
For a = 1 To Len(Texto)
DecodeDateGwt = DecodeDateGwt + (InStr(Base, Mid(Texto, a, 1)) - 1) * (Len(Base) ^ ((Len(Texto) - (a))))
Next
DecodeDateGwt = IIf(isDate, DecodeDateGwt / 1000, DecodeDateGwt)
'devuelve timestamp
End Function
If what you need to encode/decode is a date, then you need to do this before:
Call encodegwtdate(date2unix("20/02/2016"),true)
Public Function Date2Unix(ByVal vDate As Date) As Long
Date2Unix = DateDiff("s", Unix1970, vDate)
End Function
Public Function Unix2Date(vUnixDate As Long) As Date
Unix2Date = DateAdd("s", vUnixDate, Unix1970)
End Function
Hope you solve it. By the way, does anyone knows what negative numbers means?????
I am trying to read a UTF-8 string from my MySql database, which I create using:
CREATE DATABASE april
DEFAULT CHARACTER SET utf8
DEFAULT COLLATE utf8_general_ci;
I make the table of interest using:
DROP TABLE IF EXISTS `article`;
CREATE TABLE `article` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`text` longtext NOT NULL,
`date_created` timestamp DEFAULT NOW(),
PRIMARY KEY (`id`)
) CHARACTER SET utf8;
If I select * from article in the MySql command line util, I get:
OIL sands output at Nexen’s Long Lake project dropped in February.
However, when I do
ResultSet rs = st.executeQuery(QUERY);
long id = -1;
String text = null;
Timestamp date = null;
while (rs.next()) {
text = rs.getString("text");
LOGGER.debug("text=" text);
}
the output I get is:
text=OIL sands output at Nexen’s Long Lake project dropped in February.
I get my Connection via:
DriverManager.getConnection("jdbc:" + this.dbms + "://" + this.serverHost + ":" + this.serverPort + "/" + this.dbName + "?useUnicode&user=" + this.username + "&password=" + this.password);
I've also tried, instead of the useUnicode parameter:
characterEncoding=UTF-8
and
characterEncoding=utf8
I also tried, instead of the line text = rs.getString("text")
rs.getBytes("text");
String[] encodings = new String[]{"US-ASCII", "ISO-8859-1", "UTF-8", "UTF-16BE", "UTF-16LE", "UTF-16", "Latin1"};
for (String encoding : encodings) {
text = new String(temp, encoding);
LOGGER.debug(encoding + ": " + text);
}
// Which outputted:
US-ASCII: OIL sands output at Nexen��������s Long Lake project dropped in February.
ISO-8859-1: OIL sands output at Nexenââ¬â¢s Long Lake project dropped in February.
UTF-8: OIL sands output at Nexen’s Long Lake project dropped in February.
UTF-16BE: 佉䰠獡湤猠潵瑰畴琠乥硥滃ꋢ芬ꉳ⁌潮朠䱡步⁰牯橥捴牯灰敤渠䙥扲畡特�
UTF-16LE: 䥏⁌慳摮畯灴瑵愠⁴敎數썮겂蓢玢䰠湯慌敫瀠潲敪瑣搠潲灰摥椠敆牢慵祲�
UTF-16: 佉䰠獡湤猠潵瑰畴琠乥硥滃ꋢ芬ꉳ⁌潮朠䱡步⁰牯橥捴牯灰敤渠䙥扲畡特�
Latin1: OIL sands output at Nexenââ¬â¢s Long Lake project dropped in February.
I load the strings into the DB using some pre-defined sql in a file. This file is UTF-8 encoded.
mysql -u april -p -D april < insert_articles.sql
This file includes the line:
INSERT INTO article (text) value ("OIL sands output at Nexen’s Long Lake project dropped in February.");
When I print out that file within my application using:
BufferedReader reader = new BufferedReader(new FileReader(new File("/home/path/to/file/sql_article_inserts.sql")));
String str;
while((str = reader.readLine()) != null) {
LOGGER.debug("LINE: " + str);
}
I get the correct, expected output:
LINE: INSERT INTO article (text) value ("OIL sands output at Nexen’s Long Lake project dropped in February.");
Any help would be much appreciated.
Some System Details:
I am running on linux (Ubuntu)
Edits:
* Edited to specify OS
* Edited to detail output of reading sql input file.
* Edited to specify more about how the data is inserted into the DB.
* Edited to to fix typo in code, and clarify example.
Is it possible you're reading the log file using the incorrect encoding? windows-1252, I am guessing.
UTF-8: OIL sands output at Nexen’s Long Lake project dropped in February.
If this is appearing in the log, do a hex dump of the log file. If the data is UTF-8, you would expect the sequence Nexen’s to become 4E 65 78 65 6E E2 80 99 73. If some other application reads this as a native ANSI encoding, it'll decode it as Nexen’s.
To confirm, you can also dump the individual characters of the return value to see if they are correct in UTF-16:
//untested
for(char ch : text.toCharArray()) {
System.out.printf("%04x%n", (int) ch);
}
I'm assuming all data is in the BMP, so you can just look up the results in the Unicode charts.
Try setting the database itself to UTF-8. When creating the DB:
CREATE DATABASE mydb
DEFAULT CHARACTER SET utf8
DEFAULT COLLATE utf8_general_ci;
Also see MySQL reference on connection charsets and MySQL reference on configuring charsets for applications
Parameters in the JDBC URL only define how the driver should communicate with the server. If the server does not use UTF8 by default these parameters won't change it either.
Have you tried executing the following SQL query after connecting? (This should switch the current connection to UTF8 on the server-side too):
SET names utf8
There are several character encodings involved.
The terminal/cmd window that the mysql command line tool is running. (putty?)
the environment in the shell (bash) where you are running your stuff. (LC_CTYPE)
Mysql internal (used in tables) : you have defined this to UTF-8
The JVM internal (always UTF16)
The character used by the writers the logger use. Default (system property) or perhaps defined in the logging frameworks configuration.
The terminal/cmd/editor that you read the logs with. ( putty/bash?)
If the terminal settings are wrong, you might have inserted corrupted data in mysql. (If your terminal is iso-8859-1 and you read a file that is UTF-8, for instance) Assuming linux, mysql should look at the env LC_CTYPE (but I am not 100% sure that it does.)
The JDBCD driver is responsible for converting the database character encoding to the JVMs internal format (UTF16) so that should not be a problem. But you can test this with a simpel java program that inserts a hard coded string, and reads it back. Print the original and received string - they should be identical. But; If both are wrong, you have a problem with the terminals character set definition.
Use a string like "HejÅÄÖ" for some drama...
ALso, write a small program that prints the same string to a file using a printwriter that converts to UTF-8 and verify that the tool you use for reading the log prints that file correctly. If not, there terminals settings are to be suspected, again.
String test = "Test HEJ \u00C5\u00C4\u00D6 ÅÄÖ";
// here's how to define what character set to use when writing to a fileOutputStream
PrintWriter pw = new PrintWriter("test.txt","UTF8");
pw.println(test);
pw.flush();
pw.close();
System.out.println(test);
output -> Test HEJ ÅÄÖ ÅÄÖ
The contents ni the file test.txt should look the same.