I've been given the (rather daunting) task of introducing i18n to a J2EE web application using the 2.3 servlet specification. The application is very large and has been in active development for over 8 years.
As such, I want to get things right the first time so I can limit the amount of time I need to scrawl through JSPs, JavaScript files, servlets and wherever else, replacing hard-coded strings with values from message bundles.
There is no framework being used here. How can I approach supporting i18n. Note that I want to have a single JSP per view that loads text from (a) properties file(s) and not a different JSP for each supported locale.
I guess my main question is whether I can set the locale somewhere in the 'backend' (i.e. read locale from user profile on login and store value in session) and then expect that the JSP pages will be able to correctly load the specified string from the correct properties file (i.e. from messages_fr.properties when the locale is to French) as opposed to adding logic to find the correct locale in each JSP.
Any ideas how I can approach this?
There are a lot of things that need to be taken care of while internationalizing application:
Locale detection
The very first thing you need to think about is to detect end-user's Locale. Depending on what you want to support it might be easy or a bit complicated.
As you surely know, web browsers tend to send end-user's preferred language via HTTP Accept-Language header. Accessing this information in the Servlet might be as simple as calling request.getLocale(). If you are not planning to support any fancy Locale Detection workflow, you might just stick to this method.
If you have User Profiles in your application, you might want to add Preferred Language and Preferred Formatting Locale to it. In such case you would need to switch Locale after user logs in.
You might want to support URL-based language switching (for example: http://deutsch.example.com/ or http://example.com?lang=de). You would need to set valid Locale based on URL information - this could be done in various ways (i.e. URL Filter).
You might want to support language switching (selecting it from drop-down menu, or something), however I would not recommend it (unless it is combined with point 3).
JSTL approach could be sufficient if you just want to support first method or if you are not planning to add any additional dependencies (like Spring Framework).
While we are at Spring Framework it has quite a few nice features that you can use both to detect Locale (like CookieLocaleResolver, AcceptHeaderLocaleResolver, SessionLocaleResolver and LocaleChangeInterceptor) and externalizing strings and formatting messages (see spring:message tab).
Spring Framework would allow you to quite easily implement all the scenarios above and that is why I prefer it.
String externalization
This is something that should be easy, right? Well, mostly it is - just use appropriate tag. The only problem you might face is when it comes to externalizing client-side (JavaScript) texts. There are several possible approaches, but let me mention these two:
Have each JSP written array of translated strings (with message tag) and simply access that array in client code. This is easier approach but less maintainable - you would need to actually write valid strings from valid pages (the ones that actually reference your client-side scripts). I have done that before and believe me, this is not something you want to do in large application (but it is probably the best solution for small one).
Another approach may sound hard in principle but it is actually way easier to handle in the future. The idea is to centralize strings on client side (move them to some common JavaScript file). After that you would need to implement your own Servlet that will return this script upon request - the contents should be translated. You won't be able to use JSTL here, you would need to get strings from Resource Bundles directly.
It is much easier to maintain, because you would have one, central point to add translatable strings.
Concatenations
I hate to say that, but concatenations are really painful from Localizability perspective. They are very common and most people don't realize it.
So what is concatenation then?
On the principle, each English sentence need to be translated to target language. The problem is, it happens many times that correctly translated message uses different word order than its English counterpart (so English "Security policy" is translated to Polish "Polityka bezpieczeństwa" - "policy" is "polityka" - the order is different).
OK, but how it is related to software?
In web application you could concatenate Strings like this:
String securityPolicy = "Security " + "policy";
or like this:
<p><span style="font-weight:bold">Security</span> policy</p>
Both would be problematic. In the first case you would need to use MessageFormat.format() method and externalize strings as (for example) "Security {0}" and "policy", in the latter you would externalize the contents of the whole paragraph (p tag), including span tag. I know that this is painful for translators but there is really no better way.
Sometimes you have to use dynamic content in your paragraph - JSTL fmt:format tag will help you here as well (it works lime MessageFormat on the backend side).
Layouts
In localized application, it often happens that translated strings are way longer than English ones. The result could look very ugly. Somehow, you would need to fix styles. There are again two approaches:
Fix issues as they happen by adjusting common styles (and pray that it won't break other languages). This is very painful to maintain.
Implement CSS Localization Mechanism. The mechanism I am talking about should serve default, language-independent CSS file and per-language overrides. The idea is to have override CSS file for each language, so that you can adjust layouts on-demand (just for one language). In order to do that, default CSS file, as well as JSP pages must not contain !important keyword next to any style definitions. If you really have to use it, move them to language-based en.css - this would allow other languages to modify them.
Culture specific issues
Avoid using graphics, colors and sounds that might be specific for western culture. If you really need it, please provide means of Localization. Avoid direction-sensitive graphics (as this would be a problem when you try to localize to say Arabic or Hebrew). Also, do not assume that whole world is using the same numbers (i.e. not true for Arabic).
Dates and time zones
Handling dates in times in Java is to say the least not easy. If you are not going to support anything else than Gregorian Calendar, you could stick to built-in Date and Calendar classes.
You can use JSTL fmt:timeZone, fmt:formatDate and fmt:parseDate to correctly set time zone, format and parse date in JSP.
I strongly suggest to use fmt:formatDate like this:
<fmt:formatDate value="${someController.somedate}"
timeZone="${someController.detectedTimeZone}"
dateStyle="default"
timeStyle="default" />
It is important to covert date and time to valid (end user's) time zone. Also it is quite important to convert it to easily understandable format - that is why I recommend default formatting style.
BTW. Time zone detection is not something easy, as web browsers are not so nice to send anything. Instead, you can either add preferred time zone field to User preferences (if you have one) or get current time zone offset from web browser via client side script (see Date object's methods)
Numbers and currencies
Numbers as well as currencies should be converted to local format. It is done in the similar way to formatting dates (parsing is also done similarly):
<fmt:formatNumber value="1.21" type="currency"/>
Compound messages
You already have been warned not to concatenate strings. Instead you would probably use MessgageFormat. However, I must state that you should minimize use of compound messages. That is just because target grammar rules are quite commonly different, so translators might need not only to re-order the sentence (this would be resolved by using placeholders and MessageFormat.format()), but translate the whole sentence in different way based on what will be substituted. Let me give you some examples:
// Multiple plural forms
English: 4 viruses found.
Polish: Znaleziono 4 wirusy. **OR** Znaleziono 5 wirusów.
// Conjugation
English: Program encountered incorrect character | Application encountered incorrect character.
Polish: Program napotkał nieznaną literę | Aplikacja napotkała nieznaną literę.
Character encoding
If you are planning to Localize into languages that does not support ISO 8859-1 code page, you would need to support Unicode - the best way is to set page encoding to UTF-8. I have seen people doing it like this:
<%# page contentType="text/html; charset=UTF-8" %>
I must warn you: this is not enough. You actually need this declaration:
<%#page pageEncoding="UTF-8" %>
Also, you would still need to declare encoding in the page header, just to be on the safe side:
<META http-equiv="Content-Type" content="text/html;charset=UTF-8">
The list I gave you is not exhaustive but this is good starting point. Good luck :)
You can do exactly this using JSTL standard tag library with the tag. Grab a copy of the JSTL specification, read the i8N chapters, which discuss general text + date, time, currency. Very clearly written and shows you how you can do it all with tags. You can also set things like Locale programmatically
You dont(and shouldnt) need to have a separate JSP file per locale. The hard task is to figure out the keys that arent i18n-ed and move them to a file per locale, say, messages_en.properties, messages_fr.properties and so on.
Locale calculation can happen in multiple places depending on your logic. We support user locales stored in a database as well as the browser locale. Every request that comes into your application will have a "Accept-Language" header that indicates what are the languages your browser has been configured with , with preferences, i.e. Japanese first and then English. If thats the case, the application should read the messages_ja.properties and for keys that are not in that file, fallback to messages_en.properties. The same can hold true for user locales that are stored inside the database. Please note that the standard is just to switch the language in the browser and expect the content to be i18n-ed. (We initially started with storing locale in the database and then moved to support locales from the browser). Also you will need a default anyway as translators miss copying keys and values from english (main language file) to other languages, so you will need to default to english for values that are not in other files.
Ive also found mygengo very useful when giving translation job to other people who know a particular language, its saved us a lot of time.
Related
Is it possible to force the language of a Wicket message in HTML?
I have a few wicket pages and generic components that are used both when the user's locale is known and when it is not known. When the locale is not known, I am required to show messages in two official languages (english and german in this example). For now I have made language specific pages for the two languages and a default page for the "unknown" case:
MyComponent.html
MyComponent_en.hmtl
MyComponent_de.html
The "unknown" page (MyComponent.html) contains message-elements for both languages. At the moment, one of the laguages is the default one and for the common components I have had to duplicate the messages in other languages by appending the language code to the message key:
MyComponent.html:
<span lang="en"><wicket:message key="myMessageKey"/></span>
<span lang="de"><wicket:message key="myMessageKey.de"/></span>
wicket_package.properties:
myMessageKey=My hovercraft is full of eels
myMessageKey.de=Mein Luftkissenfahrzeug ist voller Aale
wicket_package_de.properties:
myMessageKey=Mein Luftkissenfahrzeug ist voller Aale
Clearly this duplication of localizations is insanity. Is there a way where I could force the language of the wicket:message in the HTML and save myself from duplicating the localizations? What I am looking for is something like <wicket:message key="myMessageKey" lang="de"/> or anything similar.
One option seems to be making my own WicketMessageResolver but I'd rather not do that work if I can avoid it.
There is no such functionality in current Wicket (9.8.0).
I'd also be hesitant to implement something like WicketMessageResolver!
But it would be quite simple to implement with non-auto component (auto-components are all created from markup - <wicket:xyz>), i.e. with a custom Panel that has two Labels which models use Localizer.get() to get the localized values.
For many projects I have worked on, programming teams work with the style of placholding every piece of static text in an xhtml file into a properties file. For example:
xhtml=
...
<h1>${messages.resourceBundle['key.to.static.text.placeholder']}</h1>
...
messages.properties=
...
key.to.static.text.placeholder=This will be the heading for this page only
...
Would anybody be able to explain what the advantage in this is?
So far, I can only see the following disadvantages:
making changes to any xhtml file requires you to hunt for the correct .properties file, and then the individual property to make the change to
if others have re-used properties, then deleting them becomes tricky as you have to be certain no other page is referencing the property, therefore after several change request rounds, properties files become large with redundant properties
if there are 1000 xhtmls, there will be 1000 .properties files to load, which is more cycles on the cpu to load and inject static pieces of text
if your using WebFlow and have flows that pass into other flows, properties have to be duplicated, meaning that sometimes you must place the same property in many different properties files to render correctly
hard to read code; if you know you want to work on the text 'This will be the heading for this page' only, you'll need to work out where that is on the xhtml from the property files first - you can't simply look at the xhtml and see clearly how the content will be laid out once rendered.
The only advantages I can see are text reuse and possibly html escaping.
Apologies if its coding 101, but I've had a hunt around Google and can't find the reasoning to the pattern.
Many Thanks
This is a common practice for internationalizing content.
You create one property file per language (or locale) and use a dynamic way off resolving which one to load depending on the context. (e.g. Language HTTP header the browser sends).
It is arguably more flexible than providing 1 jsp file per language, and can still deal with complex cases where plurals or stylistic differences might change the way you write localized text.
This is a standard JDK feature, lookup resource bundles.
You do not have to build 1 file per jsp (maybe your framework works this way?), although doing so can help the person writing the translation.
How would you use SafeHtml in combination with links?
Scenario: Our users can enter unformatted text which may contain links, e.g. I like&love http://www.stackoverflow.com. We want to safely render this text in GWT but make the links clickable, e.g. I like&love <a="http://www.stackoverflow.com">stackoverflow.com</a>. Aside rendering the text in the GWT frontend, we also want to send it via email where the links should be clickable as well.
So far, we considered the following options:
Store the complete text as HTML in the backend and let the frontend assume it's correctly encoded (I like&love <a="http://www.stackoverflow.com">stackoverflow.com</a>) -> Introduces XSS vulnerabilities
Store plain text but the links as HTML (I like&love <a="http://www.stackoverflow.com">stackoverflow.com</a>) in the backend and use HtmlSanitizer in the frontend
Store plain text and special encoding for the links (I like&love [stackoverflow.com|http://www.stackoverflow.com]) in the backend and use a custom SafeHtml generator in the frontend
To us, the third option looks the cleanest but it seems to require the most custom code since we can't leverage GWT's SafeHtml infrastructure.
Could anybody share how to best solve the problem? Is there another option that we didn't consider so far?
Why not store the text exactly as it was entered by the user, and perform any special treatment when transforming it for the output (e.g. for sending emails, creating PDFs, ...). This is the most natural approach, and you won't have to undo any special treatment e.g. when you offer the user to edit the string.
As a general rule, I would always perform encoding/escaping/transformation only for the immediate transport/storage/output target. There are very few reasons to deviate from this rule, one of them may be performance, e.g. caching a transformed value in the DB. (In these cases, I think it's best to give the DB field a specific name like 'text_htmltransformed' - this avoids 'overescaping', which can be just as harmful as no escaping.)
Note: Escaping/encoding is no replacement for input validation.
I am automating test cases for a web application using selenium 2.0 and Java. My application supports multiple languages. Some of the test cases require me to validate the text that appears in the UI like success/error messages etc.I am using a properties file to store whatever text I am referring in my tests from the UI, currently only english. For example there is locale_english.properties(see below) that contains all references in english. I am going to have multiple properties files like this for different locales like locale_chinese.properties,locale_french.properties and so on. For locales other than english, their corresponding properties file would have UTF-8 characters (e.g \u30ed) representing the native characters(see below). So If I want to test say Chinese UI, I would load "locale_chinese.properties" instead of "locale_english.properties". I am going to convert the native characters for non-english locale using perhaps native2ascii from JDK or some other way.I tested that Selenium API works well with UTF-8 characters for non-english locales
---locale_english.properties------
user.login.error= Please verify username/password
---locale_chinese.properties------
user.login.error= \u30ed\u30ef\u30eg\u30eh\u30ed
and so on.
The problem is that my locale_english.properties is growing and going out of control. It is becoming hard to manage a single properties file for one locale let alone for multiple locales. Is there a better way of handling localization in Java, particularly in situations like I am in?
Thanks!
You're right that there is a problem managing the files, but you're also right that this is the best approach. Some things are just hard :-(
Selenium (at least the Selenium RC API) does indeed support Unicode input and output, we have lots of tests that enter and confirm Cyrillic and Simple Chinese characters from C#. Since Java strings are Unicode at the core (just like C#), I expect you could simply create the file in a UTF-8-friendly editor like Notepad++ and read them straight into strings and use them directly in the Selenium API.
This is how I solved the issue for those who are interested.
a database would work better for many reasons, like growth, central location, kept outside of app and can be edited and maintained outside of app. We used a table with columns:
id (int) auto increment
id_text -- this and other columns are varchar ... except for date time for last 2
lang
translation
created_by
updated_by
created_date
updated_date
An id is a short english description of the text - like 'hello' or 'error1msg', the key in your map.
In java had a function to get the text for a particular text ... and a app level property - default language (usually en but good to keep it configurable)
Function would scan already loaded hashmap for language asked for - say "ch"
If corresponding translation was not found for this language we would return the default language translation and if that was not founf then we would return "[" + id "]" so the tester knows something is missing in data base - can go to web screen to edit translation table and add it.
I've recently inherited a internationalized and text-heavy Struts 1.1 web application. Many of the JSP files look like:
<p>
<bean:message key="alert" />
</p>
and the properties files look like:
messages.properties
alert=Please update your address and contact information.
with the appropriate translations in N other languages (messages_fr.properties, etc).
Problems:
DRY violation - I have N references to my Struts action URLs instead of 1, which makes refactoring action URLs error-prone.
Mixed concerns - My application's markup is now in more than just my JSP files, making it difficult for a web specialist to tweak the markup (using CSS, etc).
Post-translation markup - Anytime I receive newly-translated text, I must decide what to surround with the <a>...</a> markup. Easy for English but less so for unfamiliar languages.
I've considered adding placeholders in the messages file, like:
alert=Please update your {0} and {1}.
but then the words "address" and "contact information" would somehow need to be localized, wrapped with markup, and passed to my message tag - and I can't see an easy way to do it.
What can I do to improve this?
Avoid creating links within long
blocks of text. Prefer shorter text
that can act as a logically complete
and independent link.
Generally, it will lead to fewer problems. Sometimes you have to compromise your UI design to accommodate localization; sometimes you need to compromise your localization process to accommodate the UI.
Any time a developer manually manipulates post-translation strings is a source of potentially expensive bugs. Cutting/pasting or string editing can result in character corruption, misplaced strings, etc. A translation defect needs the participation of outside parties to fix which involves cost and takes time.
Thinking on it, something like this might be less ugly:
<p>Please update your address and contact information.
<br />
update address
<br />
update contact information</p>
...but I'm no UI designer.
One approach that comes to mind is that you could store the translated replacement parameters i.e. "address" and "contact information" in a separate properties file, one per locale. Then have your Action class (or probably some helper class) look up the values from the correct ResourceBundle for the current locale and pass them to the message tag.
Perhaps:
#
alert=Please update your {0}address{1} and {2}contact information{3}.
The message message tag API allows
only 5 parametric arguments
Ah! I blame my complete ignorance of the Struts API.
To quote the manual:
Some of the features in this taglib
are also available in the JavaServer
Pages Standard Tag Library (JSTL). The
Struts team encourages the use of the
standard tags over the Struts specific
tags when possible.
You could probably do this with the http://java.sun.com/jsp/jstl/fmt taglib.
<fmt:bundle basename="messages">
<fmt:message key="alert">
<fmt:param value='<a href="/">' />
<fmt:param value="</a>" />
<fmt:param value='<a href="/">' />
<fmt:param value="</a>" />
</fmt:message>
</fmt:bundle>
The downside is that this isn't valid XML and yanking the values to variables involves more indirection, lookups and verbosity. This is not a good solution.
I don't know Struts, but if it is anything like JavaServer Faces (same architect), then there is probably support for configuring a replacement control. I would either replace the existing control with a more flexible one or add a new one.
Anytime I receive newly-translated
text, I must decide what to surround
with the <a>...</a> markup.
There is no way you should be doing this and I see this as a fault in your translation process (I am an ex-localization engineer and ex-developer of localization tools). The {0} characters should be included in the files that are sent to the translators. The localization guidelines should explain the string's context and the meaning of any variables.
You can programmatically validate the property bundles on return. String-specific regex's might do the trick. It isn't outside the realms of possibility that "address" and "contact information" would swap order during translation.
The simplest solution is to redesign the messages to render:
Please update your address.
Please update your contact information.
I accept that this might not be a solution in all cases and may have your UI designer spitting teeth.