Develop Reference

Java Gradle Eclipse - UTF-8 symbols showing before build in IDE but not after build

i'm fairly new to coding and currently im trying to build my first fat JAR with gradle in eclipse. My project generates a HTML file with data i get from a JSON file. some of the strings in there use UTF-8 symbols/emojis. I managed to get them into my java objects with "Jackson" and by using the getBytes() method for the strings. When i generate the HTML from those objects in eclipse they show up the right way (when opened in browser outside of eclipse).
The problem occurs, when i build the fat jar with "gradlew jar" (i had to make some changes in build.gradle to jar to make it contain the dependencies).
Everything works fine, besides the UTF-8 symbols not showing, instead i get a "?", like the encoding is not set right.
So i think the problem is encoding settings in gradle.
What i tried so far is adding this to gradlew.bat:
set GRADLE_OPTS=-Dfile.encoding=utf-8
this to build.gradle :
tasks.withType(JavaCompile) {
options.encoding = 'UTF-8'
}
and this to gradle.Properties
systemProp.file.encoding=utf-8
like mentioned in this thread Show UTF-8 text properly in Gradle
None of those seem to help. Any ideas? Im stuck, struggle to find a solution.

-Dfile.encoding=UTF-8 sets the default encoding used at runtime when running your compiled code, e.g. java -Dfile.encoding=UTF-8 -jar .... Don't mix it up with the encoding that is used by the Java compiler to read the .java files: javac -encoding ....
To avoid to set it on the command line, you can set it also programmatically via System.setProperty("file.encoding", "UTF-8");. But better set the encoding where it used, e.g. instead of BufferedWriter writer= Files.newBufferedWriter(path); use BufferedWriter writer= Files.newBufferedWriter(path, StandardCharsets.UTF_8);.

Invalid Header Field Name when adding manifest to JAR using Eclipse

I'm using eclipse to develop a java applet and need to add an external jar to Class-Path in the manifest. I created the manifest file using notepad, used UTF-8 encoding and added the line break at the end, but attempting to add even a simple example manifest during jar creation results in the following error from Eclipse.
invalid header field name: ï>¿Manifest-Version
invalid header field name: ï>¿Manifest-Version
I have a feeling that it's some simple configuration issue, but I can't figure out what.
To test it, my manifest consists of the single line
Manifest-Version: 1.0

Notepad save your file with a special mark named 'BOM' at the beginning : 2 bytes (U+FEFF) which identify UTF-8 format.
You have to edit the file with Notepad++ (or vi) and encode it as UTF-8 (without BOM).
Next steps to make JARs:
Using eclipse
Using Apache ANT
Using Apache Maven

In Notepad++ , from the encoding menu , choose encode in UTF-8 without BOM .

In eclipse, right click in file properties and choose text file encoding to ISO-8859-1
Back in editor delete the special chars.

Why do I need to escape unicode in java source files?

Please note that I'm not asking how but why. And I don't know if it's a RCP specific problem or if it's something inherent to java.
My java source files are encoded in UTF-8.
If I define my literal strings like this :
new Language("fr", "Français"),
new Language("zh", "中文")
It works as I expect when I use the string in the application by launching it from Eclipse as an Eclipse application :
But if fails when I launch the .exe built by the "Eclipse Product Export Wizard" :
The solution I use is to escape the chars like this :
new Language("fr", "Fran\u00e7ais"), // Français
new Language("zh", "\u4e2d\u6587") // 中文
There is no problem in doing this (all my other strings are in properties files, only the languages names are hardcoded) but I'd like to understand.
I thought the compiler had to convert the java literal strings when building the bytecode. So why is the unicode escaping necessary ? Is it wrong to use use high range unicode chars in java source files ? What happens exactly to those chars at compilation and in what it is different from the handling of escaped chars ? Is the problem just related to RCP cache ?

It appears that the Eclipse Product Export Wizard is not interpreting your files as UTF-8. Perhaps you need to run Eclipse's JVM with the encoding set to UTF-8 (-Dfile.encoding=UTF8 in eclipse.ini)?
(Copypasta'd at OPs request)

When exporting a plug-in, it gets compiled through a process separate from the normal build process within the IDE. There is a known bug that the build process (PDE.Build) disregards the text encoding used by the IDE.
The export can be made to work properly by specifying the text encoding in the build.properties file of your plugin
javacDefaultEncoding.. =UTF-8

Saving JSP as UTF-8 in NetBeans

i've got some jsp files from another developers and now need to work with them. When i add to the document any UTF-8 char and want to save the document, NetBeans automatically offers me saving in ISO-8859-1.
Actually i'm getting this message from NetBeans:
The index.jsp contains characters
which will probably be damaged during
conversion to the ISO-8859-1 character
set. Do you want to save the file
using this character set? (Yes/No)
NB didn't offer me any other option like saving the file as UTF-8 (as it should be already written in).
I don't know how to save those jsp files in the character set they are already written in.
And don't tell me, that changing the content of the file itself (which is uneffective due to including headers etc. from other files) is the only way...
http://forums.netbeans.org/topic8750.html

Firstly; don't forget to consider this line at top:
<%#page contentType="text/html" pageEncoding="UTF-8"%>
Secondly;
In the NetBeans folder there is a config file. There should be a line like that:
netbeans_default_options="-J-Xms32m -J-Xmx128m -J-XX:PermSize=32m -J-XX:MaxPermSize=160m -J-Xverify:none -J-Dapple.laf.useScreenMenuBar=true"
Add this to the end of the line:
-J-Dfile.encoding=UTF-8
Thirdly:
NetBeans implements a project encoding setting.
To change the language encoding for a project:
Right-click a project node in the Projects windows and choose Properties.
Under Sources, select an encoding value from the Encoding drop-down field.
The encoding affects at least:
* how non-ASCII characters are displayed in the editor window when you open files
* Java file compilation of sources containing non-ASCII identifiers, string literals, or comments
* textual search for international characters over the project
Starting from NetBeans IDE 6.8, you can also specify the encoding that will be used at runtime. For example, this can be useful when the encoding for the operating system on which the application will run is different from your project's encoding.
To specify the encoding to be used at runtime:
In the Files window for your project, open nbproject > private > private.properties
Add the following line to the private.properties file and save changes:
runtime.encoding = < encoding >
This encoding will override the encoding setting for your project and will be used when running your application.
In general,
*.properties files always use ISO-8859-1 encoding plus \uXXXX escapes. (International characters will be displayed natively in the editor but stored as an escape on disk.)
*.xml files and some *.html files can specify their own encodings, regardless of the project encoding. For such files, the IDE's editor ignores the project encoding.
These may help you.
Sources for my answer that I used:
Link1: http://forums.netbeans.org/topic33.html
Link2: http://wiki.netbeans.org/FaqI18nProjectEncoding

Illegal Character when trying to compile java code

I have a program that allows a user to type java code into a rich text box and then compile it using the java compiler. Whenever I try to compile the code that I have written I get an error that says that I have an illegal character at the beginning of my code that is not there. This is the error the compiler is giving me:
C:\Users\Travis Michael>"\Program Files\Java\jdk1.6.0_17\bin\javac" Test.java
Test.java:1: illegal character: \187
public class Test
^
Test.java:1: illegal character: \191
public class Test
^
2 errors

The BOM is generated by, say, File.WriteAllText() or StreamWriter when you don't specify an Encoding. The default is to use the UTF8 encoding and generate a BOM. You can tell the java compiler about this with its -encoding command line option.
The path of least resistance is to avoid generating the BOM. Do so by specifying System.Text.Encoding.Default, that will write the file with the characters in the default code page of your operating system and doesn't write a BOM. Use the File.WriteAllText(String, String, Encoding) overload or the StreamWriter(String, Boolean, Encoding) constructor.
Just make sure that the file you create doesn't get compiled by a machine in another corner of the world. It will produce mojibake.

That's a byte order mark, as everyone says.
javac does not understand the BOM, not even when you try something like
javac -encoding UTF8 Test.java
You need to strip the BOM or convert your source file to another encoding. Notepad++ can convert a single files encoding, I'm not aware of a batch utility on the Windows platform for this.
The java compiler will assume the file is in your platform default encoding, so if you use this, you don't have to specify the encoding.

If using an IDE, specify the java file encoding (via the properties panel)
If NOT using an IDE, use an advanced text-editor (I can recommend Notepad++) and set the encoding to "UTF without BOM", or "ANSI", if that suits you.

In this case do the following Steps 1-7
In Android Studio
1. Menu -> Edit -> Select All
2. Menu -> Edit -> Cut
Open new Notepad.exe
In Notepad
4. Menu -> Edit -> Paste
5. Menu -> Edit -> Select All
6. Menu -> Edit -> Copy
Back In Android Studio
7. Menu -> Edit -> Paste

http://en.wikipedia.org/wiki/Byte_order_mark
The byte order mark (BOM) is a Unicode
character used to signal the
endianness (byte order) of a text file
or stream. Its code point is U+FEFF.
BOM use is optional, and, if used,
should appear at the start of the text
stream. Beyond its specific use as a
byte-order indicator, the BOM
character may also indicate which of
the several Unicode representations
the text is encoded in.
The BOM is a funky-looking character that you sometimes find at the start of unicode streams, giving a clue what the encoding is. It's usually handles invisibly by the string-handling stuff in Java, so you must have confused it somehow, but without seeing your code, it's hard to see where.
You might be able to fix it trivially by manually stripping the BOM from the string before feeding it to javac. It probably qualifies as whitespace, so try calling trim() on the input String, and feeding the output of that to javac.

That's a problem related to BOM (Byte Order Mark) character. Byte Order Mark BOM is an Unicode character used for defining a text file byte order and comes in the start of the file. Eclipse doesn't allow this character at the start of your file, so you must delete it. for this purpose, use a rich text editor like Notepad++ and save the file with encoding "UTF-8 without BOM". That should remove the problem.
I have copy pasted the some content from a website to a Notepad++ editor,
it shows the "LS" with black background. Have deleted the "LS" content and
have copy the same content from notepad++ to java file, it works fine.

I solved this by right clicking in my textEdit program file and selecting [substitutions] and un-checking smart quotes.

instead of getting Notepad++,
You can simply
Open the file with Wordpad
and then
Save As - Plain Text document

Even I was facing this issue as am using notepad++ to code. It is very convenient to type the code in notepad++. However after compiling I get an error " error: illegal character: '\u00bb'".
Solution :
Start writing the code in older version of notepad(which will be there by default in your PC) and save it. Later the modifications can be done using notepad++.
It works!!!

I had the same problem with a file i generated using the command echo echo "" > Main.java in Windows Powershell. I searched the problem and it seemed to have something to do with encoding. I checked the encoding of the file using file -i Main.java and the result was text/plain; charset=utf-16le.
Later i deleted the file and recreated it using git bash using touch Main.java and with this the file compiled successfully. I checked the file encoding using file -i command and this time the result was Main.java: text/x-c; charset=us-ascii.
Next i searched the internet and found that to create an empty file using Powershell we can use the Cmdlet New-Item. I create the file using New-Item Main.java and checked it's encoding and this time the result was Main.java: text/x-c; charset=us-ascii and this time it compiled successully.