I am reading a csv file in java, adding a new column with new information and exporting it back to a CSV file. I have a problem in reading the CSV file in UTF-8 format. I read line by line and store it in a StringBuilder, but when I print the line I can see that the information I'm reading is not in UTF-8 but in ANSI. I used both System.out.print and printstream in UTF and the information appears still in ANSI. This is my code :
BufferedReader br;
try {
br = new BufferedReader(new InputStreamReader(new FileInputStream(
"./users.csv"), "UTF8"));
String line;
while ((line = br.readLine()) != null) {
if (line.contains("none#none.com")) {
continue;
}
if (!line.contains("#") && !line.contains("FirstName")) {
continue;
}
PrintStream ps = new PrintStream(System.out, true, "UTF-8");
ps.print(line + "\n");
sbusers.append(line);
sbusers.append("\n");
sbusers2.append(line);
sbusers2.append(",");
}
br.close();
} catch (IOException e) {
System.out.println("Failed to read users file.");
} finally {
}
It prints out information like "Professor -P�s". Since the reading isn't being done correctly the output to the new file is also being exported in ANSI.
Are you sure your CSV is UTF-8 encoded? My guess is that it's not. Try using ISO-8859-1 for reading the file, but keep the output as UTF-8. (UTF8 and UTF-8 both tend to work, but you should use UTF-8 as #Marcelo suggested)
In the line:
br = new BufferedReader(new InputStreamReader(new FileInputStream("./users.csv"),"UTF8"));
Your charset should be "UTF-8" not "UTF8".
Printing to System.out using UTF encoding ????????????
Why would you do that ? System.out and the encoding it uses is determined at the OS level (it becomes the default charset in the JVM), and that's the only one you want to use on System.out.
Fist, as suggested by #Marcelo, use UTF8 instead of UTF-8:
BufferedReader in = new BufferedReader(
new InputStreamReader(
new FileInputStream("./users.csv"), "UTF8"));
Second, forget about the PrintStream, just use System.out, or better yet, a logging API. You don't need to worry about how Java will output your string to the console (number one rule about character encoding: After you've read things successfully, let Java handle the encoding and only worry about it again when you are writing to an external file / database / etc).
Third and more important, check that your file is really encoded in UTF-8, this is the cause of 99% of the encoding problems.
Make sure that you test with a real UTF-8 file (use tools like iconv to convert to UTF-8 and be sure about it).
found a potential solution(I had the same problem). Depending on the type of UTF-8 encoding you need to specify if further...
Replace:
br = new BufferedReader(new InputStreamReader(new FileInputStream(
"./users.csv"), "UTF8"));
With:
br = new BufferedReader(new InputStreamReader(new FileInputStream(
"./users.csv"), "ISO_8859_1"));
For further understanding: https://mincong.io/2019/04/07/understanding-iso-8859-1-and-utf-8/
Related
I'm having some problems with something simple and it's doing my head in. I have some entries in an Excel spreadsheet that contains various Asian characters etc. When exported as "Unicode text (*.txt)" with UTF-8 selected as encoding I can view it correctly in Notepad, but when I try to print it in the Eclipse console I get gibberish. I've tried variations as to how to read as UTF-8 and I know the console can display it:
try {
//BufferedReader in = new BufferedReader(new FileReader("testtest.txt"));
File fileDir = new File("testestet.txt");
//PrintStream out = new PrintStream(System.out, true, "UTF-8"); // tried this just in case
System.out.println("사과"); // this prints just fine
BufferedReader in = new BufferedReader(
new InputStreamReader(new FileInputStream(fileDir), StandardCharsets.UTF_8));
String line;
while((line = in.readLine()) != null)
{
System.out.println(line);
}
in.close();
}
catch (Exception e) { e.printStackTrace(); }
Any ideas? Whatever I solution I've found here has not worked. I'm wondering if excel is just borked...
Should you try this instead?
BufferedReader in = new BufferedReader(
new InputStreamReader(new FileInputStream(fileDir), StandardCharsets.UTF_8));
Heh, I found the problem while testing all StandardCharsets. UTF-16 works! I guess some of the characters I had in my Excel spreadsheet weren't supported by UTF-8 and Excel solved the problem without telling me...
i have a file which has the following string:
Vol conforme à la réglementation
However, when i read the file using SmbFileInputStream i get:
Vol conforme � la r�glementation
Could you please let me know the best way to read this file so that I get the String as I have in the original file. I am converting it to utf-8, which I am not sure is the correct way. Here is the current code that I am using:
SmbFileInputStream smbFileInputStream = new SmbFileInputStream(fileURL);
BufferedReader bufferedFileReader = new BufferedReader(new InputStreamReader(smbFileInputStream, "UTF-8"));
String line = null;
StringBuilder stringBuilder = new StringBuilder();
try {
while ((line = bufferedFileReader.readLine()) != null) {
if (!line.trim().isEmpty()) {
stringBuilder.append(line);
}
}
return stringBuilder.toString();
} finally {
bufferedFileReader.close();
}
Your file is not UTF-8 encoded. Based on the output of the baked string, it's probably ISO-8859-1 encoded or Windows cp1252 encoded, or even ISO-8859-15.
You should pass these encodings instead. It won't be quickly obvious to know which one of these encoding to use until your data contains a byte which maps to the wrong character.
The Euro symbol is a good test. It doesn't exist in ISO-8859-1 and is in different map positions in cp1252 and ISO-8859-15.
Notepad++ is an awesome tool for quickly checking files with different decodings.
I am trying to read a file containing greek words in utf8
with the following code
reader = new BufferedReader(new InputStreamReader(new FileInputStream(file), "UTF8"));
while((line = reader.readLine()) != null){
tokenizer = new StringTokenizer(line, delimiter);
while(tokenizer.hasMoreTokens()){
currentToken = tokenizer.nextToken();
map.put(currentToken, 1);
}
}
On every forum I looked for, I saw this new FileInputStream(file), "UTF8")
but still the printed results is like that ����
p.s. when i print a variable containing a greek word from inside the code, the print is successfull, that means that the problem is on file read.
any ideas?
There are some with too professionalism here. I remind you again that we are humans, not compilers! I am here again "powers" you deleted by post! I am very proud of being born in the birthplace of democracy, respecting the other discussants! You don't respect anything "guru" guys...
PS: Yeah, I know that you disseminate again down votes, but who really cares?
There is no "UTF8" charset in Java. The correct charset name is "UTF-8":
new InputStreamReader(new FileInputStream(file), "UTF-8"))
Or use StandardCharsets.UTF_8 instead to avoid any ambiguity:
new InputStreamReader(new FileInputStream(file), StandardCharsets.UTF_8))
That being said, make sure the file is actually UTF-8 encoded. If it has a UTF-8 BOM in front, you will have to either strip it off from the file itself, or manually skip it when reading the file before then reading the lines. Java readers do not recognize or skip BOMs automatically.
Use this for proper converstion - this one is from iso-8859-1 to utf-8:
public String to_utf8(String fieldvalue) throws UnsupportedEncodingException{
String fieldvalue_utf8 = new String(fieldvalue.getBytes("ISO-8859-1"), "UTF-8");
return fieldvalue_utf8;
}
I am using Marathi Wordnet.In this wordnet there are text documents including marathi words
I want to read these marathi documents in my java code.I have tried with using the BufferedReader and FileReader.But I failed.
This is the code I have tried.
FileReader fr=new FileReader("onto_txt");
BufferedReader br=new BufferedReader(fr);
String line=br.readLine();
while(line!=null){
System.out.println(line);
line=br.readLine();
}
fr.close();
br.close();
FileReader is an old utility class using the default encoding of the platform.
Assuming that the file is in UTF-8, better explicitly specify the encoding.
try (BufferedReader br = new BufferedReader(new InputStreamReader(
new FileInputStream("C:/xyz/onto_txt"), StandardCharsets.UTF_8))) {
String line = br.readLine();
while (line != null) {
System.out.println(line);
System.out.println(Arrays.toString(line.getBytes(StandardCharsets.UTF_8)));
line = br.readLine();
}
} // Closes br
Using System.out again converts the line to the encoding of the platform. That might not be able to display the String line; hence the dump of every single byte. Not very informative, but it might clarify that where ? is diplayed in the prior line, there really are Unicode characters.
Internally java String holds Unicode, and can contain any text. So you might process line as desired in the while.
I am using following code to write json to my local path which i get from my html page.Again I have to construct a html page by reading content from the saved local json file.For this I have to read this saved file from local which is plain text and give as input to java file. I got confused whether to use Buffered Reader or BufferedInputStream to read that file from local path.Please help me.
java.io.BufferedWriter jsonOut = new java.io.BufferedWriter(
new java.io.OutputStreamWriter(
new java.io.FileOutputStream(uploadDir +
_req.getParameter("filename")), "ISO-8859-1"));
BufferedReader for text.
Reason: http://tutorials.jenkov.com/java-io/bufferedreader.html
You can use BufferedReader for text but you should ensure to use the proper charset in your case (otherwise it defaults to the platform charset)
BufferedReader reader = new BufferedReader(new InputStreamReader(new FileInputStream(myFile),"ISO-8859-1"));
To read a file you can use the following code
File f = new File("your json file");
BufferedReader buf = new BufferedReader(new FileReader(f));
String line = null;
while ((line = buf.readLine()) != null) {
System.out.println("json file line " + line);
// do your changes
}