Develop Reference

Issue parsing arabic characters using java

I'm trying to migrate and parse a database to a new schema using Java.
Problem is there are some characters, specially arab, which get messed up while treating the data in Java.
Here's one of the lines where I'm getting trouble with which is in countryToParse.sql file:
(4, 'Afganistán', 1, 'Afgano', 'Afghanistan', 'AF', 'أفغانستان', 'Afghan', 'أفغاني');
After I parse it, the resulting line in countryParsed.sql is seen as:
(4, 'Afganistán', 1, 'Afgano', 'Afghanistan', 'AF', 'أ�?غانستان', 'Afghan', 'أ�?غاني');
You see how there are certain arabic characters that get messed up.
If I open the files I can check they are both coded in UTF-8.
Here's the Java code I'm using. In method writeToTextFile() I've added three ways I found to write a file using UTF-8 (not to mention I'm getting the same error using them three ways)
public class MainStackOverflow {
public static void main(String[] args) throws IOException {
String countryStr = new String(readTextFile("src/data/countryToParse.sql").getBytes(), "UTF-8");
writeToTextFile("src/data/countryParsed.sql", countryStr);
}
public static String readTextFile(String fileName) throws IOException {
String content = new String(Files.readAllBytes(Paths.get(fileName)));
return content;
}
public static void writeToTextFile(String fileName, String content) throws IOException {
/* Way 1 */
Files.write(Paths.get(fileName), content.getBytes("UTF-8"), StandardOpenOption.CREATE);
/* Way 2 */
BufferedWriter out = new BufferedWriter(new OutputStreamWriter(
new FileOutputStream(fileName), "UTF-8"));
try {
out.write(content);
} finally {
out.close();
}
/* Way 3 */
PrintWriter out1 = new PrintWriter(new File(fileName), "UTF-8");
out1.write(content);
out1.flush();
out1.close();
/* */
}
}

You forgot to set the encoding in this line:
String content = new String(Files.readAllBytes(Paths.get(fileName)));
Try simply this:
public static void main(String[] args) throws IOException {
String countryStr = new String(readTextFile("src/data/countryToParse.sql"), "UTF-8");
writeToTextFile("src/data/countryParsed.sql", countryStr);
}
public static byte[] readTextFile(String fileName) throws IOException {
return Files.readAllBytes(Paths.get(fileName));
}

file is not getting passed as an input

I'm getting NullPointerException upon passing a .xlsb file to a method. It seems that file is not getting selected but the file exists in that folder. Please advise
public class readxlsb {
public static void main(String[] args) throws Exception {
try {
String fileName1 ="C:\\abc\\TULDUBINV_EX_10768148_1.xlsb";
com.smartxls.WorkBook wb = new com.smartxls.WorkBook();
System.out.println(wb);
wb.readXLSB(fileName1);
System.out.println(wb);
} catch (Exception e) {
e.printStackTrace();
}
}
}
The constructor of the API have the following methods:
public void readXLSB(String filename) throws Exception {
this.a.q(filename);
}
public void readXLSB(InputStream in) throws Exception {
this.a.d(in);
}
public void readXLSB(InputStream in, String pass) throws Exception {
this.a.a(in, pass);
}

Why not to use jxl.Workbook, it's much easier and comprehensive. I have worked with jxl and highly rely on it. Use like-Workbook workBook = Workbook.getWorkbook(new File("C:\\abc\\TULDUBINV_EX_10768148_1.xlsb"));
Sheet sheet = workBook.getSheet(0);

Count the bytes written to file via BufferedWriter formed by GZIPOutputStream

I have a BufferedWriter as shown below:
BufferedWriter writer = new BufferedWriter(new OutputStreamWriter(
new GZIPOutputStream( hdfs.create(filepath, true ))));
String line = "text";
writer.write(line);
I want to find out the bytes written to the file with out querying file like
hdfs = FileSystem.get( new URI( "hdfs://localhost:8020" ), configuration );
filepath = new Path("path");
hdfs.getFileStatus(filepath).getLen();
as it will add overhead and I don't want that.
Also I cant do this:
line.getBytes().length;
As it give size before compression.

You can use the CountingOutputStream from Apache commons IO library.
Place it between the GZIPOutputStream and the file Outputstream (hdfs.create(..)).
After writing the content to the file you can read the number of written bytes from the CountingOutputStream instance.

If this isn't too late and you are using 1.7+ and you don't wan't to pull in an entire library like Guava or Commons-IO, you can just extend the GZIPOutputStream and obtain the data from the associated Deflater like so:
public class MyGZIPOutputStream extends GZIPOutputStream {
public MyGZIPOutputStream(OutputStream out) throws IOException {
super(out);
}
public long getBytesRead() {
return def.getBytesRead();
}
public long getBytesWritten() {
return def.getBytesWritten();
}
public void setLevel(int level) {
def.setLevel(level);
}
}

You can make you own descendant of OutputStream and count how many time write method was invoked

This is similar to the response by Olaseni, but I moved the counting into the BufferedOutputStream rather than the GZIPOutputStream, and this is more robust, since def.getBytesRead() in Olaseni's answer is not available after the stream has been closed.
With the implementation below, you can supply your own AtomicLong to the constructor so that you can assign the CountingBufferedOutputStream in a try-with-resources block, but still retrieve the count after the block has exited (i.e. after the file is closed).
public static class CountingBufferedOutputStream extends BufferedOutputStream {
private final AtomicLong bytesWritten;
public CountingBufferedOutputStream(OutputStream out) throws IOException {
super(out);
this.bytesWritten = new AtomicLong();
}
public CountingBufferedOutputStream(OutputStream out, int bufSize) throws IOException {
super(out, bufSize);
this.bytesWritten = new AtomicLong();
}
public CountingBufferedOutputStream(OutputStream out, int bufSize, AtomicLong bytesWritten)
throws IOException {
super(out, bufSize);
this.bytesWritten = bytesWritten;
}
#Override
public void write(byte[] b) throws IOException {
super.write(b);
bytesWritten.addAndGet(b.length);
}
#Override
public void write(byte[] b, int off, int len) throws IOException {
super.write(b, off, len);
bytesWritten.addAndGet(len);
}
#Override
public synchronized void write(int b) throws IOException {
super.write(b);
bytesWritten.incrementAndGet();
}
public long getBytesWritten() {
return bytesWritten.get();
}
}

Append to ObjectOutputStream iteratively

I want to add lots of data to a file. I defined the HYB class since my object contains ofdifferent types of data (String and byte[]). I used ObjectOutputStream and ObjectInputStream to write and read from the file. But my code does not print the expected result. To write my code I used code in the following pages:
How can I append to an existing java.io.ObjectStream?
ClassCastException when Appending Object OutputStream
I try to debug my code and found the problem but I could not. This is my code:
import java.io.*;
import java.io.BufferedOutputStream;
import java.util.*;
public class HYB implements Serializable
{
private static final long serialVersionUID = 1L;
private List<byte[]> data = new ArrayList<>();
public void addRow(String s,byte[] a)
{
data.add(s.getBytes()); // add encoding if necessary
data.add(a);
}
#Override public String toString()
{
StringBuilder sb = new StringBuilder();
synchronized (data)
{
for(int i=0;i<data.size();i+=2)
{
sb.append(new String(data.get(i)));
sb.append(Arrays.toString(data.get(i+1))+"\n");
}
}
return sb.toString();
}
private static void write(File storageFile, HYB hf)
throws IOException {
ObjectOutputStream oos = getOOS(storageFile);
oos.writeObject(hf);
oos.flush();
oos.close();
}
public static ObjectOutputStream getOOS(File file) throws IOException
{
if (file.exists()) {
return new AppendableObjectOutputStream(new FileOutputStream(file, true));
} else {
return new ObjectOutputStream(new FileOutputStream(file));
}
}
private static ObjectInputStream getOIS(FileInputStream fis)
throws IOException {
long pos = fis.getChannel().position();
return pos == 0 ? new ObjectInputStream(fis) :
new AppendableObjectInputStream(fis);
}
private static class AppendableObjectOutputStream extends
ObjectOutputStream {
public AppendableObjectOutputStream(OutputStream out)
throws IOException {
super(out);
}
#Override
protected void writeStreamHeader() throws IOException {
}
}
private static class AppendableObjectInputStream extends ObjectInputStream {
public AppendableObjectInputStream(InputStream in) throws IOException {
super(in);
}
#Override
protected void readStreamHeader() throws IOException {
// do not read a header
}
}
public static void main(String[] args) throws FileNotFoundException, IOException, ClassNotFoundException
{
File x=new File ("test");
HYB hf1 = new HYB();
hf1.addRow("fatemeh",new byte[] {11,12,13});
hf1.addRow("andisheh",new byte[] {14,15,16});
write(x,hf1);
HYB hf = new HYB();
hf.addRow("peter",new byte[] {1,2,3});
hf.addRow("jaqueline",new byte[] {4,5,6});
write(x,hf);
FileInputStream fis = new FileInputStream(x);
HYB hf2 = (HYB) getOIS(fis).readObject();
System.out.println(hf2);
}
}
expected results:
fatemeh[11, 12, 13]
andisheh[14, 15, 16]
peter[1, 2, 3]
jaqueline[4, 5, 6]
actual results:
fatemeh[11, 12, 13]
andisheh[14, 15, 16]

Writing the two HYB objects to the ObjectOutputStream doesn't merge them into a single HYB object; the ObjectOutputStream still contains two HYB object, of which your code reads one. If you did a second call to readObject(), the second one would be retrieved and could be printed to the screen. So you could just wrap the readObject() and println() calls in a loop that reads/writes until there's nothing else to read from the stream.

You are writing two HYB objects to the stream, but only reading one out.
You need to readObject() twice.

Changing the value of a "File" parameter without creating a new instance of a class

I have a little problem here.
I am declaring a new object "Fleet" in these methods:
public void run() throws FileNotFoundException
{
File file = new File(getFile());
Fleet fleet = new Fleet(file);
buildFleet(file, fleet);
}
private void buildFleet(File file, Fleet fleet) throws FileNotFoundException
{
fleet.addVehicle(Honda);
userMenu(fleet);
}
The last line calls the userMenu() method. In this method I need to be able to change the value of "File" inside Fleet without creating a new instance of the class.
private void userMenu(Fleet fleet) throws FileNotFoundException
{
PrintWriter pw = new PrintWriter("temp.txt");
File file = new File("temp.txt");
fleet = new Fleet(file);
this.createMenu();
choice = this.menu.getChoice();
while(choice != 8)
{
switch(choice)
{
case 1:
//Do stuff
fleet.addVehicle(Honda);
break;
}
}
Also, I am not allowed to create any new class level data.
Any suggestions?

What about a setter on your Fleet class for the file:
public class Fleet {
private File file;
...
public void setFile( File file ){
this.file = file;
}
}
You can then call this method to change the file inside your fleet object by calling
fleet.setFile( myNewFile );

Solved:
I changed:
private void userMenu() throws FileNotFoundException
{
PrintWriter pw = new PrintWriter("temp.txt");
File file = new File("temp.txt");
to:
private void userMenu(Fleet fleet) throws FileNotFoundException
{
PrintWriter pw = new PrintWriter("temp.txt");
File file = new File("temp.txt");
fleet.file = file;