Develop Reference

Weird issue with zip/gzip files

Below is a program which saves the bytes to a .png file and zips into a given name folder.
byte[] decodedBytes = Base64.decodeBase64(contents);
// System.out.println(new String(decodedBytes));
InputStream targetStream = new ByteArrayInputStream(decodedBytes);
int count;
ZipOutputStream out = new ZipOutputStream(new FileOutputStream(zipFilePath));
out.putNextEntry(new ZipEntry(pngFile));
byte[] b = new byte[1024];
while ((count = targetStream.read(b)) > 0) {
out.write(b, 0, count);
}
out.flush();
out.close();
targetStream.close();
When i open it manually using 7 zip I see the following folder structure (c:\output\nameofzipfile.zip\nameofpng.png\nameofpng). Why is this happening? What am I doing wrong? As per my understanding this should be the structure (c:\output\nameofzipfile.zip\nameofpng.png)

Worked with the following code
byte[] decoded = Base64.decodeBase64(contents);
try (FileOutputStream fos = new FileOutputStream(zipFilePath + amazonOrderId + zipFileName)) {
fos.write(decoded);
fos.close();
}
file = new File(destDirectory + amazonOrderId + pngFile);
if (file.exists()) {
file.delete();
}
try (OutputStream out = new FileOutputStream(destDirectory + amazonOrderId + pngFile)) {
try (InputStream in = new GZIPInputStream(
new FileInputStream(zipFilePath + amazonOrderId + zipFileName))) {
byte[] buffer = new byte[65536];
int noRead;
while ((noRead = in.read(buffer)) != -1) {
out.write(buffer, 0, noRead);
}
}
}

java OutOfMemoryError about FileOutputStream?

Thanks for everyone ^_^,the problem is solved:there is a single line is too big(over 400M...I download a damaged file while I didn't realize), so throw a OutOfMemoryError
I want to split a file by using java,but it always throw OutOfMemoryError: Java heap space,I searched on the whole Internet,but it looks like no help :(
ps. the file's size is 600M,and it have over 30,000,000 lines,every line is no longer than 100 chars.
(maybe you can generate a "level file" like this:{
id:0000000001,level:1
id:0000000002,level:2
....(over 30 millions)
})
pss. set the Jvm memory size larger is not work,:(
psss. I changed to another PC, problem remains/(ㄒoㄒ)/~~
no matter how large the -Xms or -Xmx I set,the outputFile's size is always same,(and the Runtime.getRuntime().totalMemory() is truely changed)
here's the stack trace:
Heap Size = 2058027008
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2882)
at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:515)
at java.lang.StringBuffer.append(StringBuffer.java:306)
at java.io.BufferedReader.readLine(BufferedReader.java:345)
at java.io.BufferedReader.readLine(BufferedReader.java:362)
at com.xiaomi.vip.tools.ptupdate.updator.Spilt.main(Spilt.java:39)
...
here's my code:
package com.updator;
import java.io.BufferedReader;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.FileReader;
public class Spilt {
public static void main(String[] args) throws Exception {
long heapSize = Runtime.getRuntime().totalMemory();
// Print the jvm heap size.
System.out.println("Heap Size = " + heapSize);
String mainPath = "/home/work/bingo/";
File mainFilePath = new File(mainPath);
FileInputStream inputStream = null;
FileOutputStream outputStream = null;
try {
if (!mainFilePath.exists())
mainFilePath.mkdir();
String sourcePath = "/home/work/bingo/level.txt";
inputStream = new FileInputStream(sourcePath);
BufferedReader bufferedReader = new BufferedReader(new FileReader(
new File(sourcePath)));
String savePath = mainPath + "tmp/";
Integer i = 0;
File file = new File(savePath + "part"
+ String.format("%0" + 5 + "d", i) + ".txt");
if (!file.getParentFile().exists())
file.getParentFile().mkdir();
file.createNewFile();
outputStream = new FileOutputStream(file);
int count = 0, total = 0;
String line = null;
while ((line = bufferedReader.readLine()) != null) {
line += '\n';
outputStream.write(line.getBytes("UTF-8"));
count++;
total++;
if (count > 4000000) {
outputStream.flush();
outputStream.close();
System.gc();
count = 0;
i++;
file = new File(savePath + "part"
+ String.format("%0" + 5 + "d", i) + ".txt");
file.createNewFile();
outputStream = new FileOutputStream(file);
}
}
outputStream.close();
file = new File(mainFilePath + "_SUCCESS");
file.createNewFile();
outputStream = new FileOutputStream(file);
outputStream.write(i.toString().getBytes("UTF-8"));
} finally {
if (inputStream != null)
inputStream.close();
if (outputStream != null)
outputStream.close();
}
}
}
I think maybe: when outputStream.close(),the memory did not release?

So you open the original file and create a BufferedReaderand a counter for the lines.
char[] buffer = new char[5120];
BufferedReader reader = Files.newBufferedReader(Paths.get(sourcePath), StandardCharsets.UTF_8);
int lineCount = 0;
Now you read into your buffer, and write the characters as they come in.
int read;
BufferedWriter writer = Files.newBufferedWriter(Paths.get(fileName), StandardCharsets.UTF_8);
while((read = reader.read(buffer, 0, 5120))>0){
int offset = 0;
for(int i = 0; i<read; i++){
char c = buffer[i];
if(c=='\n'){
lineCount++;
if(lineCount==maxLineCount){
//write the range from 0 to i to your old writer.
writer.write(buffer, offset, i-offset);
writer.close();
offset=i;
lineCount=0;
writer = Files.newBufferedWriter(Paths.get(newName), StandarCharset.UTF_8);
}
}
writer.write(buffer, offset, read-offset);
}
writer.close();
}
That should keep the memory usage lower and prevent you from reading too large of a line at once. You could go without BufferedWriters and control the memory even more, but I don't think that is necessary.

I've tested with large text file.(250Mb)
it works well.
You need to add try catch exception codes for file stream.
public class MyTest {
public static void main(String[] args) {
String mainPath = "/home/work/bingo/";
File mainFilePath = new File(mainPath);
FileInputStream inputStream = null;
FileOutputStream outputStream = null;
try {
if (!mainFilePath.exists())
mainFilePath.mkdir();
String sourcePath = "/home/work/bingo/level.txt";
inputStream = new FileInputStream(sourcePath);
Scanner scanner = new Scanner(inputStream, "UTF-8");
String savePath = mainPath + "tmp/";
Integer i = 0;
File file = new File(savePath + "part" + String.format("%0" + 5 + "d", i) + ".txt");
if (!file.getParentFile().exists())
file.getParentFile().mkdir();
file.createNewFile();
outputStream = new FileOutputStream(file);
int count = 0, total = 0;
while (scanner.hasNextLine()) {
String line = scanner.nextLine() + "\n";
outputStream.write(line.getBytes("UTF-8"));
count++;
total++;
if (count > 4000000) {
outputStream.flush();
outputStream.close();
count = 0;
i++;
file = new File(savePath + "part" + String.format("%0" + 5 + "d", i) + ".txt");
file.createNewFile();
outputStream = new FileOutputStream(file);
}
}
outputStream.close();
file = new File(mainFilePath + "_SUCCESS");
file.createNewFile();
outputStream = new FileOutputStream(file);
outputStream.write(i.toString().getBytes("UTF-8"));
} catch (FileNotFoundException e) {
System.out.println("ERROR: FileNotFoundException :: " + e.getStackTrace());
} catch (IOException e) {
System.out.println("ERROR: IOException :: " + e.getStackTrace());
} finally {
if (inputStream != null)
try {
inputStream.close();
if (outputStream != null)
outputStream.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
if the problem still occurs, change java heap memory size with following command on the shell prompt.
ex)
Xmx1g : 1Gb heap memory size,
MyTest : class name
java -Xmx1g MyTest

Reading JPEG Stream over socket gives Null characters

I am reading a .jpg file over InputStream using this code but I am receiving NULNUL...n stream after some text. Ii am reading this file link to file and link of file that I received , link is Written File link.
while ((ret = input.read(imageCharArray)) != -1) {
packet.append(new String(imageCharArray, 0, ret));
totRead += ret;
imageCharArray = new char[4096];
}
file = new File(
Environment.getExternalStorageDirectory()
+ "/FileName_/"
+ m_httpParser.filename + ".jpg");
PrintWriter printWriter = new PrintWriter(file);
// outputStream = new FileOutputStream(file); //also Used FileoutputStream for writting
// outputStream.write(packet.toString().getBytes());//
// ,
printWriter.write(packet.toString());
// outputStream.close();
printWriter.close();
}
I have also tried FileoutputStream but hardlucj for this too as commented in my code.
Edit
I have used this also. I have a content length field upto which i am reading and writing
byte[] bytes = new byte[1024];
int totalReadLength = 0;
// read untill we have bytes
while ((read = inputStream.read(bytes)) != -1
&& contentLength >= (totalReadLength)) {
outputStream.write(bytes, 0, read);
totalReadLength += read;
System.out.println(" read size ======= "
+ read + " totalReadLength = "
+ totalReadLength);
}

String is not a container for binary data, and PrintWriter isn't a way to write it. Get rid of all, all, the conversions between bytes and String and vice versa, and just transfer the bytes with input and output streams:
while ((count = in.read(buffer)) > 0)
{
out.write(buffer, 0, count);
}
If you need to constrain the number of bytes read from the input, you have to do that before calling read(), and you also have to constrain the read() correctly:
while (total < length && (count = in.read(buffer, 0, length-total > buffer.length ? buffer.length: (int)(length-total))) > 0)
{
total += count;
out.write(buffer, 0, count);
}

I tested it in my Nexus4 and it's working for me. Here is the snippet of code what I tried :
public void saveImage(String urlPath)throws Exception{
String fileName = "kumar.jpg";
File folder = new File("/sdcard/MyImages/");
// have the object build the directory structure, if needed.
folder.mkdirs();
final File output = new File(folder,
fileName);
if (output.exists()) {
output.delete();
}
InputStream stream = null;
FileOutputStream fos = null;
try {
URL url = new URL(urlPath);
stream = url.openConnection().getInputStream();
// InputStreamReader reader = new InputStreamReader(stream);
DataInputStream dis = new DataInputStream(url.openConnection().getInputStream());
byte[] fileData = new byte[url.openConnection().getContentLength()];
for (int x = 0; x < fileData.length; x++) { // fill byte array with bytes from the data input stream
fileData[x] = dis.readByte();
}
dis.close();
fos = new FileOutputStream(output.getPath());
fos.write(fileData);
} catch (Exception e) {
e.printStackTrace();
} finally {
if (stream != null) {
try {
stream.close();
} catch (IOException e) {
e.printStackTrace();
}
}
if (fos != null) {
try {
fos.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
Just Call the above function in a background thread and pass your url. It'll work for sure. Let me know if it helps.

You can check below code.
destinationFile = new File(
Environment.getExternalStorageDirectory()
+ "/FileName_/"
+ m_httpParser.filename + ".jpg");
BufferedOutputStream buffer = new BufferedOutputStream(new FileOutputStream(destinationFile));
byte byt[] = new byte[1024];
int i;
for (long l = 0L; (i = input.read(byt)) != -1; l += i ) {
buffer.write(byt, 0, i);
}
buffer.close();

PDF to Bytearray conversion in java

Below is my code to convert a PDF file to byte array
public class ByteArrayExample{
public static void main(String[] args) {
try{
BufferedReader bf = new BufferedReader(new InputStreamReader(System.in));
System.out.println("Enter File name: ");
String str = bf.readLine();
File file = new File(str);
//File length
int size = (int)file.length();
if (size > Integer.MAX_VALUE){
System.out.println("File is to larger");
}
byte[] bytes = new byte[size];
DataInputStream dis = new DataInputStream(new FileInputStream(file));
int read = 0;
int numRead = 0;
while (read < bytes.length && (numRead=dis.read(bytes, read,
bytes.length-read)) >= 0) {
read = read + numRead;
}
System.out.println("File size: " + read);
// Ensure all the bytes have been read in
if (read < bytes.length) {
System.out.println("Could not completely read: "+file.getName());
}
}
catch (Exception e){
e.getMessage();
}
}
}
Issue is this actually converts the file name into the byte array not the actual PDF file.Can anyone please help me with this.

I added this to the end to check it and it copied the PDF file. Your code is working fine
dis.close();
DataOutputStream out = new DataOutputStream(new FileOutputStream(new File("c:\\out.pdf")));
out.write(bytes);
out.close();
System.out.println("File size: " + read);
// Ensure all the bytes have been read in
if (read < bytes.length) {
System.out.println("Could not completely read: "+file.getName());
}
edit: here is my entire code, its just copied from yours. I ran it in IDE (eclipse) and entered "c:\mypdf.pdf" for the input and it copied it to out.pdf. Identical Copys. Do note that I did close both streams which I noticed you forgot to do in your code.
public class Main {
public static void main(String[] args) {
try {
BufferedReader bf = new BufferedReader(new InputStreamReader(System.in));
System.out.println("Enter File name: ");
String str = bf.readLine();
File file = new File(str);
//File length
int size = (int) file.length();
if (size > Integer.MAX_VALUE) {
System.out.println("File is to larger");
}
byte[] bytes = new byte[size];
DataInputStream dis = new DataInputStream(new FileInputStream(file));
int read = 0;
int numRead = 0;
while (read < bytes.length && (numRead = dis.read(bytes, read,
bytes.length - read)) >= 0) {
read = read + numRead;
}
dis.close();
DataOutputStream out = new DataOutputStream(new FileOutputStream(new File("c:\\out.pdf")));
out.write(bytes);
out.close();
System.out.println("File size: " + read);
// Ensure all the bytes have been read in
if (read < bytes.length) {
System.out.println("Could not completely read: " + file.getName());
}
} catch (Exception e) {
e.getMessage();
}
}
}

To convert the file into byte array

I am creating one directory i.e file and storing the bitmap images into that file,now how to convert it into byte array
File myDir = new File(root + "/saved_images");
myDir.mkdirs();
Random generator = new Random();
int n = 10000;
n = generator.nextInt(n);
String fname = "Image-"+ n +".jpg";
File file = new File (myDir, fname);
if (file.exists ()) file.delete ();
try {
FileOutputStream out = new FileOutputStream(file);
bmp.compress(Bitmap.CompressFormat.JPEG, 90, out);
out.flush();
out.close();
} catch (Exception e) {
e.printStackTrace();
}

If you just want to modify your existing code to write the image to a byte array instead of a file, then replace the try block with this code:
ByteArrayOutputStream out = new ByteArrayOutputStream();
bmp.compress(Bitmap.CompressFormat.JPEG, 90, out);
bytes = out.getBytes();
... where bytes has type byte[], and get rid of the code that generates the filename and deletes the existing file if it exists. Since you writing to a ByteArrayOutputStream, there is not need to call flush() or close() on out. (They won't do anything.)

Not exactly sure what you're trying to do, but you can try something like:
InputStream is = ...
ByteArrayOutputStream buffer = new ByteArrayOutputStream();
int nRead;
byte[] data = new byte[some huge number, power of 2 preferably];
while ((nRead = is.read(data, 0, data.length)) != -1) {
buffer.write(data, 0, nRead);
}
buffer.flush();
byte[] byteArray = buffer.toByteArray();

Just Use this to read the file where you kept.
// Returns the contents of the file in a byte array.
public static byte[] getBytesFromFile(File file) throws IOException {
InputStream is = new FileInputStream(file);
// Get the size of the file
long length = file.length();
// You cannot create an array using a long type.
// It needs to be an int type.
// Before converting to an int type, check
// to ensure that file is not larger than Integer.MAX_VALUE.
if (length > Integer.MAX_VALUE) {
// File is too large
}
// Create the byte array to hold the data
byte[] bytes = new byte[(int)length];
// Read in the bytes
int offset = 0;
int numRead = 0;
while (offset = 0) {
offset += numRead;
}
// Ensure all the bytes have been read in
if (offset
Courtesy : http://www.exampledepot.com

I have used this code for converting image file into byte araay,
Bitmap bm = BitmapFactory.decodeResource(getResources(),R.drawable.abc);
ByteArrayOutputStream bos = new ByteArrayOutputStream();
bm.compress(Bitmap.CompressFormat.JPEG, 40 , bos);
public byte[] bitmapdata = bos.toByteArray();
Log.w("Image Conversion", String.valueOf(bitmapdata.length));
String converted_txt="";
for (int i = 0; i < bitmapdata.length; i++)
{
Log.w("Image Conversion", String.valueOf(bitmapdata[i]));
ba = bitmapdata[i];
converted_txt=converted_txt+bitmapdata[i];
}
try
{
File myFile = new File("/sdcard/myImageToByteFile.jpg");
myFile.createNewFile();
fOut = new FileOutputStream(myFile);
OutputStreamWriter myOutWriter = new OutputStreamWriter(fOut);
myOutWriter.write(ba);
myOutWriter.close();
fOut.close();
}
catch (Exception e)
{
Toast.makeText(getApplicationContext(), e.getMessage(),5000).show();
}