I have reduced the strange issue in this code to the minimum. This program writes 128,000 times the bytes for (int)90000 into a file and then tries to read it back in.
set zipped=false and everything works like a charm
set zipped=true and everything works like a charm until the 496th chunk of 1024 bytes. At that point a single byte is lost and everything is shifted to the left by one byte (see output)
...
0 1 95 -112- which is byte code for int 90,000
Counters: 496 126937
1 95 -112 0- which is byte code for int 23,040,000
...
this is the code i came up with. I just can't figure out why it suddenly breaks in the middle of doing the same thing over and over. Any help/insights/explainers much appreciated.
public class TestApp7 {
static final boolean zipped = true;
static File theFile = null;
private static void writeZipData() throws Exception {
FileOutputStream fos = new FileOutputStream(theFile);
BufferedOutputStream bos = null;
if (zipped) {
GZIPOutputStream gzout = new GZIPOutputStream(fos);
bos = new BufferedOutputStream(gzout);
} else
bos = new BufferedOutputStream(fos);
byte[] bs9 = RHUtilities.toByteArray((int)90000);
for (int i=0; i<128000; i++)
bos.write(bs9);
bos.flush();
bos.close();
}
private static void readZipData() throws Exception {
byte[] buf = new byte[1024];
int chunkCounter = 0;
int intCounter = 0;
FileInputStream fin = new FileInputStream(theFile);
int rdLen = 0;
if (zipped) {
GZIPInputStream gin = new GZIPInputStream(fin);
while ((rdLen = gin.read(buf)) != -1) {
System.out.println("Counters: " + chunkCounter + " " + intCounter);
for (int i=0; i<rdLen/4; i++) {
byte[] bs = Arrays.copyOfRange(buf,(i*4),((i+1)*4));
intCounter++;
System.out.print(bs[0] + " " + bs[1] + " " + bs[2] + " " + bs[3]);
}
chunkCounter++;
}
gin.close();
} else {
while ((rdLen = fin.read(buf)) != -1) {
System.out.println("Counters: " + chunkCounter + " " + intCounter);
for (int i=0; i<rdLen/4; i++) {
byte[] bs = Arrays.copyOfRange(buf,(i*4),((i+1)*4));
intCounter++;
System.out.print(bs[0] + " " + bs[1] + " " + bs[2] + " " + bs[3]);
}
chunkCounter++;
}
}
fin.close();
}
public static void main(String args[]) {
try {
if (zipped)
theFile = new File("Test.gz");
else
theFile = new File("Test.dat");
writeZipData();
readZipData();
} catch (Throwable e) { e.printStackTrace(); }
}
}
So based on Jon's wonderful comments ... you cannot rely on .read(buffer) filling the buffer even when there are more bytes in the stream - it stops at the boundary where the BufferedOutputStream-wrapped GZIPOutputStream saved a chunk of data. just add another read to go beyond the boundary and complete the chunk
while ((rdLen = gin.read(buf)) != -1) {
if (rdLen<chunksize) {
byte[] missBytes = new byte[chunksize-rdLen];
int rdLine_miss = 0;
if ((rdLine_miss = gin.read(missBytes)) > 0)
System.arraycopy(missBytes,0,buf,rdLen,rdLine_miss);
rdLen += rdLine_miss;
}
for (int i=0; i<rdLen/4; i++) {
byte[] bs = Arrays.copyOfRange(buf,(i*4),((i+1)*4));
intCounter++;
System.out.println(bs[0] + " " + bs[1] + " " + bs[2] + " " + bs[3] + " ");
}
chunkCounter++;
}
Related
I'm trying to convert a file with a .wav extension to a double array but I am getting an error:
09-15 05:09:47.222 22358-22358/com.R1100.bluetooth D/R1100Err: unsupported audio format: '/storage/emulated/0/HeartSounds/a0002.wav'
The file really is a .wav but I have no idea why this happens.
Here is the method I used:
public static double[] read(String filename) {
byte[] data = readByte(filename);
int n = data.length;
double[] d = new double[n/2];
for (int i = 0; i < n/2; i++) {
d[i] = ((short) (((data[2*i+1] & 0xFF) << 8) + (data[2*i] & 0xFF))) / ((double) MAX_16_BIT);
}
return d;
}
// return data as a byte array
private static byte[] readByte(String filename) {
byte[] data = null;
AudioInputStream ais = null;
try {
// try to read from file
File file = new File(filename);
if (file.exists()) {
ais = AudioSystem.getAudioInputStream(file);
int bytesToRead = ais.available();
data = new byte[bytesToRead];
int bytesRead = ais.read(data);
if (bytesToRead != bytesRead)
throw new IllegalStateException("read only " + bytesRead + " of " + bytesToRead + " bytes");
}
// try to read from URL
else {
URL url = Wav.class.getResource(filename);
ais = AudioSystem.getAudioInputStream(url);
int bytesToRead = ais.available();
data = new byte[bytesToRead];
int bytesRead = ais.read(data);
if (bytesToRead != bytesRead)
throw new IllegalStateException("read only " + bytesRead + " of " + bytesToRead + " bytes");
}
}
catch (IOException e) {
throw new IllegalArgumentException("could not read '" + filename + "'", e);
}
catch (UnsupportedAudioFileException e) {
throw new IllegalArgumentException("unsupported audio format: '" + filename + "'", e);
}
return data;
}
Thanks.
I am writing code to deflate and inflate a string in base 64 encode but I am getting the following error:
Exception in thread "main" java.util.zip.ZipException: incorrect header check
at java.util.zip.InflaterOutputStream.write(InflaterOutputStream.java:284)
at java.io.FilterOutputStream.write(FilterOutputStream.java:108)
at serializer.test.SerializerTest.main(SerializerTest.java:43)
My code is:
XsltObject Xslt = new XsltObject();
Xslt.setXslt(readFile("C:\\codebase\\OverallSystem\\EBE_TEMPERED_XMLS\\bank_timestamp-0.xml"));
System.out.println("Original String Length: "+ Xslt.getXslt().length());
//JSONObject jsonObj = new JSONObject( Xslt );
// System.out.println( jsonObj );
//System.out.println( "Json Length:" + jsonObj);
DeflaterOutputStream outputStream;
for ( int i = 1; i <= 9; ++i ) {
ByteArrayOutputStream arrayOutputStream = new ByteArrayOutputStream();
outputStream = new DeflaterOutputStream(arrayOutputStream, new Deflater( i, true ));
outputStream.write(Xslt.getXslt().getBytes());
outputStream.close();
//System.out.println("Deflate (lvl=" + i + ");" + arrayOutputStream.toString("ISO-8859-1"));
System.out.println("Deflate (lvl=" + i + ");" + arrayOutputStream.toString("ISO-8859-1").length());
String temp = DatatypeConverter.printBase64Binary(arrayOutputStream.toString("UTF-8").getBytes());
System.out.println(temp);
System.out.println("Base 64 len: " + temp.length());
byte[] data =DatatypeConverter.parseBase64Binary(temp);
ByteArrayOutputStream inflateArrayOutputStream = new ByteArrayOutputStream();
InflaterOutputStream iis = new InflaterOutputStream(inflateArrayOutputStream, new Inflater());
iis.write(data);
iis.close();
System.out.println("Inflate (lvl=" + i + ");" + inflateArrayOutputStream.toString("ISO-8859-1"));
System.out.println("Inflate (lvl=" + i + ");" + inflateArrayOutputStream.toString("ISO-8859-1").length());
What am I doing wrong?
This fixed all my issues,and is all JDK usage:
package serializer.test;
import java.io.BufferedReader;
import java.io.FileReader;
import java.io.IOException;
import java.io.UnsupportedEncodingException;
import java.util.Arrays;
import java.util.zip.*;
import javax.xml.bind.DatatypeConverter;
public class DeflationApp
{
private String compressBase64(String stringToCompress, int level)
throws UnsupportedEncodingException
{
byte[] compressedData = new byte[1024];
byte[] stringAsBytes = stringToCompress.getBytes("UTF-8");
Deflater compressor = new Deflater(level, false);
compressor.setInput(stringAsBytes);
compressor.finish();
int compressedDataLength = compressor.deflate(compressedData);
byte[] bytes = Arrays.copyOf(compressedData, compressedDataLength);
return DatatypeConverter.printBase64Binary(bytes);
}
private String decompressToStringBase64(String base64String)
throws UnsupportedEncodingException, DataFormatException
{
byte[] compressedData = DatatypeConverter
.parseBase64Binary(base64String);
Inflater deCompressor = new Inflater();
deCompressor.setInput(compressedData, 0, compressedData.length);
byte[] output = new byte[100000];
int decompressedDataLength = deCompressor.inflate(output);
deCompressor.end();
return new String(output, 0, decompressedDataLength, "UTF-8");
}
public static void main(String[] args) throws DataFormatException,
IOException
{
DeflationApp m = new DeflationApp();
String strToBeCompressed = readFile(
"C:\\codebase\\OverallSystem\\MappingMapToEBECommon.xslt")
.trim();
for (int i = 1; i <= 9; ++i)
{
String compressedData = m.compressBase64(strToBeCompressed, i);
String deCompressedString = m.decompressToStringBase64(compressedData);
System.out.println("Base 64:");
System.out.println("Original Length with level("+i+"): " + strToBeCompressed.length());
System.out.println("Compressed with level("+i+"): " + compressedData.toString());
System.out.println("Compressed with level("+i+") Length: " + compressedData.toString().length());
System.out.println("Decompressed with level("+i+"): " +
+ deCompressedString.length());
System.out.println("Decompressed with level("+i+"): " + deCompressedString);
}
for (int i = 1; i <= 9; ++i)
{
byte[] compressedData = m.compress(strToBeCompressed, i);
String deCompressedString = m.decompressToString(compressedData);
System.out.println("Without Base 64:");
System.out.println("Original Length with level("+i+"): " + strToBeCompressed.length());
System.out.println("Compressed with level("+i+"): " + new String(compressedData));
System.out.println("Compressed with level("+i+") Length: " + new String(compressedData).length());
System.out.println("Decompressed with level("+i+"): " +
+ deCompressedString.length());
System.out.println("Decompressed with level("+i+"): " + deCompressedString);
}
}
private byte[] compress(String stringToCompress, int level) throws UnsupportedEncodingException
{
byte[] compressedData = new byte[1024];
byte[] stringAsBytes = stringToCompress.getBytes("UTF-8");
Deflater compressor = new Deflater(level, false);
compressor.setInput(stringAsBytes);
compressor.finish();
int compressedDataLength = compressor.deflate(compressedData);
return Arrays.copyOf(compressedData, compressedDataLength);
}
private String decompressToString(byte[] compressedData) throws UnsupportedEncodingException, DataFormatException
{
Inflater deCompressor = new Inflater();
deCompressor.setInput(compressedData, 0, compressedData.length);
byte[] output = new byte[100000];
int decompressedDataLength = deCompressor.inflate(output);
deCompressor.end();
return new String(output, 0, decompressedDataLength, "UTF-8");
}
public static String readFile(String file) throws IOException
{
BufferedReader reader = new BufferedReader(new FileReader(file));
String line = null;
StringBuilder stringBuilder = new StringBuilder();
String ls = System.getProperty("line.separator");
try
{
while ((line = reader.readLine()) != null)
{
stringBuilder.append(line);
stringBuilder.append(ls);
}
return stringBuilder.toString();
}
finally
{
reader.close();
}
}
}
I too had memory problems with DeflaterOutputStream - it works if you let it use the default constructor. This works fine :
for (Entry<String, String> entry : valueMap.entrySet()) {
String key = entry.getKey();
String value = entry.getValue();
ByteArrayOutputStream baos = new ByteArrayOutputStream();
DeflaterOutputStream dos = new DeflaterOutputStream(baos);
try {
dos.write(value.getBytes());
dos.flush();
dos.close();
}
catch (IOException e) {
throw new RuntimeException(e);
}
byte[] zipData = baos.toByteArray();
zipValueMap.put(key, zipData);
}
But change that to :
ByteArrayOutputStream baos = new ByteArrayOutputStream();
Deflater deflater = new Deflater(Deflater.BEST_SPEED);
DeflaterOutputStream dos = new DeflaterOutputStream(baos, deflater);
And that gives me memory leak in JVM C code that takes up 80g and crashes my mint system. So why would the default constructor work and yet when I pass my own deflator in it fails so badly :
Decoding DeflaterOutputStream (java 1.8_40) I find some special code in the close method :
public void close() throws IOException {
if (!closed) {
finish();
if (usesDefaultDeflater)
def.end();
out.close();
closed = true;
}
}
I guess they put that in to workaround problem with deflater.
The best solution was to call it explicitly in the loop :
try {
dos.write(value.getBytes());
dos.flush();
dos.close();
deflater.end();
}
And no more memory leak. It's a bad memory leak as well, since it is from the C side, so it never threw a JVM error, it just chewed up all the 40g of ram I had, and then started on the swap space. I had to ssh into the box and kill it.
I'm using the code below to read a Unix socket:
Boolean flag = false;
while (!flag) {
try {
File socketFile = new File("./RISP");
AFUNIXSocket sock = AFUNIXSocket.newInstance();
sock.connect(new AFUNIXSocketAddress(socketFile));
System.out.println("!!!!!!!!!!CONNECTED!!!!!!!!!");
flag = true;
BufferedReader input = new BufferedReader(new InputStreamReader(sock.getInputStream()));
String line = null;
while ((line = input.readLine())!=null) {
System.out.println(line);
}
} catch (IOException e) {
System.out.println("NOT CONNECTED....." + e);
}
try {
Thread.sleep(2000);
} catch (InterruptedException inter) {
System.out.println(inter);
}
}
I need to read the first 4 bytes of each pack and convert them from binary to integer.
I've read many posts but I'm still looking for the best solution to my problem.
Reader and Writer are designed for reading text.
For binary, you should try InputStream and OutputStream, in this case, you want DataInputStream, possibly buffered.
DataInputStream in = new DataInputStream(new BufferedInputStream(socket.getInputStream()));
int len = in.readInt(); // read big-endian.
if (LITTLE_ENDIAN)
len = Integer.reverseBytes(len);
byte[] bytes = new byte[len];
in.readFully(bytes);
seems it works
DataInputStream in = new DataInputStream(new BufferedInputStream(sock.getInputStream()));
int len = -1;
while ((len = in.readInt()) != -1) {
len = Integer.reverseBytes(len);
byte[] bytes = new byte[len];
in.readFully(bytes);
if (bytes.length > 4) {
System.out.println(" BYTE0: " + bytes[0] +
" BYTE1: " + bytes[1] +
" BYTE2: " + bytes[2] +
" BYTE2: " + bytes[3] +
" LENGHT: " + bytes.length);
}
}
please let me know is i miss something.
thanks a lot to you guys.
i have an application that want to write a file using fileoutputstream
here's the code, method patch
public static Response patch() {
try {
System.out.println("PATCH");
System.out.println(request.contentType);
String file = params.get("filename");
System.out.println("patch file: " + file);
Map<String, Header> MapOffset = request.headers;
for (Entry<String, Header> entry : MapOffset.entrySet()) {
System.out.println("Header['" + entry.getKey() + "]: "
+ entry.getValue().value());
}
Header offsetParam = MapOffset.get("offset");
Long offset = 0L;
if (offsetParam != null) {
offset = Long.parseLong(offsetParam.value());
}
InputStream input = request.body;
File f = new File(UPLOAD_DIR + System.getProperty("file.separator")
+ file);
System.out.println("address: " + f.getAbsolutePath());
System.out.println("offset: " + offset);
System.out.println("length: " + f.length());
fileBasicUpload(f, offset, input);
Response respon = new Response();
respon.status = OK;
return respon;
} catch (Exception e) {
e.printStackTrace();
return null;
}
}
and this is where i write a file
private static void fileBasicUpload(File f, Long offset, InputStream input)
throws IOException {
FileOutputStream output = null;
try {
int c = -1;
byte[] b = new byte[1024];
try {
output = new FileOutputStream(f, true);
while ((c = input.read(b)) != -1) {
output.write(b, 0, c);
}
} catch (Exception e) {
System.out.println("Error: " + e.getMessage());
}
} finally {
output.close();
}
}
but when my application called, then stream closed error is show up at while ((c = input.read(b)) != -1) that line.
i don't know how that error is called. sorry for my poor english and thanks
i found the answer. in my application i found like this
public static Response upload(File file){
System.out.println("Appliaction.upload");
response = ResumableUpload.post();
return response;
// render(response);
}
the parameter file, it must be delete, then it work!
I want to know if really 'put' has succeeded in putting the file to destination. If for any reason the file is not put in destination [maybe due to problems in destination server like space constraint, etc] I need to know that.
Code:
private static boolean putFile(String m_sLocalFile, FtpClient m_client) {
boolean success = false;
int BUFFER_SIZE = 10240;
if (m_sLocalFile.length() == 0) {
System.out.println("Please enter file name");
}
byte[] buffer = new byte[BUFFER_SIZE];
try {
File f = new File(m_sLocalFile);
int size = (int) f.length();
System.out.println("File " + m_sLocalFile + ": " + size + " bytes");
System.out.println(size);
FileInputStream in = new FileInputStream(m_sLocalFile);
OutputStream out = m_client.put(f.getName());
int counter = 0;
while (true) {
int bytes = in.read(buffer);
if (bytes < 0)
break;
out.write(buffer, 0, bytes);
counter += bytes;
System.out.println(counter);
}
out.close();
in.close();
} catch (Exception ex) {
System.out.println("Error: " + ex.toString());
}
return success;
}
I would expect it to throw an IOException. Do you have any reason to believe it doesn't? But you shouldn't be using that class directly, you should be using an ftp: URL and its URLConnection class to do the I/O with, after calling setDoOutput(true).