I have a log analyzing tool that needs to grab *.gz files from Linux servers and unzip them on both Linux and Windows clients. I am getting "Unexpected end of ZLIB input stream" in many instances, which I assume is a difference in detail in the files on Linux and Windows.
Below is my function. It's pretty basic. How do I improved it to prevent the EOF error?
The "in" symbol is a FileInputStream that is created when constructing the class that this function is part of.
public void unzip(File fileTo) throws IOException {
OutputStream out = new FileOutputStream(fileTo);
LOGGER.info("Setting up the file for outputstream : "+fileTo);
try {
in = new GZIPInputStream(in);
byte[] buffer = new byte[65536];
int noRead;
while ((noRead = in.read(buffer)) != -1) {
out.write(buffer, 0, noRead);
}
} finally {
try { out.close(); } catch (Exception e) {}
}
}
I changed from the above to this and now it works. It seems that it was trying to load the output stream before it was done loading the input stream.
public void unzip(File fileTo, String f) throws IOException,
EOFException, InterruptedException {
LOGGER.info("Setting up the file for outputstream : "+fileTo);
GZIPInputStream cIn = new GZIPInputStream(new FileInputStream(f));
OutputStream out = new FileOutputStream(fileTo);
fileTo.setReadable(true, false);
fileTo.setWritable(true, false);
byte[] buffer = new byte[65536];
int noRead;
for (int i = 10; i > 0 && cIn.available() == 1; i--) {
Thread.sleep(1000);
}
try {
while ((noRead = cIn.read(buffer)) != -1) {
out.write(buffer, 0, noRead);
}
} finally {
try { out.close();cIn.close();in.close(); } catch (Exception e) {}
}
}
Related
Question at the bottom
I'm using netty to transfer a file to another server.
I limit my file-chunks to 1024*64 bytes (64KB) because of the WebSocket protocol. The following method is a local example what will happen to the file:
public static void rechunck(File file1, File file2) {
FileInputStream is = null;
FileOutputStream os = null;
try {
byte[] buf = new byte[1024*64];
is = new FileInputStream(file1);
os = new FileOutputStream(file2);
while(is.read(buf) > 0) {
os.write(buf);
}
} catch (IOException e) {
Controller.handleException(Thread.currentThread(), e);
} finally {
try {
if(is != null && os != null) {
is.close();
os.close();
}
} catch (IOException e) {
Controller.handleException(Thread.currentThread(), e);
}
}
}
The file is loaded by the InputStream into a ByteBuffer and directly written to the OutputStream.
The content of the file cannot change while this process.
To get the md5-hashes of the file I've wrote the following method:
public static String checksum(File file) {
InputStream is = null;
try {
is = new FileInputStream(file);
MessageDigest digest = MessageDigest.getInstance("MD5");
byte[] buffer = new byte[8192];
int read = 0;
while((read = is.read(buffer)) > 0) {
digest.update(buffer, 0, read);
}
return new BigInteger(1, digest.digest()).toString(16);
} catch(IOException | NoSuchAlgorithmException e) {
Controller.handleException(Thread.currentThread(), e);
} finally {
try {
is.close();
} catch(IOException e) {
Controller.handleException(Thread.currentThread(), e);
}
}
return null;
}
So: just in theory it should return the same hash, shouldn't it? The problem is that it returns two different hashes that do not differ with every run.. file size stays the same and the content either.
When I run the method once for in: file-1, out: file-2 and again with in: file-2 and out: file-3 the hashes of file-2 and file-3 are the same! This means the method will properly change the file every time the same way.
1. 58a4a9fbe349a9e0af172f9cf3e6050a
2. 7b3f343fa1b8c4e1160add4c48322373
3. 7b3f343fa1b8c4e1160add4c48322373
Here is a little test that compares all buffers if they are equivalent. Test is positive. So there aren't any differences.
File file1 = new File("controller/templates/Example.zip");
File file2 = new File("controller/templates2/Example.zip");
try {
byte[] buf1 = new byte[1024*64];
byte[] buf2 = new byte[1024*64];
FileInputStream is1 = new FileInputStream(file1);
FileInputStream is2 = new FileInputStream(file2);
boolean run = true;
while(run) {
int read1 = is1.read(buf1), read2 = is2.read(buf2);
String result1 = Arrays.toString(buf1), result2 = Arrays.toString(buf2);
boolean test = result1.equals(result2);
System.out.println("1: " + result1);
System.out.println("2: " + result2);
System.out.println("--- TEST RESULT: " + test + " ----------------------------------------------------");
if(!(read1 > 0 && read2 > 0) || !test) run = false;
}
} catch (IOException e) {
e.printStackTrace();
}
Question: Can you help me chunking the file without changing the hash?
while(is.read(buf) > 0) {
os.write(buf);
}
The read() method with the array argument will return the number of files read from the stream. When the file doesn't end exactly as a multiple of the byte array length, this return value will be smaller than the byte array length because you reached the file end.
However your os.write(buf); call will write the whole byte array to the stream, including the remaining bytes after the file end. This means the written file gets bigger in the end, therefore the hash changed.
Interestingly you didn't make the mistake when you updated the message digest:
while((read = is.read(buffer)) > 0) {
digest.update(buffer, 0, read);
}
You just have to do the same when you "rechunk" your files.
Your rechunk method has a bug in it. Since you have a fixed buffer in there, your file is split into ByteArray-parts. but the last part of the file can be smaller than the buffer, which is why you write too many bytes in the new file. and that's why you do not have the same checksum anymore. the error can be fixed like this:
public static void rechunck(File file1, File file2) {
FileInputStream is = null;
FileOutputStream os = null;
try {
byte[] buf = new byte[1024*64];
is = new FileInputStream(file1);
os = new FileOutputStream(file2);
int length;
while((length = is.read(buf)) > 0) {
os.write(buf, 0, length);
}
} catch (IOException e) {
Controller.handleException(Thread.currentThread(), e);
} finally {
try {
if(is != null)
is.close();
if(os != null)
os.close();
} catch (IOException e) {
Controller.handleException(Thread.currentThread(), e);
}
}
}
Due to the length variable, the write method knows that until byte x of the byte array, only the file is off, then there are still old bytes in it that no longer belong to the file.
This question already has answers here:
Java multiple file transfer over socket
(3 answers)
Closed 6 years ago.
I am first transferring a file from a client to my master, the stores the byte array and then sends to the slave. Where the slave stores the byte array. But when The file is sent properly from client to master but when I send the byte array to the slave it to the slave the read method in input stream constantly reads 0.
// This method writes the file to the master
public void writeFile(File file) {
try {
this.write(String.valueOf(file.length()));
byte[] bytearray = new byte[(int) file.length()];
FileInputStream fin = new FileInputStream(file);
BufferedInputStream bin = new BufferedInputStream(fin);
bin.read(bytearray, 0, bytearray.length);
BufferedOutputStream bos;
OutputStream os = socket.getOutputStream();
bos= new BufferedOutputStream(os);
bos.write(bytearray, 0, bytearray.length);
bos.flush();
} catch (IOException e) {
e.printStackTrace();
}
}
//This method reads the file into the master as a byte array and the byte array from the master into slave
public byte[] readFile() {
byte[] bytearray = null;
try {
int currentTot = 0;
int filesize = Integer.parseInt(this.read());
System.out.println(filesize);
bytearray = new byte[filesize];
InputStream is = socket.getInputStream();
int bytesRead;
bytesRead = is.read(bytearray, 0, bytearray.length);
currentTot = bytesRead;
int count = 0;
do {
bytesRead = is.read(bytearray, currentTot, (bytearray.length - currentTot));
if (bytesRead > 0) {
currentTot += bytesRead;
count = 0;
} else {
count++;
System.out.println("count " + count);
}
} while (bytesRead > -1);
System.out.println(currentTot);
// bos.write(bytearray, 0, currentTot);
// bos.flush();
// bos.close();
} catch (IOException e) {
e.printStackTrace();
}
return bytearray;
}
//This method writes from the master to the slave
public void writeByte(byte[] m) {
this.write(String.valueOf(m.length));
System.out.println("File side inside sender" + m.length);
// byte[] bytearray = m;
OutputStream os;
try {
os = socket.getOutputStream();
os.write(m, 0, m.length);
os.flush();
//os.close();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
Interestingly if I close my output stream after I send my byte array from my master it works well. But I cannot close stream because the slave needs to communicate with the master further. Thanks in advance.
public void write(String output) {
if (pw == null)
this.openWriter();
pw.println(output);
}
public String read() {
try {
if (br == null) {
if (this.socket != null)
br = new BufferedReader(new InputStreamReader(this.socket.getInputStream()));
}
return br.readLine();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return null;
}
You're misreading the file length in the receiver. You are getting zero, so you're constructing a zero length byte array, so read() returns zero.
You need to send the length via DataOutputStream.writeLong() and read it via DataInputStream.readLong(). And then your sending and receiving code is all wrong as well. See my answer here for complete code.
I have two problems with my java server-client file communication,
I have the CLIENT sends files over to the server and the SERVER receives the files.
My 2 issues are:
1) whenever i send a file, it is 8 bytes less (i do not know why)
2) the file transfer is not complete (with 8 bytes less) unless i close the socket, which i do not want. i want my connection to be persistent, so how can i send a EOF from the client to the server.
here is my client who sends files
public void sendFiles(String file) {
try {
File myFile = new File(file);
long length = myFile.length();
byte[] buffer = new byte[8192];
System.out.println(length);
FileInputStream in = new FileInputStream(myFile);
BufferedInputStream bis = new BufferedInputStream(in);
BufferedOutputStream outF = new BufferedOutputStream(sock.getOutputStream());
out.print("%SF%" + length + "$" + myFile.getName() + "#");
out.flush();
int count;
while ((count = in.read(buffer)) > 0) {
outF.write(buffer, 0, count);
}
outF.flush();
in.close();
bis.close();
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
SERVER who receives files.
I'm passing the name and the length of the file but only using the name of the file. however, i don't know if i need to use the length of the file or not, in case of EOF or something. Please advice
Also, The code hangs in
while ((count = this.sock.getInputStream().read(buffer)) > 0) {
due to no EOF which i do not know how to implement
public void recvFile(String fileName, int length) {
try {
byte[] buffer = new byte[8192];
FileOutputStream outF = new FileOutputStream("/Users/Documents" +fileName);
BufferedOutputStream bos = new BufferedOutputStream(outF);
int count = length;
while ((count = this.sock.getInputStream().read(buffer)) > 0) {
bos.write(buffer, 0, count);
}
bos.close();
} catch (IOException ex) {
ex.printStackTrace();
}
}
UPDATE: I have removed the flush() as advised that it is not needed. Also, i have tested this code in a different class and it worked but it doesn't work here with client-server chat. Could anyone tell me why?
Any help or hints would be appreciated.
Thank you.
I would suggest to you send the file size first and/or properties of the file... You can try HTTP which is wide use for this task...
Another suggestion would be for you to open another connection on other TCP port just to send the file (this is actually how FTP sends files)
I suspect the problem you have is in code you haven't shown.
In this example you can send multiple messages or files over the same stream.
import java.io.*;
import java.net.ServerSocket;
import java.net.Socket;
import java.nio.channels.SocketChannel;
/**
* Created by peter on 1/25/15.
*/
public class DataSocket implements Closeable {
private final Socket socket;
private final DataOutputStream out;
private final DataInputStream in;
public DataSocket(Socket socket) throws IOException {
this.socket = socket;
this.out = new DataOutputStream(new BufferedOutputStream(socket.getOutputStream()));
this.in = new DataInputStream(new BufferedInputStream(socket.getInputStream()));
}
#Override
public void close() throws IOException {
out.flush();
socket.close();
}
// message format is length as UTF-8 encoded name, 32-bit int followed by data.
public void writeMessage(String description, byte[] bytes) throws IOException {
out.writeUTF(description);
out.writeInt(bytes.length);
out.write(bytes);
out.flush();
}
public byte[] readMessage(String[] description) throws IOException {
description[0] = in.readUTF();
int length = in.readInt();
byte[] bytes = new byte[length];
in.readFully(bytes);
return bytes;
}
public void writeFile(File file) throws IOException {
long length = file.length();
if (length > Integer.MAX_VALUE) throw new IllegalArgumentException("length=" + length);
out.writeUTF(file.toString());
out.writeInt((int) length);
byte[] buffer = new byte[(int) Math.min(length, 32 * 1024)];
try (FileInputStream fis = new FileInputStream(file)) {
for (int len; (len = fis.read(buffer)) > 0; ) {
out.write(buffer, 0, len);
}
}
out.flush();
}
public void readFile(File dir) throws IOException {
String fileName = in.readUTF();
int length = in.readInt();
byte[] buffer = new byte[(int) Math.min(length, 32 * 1024)];
try (FileOutputStream fos = new FileOutputStream(new File(dir, fileName))) {
while (length > 0) {
int len = in.read(buffer);
fos.write(buffer, 0, len);
length -= len;
}
}
}
// todo convert to a unit test
public static void main(String[] args) throws IOException {
// port 0 opens on a random free port.
ServerSocket sc = new ServerSocket(0);
DataSocket ds1 = new DataSocket(new Socket("localhost", sc.getLocalPort()));
DataSocket ds2 = new DataSocket(sc.accept());
sc.close();
// now ds1 and ds2 are connected.
File f = File.createTempFile("deleteme","");
f.deleteOnExit();
try (FileOutputStream fos = new FileOutputStream(f)) {
fos.write(new byte[10001]);
}
// send a request
ds1.writeMessage("Send me the file", new byte[0]);
String[] desc = { null };
byte[] data = ds2.readMessage(desc);
if (!desc[0].equals("Send me the file")) throw new AssertionError();
// return a response
ds2.writeFile(f);
f.delete();
if (f.exists()) throw new AssertionError();
ds1.readFile(new File(""));
if (f.length() != 10001) throw new AssertionError("length="+f.length());
ds1.close();
ds2.close();
System.out.println("Copied a "+f.length()+" file over TCP");
}
}
I'm trying to extract 2 jar files from the currently running jar however they always end up at 2kb even though their sizes are 104kb and 1.7m, Heres what I've got
public static boolean extractFromJar(String fileName, String dest) {
if (Configuration.getRunningJarPath() == null) {
return false;
}
File file = new File(dest + fileName);
if (file.exists()) {
return false;
}
if (file.isDirectory()) {
file.mkdir();
return false;
}
try {
JarFile jar = new JarFile(Configuration.getRunningJarPath());
Enumeration<JarEntry> e = jar.entries();
while (e.hasMoreElements()) {
JarEntry je = e.nextElement();
InputStream in = new BufferedInputStream(jar.getInputStream(je));
OutputStream out = new BufferedOutputStream(
new FileOutputStream(file));
copyInputStream(in, out);
}
return true;
} catch (Exception e) {
Methods.debug(e);
return false;
}
}
private final static void copyInputStream(InputStream in, OutputStream out)
throws IOException {
while (in.available() > 0) {
out.write(in.read());
}
out.flush();
out.close();
in.close();
}
This should work better then relying on InputStream.available() method:
private final static void copyInputStream(InputStream in, OutputStream out)
throws IOException {
byte[] buff = new byte[4096];
int n;
while ((n = in.read(buff)) > 0) {
out.write(buff, 0, n);
}
out.flush();
out.close();
in.close();
}
available() method is not reliable to read data as it is just an estimate, as per its documentation.
You need to depend on read() method until read a non -ve.
byte[] contentBytes = new byte[ 4096 ];
int bytesRead = -1;
while ( ( bytesRead = inputStream.read( contentBytes ) ) > 0 )
{
out.write( contentBytes, 0, bytesRead );
} // while available
You can go through a discussion on what the problems with available() is at here.
I'm not sure about extracting jars, but every jar is actually a zip file, so you can try unzip it.
you can findout about unziping in java here:
How to unzip files recursively in Java?
I know that the canonical way of handling the output/error streams of an external Process in Java, is to use two extra-threads in order to pull the data from the output and error streams lest the process might be blocked.
Now what about the following?
public static void main(String[] args) throws IOException, InterruptedException {
ProcessBuilder processBuilder = new ProcessBuilder(args);
Process process = processBuilder.start();
InputStream outputStream = null, errorStream = null;
ByteArrayOutputStream outputBuffer = new ByteArrayOutputStream();
ByteArrayOutputStream errorBuffer = new ByteArrayOutputStream();
try {
outputStream = process.getInputStream();
errorStream = process.getErrorStream();
byte[] tmp = new byte[1024];
while (true) {
int outputBytes = readAvailablOnce(outputStream, outputBuffer, tmp);
int errorBytes = readAvailablOnce(errorStream, errorBuffer, tmp);
if (outputBytes == 0 && errorBytes == 0) {
try {
process.exitValue();
break;
} catch (IllegalThreadStateException e) {
// keep on looping
}
}
}
readAvailableAll(outputStream, outputBuffer, tmp);
readAvailableAll(errorStream, errorBuffer, tmp);
} finally {
closeQuietly(outputStream);
closeQuietly(errorStream);
}
System.out.println(outputBuffer.toString("ASCII"));
System.err.println(errorBuffer.toString("ASCII"));
System.err.println("exit code: " + process.exitValue());
}
private static void closeQuietly(InputStream in) {
if (in != null) {
try {
in.close();
} catch (IOException e) {
// ignored
}
}
}
private static int readAvailablOnce(
InputStream inputStream, OutputStream outputStream, byte[] buffer)
throws IOException {
int bytesRead = 0;
if (inputStream.available() > 0) {
bytesRead = inputStream.read(buffer);
outputStream.write(buffer, 0, bytesRead);
}
return bytesRead;
}
private static void readAvailableAll(
InputStream inputStream, OutputStream outputStream, byte[] buffer)
throws IOException {
if (inputStream.available() > 0) {
int bytesRead = 0;
while ((bytesRead = inputStream.read(buffer)) >= 0) {
outputStream.write(buffer, 0, bytesRead);
}
}
}
This actually runs fine in the few examples I tried (running "dir", "ps aux", etc).
It also has the disadvantage of not allowing you to easily handle the outputs line by line (here you buffer everything before doing anything), unless you start doing some more or less complicated stuff with Buffers and CharsetDecoders.
Still, it looks useful for anything without unacceptably big output (although nothing forces us to buffer the whole output before using it).
I only tried the 1.5 and 1.6 JVM (Windows XP and Linux).
Also, this code makes the assumption that the final output bits of the process will be readily available for reading (InputStream.available() > 0).
Anyone would know what (or if something) is wrong with this code (or have a better idea)?
I would use ProcessBuilder.redirectErrorStream(true) which allows you to read one stream in the current thread. You wouldn't need any background threads.