I received this python script that generates a file checksum:
import sys,os
if __name__=="__main__":
#filename=os.path.abspath(sys.argv[1])
#filename=r"H:\Javier Ortiz\559-7 From Pump.bin"
cksum=0
offset=0
pfi=open(filename,'rb')
while 1:
icks=0
chunk=pfi.read(256)
if not chunk: break #if EOF exit loop
for iter in chunk:
icks+=ord(iter)
print ord(iter)
cksum=(cksum+icks) & 0xffff
pfi.close()
print "cksum=0x%4.4x"%cksum
And I'm trying to convert it to Java but I'm not geting the same results.
Here's my Java code:
import java.io.BufferedInputStream;
import java.io.DataInputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
public class ChecksumCalculator {
private ChecksumCalculator() {
}
public static int getChecksum(File file) {
int cksum = 0;
FileInputStream fis = null;
BufferedInputStream bis = null;
DataInputStream dis = null;
try {
fis = new FileInputStream(file);
// Here BufferedInputStream is added for fast reading.
bis = new BufferedInputStream(fis);
dis = new DataInputStream(bis);
byte[] buffer = new byte[256];
// dis.available() returns 0 if the file does not have more lines.
while (dis.read(buffer) != -1) {
int icks = 0;
for (byte b : buffer) {
icks += b & 0xff;
System.out.println(b & 0xff);
}
cksum = (cksum + icks) & 0xffff;
System.out.println("Checksum: " + cksum);
}
// dispose all the resources after using them.
fis.close();
bis.close();
dis.close();
return cksum;
} catch (FileNotFoundException e) {
e.printStackTrace();
return -1;
} catch (IOException e) {
e.printStackTrace();
return -1;
}
}
static public void main(String[] s) {
System.out.println("0x" + getChecksum(new File("H:\\Javier Ortiz\\559-7 From Pump.bin")));
}
}
But I get different results on a file. For example if I run it on a plain txt file containing only the word test it gives out the following result:
python: cksum=0x01c0
java: cksum=0x448
Any idea?
Your Python version prints the checksum in hex, while your Java version prints it in decimal. You should make your Java version print in hex, too. 0x1c0 == 448.
To use the cksum=0x%4.4x format string as you had in your Python version, use this:
System.out.printf("cksum=0x%4.4x%n", ...);
or even better
System.out.printf("cksum=%#04x%n", ...);
Also, you don't need a DataInputStream for this. Just use bis.read(buffer) instead of dis.read(buffer).
1C016 = 44810
I think that's your problem.
dis.read(buffer) returns the number of bytes that was actually read. For the last chunk, it will probably be less than 256. So the for loop shouldn't always be performed 256 times - it should be performed as many times as the actual byte count that was read from the stream.
I'm not a Python developer, but it doesn't look like ord(icks) in Python does the same as b & 0xff in Java.
Keep in mind that all Java types are signed; this might affect the calculation.
Also, although this doesn't affect correctness - it's a good practice to clean all the resources (i.e. to close the streams) in a finally block.
Related
I am using Android Studio and Oracle Java 8. I am trying to get all bytes from a file and pass them to a byte array. The code below acts like it does not see import java.io.File;
I get the error message:
cannot resolve method getBytesFromFile(java.io.File)
code
import java.io.File;
// ...
File path = new File(
Environment.getExternalStorageDirectory().getAbsolutePath()
+ "/zTest-Records/");
path.mkdirs();
try {
recordingFile = File.createTempFile("recording", ".pcm", path);
} catch (IOException e) {
throw new RuntimeException("Couldn't create pcm file", e);
}
// NOTE: The code below gives error message: cannot resolve method 'getBytesFromFile(java.io.File)'
byte[] data = getBytesFromFile(recordingFile);
That function is not defined, perhaps you copy paste this code from somewhere.
A quick google search points to this link:
https://code.google.com/p/picturesque/source/browse/myClasses/GetBytesFromFile.java?r=1d9332c4c969b4d35847c10f7c83b04c1ccb834f
package myClasses;
import java.util.*;
import java.io.*;
public class GetBytesFromFile {
public static byte[] getBytesFromFile(File file) throws IOException {
InputStream is = new FileInputStream(file);
// Get the size of the file
long length = file.length();
// You cannot create an array using a long type.
// It needs to be an int type.
// Before converting to an int type, check
// to ensure that file is not larger than Integer.MAX_VALUE.
if (length > Integer.MAX_VALUE) {
// File is too large
}
// Create the byte array to hold the data
byte[] bytes = new byte[(int)length];
// Read in the bytes
int offset = 0;
int numRead = 0;
while (offset < bytes.length
&& (numRead = is.read(bytes, offset, Math.min(bytes.length - offset, 512*1024))) >= 0) {
offset += numRead;
}
// Ensure all the bytes have been read in
if (offset < bytes.length) {
throw new IOException("Could not completely read file "+file.getName());
}
// Close the input stream and return bytes
is.close();
return bytes;
}
}
You probably need to add this class to your project
I'm generating random ints and trying to write them down to a file. The problem is when I open the file I've created I don't find my ints but a set of symbols like squares etc... Is it a problem of encoding ?
import java.io.DataOutputStream;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
public class GenerateBigList {
public static void main(String[] args) {
//generate in memory big list of numbers in [0, 100]
List list = new ArrayList<Integer>(1000);
for (int i = 0; i < 1000; i++) {
Double randDouble = Math.random() * 100;
int randInt = randDouble.intValue();
list.add(randInt);
}
//write it down to disk
File file = new File("tmpFileSort.txt");
try {
FileOutputStream fos = new FileOutputStream("C:/tmp/tmpFileSort.txt");
DataOutputStream dos = new DataOutputStream(fos);
writeListInteger(list, dos);
dos.close();
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
private static void writeListInteger(List<Integer> list, DataOutputStream dos) throws IOException {
for (Integer elt : list) {
dos.writeInt(elt);
}
}
}
A partial copy paste from the created file:
/ O a C ? 6 N
From the doc:
public final void writeInt(int v) throws IOException
Writes an int to the underlying output stream as four bytes, high byte first. If no exception is thrown, the counter written is incremented by 4.
There is no encoding problem. That is what you see when you open a binary file with a text editor. Try to open with a hex editor.
Those "symbols" are your ints. That's what binary files look like if you open them in a text editor. Notice that the file is exactly 4000 bytes in size, and you wrote 1000 ints, at 4 bytes each.
If you read the file in with a DataInputStream you will get the original values back:
try (DataInputStream dis = new DataInputStream(
new BufferedInputStream(new FileInputStream("C:/tmp/tmpFileSort.txt")))) {
for (int i = 0; i < 1000; i++) {
System.out.println(dis.readInt());
}
} catch (IOException e) {
throw new RuntimeException(e);
}
It writes binary, not text. Your expectations are misplaced. If you want text, use a Writer.
I am working a project in which I have to play with some file reading writing tasks. I have to read 8 bytes from a file at one time and perform some operations on that block and then write that block to second file, then repeat the cycle until first file is completely read in chuncks of 8 bytes everytime and the after manipulation the data should be added/appended to the second. However, in doing so, I am facing some problems. Following is what I am trying:
private File readFromFile1(File file1) {
int offset = 0;
long message= 0;
try {
FileInputStream fis = new FileInputStream(file1);
byte[] data = new byte[8];
file2 = new File("file2.txt");
FileOutputStream fos = new FileOutputStream(file2.getAbsolutePath(), true);
DataOutputStream dos = new DataOutputStream(fos);
while(fis.read(data, offset, 8) != -1)
{
message = someOperation(data); // operation according to business logic
dos.writeLong(message);
}
fos.close();
dos.close();
fis.close();
} catch (IOException e) {
System.out.println("Some error occurred while reading from File:" + e);
}
return file2;
}
I am not getting the desired output this way. Any help is appreciated.
Consider the following code:
private File readFromFile1(File file1) {
int offset = 0;
long message = 0;
File file2 = null;
try {
FileInputStream fis = new FileInputStream(file1);
byte[] data = new byte[8]; //Read buffer
byte[] tmpbuf = new byte[8]; //Temporary chunk buffer
file2 = new File("file2.txt");
FileOutputStream fos = new FileOutputStream(file2.getAbsolutePath(), true);
DataOutputStream dos = new DataOutputStream(fos);
int readcnt; //Read count
int chunk; //Chunk size to write to tmpbuf
while ((readcnt = fis.read(data, 0, 8)) != -1) {
//// POINT A ////
//Skip chunking system if an 8 byte octet is read directly.
if(readcnt == 8 && offset == 0){
message = someOperation(tmpbuf); // operation according to business logic
dos.writeLong(message);
continue;
}
//// POINT B ////
chunk = Math.min(tmpbuf.length - offset, readcnt); //Determine how much to add to the temp buf.
System.arraycopy(data, 0, tmpbuf, offset, chunk); //Copy bytes to temp buf
offset = offset + chunk; //Sets the offset to temp buf
if (offset == 8) {
message = someOperation(tmpbuf); // operation according to business logic
dos.writeLong(message);
if (chunk < readcnt) {
System.arraycopy(data, chunk, tmpbuf, 0, readcnt - chunk);
offset = readcnt - chunk;
} else {
offset = 0;
}
}
}
//// POINT C ////
//Process remaining bytes here...
//message = foo(tmpbuf);
//dos.writeLong(message);
fos.close();
dos.close();
fis.close();
} catch (IOException e) {
System.out.println("Some error occurred while reading from File:" + e);
}
return file2;
}
In this excerpt of code, what I did was:
Modify your reading code to include the amount of bytes actually read from the read() method (noted readcnt).
Added a byte chunking system (the processing does not happen until there are at least 8 bytes in the chunking buffer).
Allowed for separate processing of the final bytes (that do not make up a 8 byte octet).
As you can see from the code, the data being read is first stored in a chunking buffer (denoted tmpbuf) until at least 8 bytes are available. This will happen only if 8 bytes are not always available (If 8 bytes are available directly and nothing is chunked, directly process. See "Point A" in code). This is done as a form of optimization to prevent excess array copies.
The chunking system uses offsets which increment every time bytes are written to tmpbuf until it reaches a value of 8 (it will not go over as the Math.min() method used in the assignment of 'chunk' will limit the value). Upon offset == 8, proceed to execute the processing code.
If that particular read produced more bytes than actually processed, continue writing them to tmpbuf, from the beginning again, whilst setting offset appropriately, otherwise set offset to 0.
Repeat cycle.
The code will leave the last few bytes of data that do not fit in an octet in the array tmpbuf with the offset variable indicating how much has actually been written. This data can then be processed separately at point C.
Seems a lot more complicating than it should be, and there probably is a better solution (possibly using existing java library methods), but off the top of my head, this is what I got. Hope this is clear enough for you to understand.
You could use the following, it uses NIO and especially the ByteBuffer class for the long handling. You can of course implement it the standard java way, but since i am a NIO fan, here is a possible solution.
The major problem in your code is that while(fis.read(data, offset, 8) != -1) will read up to 8 bytes, and not always 8 bytes, plus reading in such small portions is not very efficient.
I have put some comments in my code, if something is unclear please leave a comment. My someOperation(...) function just copies the next long value from the buffer.
Update:
added finally block to close the files.
import java.io.File;
import java.io.IOException;
import java.nio.ByteBuffer;
import java.nio.channels.FileChannel;
import java.nio.file.StandardOpenOption;
public class TestFile {
static final int IN_BUFFER_SIZE = 1024 * 8;
static final int OUT_BUFFER_SIZE = 1024 *9; // make the out-buffer > in-buffer, i am lazy and don't want to check for overruns
static final int MIN_READ_BYTES = 8;
static final int MIN_WRITE_BYTES = 8;
private File readFromFile1(File inFile) {
final File outFile = new File("file2.txt");
final ByteBuffer inBuffer = ByteBuffer.allocate(IN_BUFFER_SIZE);
final ByteBuffer outBuffer = ByteBuffer.allocate(OUT_BUFFER_SIZE);
FileChannel readChannel = null;
FileChannel writeChannel = null;
try {
// open a file channel for reading and writing
readChannel = FileChannel.open(inFile.toPath(), StandardOpenOption.READ);
writeChannel = FileChannel.open(outFile.toPath(), StandardOpenOption.CREATE, StandardOpenOption.WRITE);
long totalReadByteCount = 0L;
long totalWriteByteCount = 0L;
boolean readMore = true;
while (readMore) {
// read some bytes into the in-buffer
int readOp = 0;
while ((readOp = readChannel.read(inBuffer)) != -1) {
totalReadByteCount += readOp;
} // while
// prepare the in-buffer to be consumed
inBuffer.flip();
// check if there where errors
if (readOp == -1) {
// end of file reached, read no more
readMore = false;
} // if
// now consume the in-buffer until there are at least MIN_READ_BYTES in the buffer
while (inBuffer.remaining() >= MIN_READ_BYTES) {
// add data to the write buffer
outBuffer.putLong(someOperation(inBuffer));
} // while
// compact the in-buffer and prepare for the next read, if we need to read more.
// that way the possible remaining bytes of the in-buffer can be consumed after leaving the loop
if (readMore) inBuffer.compact();
// prepare the out-buffer to be consumed
outBuffer.flip();
// write the out-buffer until the buffer is empty
while (outBuffer.hasRemaining())
totalWriteByteCount += writeChannel.write(outBuffer);
// prepare the out-buffer for writing again
outBuffer.flip();
} // while
// error handling
if (inBuffer.hasRemaining()) {
System.err.println("Truncated data! Not a long value! bytes remaining: " + inBuffer.remaining());
} // if
System.out.println("read total: " + totalReadByteCount + " bytes.");
System.out.println("write total: " + totalWriteByteCount + " bytes.");
} catch (IOException e) {
System.out.println("Some error occurred while reading from File: " + e);
} finally {
if (readChannel != null) {
try {
readChannel.close();
} catch (IOException e) {
System.out.println("Could not close read channel: " + e);
} // catch
} // if
if (writeChannel != null) {
try {
writeChannel.close();
} catch (IOException e) {
System.out.println("Could not close write channel: " + e);
} // catch
} // if
} // finally
return outFile;
}
private long someOperation(ByteBuffer bb) {
// consume the buffer, do whatever you want with the buffer.
return bb.getLong(); // consumes 8 bytes of the buffer.
}
public static void main(String[] args) {
TestFile testFile = new TestFile();
File source = new File("input.txt");
testFile.readFromFile1(source);
}
}
Recently, I wrote a simple client server program for file transfer over standard TCP sockets. The average throughput was around 2.2Mbps over WiFi channel. My question is:
Is it possible to transfer a large file (say 5 GB) over multiple data IO streams so that each stream could transfer several parts of the same file in a parallel manner (different threads could be used for this purpose)? These file parts could be re-assembled at the receiving end.
I tried to split a small file and transfered it over a dataoutputstream. The first segment works fine, but I don't know how to read a file input stream in selective manner (I also tried mark() and reset() methods for selective reading but no use)
Here is my code (for testing purpose, I have redirected the output to fileoutputstream):
public static void main(String[] args) {
// TODO Auto-generated method stub
final File myFile=new File("/home/evinish/Documents/Android/testPicture.jpg");
long N=myFile.length();
try {
FileInputStream in=new FileInputStream(myFile);
FileOutputStream f0=new FileOutputStream("/home/evinish/Documents/Android/File1.jpg");
FileOutputStream f1=new FileOutputStream("/home/evinish/Documents/Android/File2.jpg");
FileOutputStream f2=new FileOutputStream("/home/evinish/Documents/Android/File3.jpg");
byte[] buffer=new byte[4096];
int i=1, noofbytes;
long acc=0;
while(acc<=(N/3)) {
noofbytes=in.read(buffer, 0, 4096);
f0.write(buffer, 0, noofbytes);
acc=i*noofbytes;
i++;
}
f0.close();
I got the first segment of my file (this can be copied to a DataOutputStream in one thread). Can any one suggest, how to read remaining part of the file (after N/3 byte) in a segment of N/3 so that three streams could be used in three threads for concurrent operation?
Here is the code to merge file segments at receiver end:
package com.mergefilespackage;
import java.io.BufferedInputStream;
import java.io.BufferedOutputStream;
import java.io.Closeable;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
public class MergeFiles {
/**
* #param args
*/
public static void main(String[] args) throws Exception{
// TODO Auto-generated method stub
IOCopier.joinFiles(new File("/home/evinish/Documents/Android/File1.jpg"), new File[] {
new File("/home/evinish/Documents/Android/File2.jpg"), new File("/home/evinish/Documents/Android/File3.jpg")});
}
}
class IOCopier {
public static void joinFiles(File destination, File[] sources)
throws IOException {
OutputStream output = null;
try {
output = createAppendableStream(destination);
for (File source : sources) {
appendFile(output, source);
}
} finally {
IOUtils.closeQuietly(output);
}
}
private static BufferedOutputStream createAppendableStream(File destination)
throws FileNotFoundException {
return new BufferedOutputStream(new FileOutputStream(destination, true));
}
private static void appendFile(OutputStream output, File source)
throws IOException {
InputStream input = null;
try {
input = new BufferedInputStream(new FileInputStream(source));
IOUtils.copy(input, output);
} finally {
IOUtils.closeQuietly(input);
}
}
}
class IOUtils {
private static final int BUFFER_SIZE = 1024 * 4;
public static long copy(InputStream input, OutputStream output)
throws IOException {
byte[] buffer = new byte[BUFFER_SIZE];
long count = 0;
int n = 0;
while (-1 != (n = input.read(buffer))) {
output.write(buffer, 0, n);
count += n;
}
return count;
}
public static void closeQuietly(Closeable output) {
try {
if (output != null) {
output.close();
}
} catch (IOException ioe) {
ioe.printStackTrace();
}
}
}
Any help would be highly appreciated! Thanks in advance!
You can't get any more speed over the same link with more sockets. Each socket sends a certain number of packets, each of a certain size. As we double the number of sockets, the number of packets/sec*socket is halved, and then decreased even more due to collisions, overhead, and contention. Packets start to bump, jumble, and otherwise panic. The OS cannot handle the pandemonium of lost ACKs, and the WiFi card struggles to transmit at such a rate. It is losing its low-level acks as well. As packets get lost, a desperate TCP stack dials down the transmit rate. If this were to be able to come up due to signal improvement, it's now stuck at the lower speed due to silly window syndrome or another form of TCP deadlock.
Any attempt of WiFi to get any higher speeds out of wider carrier bands, MiMo, or multiple paths, has already been realized as gains, even with one socket. You can't take it any farther.
Now, wait. We're quite below WiFi speed, aren't we? Of course, we need to use buffering!
Make sure you create BufferedWriter and BufferedReader objects from your socket's getInputStream or getOutputStream methods. Then write to/read from those buffers. Your speed may increase somewhat.
You could get the byte array of the FileInputStream and split it every 10 KB (every 10.000 bytes).
Then send these parts through the streams in order.
On the server you can put the arrays together again and read the file from this giant byte array.
I would like to redirect/hide the sysout generated by the following code :
Tools tool = new ToolsImpl();
HashCode hash = tool.computeHashCodes(dir);
The difficult part is : the method computeHashCodes is stored in a jar.
I've tried the following code :
PrintStream printStreamOriginal=System.out;
System.out.println("sysout deactivated");
System.setOut(new PrintStream(new OutputStream() {
public void write(int b) {}
}));
System.out.println("Text to delete");
Tools tool = new ToolsImpl();
HashCode hash = tool.computeHashCodes(dir);
System.setOut(printStreamOriginal);
System.out.println("sysout reactivated");
The "text to delete" is indeed deleted, but the sysout generated by the ".computeHashCodes" is not. Does someone know how to hide this sysout ?
Thx in advance,
Mike
The code may be writing to System.err instead.
Try the same exercise but with System.err instead of System.out.
your solution works fine when using System.out, so I'm guessing that the code you want to "block" doesn't use System.out for the output. Try to find how the output is done, so you can "block" it.
See here:
Writing to jars
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.util.jar.JarEntry;
import java.util.jar.JarOutputStream;
import java.util.jar.Manifest;
public class CreateJarFile {
public static int BUFFER_SIZE = 10240;
protected void createJarArchive(File archiveFile, File[] tobeJared) {
try {
byte buffer[] = new byte[BUFFER_SIZE];
// Open archive file
FileOutputStream stream = new FileOutputStream(archiveFile);
JarOutputStream out = new JarOutputStream(stream, new Manifest());
for (int i = 0; i < tobeJared.length; i++) {
if (tobeJared[i] == null || !tobeJared[i].exists()
|| tobeJared[i].isDirectory())
continue; // Just in case...
System.out.println("Adding " + tobeJared[i].getName());
// Add archive entry
JarEntry jarAdd = new JarEntry(tobeJared[i].getName());
jarAdd.setTime(tobeJared[i].lastModified());
out.putNextEntry(jarAdd);
// Write file to archive
FileInputStream in = new FileInputStream(tobeJared[i]);
while (true) {
int nRead = in.read(buffer, 0, buffer.length);
if (nRead <= 0)
break;
out.write(buffer, 0, nRead);
}
in.close();
}
out.close();
stream.close();
System.out.println("Adding completed OK");
} catch (Exception ex) {
ex.printStackTrace();
System.out.println("Error: " + ex.getMessage());
}
}
}
Thanks everyone, I finally managed to avoid displaying the sysout.
When Magodiez advised me to find how the output was done, I thought I couldn't do that because I didn't have access to the source code ; but then I realised I just had to decompile the code.
So I decompiled it with Java Decompiler, and then I saw how the output was done :
LOGGER.log(Level.INFO, str2);
Then I resolved my problem by using the following line :
java.util.logging.Logger.getLogger("log.tools").setLevel(Level.SEVERE);
It is actually what I really wanted, now only the SEVERE messages will be printed on the sysout.
Thanks again !