I keep getting the following error when I run my program:
java.lang.OutOfMemoryError : Java heap space
I tried fixing this by adding -Xms512M -Xmx1524M to both Program arguments and VM arguments (I use eclipse), but this doesn't seem to prevent the error.
If the solution is adding more memory in eclipse run configuration, the question is > is this going to be exported too? or is it just for me, how can I make sure this doesn't happen on other computers?
Current code
private static void checkSumGen() throws IOException NoSuchAlgorithmException
{
File plFolder = new File(".\\Plugins");
File[] listOfFiles = plFolder.listFiles();
List<String> listClone = new ArrayList<String>();
for (int i = 0; i < listOfFiles.length; i++)
{
File file = listOfFiles[i];
String fileloc = file.getAbsolutePath();
MessageDigest md = MessageDigest.getInstance("SHA-256");
FileInputStream fis = new FileInputStream(fileloc);
byte[] dataBytes = new byte[1024];
int nread = 0;
while ((nread = fis.read(dataBytes)) != -1)
{
md.update(dataBytes, 0, nread);
} ;
byte[] mdbytes = md.digest();
// convert the byte to hex format method 1
StringBuffer sb = new StringBuffer();
for (int i1 = 0; i1 < mdbytes.length; i++)
{
sb.append(Integer.toString((mdbytes[i1] & 0xff) + 0x100, 16).substring(1));
}
System.out.println("Hex format : " + sb.toString());
// convert the byte to hex format method 2
StringBuffer hexString = new StringBuffer();
for (int i2 = 0; i2 < mdbytes.length; i++)
{
hexString.append(Integer.toHexString(0xFF & mdbytes[i2]));
}
System.out.println("Hex format : " + hexString.toString());
}
}
You have an infinite loop in your code that causes the OutOfMemoryError.
In:
for (int i1 = 0; i1 < mdbytes.length; i++)
you increment i instead of i1 causing i1 < mdbytes.length to be true forever.
Related
I am creating a method that will take in a file and split it into shardCount pieces and generate a parity file.
When I run this method, it appears that I am writing out extra data into my parity file. This is my first time using FileChannel and ByteBuffers, so I'm not certain I completely understand how to use them despite staring at the documentation for about 8 hours.
This code is a simplified version of the parity section.
public static void splitAndGenerateParityFile(File file, int shardCount, String fileID) throws IOException {
RandomAccessFile rin = new RandomAccessFile(file, "r");
FileChannel fcin = rin.getChannel();
//Create parity files
File parity = new File(fileID + "_parity");
if (parity.exists()) throw new FileAlreadyExistsException("Could not create parity file! File already exists!");
RandomAccessFile parityRAF = new RandomAccessFile(parity, "rw");
FileChannel parityOut = parityRAF.getChannel();
long bytesPerFile = (long) Math.ceil(rin.length() / shardCount);
//Make buffers for each section of the file we will be reading from
for (int i = 0; i < shardCount; i++) {
ByteBuffer bb = ByteBuffer.allocate(1024);
shardBuffers.add(bb);
}
ByteBuffer parityBuffer = ByteBuffer.allocate(1024);
//Generate parity
boolean isParityBufferEmpty = true;
for (long i = 0; i < bytesPerFile; i++) {
isParityBufferEmpty = false;
int pos = (int) (i % 1024);
byte p = 0;
if (pos == 0) {
//Read chunk of file into each buffer
for (int j = 0; j < shardCount; j++) {
ByteBuffer bb = shardBuffers.get(j);
bb.clear();
fcin.read(bb, bytesPerFile * j + i);
bb.rewind();
}
//Dump parity buffer
if (i > 0) {
parityBuffer.rewind();
parityOut.write(parityBuffer);
parityBuffer.clear();
isParityBufferEmpty = true;
}
}
//Get parity
for (ByteBuffer bb : shardBuffers) {
if (pos >= bb.limit()) break;
p ^= bb.get(pos);
}
//Put parity in buffer
parityBuffer.put(pos, p);
}
if (!isParityBufferEmpty) {
parityBuffer.rewind();
parityOut.write(parityBuffer);
parityBuffer.clear();
}
fcin.close();
rin.close();
parityOut.close();
parityRAF.close();
}
Please let me know if there is anything wrong with either the parity algorithm or the file IO, or if there's anything I can do to optimize this. I'm happy to hear about other (better) ways of doing file IO.
Here is the solution I found (though it may need more tuning):
public static void splitAndGenerateParityFile(File file, int shardCount, String fileID) throws IOException {
int BUFFER_SIZE = 4 * 1024 * 1024;
RandomAccessFile rin = new RandomAccessFile(file, "r");
FileChannel fcin = rin.getChannel();
//Create parity files
File parity = new File(fileID + "_parity");
if (parity.exists()) throw new FileAlreadyExistsException("Could not create parity file! File already exists!");
RandomAccessFile parityRAF = new RandomAccessFile(parity, "rw");
FileChannel parityOut = parityRAF.getChannel();
//Create shard files
ArrayList<File> shards = new ArrayList<>(shardCount);
for (int i = 0; i < shardCount; i++) {
File f = new File(fileID + "_part_" + i);
if (f.exists()) throw new FileAlreadyExistsException("Could not create shard file! File already exists!");
shards.add(f);
}
long bytesPerFile = (long) Math.ceil(rin.length() / shardCount);
ArrayList<ByteBuffer> shardBuffers = new ArrayList<>(shardCount);
//Make buffers for each section of the file we will be reading from
for (int i = 0; i < shardCount; i++) {
ByteBuffer bb = ByteBuffer.allocate(BUFFER_SIZE);
shardBuffers.add(bb);
}
ByteBuffer parityBuffer = ByteBuffer.allocate(BUFFER_SIZE);
//Generate parity
boolean isParityBufferEmpty = true;
for (long i = 0; i < bytesPerFile; i++) {
isParityBufferEmpty = false;
int pos = (int) (i % BUFFER_SIZE);
byte p = 0;
if (pos == 0) {
//Read chunk of file into each buffer
for (int j = 0; j < shardCount; j++) {
ByteBuffer bb = shardBuffers.get(j);
bb.clear();
fcin.position(bytesPerFile * j + i);
fcin.read(bb);
bb.flip();
}
//Dump parity buffer
if (i > 0) {
parityBuffer.flip();
while (parityBuffer.hasRemaining()) {
parityOut.write(parityBuffer);
}
parityBuffer.clear();
isParityBufferEmpty = true;
}
}
//Get parity
for (ByteBuffer bb : shardBuffers) {
if (!bb.hasRemaining()) break;
p ^= bb.get();
}
//Put parity in buffer
parityBuffer.put(p);
}
if (!isParityBufferEmpty) {
parityBuffer.flip();
parityOut.write(parityBuffer);
parityBuffer.clear();
}
fcin.close();
rin.close();
parityOut.close();
parityRAF.close();
}
As suggested by VGR, I replaced rewind() with flip(). I also switched to relative operations instead of absolute. I don't think the absolute methods adjust the cursor position or the limit, so that was likely the cause of the error. I also changed the buffer size to 4MB as I am interested in generating the parity for large files.
I have below code which is used to write the list array into a dat file , I ran on OnePlus2 and the IDE is Android Studio 1.3.
private void writeToFile(List<Short> list) throws IOException {
String stringTransform = transform(list);
String str = new String(stringTransform.getBytes(), "ascii");
byte[] bytes = new byte[str.length() / 8];
char chatAt;
for (int i = 0; i < str.length() / 8; i++) {
for (int j = 0; j < 8; j++) {
chatAt = str.charAt(i * 8 + j);
if (chatAt == '1') {
byte b = (byte) (0x80 >> j);
bytes[i] = (byte) (bytes[i] | b);
}
}
}
FileOutputStream fos = new FileOutputStream("/sdcard/1.dat");
fos.write(bytes);
fos.close();
fos.flush();
}
private String transform(List<Short> list) {
StringBuilder sb = new StringBuilder(list.size());
for (Short integer : list) {
sb.append(integer);
}
return sb.toString();
}
However , I input the same data in different time , and the dat file which is generated will show different content , as the pictures show:
This is not about your code. This is about program and encoding you using. Try to change encoding in your editor. If it is binary file I would recommend sublimetext 3 with HexViewer plugin:
everyone! I recently learned about variable byte encoding.
for example, if a file contains this sequence of number: 824 5 214577
applying variable byte encoding this sequence would be encoded as 000001101011100010000101000011010000110010110001.
Now I want to know how to write that in another file such that to produce a kind of compressed file from the original. and similarly how to read it. I'm using JAVA .
Have tried this:
LinkedList<Integer> numbers = new LinkedList<Integer>();
numbers.add(824);
numbers.add(5);
numbers.add(214577);
String code = VBEncoder.encodeToString(numbers);//returns 000001101011100010000101000011010000110010110001 into code
File file = new File("test.compressed");
DataOutputStream out = new DataOutputStream(new BufferedOutputStream(new FileOutputStream(file)));
out.writeBytes(code);
out.flush();
this just writes the binary representation into the file..and this is not what I'm expecting.
I have also tried this:
LinkedList<Integer> code = VBEncoder.encode(numbers);//returns linked list of Byte(i give its describtion later)
File file = new File("test.compressed");
DataOutputStream out = new DataOutputStream(new BufferedOutputStream(new FileOutputStream(file)));
for(Byte b:code){
out.write(b.toInt());
System.out.println(b.toInt());
}
out.flush();
// he goes the describtion of the class Byte
class Byte {
int[] abyte;
Byte() {
abyte = new int[8];
}
public void readInt(int n) {
String bin = Integer.toBinaryString(n);
for (int i = 0; i < (8 - bin.length()); i++) {
abyte[i] = 0;
}
for (int i = 0; i < bin.length(); i++) {
abyte[i + (8 - bin.length())] = bin.charAt(i) - 48;
}
}
public void switchFirst() {
abyte[0] = 1;
}
public int toInt() {
int res = 0;
for (int i = 0; i < 8; i++) {
res += abyte[i] * Math.pow(2, (7 - i));
}
return res;
}
public static Byte fromString(String codestring) {
Byte b = new Byte();
for(int i=0; i < 8; i++)
b.abyte[i] = (codestring.charAt(i)=='0')?0:1;
return b;
}
public String toString() {
String res = "";
for (int i = 0; i < 8; i++) {
res += abyte[i];
}
return res;
}
}
its prints this in the console:
6
184
133
13
12
177
this second attempt seems to work...the output file size is 6 bytes while for the first attemps it was 48 bytes.
but the problem in the second attempt is that I can't successfully read back the file.
InputStreamReader inStream = new InputStreamReader(new FileInputStream(file));
int c = -1;
while((c = inStream.read()) != -1){
System.out.println( c );
}
i get this:
6
184
8230
13
12
177
..so maybe I'm doing it the wrong way: expecting to receive some good advice from you. thanks!
It is solved; I was just not reading the file the right way:below is the right way:
DataInputStream inStream = null;
inStream = new DataInputStream(new BufferedInputStream(newFileInputStream(file)));
int c = -1;
while((c = inStream.read()) != -1){
Byte b = new Byte();
b.readInt(c);
System.out.println( c +":" + b.toString());
}
now I get this as the result:
6:00000110
184:10111000
133:10000101
13:00001101
12:00001100
177:10110001
Now the importance of writing the original sequence of integers into variable encoded bytes reduces the size of the file; if we normally write this sequence of integers in the file, its size would be 12 bytes (3 * 4 bytes). but now it is just 6 bytes.
int c = -1;
LinkedList<Byte> bytestream = new LinkedList<Byte>();
while((c = inStream.read()) != -1){
Byte b = new Byte();
b.readInt(c);
bytestream.add(b);
}
LinkedList<Integer> numbers = VBEncoder.decode(bytestream);
for(Integer number:numbers) System.out.println(number);
//
//here goes the code of VBEncoder.decode
public static LinkedList<Integer> decode(LinkedList<Byte> code) {
LinkedList<Integer> numbers = new LinkedList<Integer>();
int n = 0;
for (int i = 0; !(code.isEmpty()); i++) {
Byte b = code.poll();
int bi = b.toInt();
if (bi < 128) {
n = 128 * n + bi;
} else {
n = 128 * n + (bi - 128);
numbers.add(n);
n = 0;
}
}
return numbers;
}
I get back the sequence:
824
5
214577
how can i write less python2 code to achieve the same thing，need encryption results are the same.
b[i++] = (byte) 172; In what way cast int to byte in python2....
public static String encryptPassword(String content) {
String resultString = "";
String appkey = "ftjf,ckdfkl";
byte[] a = appkey.getBytes();
byte[] datSource = content.getBytes();
byte[] b = new byte[a.length + 4 + datSource.length];
int i;
for (i = 0; i < datSource.length; i++) {
b[i] = datSource[i];
}
b[i++] = (byte) 172;
b[i++] = (byte) 163;
b[i++] = (byte) 161;
b[i++] = (byte) 163;
for (int k = 0; k < a.length; k++) {
b[i] = a[k];
i++;
}
try {
MessageDigest md5 = MessageDigest.getInstance("MD5");
md5.update(b);
resultString = new HexBinaryAdapter().marshal(md5.digest());
} catch (Exception e) {
}
return resultString.toLowerCase();
}
public static void main(String[] args) {
System.out.print(encryptPassword("123456"));
}
Use chr to get the char of a byte in python.
from Crypto.Hash import MD5
key = 'ftjf,ckdfkl'
src= '123'
b = src+chr(172)+chr(163)+chr(161)+chr(163)+key
md5 = MD5.new()
md5.update(b)
text = md5.hexdigest()
print text
Given the src '123',output is:
eedb36cc3204923459dcf891224a0c1d
I needed a method that would convert hex to ascii, and most seem to be a variation of the following:
public String hexToAscii(String hex) {
StringBuilder sb = new StringBuilder();
StringBuilder temp = new StringBuilder();
for(int i = 0; i < hex.length() - 1; i += 2){
String output = hex.substring(i, (i + 2));
int decimal = Integer.parseInt(output, 16);
sb.append((char)decimal);
temp.append(decimal);
}
return sb.toString();
}
The idea is to look at
hexToAscii("51d37bdd871c9e1f4d5541be67a6ab625e32028744d7d4609d0c37747b40cd2d");
If I print the result out, I get
-Í#{t7?`Ô×D?2^b«¦g¾AUM??Ý{ÓQ.
This is not the result I am needing though. A friend got the correct result in PHP which was the string reverse of the following:
QÓ{݇žMUA¾g¦«b^2‡D×Ô`7t{#Í-
There are clearly characters that his hexToAscii function is encoding whereas mine is not.
Not really sure why this is the case, but how can I implement this version in Java?
Assuming your input string is in, I would use a method like this
public static byte[] decode(String in) {
if (in != null) {
in = in.trim();
List<Byte> bytes = new ArrayList<Byte>();
char[] chArr = in.toCharArray();
int t = 0;
while (t + 1 < chArr.length) {
String token = "" + chArr[t] + chArr[t + 1];
// This subtracts 128 from the byte value.
int b = Byte.MIN_VALUE
+ Integer.valueOf(token, 16);
bytes.add((byte) b);
t += 2;
}
byte[] out = new byte[bytes.size()];
for (int i = 0; i < bytes.size(); ++i) {
out[i] = bytes.get(i);
}
return out;
}
return new byte[] {};
}
And then you could use it like this
new String(decode("51d37bdd871c9e1f4d5541be67a6ab625e"
+"32028744d7d4609d0c37747b40cd2d"))
How about trying like this:
public static void main(String[] args) {
String hex = "51d37bdd871c9e1f4d5541be67a6ab625e32028744d7d4609d0c37747b40cd2d";
StringBuilder output = new StringBuilder();
for (int i = 0; i < hex.length(); i+=2) {
String str = hex.substring(i, i+2);
output.append((char)Integer.parseInt(str, 16));
}
System.out.println(output);
}