I'm generating random ints and trying to write them down to a file. The problem is when I open the file I've created I don't find my ints but a set of symbols like squares etc... Is it a problem of encoding ?
import java.io.DataOutputStream;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
public class GenerateBigList {
public static void main(String[] args) {
//generate in memory big list of numbers in [0, 100]
List list = new ArrayList<Integer>(1000);
for (int i = 0; i < 1000; i++) {
Double randDouble = Math.random() * 100;
int randInt = randDouble.intValue();
list.add(randInt);
}
//write it down to disk
File file = new File("tmpFileSort.txt");
try {
FileOutputStream fos = new FileOutputStream("C:/tmp/tmpFileSort.txt");
DataOutputStream dos = new DataOutputStream(fos);
writeListInteger(list, dos);
dos.close();
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
private static void writeListInteger(List<Integer> list, DataOutputStream dos) throws IOException {
for (Integer elt : list) {
dos.writeInt(elt);
}
}
}
A partial copy paste from the created file:
/ O a C ? 6 N
From the doc:
public final void writeInt(int v) throws IOException
Writes an int to the underlying output stream as four bytes, high byte first. If no exception is thrown, the counter written is incremented by 4.
There is no encoding problem. That is what you see when you open a binary file with a text editor. Try to open with a hex editor.
Those "symbols" are your ints. That's what binary files look like if you open them in a text editor. Notice that the file is exactly 4000 bytes in size, and you wrote 1000 ints, at 4 bytes each.
If you read the file in with a DataInputStream you will get the original values back:
try (DataInputStream dis = new DataInputStream(
new BufferedInputStream(new FileInputStream("C:/tmp/tmpFileSort.txt")))) {
for (int i = 0; i < 1000; i++) {
System.out.println(dis.readInt());
}
} catch (IOException e) {
throw new RuntimeException(e);
}
It writes binary, not text. Your expectations are misplaced. If you want text, use a Writer.
Related
I have a file where data keeps appending. I am using java to read that file and process the data. To get the latest data, I am storing the offset till where I have read the file and continue reading from that offset when java process runs next.
RandomAccessFile f = new RandomAccessFile("file.txt","r");
f.seek(offset)
The problem here is performance. Its around 300 times slower than BufferedReader. Is it possible to resume reading from particular line using BufferedReader?
import java.io.RandomAccessFile;
import java.io.FileReader;
import java.io.IOException;
public class ReadFileExample {
public static void main(String[] args) {
RandomAccessFile objReader = null;
try {
String strCurrentLine;
long startTime = System.currentTimeMillis();
objReader = new RandomAccessFile("auditlog-2018-12-21.txt", "r");
while ((strCurrentLine = objReader.readLine()) != null) {
}
System.out.println(System.currentTimeMillis()-startTime);
} catch (Exception e) {
e.printStackTrace();
} finally {
try {
if (objReader != null)
objReader.close();
} catch (IOException ex) {
ex.printStackTrace();
}
}
}
}
Reading 30M file having 100,000 lines takes 12 s while replacing RandomAccessFile with BufferReader takes less than 400ms.
you can try the below code
try {
BufferedReader reader = Files.newBufferedReader(Paths.get("file.txt"), StandardCharsets.UTF_8);
List<String> line = reader.lines().skip(31).limit(1).collect(Collectors.toList());
line.stream().forEach(System.out::println);
}catch(Exception e){
}
I had several attempts of using java.io in several ways, but I could never get it to work. My idea was to store points earned in a file named save_data.txt, then retrieve the 3 highest integers in that list and display them on a leaderboard.
import java.io.BufferedReader;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.IOException;
import java.util.ArrayList;
public class TextFind {
public static void main(String[] args) {
ArrayList<Integer> list = new ArrayList<Integer>();
File file = new File("save_data.txt");
BufferedReader reader = null;
try {
reader = new BufferedReader(new FileReader(file));
String text = null;
while((text = reader.readLine()) != null) {
list.add(Integer.parseInt(text));
}
}catch (FileNotFoundException e) {
e.printStackTrace();
}catch (IOException e) {
e.printStackTrace();
} finally {
try {
if(reader != null) {
reader.close();
}
}catch(IOException e) {
}
}
}
}
I took this and called it when the game stopped running. It seems to do nothing.
You are not that far off actually. Are there values in your save_date.txt file? Here is some example using Java 8:
public static void main(String[] args) {
List<Integer> highScore = Arrays.asList(1, 2); // Dummy values
Path filePath = Paths.get("save_data.txt"); // Your saved data
// Transform available high scores to a single String using the System line separator to separated the values and afterwards transform the String to bytes ...
byte[] bytes = highScore.stream().map(Object::toString).collect(Collectors.joining(System.lineSeparator())).getBytes();
try {
// Write those high score bytes to a file ...
Files.write(filePath, bytes, StandardOpenOption.CREATE);
} catch (IOException e) {
e.printStackTrace();
}
List<String> lines = Collections.emptyList();
try {
// Read all available high scores lines from the file ...
lines = Files.readAllLines(filePath);
} catch (IOException e) {
e.printStackTrace();
}
int skipLines = Math.max(lines.size() - 3, 0); // You only want the three highest values so we use the line count to determine the amount of values that may be skipped and we make sure that the value may not be negative...
// Stream through all available lines stored in the file, transform the String objects to Integer objects, sort them, skip all values except the last three and sort their order descending
highScore = lines.stream().map(Integer::valueOf).sorted().skip(skipLines).sorted(Comparator.reverseOrder()).collect(Collectors.toList());
// Print the result
highScore.forEach(System.out::println);
}
I am doing my bachelor thesis on the topic of cryptography and its costs.
A part of that is to compare different crypto Algorithms and Cipher Modes in terms of Runtime and Resource cost.
For that I wrote a small tool that should work in four steps:
Read Input File
Encrypt the Input File and write it to a new file.
Read and decrypt the just written encrypted file.
Write another copy of the decrypted file onto file system.
Compare initial input file and decrypted file to see if they are equal.
It works just fine with a small .txt input File. But for some reason it doesn’t work with any other kind of file. If I take an Image as input File, the first few pixels are good, the rest ends up corrupted.
So as far as I understand the problem should be somehow when I initialize the Cipher or when I use the Streams.
I also tried to out comment the lines encrypting and decrypting and it also works then in just making plain copies of the input file which are identical.
Any suggestions are welcome, I will try to test them out asap and report back what results showed up.
I do apologies for the “Hungarian notations”. The p is just use for public and the l for local. It is the way we do it in our company.
So here is my class:
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.security.InvalidAlgorithmParameterException;
import java.security.InvalidKeyException;
import java.security.Key;
import java.security.NoSuchAlgorithmException;
import java.util.Arrays;
import java.util.Date;
import javax.crypto.Cipher;
import javax.crypto.CipherInputStream;
import javax.crypto.CipherOutputStream;
import javax.crypto.KeyGenerator;
import javax.crypto.NoSuchPaddingException;
import javax.crypto.spec.IvParameterSpec;
public class AES_Cipher_Test {
public String pLocalRef = "E:\\Test.txt";
public String pLocalRefOutput = "E:\\Test-crypted.txt";
public String pLocalCopyOutput = "E:\\Test-Neu.txt";
public Key pKeyAES = null;
public int pBitKey = 128;
public Cipher pCipher;
public FileOutputStream pFos;
public FileInputStream pFis;
public CipherOutputStream pCos;
public CipherInputStream pCis;
public File pInputFile = new File(this.pLocalRef);
public File pOutputFile = new File(this.pLocalRefOutput);
public File pGeneratedFile = new File(this.pLocalCopyOutput);
public AES_Cipher_Test() {
crypt_decrypt_write_File();
}
public void crypt_decrypt_write_File() {
byte[] lLoadedFile = null;
byte[] lGeneratedFileByte = null;
try {
// generate new random AES Key
KeyGenerator lKeygen = KeyGenerator.getInstance("AES");
lKeygen.init(this.pBitKey);
this.pKeyAES = lKeygen.generateKey();
// read input File
this.pFis = new FileInputStream(this.pInputFile);
FileInputStream tempStream = new FileInputStream(this.pInputFile);
int count = 0;
while (tempStream.read() != -1){
count ++;
}
lLoadedFile = new byte[count]; // new byte[this.pFis.available()]
this.pFis.read(lLoadedFile);
System.err.println("lLoadedFile.legth " + lLoadedFile.length);
this.pFis.close();
//init Cipher with AES Encrypt Mode CFB8 oder CTR
this.pCipher = Cipher.getInstance("AES/CTR/PKCS5Padding");
this.pCipher.init(Cipher.ENCRYPT_MODE, this.pKeyAES);
// build cipher stream from FileOutputStream
this.pFos = new FileOutputStream(this.pOutputFile);
this.pCos = new CipherOutputStream(this.pFos, this.pCipher);
//write encrypted Data to stream
this.pCos.write(lLoadedFile);
this.pCos.close();
this.pFos.close();
// init Cipher for decrypt Mode
this.pCipher.init(Cipher.DECRYPT_MODE, this.pKeyAES, new IvParameterSpec(this.pCipher.getIV()));
// read just written localFile and decrypt
this.pFis = new FileInputStream(this.pOutputFile);
tempStream = new FileInputStream(this.pOutputFile);
count = 0;
while (tempStream.read() != -1){
count ++;
}
byte[] lBytes = new byte[count];// new byte[this.pFis.available()]
this.pCis = new CipherInputStream(this.pFis, this.pCipher);
int lBytesRead = this.pCis.read(lBytes);
while (lBytesRead > -1) {
lBytesRead = this.pCis.read(lBytes);
}
this.pCis.close();
this.pFis.close();
System.err.println("lBytes.length " + lBytes.length);
// write new not crypted File to see if procedure works
this.pFos = new FileOutputStream(this.pLocalCopyOutput);
this.pFos.write(lBytes);
this.pFos.close();
//compare Input File and Output File
this.pFis = new FileInputStream(this.pGeneratedFile);
tempStream = new FileInputStream(this.pGeneratedFile);
count = 0;
while (tempStream.read() != -1){
count ++;
}
lGeneratedFileByte = new byte[count]; // new byte[this.pFis.available()]
int i = this.pFis.read(lGeneratedFileByte);
this.pFis.close();
System.err.println("lGeneratedFileByte.length " + i);
System.err.println("Test if initial File and new File are identical = " + Arrays.equals(lGeneratedFileByte, lLoadedFile));
} catch (FileNotFoundException e) {
throw new RuntimeException("FILE_DOES_NOT_EXIST", e);
} catch (NoSuchAlgorithmException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (NoSuchPaddingException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (InvalidKeyException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (InvalidAlgorithmParameterException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
public static void main(String args[]) {
System.err.println("Start AES_Cipher_Test");
long start = new Date().getTime();
new AES_Cipher_Test();
long runningTime = new Date().getTime() - start;
System.err.println("End AES_Cipher_Test");
System.err.println("Runtime: " + runningTime);
}
}
Usual series of mistakes.
read() isn't specified to fill the buffer. It is only specified to transfer at least one byte, or else return -1 indicating end of stream. You have to loop:
while ((count = in.read(buffer)) > 0)
{
out.write(buffer, 0, count);
}
Your existing loop while (!(lBytesRead < lBytes.length)) is basically nonsense.
available() explicitly is not the total number of bytes in the stream, and any usage of it to allocate a buffer of such a size is explicitly stated in the Javadoc to be incorrect. Again, you have to loop, see above. There are few if any uses of available(), and this isn't one of them.
Please have a look at the following code
package normal;
import java.io.*;
import java.util.*;
public class ReGenerateFiles
{
private File inputFolder, outputFolder;
private String outputFormat, nameOfTheFile;
private FileInputStream fis;
private FileOutputStream fos;
List <Byte> al;
private byte[] buffer;
private int blockNumber = 1;
public ReGenerateFiles(File inputFile, File outputFile, String nameOfTheFile, String outputFormat)
{
this.inputFolder = inputFile;
this.outputFolder = outputFile;
this.nameOfTheFile = nameOfTheFile;
this.outputFormat = outputFormat;
}
public void reGenerate() throws IOException
{
File file[] = inputFolder.listFiles();
for(int i=0;i<file.length;i++)
{
System.out.println(file[i]);
}
for(int i=0;i<file.length;i++)
{
try
{
buffer = new byte[5000000];
int read = fis.read(buffer, 0, buffer.length);
fis.close();
writeBlock();
}
catch(IOException io)
{
io.printStackTrace();
}
finally
{
fis.close();
fos.close();
}
}
}
private void writeBlock()throws IOException
{
try
{
fos = new FileOutputStream(outputFolder+"/"+nameOfTheFile+"."+outputFormat,true);
fos.write(buffer, 0, buffer.length);
fos.flush();
fos.close();
}
catch(Exception e)
{
fos.close();
e.printStackTrace();
}
finally
{
fos.close();
}
}
}
In here, I am trying to re-assemble a splited file, back to its original file. This is how I split it (The selected answer)
Now, when I am trying to re-assemble it, I am getting an error saying something similar to "Cannot access kalimba.mp3. It is used by another program". This happens after executing 93 splitted files (there are 200 more). Why it is happening, even though I have make sure the streams are closed inside finally block?
Another question, I have assigned 500000 as the byte array value. I did this because I failed to set the byte array according to the original size of the particular file which is about to process. I assigned the byte array value as
buffer = new byte[(byte)file[i].length();
before, but it didn't work.
Please help me to solve these two issues and re-assemble the splitted file back, correctly.
That is because the buffer never fitted to the size of the actual content. It is always 50000, and the actually size vary. Thats the issue
I received this python script that generates a file checksum:
import sys,os
if __name__=="__main__":
#filename=os.path.abspath(sys.argv[1])
#filename=r"H:\Javier Ortiz\559-7 From Pump.bin"
cksum=0
offset=0
pfi=open(filename,'rb')
while 1:
icks=0
chunk=pfi.read(256)
if not chunk: break #if EOF exit loop
for iter in chunk:
icks+=ord(iter)
print ord(iter)
cksum=(cksum+icks) & 0xffff
pfi.close()
print "cksum=0x%4.4x"%cksum
And I'm trying to convert it to Java but I'm not geting the same results.
Here's my Java code:
import java.io.BufferedInputStream;
import java.io.DataInputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
public class ChecksumCalculator {
private ChecksumCalculator() {
}
public static int getChecksum(File file) {
int cksum = 0;
FileInputStream fis = null;
BufferedInputStream bis = null;
DataInputStream dis = null;
try {
fis = new FileInputStream(file);
// Here BufferedInputStream is added for fast reading.
bis = new BufferedInputStream(fis);
dis = new DataInputStream(bis);
byte[] buffer = new byte[256];
// dis.available() returns 0 if the file does not have more lines.
while (dis.read(buffer) != -1) {
int icks = 0;
for (byte b : buffer) {
icks += b & 0xff;
System.out.println(b & 0xff);
}
cksum = (cksum + icks) & 0xffff;
System.out.println("Checksum: " + cksum);
}
// dispose all the resources after using them.
fis.close();
bis.close();
dis.close();
return cksum;
} catch (FileNotFoundException e) {
e.printStackTrace();
return -1;
} catch (IOException e) {
e.printStackTrace();
return -1;
}
}
static public void main(String[] s) {
System.out.println("0x" + getChecksum(new File("H:\\Javier Ortiz\\559-7 From Pump.bin")));
}
}
But I get different results on a file. For example if I run it on a plain txt file containing only the word test it gives out the following result:
python: cksum=0x01c0
java: cksum=0x448
Any idea?
Your Python version prints the checksum in hex, while your Java version prints it in decimal. You should make your Java version print in hex, too. 0x1c0 == 448.
To use the cksum=0x%4.4x format string as you had in your Python version, use this:
System.out.printf("cksum=0x%4.4x%n", ...);
or even better
System.out.printf("cksum=%#04x%n", ...);
Also, you don't need a DataInputStream for this. Just use bis.read(buffer) instead of dis.read(buffer).
1C016 = 44810
I think that's your problem.
dis.read(buffer) returns the number of bytes that was actually read. For the last chunk, it will probably be less than 256. So the for loop shouldn't always be performed 256 times - it should be performed as many times as the actual byte count that was read from the stream.
I'm not a Python developer, but it doesn't look like ord(icks) in Python does the same as b & 0xff in Java.
Keep in mind that all Java types are signed; this might affect the calculation.
Also, although this doesn't affect correctness - it's a good practice to clean all the resources (i.e. to close the streams) in a finally block.