Suppose each time the buffer of the input-steam read 1000 bytes. There are some start signs and the video name at the beginning of the buffer, before the actual video content, like 100 byte. I don't want to write them into the result buffer. So the first time write 101-999 to the buffer. And the second time I hope to write 1000-1999. Currently, it write 0-999 again, and the result video has an 900 extra bytes before the actually video contents.
Is there anyway to write the buffer skipping the first buffer length? thanks!
I use this code for skipping bytes from a ByteBuffer:
import java.nio.ByteBuffer;
public class Utility {
public static void skip(ByteBuffer bb, int skip) {
bb.position( bb.position() + skip);
}
}
Sophia, you really do need to include example code so people can help, but I see from your tags you are likely asking about NIO's ByteBuffer.
What you want to do is skip the content you don't want by way of the ByteBuffer.position(int) method - there is no magic in the ByteBuffer impl, it is a backing data store (either a byte[] or direct memory reference to OS) and a series of int pointers that refer to conceptual positions in the buffer (start, end, limit, etc.) -- you just want to make sure you "skip" the bytes you don't want, which can be done by moving the position beyond it so the next operation to write out the entire buffer will start from position and go to limit.
Related
I am new to Random File Access, and I have encountered one issue - as far as I have understood, RandomAccessFile class provides a Random Access file for reading/writing. I can use seek() method to move to preferable position and start reading or wrting, but does not matter in this case. It is completely the random access? But in FileInputStream I have the same ability
read(bute[] byte, int off, int len)
this method provides me reading from some particular place. So, what is the difference? (I guess, InputStream read all file, but just go through all symbols before off position, but it only my guess).
Looking at the documentation of the read method:
https://docs.oracle.com/javase/7/docs/api/java/io/FileInputStream.html#read(byte[],%20int,%20int)
it states that off is "the start offset in the destination array b". So using this call, you can read the next len bytes from the stream and put them is a certain place in your memory buffer. This does not allow you to skip forward like the seek method of a random access file.
The read method you mention does not let you read from any particular place. It always reads from the "next" position in the stream, where it left off, and it puts the read bytes into the byte array at position off. off is the offset in the output, not the input.
I am trying to encrypt a file(txt, pdf, doc) using Google Tink - streaming AEAD encryption, below is the Java code which I am trying to execute. But all I get is 1 KB output encrypted file and no errors. All Input files whether 2 MB or more than 10 MB, output file will be always of 1 KB. I am unable to figure out what could be going wrong, can someone please help.
TinkConfig.register();
final int chunkSize = 256;
KeysetHandle keysetHandle = KeysetHandle.generateNew(
StreamingAeadKeyTemplates.AES128_CTR_HMAC_SHA256_4KB);
// 2. Get the primitive.
StreamingAead streamingAead = keysetHandle.getPrimitive(StreamingAead.class);
// 3. Use the primitive to encrypt some data and write the ciphertext to a file,
FileChannel ciphertextDestination =
new FileOutputStream("encyptedOutput.txt").getChannel();
String associatedData = "Tinks34";
WritableByteChannel encryptingChannel =
streamingAead.newEncryptingChannel(ciphertextDestination, associatedData.getBytes());
ByteBuffer buffer = ByteBuffer.allocate(chunkSize);
InputStream in = new FileInputStream("FileToEncrypt.txt");
while (in.available() > 0) {
in.read(buffer.array());
System.out.println(in);
encryptingChannel.write(buffer);
}
encryptingChannel.close();
in.close();
System.out.println("completed");
This is all about understanding ByteBuffer and how it operates. Let me explain.
in.read(buffer.array());
This writes data to the underlying array, but since array is decoupled from the state of the original buffer, the position of the buffer is not advanced. This is not good, as the next call:
encryptingChannel.write(buffer);
will now think that the position is 0. The limit hasn't changed either and is therefore still set to the capacity: 256. That means the result of the write operation is to write 256 bytes and set the position to the limit (the position).
Now the read operation still operates on the underlying byte array, and that's still 256 bytes in size. So all next read operations take place perfectly. However, all the write operations will assume that there are no bytes to be written, as the position remains at 256.
To use ByteBuffer you can use FileBuffer.read. Then you need to flip the buffer before writing the read data. Finally, after writing you need to clear the buffer's position (and limit, but that only changes on the last read) to prepare the buffer for the next read operation. So the order is commonly read, flip, write, clear for instances of Buffer.
Don't mix Channels and I/O streams, it will makes your life unnecessarily complicated, and learning how to use ByteBuffer is hard enough all by itself.
I have a ByteBuffer object called msg with the intended message length in the first four bytes, which I read as follows:
int msgLen = msg.getInt();
LOG.debug("Message size: " + msgLen);
If the msgLen is less than some threshold value, I have a partial message and need to cache. In this case, I'd like to put those first four bytes back into the beginning of the message; that is, put the message back together to be identical to pre-reading. For example:
if (msgLen < threshold) {
msg.rewind();
msg.put(msgLen);
Unfortunately, this does not seem to be the correct way to do this. I've tried many combinations of flip, put, and rewind, but must be misunderstanding.
How would I put the bytes back into the write buffer in their original order?
Answer was posted by Andremoniy in comments section. Read operations do not consume bytes in the buffer, so msg.rewind() was adequate. This didn't work in my case because of some other logic in the program, and I incorrectly associated that with a problem at the buffer level.
I have a very small threaded application, which is collecting small chunks of data in arrays (because it is sound data, and Java wants that to be an array) and trying to put it into an ArrayList for storage. All of that is effectively the front half of a producer/consumer pattern.
Problem: It doesn't seem to work.
On the producer end, I have this code:
public synchronized void run() {
// do a whole bunch of audio set-up
try {
// more audio stuff
while (true) {
if (producing) {
byte[] data = new byte[line.getBufferSize() ];
numBytesRead = line.read(data, 0, data.length);
System.out.println("Producer: Size of dat[] is " + data.length);
// Save this chunk of data.
buffer.addData(data);
}
This seems straightforward, aside from the audio stuff and bookkeeping.
In the buffer class, I have:
public class Buffer {
ArrayList list ;
public void addData(byte[] data) {
list.addAll(Arrays.asList(data));
}
This also seems straightforward.
Here is the problem: If my array is of length (say) 1024, and the elements are all there (which I've verified that they are) I would expect the size of the ArrayList to grow by 1024 every time I add data. It doesn't. It grows by 1, as though I was making either an ArrayList of ArrayLists or an ArrayList of Arrays, rather than the Arraylist of elements I desire.
I suspect I'm going to have this problem on the flip side as well, where I might have an ArrayList of tens of thousands of bytes, and want to retrieve an array of the first 1024 elements.
I cannot help but think I'm missing something very simple. Can anyone shed light on why this is not working? (Or if there is some fundamentally better way to do what I'm trying to do?)
Arrays.asList() will not perform the conversion from byte to Byte, it will return a list containing 1 element; the byte[] you pass in.
If your aim is to add Byte objects for every byte, you will have to do that yourself in a loop. Note that this will use much more memory than passing byte[]s however.
Also note that it is not garuanteed that, even if the input stream has more than enough data left, that you will read data.length bytes every time (result of buffer sizes, concurency, etc.) so you run the risk of passing a bunch of 0 bytes at the end of your buffer if you read less bytes than you asked for.
byte[] data should be Byte[]. You must use object, not primitive.
I cannot help but think I'm missing something very simple. Can anyone shed light on why this is not working? (Or if there is some fundamentally better way to do what I'm trying to do?)
Storing byte data in an ArrayList<Byte> has a lot of memory overhead, compared to a byte[] array. if you're working with a large amount of data, you may want to use a byte[] array for storage as well. Take a look at the source code for ByteArrayOutputStream - I don't know if will work for you as-is, but you might be able to create a similar sort of class that manages an expanding byte array.
I'm using a FileReader wrapped in a LineNumberReader to index a large text file for speedy access later on. Trouble is I can't seem to find a way to read a specific line number directly. BufferedReader supports the skip() function, but I need to convert the line number to a byte offset (or index the byte offset in the first place).
I took a crack at it using RandomAccessFile, and while it worked, it was horribly slow during the initial indexing. BufferedReader's speed is fantastic, but... well, you see the problem.
Some key info:
The file can be any size (currently 35,000 lines)
It's stored on Android's internal filesystem (via getFilesDir() to be exact)
The formatting is not fixed width, unfortunately (hence the need to read by line)
Any ideas?
Describes an extended RandomAccessFile with buffering semantics
Trouble is I can't seem to find a way to read a specific line number directly
Unless you know the length of each line you can't read it directly
There is no shortcut, you will need to read then entire file up front and calculate the offsets manualy.
I would just use a BufferedReader and then get the length of each string and add 1 (or 2?) for the EOL string.
Consider saving an file index along with the large text file. If this file is something you are generating, either on your server, or on the device, it should be trivial to generate an index once and distribute and/or save it along with the file.
I'd recommend an int[] where each value is the absolute offset in bytes for the n*(index+1) th line. So you could have an array of size 35,000 with the start of each line, or an array of size 350, with the start of every 100th line.
Here's an example assuming you have an index file containing an raw sequence of int values:
public String getLineByNumber(RandomAccessFile index,
RandomAccessFile data,
int lineNum) {
index.seek(lineNum*4);
data.seek(index.readInt());
return data.readLine();
}
I took a crack at it using
RandomAccessFile, and while it worked,
it was horribly slow during the
initial indexing
You've started the hard part already. Now for the harder part.
BufferedReader's speed is fantastic,
but...
Is there something in your use of RandomAccessFile that made it slower than it has to be? How many bytes did you read at a time? If you read one byte at a time it will be sloooooow. IF you read in an array of bytes at a time, you can speed things up and use the byte array as a buffer.
Just wrapping up the previous comments :
Either you use RandomAccessFile to first count byte and second parse what you read to find lines by hand OR you use a LineNumberReader to first read lines by lines and count the bytes of each line of char (2 bytes in utf 16 ?) by hand.