I have a ByteBuffer that contains a large file (100 MB):
java.nio.ByteBuffer byteBuffer = ByteBuffer.wrap(multipartFile.getBytes());
writeChannel.write(byteBuffer);
writeChannel.closeFinally();
I can only legally write 1 MB to the writeChannel at a time.
How do I slice up the contents of the ByteBuffer and write only a 1 MB slice at a time into the writeChannel?
You can use ByteBuffer#slice() to get a duplicate view of your base ByteBuffer instance, then bump the position along to expose a sliding window of content. Alternately, you can just do the same to your base buffer if you don't need to expose it to any other consumers.
You can change the starting position of your view of the content via the single-argument Buffer#position(int) method, and change the end position of your view via Buffer#limit(int). So long as you're mindful not to push the view beyond the limit of the underlying buffer, you can do the following:
final ByteBuffer view = base.slice();
for (int start = base.position(), end = base.limit(), stride = 1000000;
start != end;
start = view.limit())
consume(view.position(start)
.limit(start + Math.min(end - start, stride)));
I didn't test it, but it looks correct. It's possible to rewrite to avoid the initial setting of the position, which isn't strictly necessary here, but it incurs either some repetition or more awkward special case treatment of the first time through.
I left it this way to preserve the basic for loop structure.
As far as I know, write() on the writeChannel (which for the name I supposed is of type SocketChannel) will "attempt to write up to r bytes to the channel, where r is the number of bytes remaining in the buffer, that is, dst.remaining(), at the moment this method is invoked". (according to this)
The description of ByteBuffer.remaining() says that this method will "Return the number of elements between the current position and the limit."
So my guessing is not that you can't write the entire ByteBuffer, but that your code should call flip() on the ByteBuffer object so it become:
java.nio.ByteBuffer byteBuffer = ByteBuffer.wrap(multipartFile.getBytes());
byteBuffer.flip();
writeChannel.write(byteBuffer);
writeChannel.closeFinally();
As its said in Buffer.flip(): "After a sequence of channel-read or put operations, invoke this method to prepare for a sequence of channel-write or relative get operations"
Related
I am new to Random File Access, and I have encountered one issue - as far as I have understood, RandomAccessFile class provides a Random Access file for reading/writing. I can use seek() method to move to preferable position and start reading or wrting, but does not matter in this case. It is completely the random access? But in FileInputStream I have the same ability
read(bute[] byte, int off, int len)
this method provides me reading from some particular place. So, what is the difference? (I guess, InputStream read all file, but just go through all symbols before off position, but it only my guess).
Looking at the documentation of the read method:
https://docs.oracle.com/javase/7/docs/api/java/io/FileInputStream.html#read(byte[],%20int,%20int)
it states that off is "the start offset in the destination array b". So using this call, you can read the next len bytes from the stream and put them is a certain place in your memory buffer. This does not allow you to skip forward like the seek method of a random access file.
The read method you mention does not let you read from any particular place. It always reads from the "next" position in the stream, where it left off, and it puts the read bytes into the byte array at position off. off is the offset in the output, not the input.
I am trying to encrypt a file(txt, pdf, doc) using Google Tink - streaming AEAD encryption, below is the Java code which I am trying to execute. But all I get is 1 KB output encrypted file and no errors. All Input files whether 2 MB or more than 10 MB, output file will be always of 1 KB. I am unable to figure out what could be going wrong, can someone please help.
TinkConfig.register();
final int chunkSize = 256;
KeysetHandle keysetHandle = KeysetHandle.generateNew(
StreamingAeadKeyTemplates.AES128_CTR_HMAC_SHA256_4KB);
// 2. Get the primitive.
StreamingAead streamingAead = keysetHandle.getPrimitive(StreamingAead.class);
// 3. Use the primitive to encrypt some data and write the ciphertext to a file,
FileChannel ciphertextDestination =
new FileOutputStream("encyptedOutput.txt").getChannel();
String associatedData = "Tinks34";
WritableByteChannel encryptingChannel =
streamingAead.newEncryptingChannel(ciphertextDestination, associatedData.getBytes());
ByteBuffer buffer = ByteBuffer.allocate(chunkSize);
InputStream in = new FileInputStream("FileToEncrypt.txt");
while (in.available() > 0) {
in.read(buffer.array());
System.out.println(in);
encryptingChannel.write(buffer);
}
encryptingChannel.close();
in.close();
System.out.println("completed");
This is all about understanding ByteBuffer and how it operates. Let me explain.
in.read(buffer.array());
This writes data to the underlying array, but since array is decoupled from the state of the original buffer, the position of the buffer is not advanced. This is not good, as the next call:
encryptingChannel.write(buffer);
will now think that the position is 0. The limit hasn't changed either and is therefore still set to the capacity: 256. That means the result of the write operation is to write 256 bytes and set the position to the limit (the position).
Now the read operation still operates on the underlying byte array, and that's still 256 bytes in size. So all next read operations take place perfectly. However, all the write operations will assume that there are no bytes to be written, as the position remains at 256.
To use ByteBuffer you can use FileBuffer.read. Then you need to flip the buffer before writing the read data. Finally, after writing you need to clear the buffer's position (and limit, but that only changes on the last read) to prepare the buffer for the next read operation. So the order is commonly read, flip, write, clear for instances of Buffer.
Don't mix Channels and I/O streams, it will makes your life unnecessarily complicated, and learning how to use ByteBuffer is hard enough all by itself.
I have the following piece of code:
byte[] payloadArray = getPayload();
int size = (HEADER_SIZE+payloadArray.length);
ByteBuffer cmdBuffer = ByteBuffer.allocate(HEADER_SIZE+payloadArray.length);
//create command
ByteBuffer lengthBuf = ByteBuffer.allocate(2);
lengthBuf.order(ByteOrder.BIG_ENDIAN);
lengthBuf.putChar((char)(size-2));
cmdBuffer.put(lengthBuf);
cmdBuffer.put(getFlag());
After the execution of the last command the first two bytes of cmdBuffer should show the value from getFlag() and lengthBuf. Though, this is not visible inside cmdBuffer.
I am not sure what is the issue here. Could someone please help?
I suspect the reason is that there are no remaining bytes in lengthBuf.
From the JavaDoc on putChar(char):
Writes two bytes containing the given char value, in the current byte order, into this buffer at the current position, and then increments the position by two.
From the JavaDoc on put(ByteBuffe):
...
Otherwise, this method copies n = src.remaining() bytes from the given buffer into this buffer, starting at each buffer's current position. The positions of both buffers are then incremented by n.
So the put operation reads from the current position of lengthBuf which is the end and there are no bytes left.
Try lengthBuf.reset() after putChar(...) which should reset the position only. Please note that depending on the buffer implementation reset() might not be supported, but ByteBuffer.allocate() creates a HeapByteBuffer which supports that operation.
Edit:
Oleg Estekhin noted that you'd better use lengthBuf.flip() instead of lengthBuf.reset().
From the JavaDoc:
After a sequence of channel-read or put operations, invoke this method to prepare for a sequence of channel-write or relative get operations.
I have some questions on java.nio.Buffer. Basically, my question starts with if a flip() call is always needed to switch between read and write, or is it only needed for slow I/O, e.g. in the case of write then read, to ensure the data are completely written before they are read. My particular question is with mappedByteBuffer. It looks like if the file exists and is of the size that I know, I can just use the position(int newPosition) call to navigate to any part of the file, and perform read or write, i.e. basically use the buffer as a chunk of memory forgetting about the concepts of mark or limit. Is this true?
Consider the following example. If I have a file that contains integer 1, then 2 from the beginning, it seems I can put another integer 3 at position 0, the rewind and read 3 and 2 out from the buffer. Shouldn't the limit stop me from the second getInt like in a normal non-mmap buffer? When do I ever need to call flip() to switch between a write and read of a mappedByteBuffer? Thanks!
final int FILESIZE = 1024;
RandomAccessFile fileHandle;
FileChannel fileChannel;
File testFile = new File("c:/temp/testbbrw.dat");
fileHandle = new RandomAccessFile(testFile, "rw");
fileChannel = fileHandle.getChannel();
MappedByteBuffer mbb = fileChannel.map(FileChannel.MapMode.READ_WRITE, 0, FILESIZE);
int pos, data;
mbb.position(0);
mbb.putInt(3);
mbb.position(0);
data=mbb.getInt(); //I get 3
data=mbb.getInt(); //I get 2, which was written to the file before this program runs
mbb.force();
fileHandle.close();
This is what Buffer.flip does
347 public final Buffer flip() {
348 limit = position;
349 position = 0;
350 mark = -1;
351 return this;
352 }
It is preparing the buffer, so that the next read operations on the buffer start at position 0 and end at the current limit. means you tell it, that you are done with changing the buffer and ready to move or copy it somewhere else (which means reading it)
my question starts with if a flip() call is always needed to switch between read and write, or is it only needed for slow I/O, e.g. in the case of write then read, to ensure the data are completely written before they are read.
A Buffer of any description starts out in a state where you can read into it, or put to it, which is the same thing.
flip() puts it into a state where you can write from it, or get from it, which is the same thing.
Despite its (extremely stupid) name, flip() is not the inverse of flip(). Its only inverses are compact() and clear().
For clarity I find it best to always leave a Buffer in the readable state, and only flip it into the writable state when I need to, and get it back to readable immediately afterwards.
This is for I/O.
If you are doing just get() and put() I'm not sure I would use flip() at all, and as this is a MappedByteBuffer I certainly wouldn't be calling clear() or compact(), both of which could do terrible things to the file, and which also rules out using flip().
The design of the Buffer API in Java is confusing and counter-intuitive compared to a typical cyclic, finite buffer. Making things worse is poor choice of terms in the documentation, compounded by equivocal uses of read/write and put/get terms, the former referring to an external operation (typically by a Channel) using a Buffer, and the latter to operations provided by a Buffer.
Java Buffers
On creation, a new buffer is "empty", ready to be filled. It may be immediately supplied with some content in the constructor, but it remains in the "filling" state.
The flip() method "flips" the logical state of the buffer from being filled to being emptied. Rather idiotically flip() does not reverse itself, even though in normal English it would usually describe a logically reversible action. Indeed, looking at the code, calling it twice without an intervening clear or compact sets the buffer into an invalid state, causing other methods to return nonsense. [1]
The clear() and compact() methods are the logical inverse of flip(), restoring the buffer to a "filling" state, with the former also emptying it and the latter maintaining the remaining content.
The general recommendation is to keep any given buffer in a consistent state always, with a try/finally; for example:
ByteBuffer wrap(ByteBuffer src, ByteBuffer tgt) {
// assume buffers are *always* kept in the "filling" state
try {
src.flip(); // change `src` to "emptying"; assume tgt already filling
// transfer some or all of `src` to `tgt`
}
finally {
if(src.remaining()) { src.compact(); } // revert `src` to "filling" without discarding remaining data
else { src.clear(); } // compact() is (usually) less efficient than clearing
}
Typical Finite Cyclic Buffers
Where Java Buffers are most counter-intuitive is due to the fact that most cyclic-buffer implementations are concurrently read/write capable, such that they maintain three values, head, tail and capacity (the last often inferred from the backing array if the language permits it) and they simply permit the values to wrap. The head is where data is read from, and the tail is where it is written to. When the end of the underlying array is reached the value of head/tail is simply set to zero (i.e. it cycles around).
When head == tail, the buffer is empty. When inc(tail) == head, the buffer is full, and the current content length is arrived at by head <= tail ? (tail - head) : (capacity - head + tail). The size of the backing array is generally capacity+1 such that the the tail index is not equal to head when the buffer is full (which would be ambiguous without a separate flag).
This makes the internal index handling slightly more complex, but for the benefit not having to flip-flop state and never needing to "compact" the data back to the start of the internal array (though most implementations will reset the start/end indices to zero whenever the buffer is emptied).
Typically, this also translates to a trade-off when reading in that two array copies may be needed; first from head to the end of the array and then from beginning of the array to tail. Three copy operations may be needed when the target is also a buffer and wraps during write (but the extra copy is hidden within the put method).
Suppositions
My best guess is that Java defines Buffers in this manner so that all reads and writes to the buffer occur in contiguous blocks. This, presumably, enables downstream/internal optimizations when dealing with things like sockets, memory maps and channels, avoiding intermediary copies needing to be made.
Just guessing here.
Notes
[1] Invalid because double flipping will cause the limit to be set to 0, not capacity, which will in turn cause BufferOverflowException for most internal methods due to limit - position being <= 0 or position >= limit. For example:
511 final int nextPutIndex() { // package-private
512 if (position >= limit)
513 throw new BufferOverflowException();
514 return position++;
515 }
516
517 final int nextPutIndex(int nb) { // package-private
518 if (limit - position < nb)
519 throw new BufferOverflowException();
520 int p = position;
521 position += nb;
522 return p;
523 }
524
525 /**
526 * Checks the given index against the limit, throwing an {#link
527 * IndexOutOfBoundsException} if it is not smaller than the limit
528 * or is smaller than zero.
529 */
530 final int checkIndex(int i) { // package-private
531 if ((i < 0) || (i >= limit))
532 throw new IndexOutOfBoundsException();
533 return i;
534 }
Suppose each time the buffer of the input-steam read 1000 bytes. There are some start signs and the video name at the beginning of the buffer, before the actual video content, like 100 byte. I don't want to write them into the result buffer. So the first time write 101-999 to the buffer. And the second time I hope to write 1000-1999. Currently, it write 0-999 again, and the result video has an 900 extra bytes before the actually video contents.
Is there anyway to write the buffer skipping the first buffer length? thanks!
I use this code for skipping bytes from a ByteBuffer:
import java.nio.ByteBuffer;
public class Utility {
public static void skip(ByteBuffer bb, int skip) {
bb.position( bb.position() + skip);
}
}
Sophia, you really do need to include example code so people can help, but I see from your tags you are likely asking about NIO's ByteBuffer.
What you want to do is skip the content you don't want by way of the ByteBuffer.position(int) method - there is no magic in the ByteBuffer impl, it is a backing data store (either a byte[] or direct memory reference to OS) and a series of int pointers that refer to conceptual positions in the buffer (start, end, limit, etc.) -- you just want to make sure you "skip" the bytes you don't want, which can be done by moving the position beyond it so the next operation to write out the entire buffer will start from position and go to limit.