Is there a good way to combine ByteBuffer & FloatBuffer ?
For example I get byte[] data and I need to convert it to float[] data and vice versa :
byte[] to float[] (java.lang.UnsupportedOperationException) :
byte[] bytes = new bytes[N];
ByteBuffer.wrap(bytes).asFloatBuffer().array();
float[] to byte[] (works) :
float[] floats = new float[N];
FloatBuffer floatBuffer = FloatBuffer.wrap(floats);
ByteBuffer byteBuffer = ByteBuffer.allocate(floatBuffer.capacity() * 4);
byteBuffer.asFloatBuffer().put(floats);
byte[] bytes = byteBuffer.array();
array() is an optional operation for ByteBuffer and FloatBuffer, to be supported only when the backing Buffer is actually implemented on top of an array with the appropriate type.
Instead, use get to read the contents of the buffer into an array, when you don't know how the buffer is actually implemented.
To add to Louis's answer, arrays in Java are limited in that they must be an independent region of memory. It is not possible to have an array that is a view of another array, whether to point at some offset in another array or to reinterpret the bytes of another array as a different type.
Buffers (ByteBuffer, FloatBuffer, etc) were created to overcome this limitation. They are equivalent to arrays in that they compile into machine instructions that are as fast as array accesses, despite requiring the programmer to use what appear to be function calls.
For top performance and minimum memory usage, you should use ByteBuffer.wrap(bytes).asFloatBuffer() and then call get() and put().
To get a float array, you must allocate a new array and copy the data into it with ByteBuffer.wrap(bytes).asFloatBuffer().get(myFloatArray).
The array method of a ByteBuffer is not something that anyone should normally use. It will fail unless the Buffer is wrapping an array (instead of pointing to some memory-mapped region like a file or raw non-GC memory) and the array is of the same type as the buffer.
Related
I have two ByteBuffers, they are completely identical but somehow behave differently when used in the GL11.glTexImage2D(); method.
My code:
IntBuffer width = BufferUtils.createIntBuffer(1);
IntBuffer height = BufferUtils.createIntBuffer(1);
IntBuffer comp = BufferUtils.createIntBuffer(1);
ByteBuffer data = STBImage.stbi_load("dsc8x12.bmp", width, height, comp, 4);
byte[] bytes = new byte[data.limit()];
for (int i = 0; i < bytes.length; i++) {
bytes[i] = data.get(i);
}
ByteBuffer data2 = ByteBuffer.wrap(bytes);
System.out.println(data.equals(data2));
GL11.glTexImage2D(GL11.GL_TEXTURE_2D, 0, GL11.GL_RGBA, 1024, 12, 0, GL11.GL_RGBA, GL11.GL_UNSIGNED_BYTE, data);
The console output is true
Here is how the window looks when the final argument in the GL11.glTexImage2D(); is either data or data2
When data is used:
When data2 is used:
The reason why you get two different results with the two different ByteBuffers is that LWJGL only works with direct NIO Buffers, that is, ByteBuffers that are not backed by arrays.
The reason for this particular output that you are getting is two-fold:
the non-direct ByteBuffer you supply (the one being created by .wrap(array)) has an internal address field value of 0. This is what LWJGL looks at when it actually calls the native OpenGL function. It reads that address field value and expects this to be the virtual memory address of the client memory to upload to the texture.
Now, there are some OpenGL functions that will just SEGFAULT when given an invalid pointer, such as zero. However, in this case, the glTexImage2D function is actually semantically overloaded, as when it receives the zero address as the last argument, it will only allocate texture memory of the requested size (here the remaining size of the ByteBuffer) and not move any client memory to the server (i.e. the GPU).
The ByteBuffer returned by STBImage.stbi_load is direct, and hence, the correct texture memory from the client's virtual memory address space is uploaded to OpenGL.
So, in essence: When you use LWJGL, you must always only use direct NIO Buffers, not ones that are wrappers of arrays!
I have a byte array. But I want array length mod 4=0. So I want to add 00 after array until pass condition.
I find many solutions to append array
byte[] arr_combined = new byte[arr_1.length + arr_2.length];
System.arraycopy(arr_1, 0, arr_combined, 0, arr_1.length);
System.arraycopy(arr_2, 0, arr_combined, arr_1.length, arr_2.length);
But I do not want to create new byte array. I only want append byte after a byte array. Thanks
As it was already mentioned, Java arrays have fixed size which cannot be changed after array is created. In your case it's probably ok to use ByteArrayOutputStream:
ByteArrayOutputStream out = new ByteArrayOutputStream();
out.write(arr_1);
out.write(arr_2);
byte[] arr_combined = out.toByteArray();
This way you work with primitive byte type (unlike List<Byte>) which will be more efficient. The drawback is that ByteArrayOutputStream is synchronized, so there's some overhead for synchronization (though I guess it's much lower than boxing/unboxing overhead when using List<Byte>). Another drawback is that ByteArrayOutputStream.write is declared to throw IOException. In practice it's never thrown, but you have to catch/rethrow it.
If you don't mind using third-party libraries, there are plenty ways to create convenient wrappers over the primitive types which don't have these drawbacks. For example, TByteArrayList from Trove library.
If you have an array you can't modify the length of it. You should use an ArrayList:
List<Byte> bytes = new ArrayList<Byte>();
Here you can add a new element wherever you want just using add method so, supposing that b it's a byte variable:
bytes.add(b);
So, in this way, you can add the number of 00 that you want regardless of the length of the ArrayList.
EDIT: I saw now on this question that you can convert the ArrayList when you have complete it to an array, like this:
Byte[] arrayBytes = bytes.toArray(new Byte[bytes.size()]);
I expect it helps to you!
I'm trying to slice a byte array to prune the first part of the array. I'm using ByteBuffer but it does not behave like I would expect.
byte[] myArray = new byte[10];
ByteBuffer buf = ByteBuffer.wrap(myArray);
buf.position(5);
ByteBuffer slicedBuf = buf.slice();
byte[] newArray = slicedBuf.array();
I would expect the size of newArray to be 5, containing only the last portion of my ByteBuffer. Instead, the full byte array is returned. I understand that this is because the "backing buffer" is the same all along.
How can I slice to have only the desired part of the array?
EDIT: Added context
The bytes are received from network. The buffer is formed like this :
[ SHA1 hash ] [ data... lots of it ]
I already have a function that takes a byte array as a parameter and calculate the SHA1 hash. What I want is to slice the full buffer to pass only the data without the expected hash.
You can use the Arrays.copyOfRange method. For example:
// slice from index 5 to index 9
byte[] slice = Arrays.copyOfRange(myArray, 5, 10);
The ByteBuffer you created is being backed by that array. When you call slice() you effectively receive a specific view of that data:
Creates a new byte buffer whose content is a shared subsequence of this buffer's content.
So calling array() on that returned ByteBuffer returns the backing array in its entirety.
To extract all the bytes from that view, you could do:
byte[] bytes = new byte[slicedBuf.remaining()];
slicedBuf.read(bytes);
The bytes from that view would be copied to the new array.
Edit to add from comments below: It's worth noting that if all you're interested in doing is copying bytes from one byte[] to another byte[], there's no reason to use a ByteBuffer; simply copy the bytes.
I'm reading a binary file and storing each record into a byte[]. Now I'd like to collect these records into a Java Vector. (So that it can grow indefinitely.) But Vector takes Objects, not primitives (or arrays of primitives, as far as I can tell).
Is there way to "box" an array of primitives, or am I going to have to rewrite my code to turn my arrays into Arrays and my bytes into Bytes?
I tried concatenating the bytes into a String, but that failed miserable, due to String.append()'s propensity to treat my bytes as ints and convert them into String-y decimal representations!!
byte[] is-an Object (all arrays are, even primitive ones). There is nothing stopping you from adding a byte[] to a Vector.
Vector<byte[]> records = new Vector<byte[]>();
byte[] firstRecord = readRecord();
records.add(firstRecord);
Though it doesn't smell like a good design. Also, you should favour passing around List (the interface) over passing around a Vector (a concrete implementation of List).
You can add all the bytes in a byte[] to a Vector<Byte> by looping through each byte.
However, I wouldn't suggest you use Vector as it is a legacy class which was replaced in Java 1.2 (1998)
You can use an ArrayList instead, but this will use 4-16 times as much memory as the original byte[].
If you cannot use TByteArrayList, I suggest you use ByteArrayOutputStream and ByteArrayInputStream.
If you absolutely cannot convert the byte's into Bytes, then you might look into a primitive collection library such as Colt. It was written for high performance scientific stuff but it has primitive collection types that you can use.
You can do:
byte[] byteArr = new byte[]{0x41, 0x43};
List<byte[]> listBytes = Arrays.asList(byteArr); // to get a list
List<byte[]> list = new Vector<byte[]>(listBytes); // to instantiate a vector
System.out.println(Arrays.toString(list.get(0)));
Update (Based on your comments)
List<byte[]> l = new Vector<byte[]>(); // to instantiate a vector
l.add(new byte[]{0x51, 0x52});
System.out.println(Arrays.toString(l.get(0)));
OUTPUT
[81, 82]
I catch what u mean,
Yes it is impossible to put array of somethings into a Vector or even into ArrayList as one element, let me explain why the following code is completely right but we misunderstand it
Vector<byte[]> records = new Vector<byte[]>();
byte[] firstRecord = readRecord();
records.add(firstRecord);
The third line of this code doesn't put the array into the Vector but instead it puts the reference firstRecord into that Vector. Then if we change the contents of firstRecord after putting it in the vector, what happen is that we change the content of the Vector because we have two references to the same thing.
Is this:
ByteBuffer buf = ByteBuffer.allocate(1000);
...the only way to initialize a ByteBuffer?
What if I have no idea how many bytes I need to allocate..?
Edit: More details:
I'm converting one image file format to a TIFF file. The problem is the starting file format can be any size, but I need to write the data in the TIFF to little endian. So I'm reading the stuff I'm eventually going to print to the TIFF file into the ByteBuffer first so I can put everything in Little Endian, then I'm going to write it to the outfile. I guess since I know how long IFDs are, headers are, and I can probably figure out how many bytes in each image plane, I can just use multiple ByteBuffers during this whole process.
The types of places that you would use a ByteBuffer are generally the types of places that you would otherwise use a byte array (which also has a fixed size). With synchronous I/O you often use byte arrays, with asynchronous I/O, ByteBuffers are used instead.
If you need to read an unknown amount of data using a ByteBuffer, consider using a loop with your buffer and append the data to a ByteArrayOutputStream as you read it. When you are finished, call toByteArray() to get the final byte array.
Any time when you aren't absolutely sure of the size (or maximum size) of a given input, reading in a loop (possibly using a ByteArrayOutputStream, but otherwise just processing the data as a stream, as it is read) is the only way to handle it. Without some sort of loop, any remaining data will of course be lost.
For example:
final byte[] buf = new byte[4096];
int numRead;
// Use try-with-resources to auto-close streams.
try(
final FileInputStream fis = new FileInputStream(...);
final ByteArrayOutputStream baos = new ByteArrayOutputStream()
) {
while ((numRead = fis.read(buf)) > 0) {
baos.write(buf, 0, numRead);
}
final byte[] allBytes = baos.toByteArray();
// Do something with the data.
}
catch( final Exception e ) {
// Do something on failure...
}
If you instead wanted to write Java ints, or other things that aren't raw bytes, you can wrap your ByteArrayOutputStream in a DataOutputStream:
ByteArrayOutputStream baos = new ByteArrayOutputStream();
DataOutputStream dos = new DataOutputStream(baos);
while (thereAreMoreIntsFromSomewhere()) {
int someInt = getIntFromSomewhere();
dos.writeInt(someInt);
}
byte[] allBytes = baos.toByteArray();
Depends.
Library
Converting file formats tends to be a solved problem for most problem domains. For example:
Batik can transcode between various image formats (including TIFF).
Apache POI can convert between office spreadsheet formats.
Flexmark can generate HTML from Markdown.
The list is long. The first question should be, "What library can accomplish this task?" If performance is a consideration, your time is likely better spent optimising an existing package to meet your needs than writing yet another tool. (As a bonus, other people get to benefit from the centralised work.)
Known Quantities
Reading a file? Allocate file.size() bytes.
Copying a string? Allocate string.length() bytes.
Copying a TCP packet? Allocate 1500 bytes, for example.
Unknown Quantities
When the number of bytes is truly unknown, you can do a few things:
Make a guess.
Analyze example data sets to buffer; use the average length.
Example
Java's StringBuffer, unless otherwise instructed, uses an initial buffer size to hold 16 characters. Once the 16 characters are filled, a new, longer array is allocated, and then the original 16 characters copied. If the StringBuffer had an initial size of 1024 characters, then the reallocation would not happen as early or as often.
Optimization
Either way, this is probably a premature optimization. Typically you would allocate a set number of bytes when you want to reduce the number of internal memory reallocations that get executed.
It is unlikely that this will be the application's bottleneck.
The idea is that it's only a buffer - not the whole of the data. It's a temporary resting spot for data as you read a chunk, process it (possibly writing it somewhere else). So, allocate yourself a big enough "chunk" and it normally won't be a problem.
What problem are you anticipating?