I have a very small threaded application, which is collecting small chunks of data in arrays (because it is sound data, and Java wants that to be an array) and trying to put it into an ArrayList for storage. All of that is effectively the front half of a producer/consumer pattern.
Problem: It doesn't seem to work.
On the producer end, I have this code:
public synchronized void run() {
// do a whole bunch of audio set-up
try {
// more audio stuff
while (true) {
if (producing) {
byte[] data = new byte[line.getBufferSize() ];
numBytesRead = line.read(data, 0, data.length);
System.out.println("Producer: Size of dat[] is " + data.length);
// Save this chunk of data.
buffer.addData(data);
}
This seems straightforward, aside from the audio stuff and bookkeeping.
In the buffer class, I have:
public class Buffer {
ArrayList list ;
public void addData(byte[] data) {
list.addAll(Arrays.asList(data));
}
This also seems straightforward.
Here is the problem: If my array is of length (say) 1024, and the elements are all there (which I've verified that they are) I would expect the size of the ArrayList to grow by 1024 every time I add data. It doesn't. It grows by 1, as though I was making either an ArrayList of ArrayLists or an ArrayList of Arrays, rather than the Arraylist of elements I desire.
I suspect I'm going to have this problem on the flip side as well, where I might have an ArrayList of tens of thousands of bytes, and want to retrieve an array of the first 1024 elements.
I cannot help but think I'm missing something very simple. Can anyone shed light on why this is not working? (Or if there is some fundamentally better way to do what I'm trying to do?)
Arrays.asList() will not perform the conversion from byte to Byte, it will return a list containing 1 element; the byte[] you pass in.
If your aim is to add Byte objects for every byte, you will have to do that yourself in a loop. Note that this will use much more memory than passing byte[]s however.
Also note that it is not garuanteed that, even if the input stream has more than enough data left, that you will read data.length bytes every time (result of buffer sizes, concurency, etc.) so you run the risk of passing a bunch of 0 bytes at the end of your buffer if you read less bytes than you asked for.
byte[] data should be Byte[]. You must use object, not primitive.
I cannot help but think I'm missing something very simple. Can anyone shed light on why this is not working? (Or if there is some fundamentally better way to do what I'm trying to do?)
Storing byte data in an ArrayList<Byte> has a lot of memory overhead, compared to a byte[] array. if you're working with a large amount of data, you may want to use a byte[] array for storage as well. Take a look at the source code for ByteArrayOutputStream - I don't know if will work for you as-is, but you might be able to create a similar sort of class that manages an expanding byte array.
Related
I wanted to know how I can get the data send through an APDU command if buffer[ISO7816.OFFSET_LC] is >=3.
Actually I am using
if (numBytes == 1) {
shortAmount = (short) buffer[ISO7816.OFFSET_CDATA];
} else if (numBytes == 2) {
shortAmount = (short) Util.getShort(buffer, ISO7816.OFFSET_CDATA);
} else if(numBytes == 3) {
//how to get the all data contained in the APDU?
}
For most Java Card implementations there is no support for the int base type. This means you cannot store more than 16 bits in one single variable.
You can however store it in an array. And - when you think about it - it already is. The APDU buffer is nothing other than a byte array in transient memory (RAM, usually) *1.
So how you handle the APDU data in the APDU buffer is up to you:
you could parse structures inside the APDU buffer;
you could copy it to another array, either persistent (EEPROM / Flash using new byte[size]) or transient memory (RAM, through JCSystem.makeTransientByteArray())
you could convert it to shorts using Util.getShort() and store the result in a short array similar to a byte array;
you could use it as input to any API that allows input from a byte array, such as one of the Key#setValue commands, OwnerPIN#check(), signature verification etc. etc..
Or you can perform any combination of above.
If you want to perform calculations with larger values you'll have to implement those or have access to special libraries. For 32 bit integer calculations have a look at my X-mas special answer. This also shows that any kind of calculations on the data are possible, even if you have to implement them yourself. Java Card is Turing-complete, so as long as you don't run out of CPU time or memory you can do any possible calculation on it.
*1 OK, the APDU buffer is a rather special array in the way it is handled by the Java Card system, but during processing of data it just acts as a normal Java Card byte array.
Store the data in to a byte array.
Key point is, you need to process the data as a byte array if its size is >= 3. For processing however, you can develop your own API's or you can use the available one.
If you want to interpret the data sent through an APDU as an integer (== 4 bytes), then you need to store the data in to a byte array. Now depending upon the use case, you can use a JCInteger class provided by "Maarten Bodewes" to further process the data.
Key point is, you need to process the data as a byte array if its size is >= 3. For processing however, you can develop your own API's or you can use the available one.
I have an array of longs I want to write to disk. The most efficient disk I/O functions take in byte arrays, for example:
FileOutputStream.write(byte[] b, int offset, int length)
...so I want to begin by converting my long[] to byte[] (8 bytes for each long). I'm struggling to find a clean way to do this.
Direct typecasting doesn't seem allowed:
ConversionTest.java:6: inconvertible types
found : long[]
required: byte[]
byte[] byteArray = (byte[]) longArray;
^
It's easy to do the conversion by iterating over the array, for example:
ByteBuffer bytes = ByteBuffer.allocate(longArray.length * (Long.SIZE/8));
for( long l: longArray )
{
bytes.putLong( l );
}
byte[] byteArray = bytes.array();
...however that seems far less efficient than simply treating the long[] as a series of bytes.
Interestingly, when reading the file, it's easy to "cast" from byte[] to longs using Buffers:
LongBuffer longs = ByteBuffer.wrap(byteArray).asLongBuffer();
...but I can't seem to find any functionality to go the opposite direction.
I understand there are endian considerations when converting from long to byte, but I believe I've already addressed those: I'm using the Buffer framework shown above, which defaults to big endian, regardless of native byte order.
No, there is not a trivial way to convert from a long[] to a byte[].
Your best option is likely to wrap your FileOutputStream with a BufferedOutputStream and then write out the individual byte values for each long (using bitwise operators).
Another option is to create a ByteBuffer and put your long values into the ByteBuffer and then write that to a FileChannel. This handles the endianness conversion for you, but makes the buffering more complicated.
Concerning the efficiency, many details will, in fact, hardly make a difference. The hard disk is by far the slowest part involved here, and in the time that it takes to write a single byte to the disk, you could have converted thousands or even millions of bytes to longs. Every performance test here will not tell you anything about the performance of the implementation, but about the performance of the hard disk. In doubt, one should make dedicated benchmarks comparing the different conversion strategies, and comparing the different writing methods, respectively.
Assuming that the main goal is a functionality that allows a convenient conversion and does not impose an unnecessary overhead, I'd like to propose the following approach:
One can create a ByteBuffer of sufficient size, view this as a LongBuffer, use the bulk LongBuffer#put(long[]) method (which takes care of endianness conversions, of necessary, and does this as efficient as it can be), and finally, write the original ByteBuffer (which is now filled with the long values) to the file, using a FileChannel.
Following this idea, I think that this method is convenient and (most likely) rather efficient:
private static void bulkAndChannel(String fileName, long longArray[])
{
ByteBuffer bytes =
ByteBuffer.allocate(longArray.length * Long.BYTES);
bytes.order(ByteOrder.nativeOrder()).asLongBuffer().put(longArray);
try (FileOutputStream fos = new FileOutputStream(fileName))
{
fos.getChannel().write(bytes);
}
catch (IOException e)
{
e.printStackTrace();
}
}
(Of course, one could argue about whether allocating a "large" buffer is the best idea. But thanks to the convenience methods of the Buffer classes, this could easily and with reasonable effort be modified to write "chunks" of data with an appropriate size, for the case that one really wants to write a huge array and the memory overhead of creating the corresponding ByteBuffer would be prohibitively large)
OP here.
I have thought of one approach: ByteBuffer.asLongBuffer() returns an instance of ByteBufferAsLongBufferB, a class which wraps ByteBuffer in an interface for treating the data as longs while properly managing endianness. I could extend ByteBufferAsLongBufferB, and add a method to return the raw byte buffer (which is protected).
But this seems so esoteric and convoluted I feel there must be an easier way. Either that, or something in my approach is flawed.
I want to optimize my code by using ByteBuffer in place of String. What I am getting is String[]. I am doing formatting on each element of it.
e.g. String strAry[] = {"Help", "I", "am", "trapped", "in", "a", "fortune", "cookie", "factory"};
is my String array, I am writing content of it to a .csv file in
format "StrArray[0]";"StrArray[1]";"StrArray2";"StrArray[3]"; so on...
which is internally creating multiple Strings and this code is running into loop for hundreds n thousands of time some time.
I want to implement ByteBuffer. While creating
ByteBuffer bbuf = ByteBuffer.allocate(bufferSize); I need to specify buffer size here.
I dont want to iterate over each element of String [] to calculate its byteSize.
Any help is appreciated.
Couple of notes:
Data structure usage
I think you should be using CharBuffer and not ByteBuffer. CharBuffer is requiring the number of characters and not bytes.
Buffers from Java NIO are always used as buffers, that means there is a possibility that you will need to read into them multiple times.
If you need to have the whole content in memory, buffers are not the data structure for this use case.
You don't have to know the exact size for a buffer, the allocated size is the maximal capacity of the buffer.
StringBuilder is a mutable data structure for string processing. You might consider using it instead.
You don't have to know the exact size.
Computation of final size
might be done using Stream API (Java 8) or similar utility methods.
Suppose each time the buffer of the input-steam read 1000 bytes. There are some start signs and the video name at the beginning of the buffer, before the actual video content, like 100 byte. I don't want to write them into the result buffer. So the first time write 101-999 to the buffer. And the second time I hope to write 1000-1999. Currently, it write 0-999 again, and the result video has an 900 extra bytes before the actually video contents.
Is there anyway to write the buffer skipping the first buffer length? thanks!
I use this code for skipping bytes from a ByteBuffer:
import java.nio.ByteBuffer;
public class Utility {
public static void skip(ByteBuffer bb, int skip) {
bb.position( bb.position() + skip);
}
}
Sophia, you really do need to include example code so people can help, but I see from your tags you are likely asking about NIO's ByteBuffer.
What you want to do is skip the content you don't want by way of the ByteBuffer.position(int) method - there is no magic in the ByteBuffer impl, it is a backing data store (either a byte[] or direct memory reference to OS) and a series of int pointers that refer to conceptual positions in the buffer (start, end, limit, etc.) -- you just want to make sure you "skip" the bytes you don't want, which can be done by moving the position beyond it so the next operation to write out the entire buffer will start from position and go to limit.
I'm reading a binary file and storing each record into a byte[]. Now I'd like to collect these records into a Java Vector. (So that it can grow indefinitely.) But Vector takes Objects, not primitives (or arrays of primitives, as far as I can tell).
Is there way to "box" an array of primitives, or am I going to have to rewrite my code to turn my arrays into Arrays and my bytes into Bytes?
I tried concatenating the bytes into a String, but that failed miserable, due to String.append()'s propensity to treat my bytes as ints and convert them into String-y decimal representations!!
byte[] is-an Object (all arrays are, even primitive ones). There is nothing stopping you from adding a byte[] to a Vector.
Vector<byte[]> records = new Vector<byte[]>();
byte[] firstRecord = readRecord();
records.add(firstRecord);
Though it doesn't smell like a good design. Also, you should favour passing around List (the interface) over passing around a Vector (a concrete implementation of List).
You can add all the bytes in a byte[] to a Vector<Byte> by looping through each byte.
However, I wouldn't suggest you use Vector as it is a legacy class which was replaced in Java 1.2 (1998)
You can use an ArrayList instead, but this will use 4-16 times as much memory as the original byte[].
If you cannot use TByteArrayList, I suggest you use ByteArrayOutputStream and ByteArrayInputStream.
If you absolutely cannot convert the byte's into Bytes, then you might look into a primitive collection library such as Colt. It was written for high performance scientific stuff but it has primitive collection types that you can use.
You can do:
byte[] byteArr = new byte[]{0x41, 0x43};
List<byte[]> listBytes = Arrays.asList(byteArr); // to get a list
List<byte[]> list = new Vector<byte[]>(listBytes); // to instantiate a vector
System.out.println(Arrays.toString(list.get(0)));
Update (Based on your comments)
List<byte[]> l = new Vector<byte[]>(); // to instantiate a vector
l.add(new byte[]{0x51, 0x52});
System.out.println(Arrays.toString(l.get(0)));
OUTPUT
[81, 82]
I catch what u mean,
Yes it is impossible to put array of somethings into a Vector or even into ArrayList as one element, let me explain why the following code is completely right but we misunderstand it
Vector<byte[]> records = new Vector<byte[]>();
byte[] firstRecord = readRecord();
records.add(firstRecord);
The third line of this code doesn't put the array into the Vector but instead it puts the reference firstRecord into that Vector. Then if we change the contents of firstRecord after putting it in the vector, what happen is that we change the content of the Vector because we have two references to the same thing.