Develop Reference

conversion of byte array to string causing OOM

In my application i m storing strings using randomaccessfile and while reading back the string i need to convert byte array to string, which is causing OOM. Is there a better way to convert other than this
str = new String(b, "UTF-8");
where b is byte array

Is there a better way to convert other than new String(bytes, "UTF-8") ?
This is actually a rather complicated question.
This constructor cannot simply incorporate the byte[] into the string:
Prior to Java 9, it is always necessary to decode the byte array to a UTF-16 coded array of char. So the constructor is liable to allocate roughly double the memory used by the source byte[].
With Java 9 you have the option of using a new compact representation for String. If you do the AND if the UTF-8 encoded byte array only contains code-points in Unicode code-plane zero (\u0000 to \u00ff) then the String value is a byte[]. However, even in this case the constructor must copy the bytes to a new byte[].
In both cases, there is no more space-efficient way to create a String from a byte[]. Furthermore, I don't think there is a more space-efficient way do the conversion starting with a stream of bytes and a character count. (I am excluding things like modifying the lava.lang.* implementation, or breaking abstraction using reflection.)
Bottom line: when converting a byte[] to a String you should allow at least twice as much contiguous free memory as the original byte[] if you want your code to work on older JVMs.

How to read different objects from byte[]

I have a byte[] with 3 different objects. How can I read from the byte[] and separate the objects?
My code:
public byte[] toByteArray() {
byte[] bytes;
byte[] sb = start.toString().getBytes();
byte[] gb = goal.toString().getBytes();
byte[] mb = gameBoard.toString().getBytes();
bytes = new byte[sb.length + gb.length + mb.length];
System.arraycopy(sb, 0, bytes, 0, sb.length);
System.arraycopy(gb, 0, bytes, sb.length, gb.length);
System.arraycopy(mb, 0, bytes, gb.length, mb.length);
return bytes;
}

Seems like you are talking about Java not JavaScript.
I recommend you to have a look at binary serialization which I guess is what you are looking for: Saving to binary/serialization java

If you store your data like this, it will be a very difficult task to read them.
I recommend using some build-in object-to-byte[] (and back) conversions like Serializable.
Also, to store several object inside one byte[] array, have a look into ObjectOutputStream

First of all you will need an actual byte[] where stuff can be read from. There are some issues about what you are trying.
toString() usually is not fit to get some data you can reconstruct the object from. It might work with an integer, get a bit messed up with floating point, and be outright impossible with complex objects which only tell about their type and id. (as Davide comment pointed)
There are no cues about where one object starts and ends. Even worse: you might have messed up the start position of 3rd object.
The JRE has a built-in serialization.
Other people use XML or JSON when they need to interact with something else. You might even implement your own flavor of java.text.Format which is able to format and parse your objects. Pick your poison.

Java: Efficiently converting an array of longs to an array of bytes

I have an array of longs I want to write to disk. The most efficient disk I/O functions take in byte arrays, for example:
FileOutputStream.write(byte[] b, int offset, int length)
...so I want to begin by converting my long[] to byte[] (8 bytes for each long). I'm struggling to find a clean way to do this.
Direct typecasting doesn't seem allowed:
ConversionTest.java:6: inconvertible types
found : long[]
required: byte[]
byte[] byteArray = (byte[]) longArray;
^
It's easy to do the conversion by iterating over the array, for example:
ByteBuffer bytes = ByteBuffer.allocate(longArray.length * (Long.SIZE/8));
for( long l: longArray )
{
bytes.putLong( l );
}
byte[] byteArray = bytes.array();
...however that seems far less efficient than simply treating the long[] as a series of bytes.
Interestingly, when reading the file, it's easy to "cast" from byte[] to longs using Buffers:
LongBuffer longs = ByteBuffer.wrap(byteArray).asLongBuffer();
...but I can't seem to find any functionality to go the opposite direction.
I understand there are endian considerations when converting from long to byte, but I believe I've already addressed those: I'm using the Buffer framework shown above, which defaults to big endian, regardless of native byte order.

No, there is not a trivial way to convert from a long[] to a byte[].
Your best option is likely to wrap your FileOutputStream with a BufferedOutputStream and then write out the individual byte values for each long (using bitwise operators).
Another option is to create a ByteBuffer and put your long values into the ByteBuffer and then write that to a FileChannel. This handles the endianness conversion for you, but makes the buffering more complicated.

Concerning the efficiency, many details will, in fact, hardly make a difference. The hard disk is by far the slowest part involved here, and in the time that it takes to write a single byte to the disk, you could have converted thousands or even millions of bytes to longs. Every performance test here will not tell you anything about the performance of the implementation, but about the performance of the hard disk. In doubt, one should make dedicated benchmarks comparing the different conversion strategies, and comparing the different writing methods, respectively.
Assuming that the main goal is a functionality that allows a convenient conversion and does not impose an unnecessary overhead, I'd like to propose the following approach:
One can create a ByteBuffer of sufficient size, view this as a LongBuffer, use the bulk LongBuffer#put(long[]) method (which takes care of endianness conversions, of necessary, and does this as efficient as it can be), and finally, write the original ByteBuffer (which is now filled with the long values) to the file, using a FileChannel.
Following this idea, I think that this method is convenient and (most likely) rather efficient:
private static void bulkAndChannel(String fileName, long longArray[])
{
ByteBuffer bytes =
ByteBuffer.allocate(longArray.length * Long.BYTES);
bytes.order(ByteOrder.nativeOrder()).asLongBuffer().put(longArray);
try (FileOutputStream fos = new FileOutputStream(fileName))
{
fos.getChannel().write(bytes);
}
catch (IOException e)
{
e.printStackTrace();
}
}
(Of course, one could argue about whether allocating a "large" buffer is the best idea. But thanks to the convenience methods of the Buffer classes, this could easily and with reasonable effort be modified to write "chunks" of data with an appropriate size, for the case that one really wants to write a huge array and the memory overhead of creating the corresponding ByteBuffer would be prohibitively large)

OP here.
I have thought of one approach: ByteBuffer.asLongBuffer() returns an instance of ByteBufferAsLongBufferB, a class which wraps ByteBuffer in an interface for treating the data as longs while properly managing endianness. I could extend ByteBufferAsLongBufferB, and add a method to return the raw byte buffer (which is protected).
But this seems so esoteric and convoluted I feel there must be an easier way. Either that, or something in my approach is flawed.

Adding elements of an Array to an ArrayList

I have a very small threaded application, which is collecting small chunks of data in arrays (because it is sound data, and Java wants that to be an array) and trying to put it into an ArrayList for storage. All of that is effectively the front half of a producer/consumer pattern.
Problem: It doesn't seem to work.
On the producer end, I have this code:
public synchronized void run() {
// do a whole bunch of audio set-up
try {
// more audio stuff
while (true) {
if (producing) {
byte[] data = new byte[line.getBufferSize() ];
numBytesRead = line.read(data, 0, data.length);
System.out.println("Producer: Size of dat[] is " + data.length);
// Save this chunk of data.
buffer.addData(data);
}
This seems straightforward, aside from the audio stuff and bookkeeping.
In the buffer class, I have:
public class Buffer {
ArrayList list ;
public void addData(byte[] data) {
list.addAll(Arrays.asList(data));
}
This also seems straightforward.
Here is the problem: If my array is of length (say) 1024, and the elements are all there (which I've verified that they are) I would expect the size of the ArrayList to grow by 1024 every time I add data. It doesn't. It grows by 1, as though I was making either an ArrayList of ArrayLists or an ArrayList of Arrays, rather than the Arraylist of elements I desire.
I suspect I'm going to have this problem on the flip side as well, where I might have an ArrayList of tens of thousands of bytes, and want to retrieve an array of the first 1024 elements.
I cannot help but think I'm missing something very simple. Can anyone shed light on why this is not working? (Or if there is some fundamentally better way to do what I'm trying to do?)

Arrays.asList() will not perform the conversion from byte to Byte, it will return a list containing 1 element; the byte[] you pass in.
If your aim is to add Byte objects for every byte, you will have to do that yourself in a loop. Note that this will use much more memory than passing byte[]s however.
Also note that it is not garuanteed that, even if the input stream has more than enough data left, that you will read data.length bytes every time (result of buffer sizes, concurency, etc.) so you run the risk of passing a bunch of 0 bytes at the end of your buffer if you read less bytes than you asked for.

byte[] data should be Byte[]. You must use object, not primitive.

I cannot help but think I'm missing something very simple. Can anyone shed light on why this is not working? (Or if there is some fundamentally better way to do what I'm trying to do?)
Storing byte data in an ArrayList<Byte> has a lot of memory overhead, compared to a byte[] array. if you're working with a large amount of data, you may want to use a byte[] array for storage as well. Take a look at the source code for ByteArrayOutputStream - I don't know if will work for you as-is, but you might be able to create a similar sort of class that manages an expanding byte array.

How can I store byte[]'s in a Vector in Java?

I'm reading a binary file and storing each record into a byte[]. Now I'd like to collect these records into a Java Vector. (So that it can grow indefinitely.) But Vector takes Objects, not primitives (or arrays of primitives, as far as I can tell).
Is there way to "box" an array of primitives, or am I going to have to rewrite my code to turn my arrays into Arrays and my bytes into Bytes?
I tried concatenating the bytes into a String, but that failed miserable, due to String.append()'s propensity to treat my bytes as ints and convert them into String-y decimal representations!!

byte[] is-an Object (all arrays are, even primitive ones). There is nothing stopping you from adding a byte[] to a Vector.
Vector<byte[]> records = new Vector<byte[]>();
byte[] firstRecord = readRecord();
records.add(firstRecord);
Though it doesn't smell like a good design. Also, you should favour passing around List (the interface) over passing around a Vector (a concrete implementation of List).

You can add all the bytes in a byte[] to a Vector<Byte> by looping through each byte.
However, I wouldn't suggest you use Vector as it is a legacy class which was replaced in Java 1.2 (1998)
You can use an ArrayList instead, but this will use 4-16 times as much memory as the original byte[].
If you cannot use TByteArrayList, I suggest you use ByteArrayOutputStream and ByteArrayInputStream.

If you absolutely cannot convert the byte's into Bytes, then you might look into a primitive collection library such as Colt. It was written for high performance scientific stuff but it has primitive collection types that you can use.

You can do:
byte[] byteArr = new byte[]{0x41, 0x43};
List<byte[]> listBytes = Arrays.asList(byteArr); // to get a list
List<byte[]> list = new Vector<byte[]>(listBytes); // to instantiate a vector
System.out.println(Arrays.toString(list.get(0)));
Update (Based on your comments)
List<byte[]> l = new Vector<byte[]>(); // to instantiate a vector
l.add(new byte[]{0x51, 0x52});
System.out.println(Arrays.toString(l.get(0)));
OUTPUT
[81, 82]

I catch what u mean,
Yes it is impossible to put array of somethings into a Vector or even into ArrayList as one element, let me explain why the following code is completely right but we misunderstand it
Vector<byte[]> records = new Vector<byte[]>();
byte[] firstRecord = readRecord();
records.add(firstRecord);
The third line of this code doesn't put the array into the Vector but instead it puts the reference firstRecord into that Vector. Then if we change the contents of firstRecord after putting it in the vector, what happen is that we change the content of the Vector because we have two references to the same thing.