Develop Reference

Convert byte stream to byte array without extra space

I have a ByteArrayOutputStream that has large amounts of data written into, which is ultimately converted into a byte array and written to a cache:
try {
ByteArrayOutputStream baos = new ByteArrayOutputStream();
try (JsonGenerator jg = mapper.getFactory().createGenerator(baos)) {
for (Object result : results) {
jg.writeObject(result);
}
}
localCache.put(cacheKey, baos.toByteArray());
}
catch (IOException e) {
throw Throwables.propagate(e);
}
Here baos.toByteArray() creates a whole new copy of the data in memory, which I'm trying to avoid. Is there a way to convert the stream to a byte array without using the extra memory?

The internal buffer and current count are protected fields documented in the Javadoc. This means you should be OK to subclass ByteArrayOutputStream and provide a byte[] getBuffer() method to access the buffer directly. Use the existing size() method to determine how much data is present.
public class MyBAOS extends ByteArrayOutputStream
{
public MyBAOS() { super(); }
public MyBAOS(int size) { super(size); }
public byte[] getBuffer() { return buf; }
}

Decompressing byte[] using LZ4

I am using LZ4 for compressing and decompressing a string.I have tried the following way
public class CompressionDemo {
public static byte[] compressLZ4(LZ4Factory factory, String data) throws IOException {
final int decompressedLength = data.getBytes().length;
LZ4Compressor compressor = factory.fastCompressor();
int maxCompressedLength = compressor.maxCompressedLength(decompressedLength);
byte[] compressed = new byte[maxCompressedLength];
compressor.compress(data.getBytes(), 0, decompressedLength, compressed, 0, maxCompressedLength);
return compressed;
}
public static String deCompressLZ4(LZ4Factory factory, byte[] data) throws IOException {
LZ4FastDecompressor decompressor = factory.fastDecompressor();
byte[] restored = new byte[data.length];
decompressor.decompress(data,0,restored, 0,data.length);
return new String(restored);
}
public static void main(String[] args) throws IOException, DataFormatException {
String string = "kjshfhshfashfhsakjfhksjafhkjsafhkjashfkjhfjkfhhjdshfhhjdfhdsjkfhdshfdskjfhksjdfhskjdhfkjsdhfk";
LZ4Factory factory = LZ4Factory.fastestInstance();
byte[] arr = compressLZ4(factory, string);
System.out.println(arr.length);
System.out.println(deCompressLZ4(factory, arr) + "decom");
}
}
it is giving following excpetion
Exception in thread "main" net.jpountz.lz4.LZ4Exception: Error decoding offset 92 of input buffer
The problem here is that decompressing is working only if i pass the actual String byte[] length i.e
public static String deCompressLZ4(LZ4Factory factory, byte[] data) throws IOException {
LZ4FastDecompressor decompressor = factory.fastDecompressor();
byte[] restored = new byte[data.length];
decompressor.decompress(data,0,restored, 0,"kjshfhshfashfhsakjfhksjafhkjsafhkjashfkjhfjkfhhjdshfhhjdfhdsjkfhdshfdskjfhksjdfhskjdhfkjsdhfk".getBytes().length);
return new String(restored);
}
It is expecting the actual string byte[] size.
Can someone help me with this

As the compression and decompressions may happen on different machines, or the machine default character encoding is not one of the Unicode formats, one should indicate the encoding too.
For the rest it is using the actual compression and decompression lengths, and better store the size of the uncompressed data too, in plain format, so it may be extracted prior to decompressing.
public static byte[] compressLZ4(LZ4Factory factory, String data) throws IOException {
byte[] decompressed = data.getBytes(StandardCharsets.UTF_8).length;
LZ4Compressor compressor = factory.fastCompressor();
int maxCompressedLength = compressor.maxCompressedLength(decompressed.length);
byte[] compressed = new byte[4 + maxCompressedLength];
int compressedSize = compressor.compress(decompressed, 0, decompressed.length,
compressed, 4, maxCompressedLength);
ByteBuffer.wrap(compressed).putInt(decompressed.length);
return Arrays.copyOf(compressed, 0, 4 + compressedSize);
}
public static String deCompressLZ4(LZ4Factory factory, byte[] data) throws IOException {
LZ4FastDecompressor decompressor = factory.fastDecompressor();
int decrompressedLength = ByteBuffer.wrap(data).getInt();
byte[] restored = new byte[decrompressedLength];
decompressor.decompress(data, 4, restored, 0, decrompressedLength);
return new String(restored, StandardCharsets.UTF_8);
}
It should be told, that String is not suited for binary data, and your compression/decompression is for text handling only. (String contains Unicode text in the form of UTF-16 two-byte chars. Conversion to binary data always involves a conversion with the encoding of the binary data. That costs in memory, speed and possible data corruption.)

I just faced the same error on Android and resolved it based on issue below:
https://github.com/lz4/lz4-java/issues/68
In short make sure you are using the same factory for both operations (compression + decompression) and use Arrays.copyOf() as below:
byte[] compress(final byte[] data) {
LZ4Factory lz4Factory = LZ4Factory.safeInstance();
LZ4Compressor fastCompressor = lz4Factory.fastCompressor();
int maxCompressedLength = fastCompressor.maxCompressedLength(data.length);
byte[] comp = new byte[maxCompressedLength];
int compressedLength = fastCompressor.compress(data, 0, data.length, comp, 0, maxCompressedLength);
return Arrays.copyOf(comp, compressedLength);
}
byte[] decompress(final byte[] compressed) {
LZ4Factory lz4Factory = LZ4Factory.safeInstance();
LZ4SafeDecompressor decompressor = lz4Factory.safeDecompressor();
byte[] decomp = new byte[compressed.length * 4];//you might need to allocate more
decomp = decompressor.decompress(Arrays.copyOf(compressed, compressed.length), decomp.length);
return decomp;
Hope this will help.

restored byte[] length is to small, you should not use compressed data.length, instead you should use data[].length * 3 or more than 3.

I resoved like this:
public static byte[] decompress( byte[] finalCompressedArray,String ... extInfo) {
int len = finalCompressedArray.length * 3;
int i = 5;
while (i > 0) {
try {
return decompress(finalCompressedArray, len);
} catch (Exception e) {
len = len * 2;
i--;
if (LOGGER.isInfoEnabled()) {
LOGGER.info("decompress Error: extInfo ={} ", extInfo, e);
}
}
}
throw new ItemException(1, "decompress error");
}
/**
* 解压一个数组
*
* #param finalCompressedArray 压缩后的数据
* #param length 原始数据长度, 精确的长度,不能大,也不能小。
* #return
*/
private static byte[] decompress(byte[] finalCompressedArray, int length) {
byte[] desc = new byte[length ];
int decompressLen = decompressor.decompress(finalCompressedArray, desc);
byte[] result = new byte[decompressLen];
System.arraycopy(desc,0,result,0,decompressLen);
return result;
}

Why does Java serialization take up so much space?

I tried serializing instances of Byte and Integer and was shocked by how much space they took up when they were received on the other end. Why is it that it only takes 4 bytes to make an Integer, but it takes up over 10 times that many bytes upon serialization? I mean in C++, a final class has a 64 bit class identifier, plus its contents. Going off that logic, I would expect an Integer to take up 64 + 32, or 96 bits when serialized.
import java.io.*;
public class Test {
public static void main (String[] ar) throws Exception {
ByteArrayOutputStream bos = new ByteArrayOutputStream();
ObjectOutput out = new ObjectOutputStream(bos);
out.writeObject(new Integer(32));
byte[] yourBytes = bos.toByteArray();
System.out.println("length: " + yourBytes.length + " bytes");
}
}
Output:
length: 81 bytes
Update:
public static void main(String[] args) throws IOException {
{
ByteArrayOutputStream bos1 = new ByteArrayOutputStream();
ObjectOutput out1 = new ObjectOutputStream(bos1);
out1.writeObject(new Boolean(false));
byte[] yourBytes = bos1.toByteArray();
System.out.println("1 Boolean length: " + yourBytes.length);
}
ByteArrayOutputStream bos = new ByteArrayOutputStream();
ObjectOutput out = new ObjectOutputStream(bos);
for (int i = 0; i < 1000; ++i) {
out.writeObject(new Boolean(true)); // 47 bytes
}
byte[] yourBytes = bos.toByteArray();
System.out.println("1000 Booleans length: " + yourBytes.length); // 7040 bytes
final int count = 1000;
ArrayList<Boolean> listBoolean = new ArrayList<>(count);
listBoolean.addAll(Collections.nCopies(count, Boolean.TRUE));
System.out.printf("ArrayList: %d%n", sizeOf(listBoolean)); // 5096 bytes
Boolean[] arrayBoolean = new Boolean[count];
Arrays.fill(arrayBoolean, true);
System.out.printf("Boolean[]: %d%n", sizeOf(arrayBoolean)); // 5083 bytes
boolean[] array = new boolean[count];
Arrays.fill(array, true);
System.out.printf("boolean[]: %d%n", sizeOf(array)); // 1027 bytes
BitSet bits = new BitSet(count);
bits.set(0, count);
System.out.printf("BitSet: %d%n", sizeOf(bits)); // 201 bytes
}
static int sizeOf(Serializable obj) throws IOException {
ByteArrayOutputStream bytesOut = new ByteArrayOutputStream();
ObjectOutputStream objsOut = new ObjectOutputStream(bytesOut);
objsOut.writeObject(obj);
return bytesOut.toByteArray().length;
}
Output:
1 Boolean length: 47 (47 bytes per Boolean)
1000 Booleans length: 7040 (7 bytes per Boolean)
ArrayList: 5096 (5 bytes per Boolean)
Boolean[]: 5083 (5 bytes per Boolean)
boolean[]: 1027 (1 bytes per boolean)
BitSet: 201 (1/5 of 1 byte per boolean)

Though Radiodef has clarified why the size of the serialized object is huge, i would like to make another point here so we don't forget the optimization present in the underlying java's serialization algorithm (almost in all algorithms).
When you write another Integer object (or any object which is already written), you would not see similar size (i mean the size would not be 81 * 2 = 162 bytes) in this case,
ObjectOutput out = new ObjectOutputStream(bos);
out.writeObject(new Integer(32));
out.writeObject(new Integer(65));
byte[] yourBytes = bos.toByteArray();
System.out.println("length: " + yourBytes.length + " bytes");
The way it works is that, when an instance (object) of class is requested for serialization for the first time, it writes the information about the whole class. i.e including class name, it writes the name of each fields present in the class. That's why the number of bytes are more. This is basically to handle the class evaluation cases properly.
While it sends the meta data of the class for first time, it also caches the same information into the local cache called value-cache or indirection table. So next time when another instance of same class is requested for serialization (remember the cache is applicable only at stream level, or before reset() is called), it just writes only a marker (just 4 bytes of information) so that the size would be less.

java.lang.Byte and java.lang.Integer are objects, so at the very least the qualified names of their classes need to also be stored for them to be deserialized. Also the serialVersionUID needs to be stored, etc. We can easily see how this extra information inflates the size quickly.
If you want to learn about the serialization format, there is an article about it at JavaWorld: http://www.javaworld.com/article/2072752/the-java-serialization-algorithm-revealed.html.
If you're concerned about the size of serialized data, pick a format which is more compact:
import java.util.*;
import java.io.*;
class Example {
public static void main(String[] args) throws IOException {
final int count = 1000;
ArrayList<Boolean> list = new ArrayList<>(count);
list.addAll(Collections.nCopies(count, Boolean.TRUE));
System.out.printf("ArrayList: %d%n", sizeOf(list));
boolean[] array = new boolean[count];
Arrays.fill(array, true);
System.out.printf("boolean[]: %d%n", sizeOf(array));
BitSet bits = new BitSet(count);
bits.set(0, count);
System.out.printf("BitSet: %d%n", sizeOf(bits));
}
static int sizeOf(Serializable obj) throws IOException {
ByteArrayOutputStream bytesOut = new ByteArrayOutputStream();
ObjectOutputStream objsOut = new ObjectOutputStream(bytesOut);
objsOut.writeObject(obj);
return bytesOut.toByteArray().length;
}
}
ArrayList: 5096
boolean[]: 1027
BitSet: 201
Example on Ideone.

C#-Server and Java-Client: TCP Socket Communication Issues

I have wrote a server program in C# using TCPListner and a client program in Java using socket but I fail to send complex objects from Java client to C# server.
When I send a simple string from Java client to C# server by converting the string into byte array,
it always show some invalid characters at the start of message when converted back to String (using Encoding.utf8.getstring(bytesArray) ) in C# server. When I pass a String from C# to Java Client it shows invalid Header error.
Please help me if any one have any alternative or know abut any free API which can solve my problem. I have tried Java-cs-bridge to send complex objects but it always show Exception on C# server.
Here is the code:
C# Server Code - Main Function
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Net;
using System.Net.Sockets;
using System.IO;
namespace netSocketServer
{
class Program
{
static void Main(string[] args)
{
TcpListener server = new TcpListener(IPAddress.Any, 8888);
var IP = Dns.GetHostEntry(Dns.GetHostName()).AddressList.Where(ip =>ip.AddressFamily == AddressFamily.InterNetwork).Select(ip =>ip).FirstOrDefault();
server.Start();
Console.WriteLine("Server is Running at " + IP.ToString());
TcpClient clientSocket = server.AcceptTcpClient();
Console.WriteLine("Client Connected ... ");
Writer wr = new Writer(clientSocket);
wr.start();
Reader r = new Reader(clientSocket);
r.start();
Console.Read();
}
}
}
C# Server Reader Class
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading;
using System.Net.Sockets;
using System.Net;
using System.IO;
namespace netSocketServer
{
class Reader
{
TcpClient socket;
NetworkStream ns;
public Reader(TcpClient s)
{
socket = s;
ns = socket.GetStream() ;
}
public void start()
{
new Thread(
t => {
while (true)
{
try
{
int size = ns.ReadByte();
byte[] buff = new byte[size];
ns.Read(buff,0,size);
String message = Encoding.UTF8.getString(buff);
Console.WriteLine("Message from Client : {0}",message);
ns.Flush();
}
catch (Exception e)
{
Console.WriteLine("Client Disconnected : " + e.Message);
}
}
}).Start();
}
}
}
C# Server Writer Class
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Net.Sockets;
using System.Text;
using System.Threading;
namespace netSocketServer
{
class Writer
{
TcpClient socket;
NetworkStream ns;
public Writer(TcpClient s)
{
socket = s;
ns = socket.GetStream();
}
public void start()
{
new Thread(
t => {
while (true)
{
try
{
Console.Write("Please Enter your Message : ");
string Message = Console.ReadLine();
byte[] buff = Encoding.UTF8.GetBytes(Message);
byte size = (byte)Message.Length;
ns.WriteByte(size);
ns.Write(buff, 0, buff.Length);
ns.Flush();
}
catch(IOException e)
{
Console.WriteLine("Client Disconnected : " + e.Message);
socket.Close();
Thread.CurrentThread.Abort();
Console.WriteLine("Press any key to Closse Server .... ");
}
}
}).Start();
}
}
}
Java Client - Main Function
package javaclient.net;
import java.io.IOException;
import java.net.Socket;
import java.util.Scanner;
/**
*
* #author Numan
*/
public class JavaClientNet {
/**
* #param args the command line arguments
*/
public static void main(String[] args)
{
Socket socket;
Read r;
Writer wr;
Scanner s = new Scanner(System.in);
try
{
// TODO code application logic here
System.out.print("Please Enter Server IP : ");
socket = new Socket(s.next(), 8888);
wr = new Writer(socket);
wr.start();
r = new Read(socket);
r.start();
}
catch (IOException ex)
{
System.out.println(ex.getMessage());
}
}
}
Java Client - Reader Class
package javaclient.net;
import java.io.IOException;
import java.io.ObjectInputStream;
import java.net.Socket;
/**
*
* #author Numan
*/
public class Read extends Thread
{
Socket socket;
ObjectInputStream inStream;
Read(Socket s)
{
socket = s;
try {
inStream = new ObjectInputStream(socket.getInputStream());
}
catch (IOException ex)
{
System.out.println(ex.getMessage());
}
}
#Override
public void run()
{
while(true)
{
try
{
String str;
byte size = inStream.readByte();
byte[] buf = new byte[size];
inStream.read(buf);
str = new String(buf);
System.out.println("Message form Server : "+str);
}
catch(IOException e)
{
System.out.println(e.getMessage());
Thread.currentThread().stop();
}
}
}
}
Java Client - Writer Class
package javaclient.net;
import java.io.IOException;
import java.io.ObjectOutputStream;
import java.net.Socket;
import java.util.Scanner;
import javacsconverter.core.tobyte.ToByteConvertHelper;
/**
*
* #author Numan
*/
public class Writer extends Thread
{
Socket socket;
ObjectOutputStream outStream;
Scanner scanner = new Scanner(System.in);
Writer(Socket s)
{
socket =s;
try
{
outStream = new ObjectOutputStream(socket.getOutputStream());
}
catch (IOException ex)
{
System.out.println(ex.getMessage());
}
}
#Override
public void run()
{
while(true)
{
try
{
System.out.print("Please Enter Your Message : ");
String str = scanner.nextLine();
byte[] buff = str.getBytes();
outStream.write(buff);
outStream.flush();
}
catch (IOException ex)
{
System.out.println(ex.getMessage());
}
}
}
}

General notes
Please do not abort the threads (both C# and Java).
C# Server
Program class
There is a data race because the static Console class is used by multiple threads:
Main thread: the Program.Main() method calls the Console.Read() method;
Worker thread: the Writer.start() method calls the Console.ReadLine() method.
Please consider replacing the Console.Read() method call of the Program.Main() method with something different, for example, Thread.Sleep(Timeout.Infinite).
Reader class
There is a mistake — the Stream.Read() method is not guaranteed to read the array of the specified "size" at once (one call), the return value should be used to determine the actual number of bytes read. Let's see the original implementation:
int size = ns.ReadByte();
byte[] buff = new byte[size];
// The Stream.Read() method does not guarantee to read the **whole array** "at once".
// Please use the return value of the method.
ns.Read(buff, 0, size);
String message = Encoding.UTF8.GetString(buff);
Corrected version:
/// <summary>
/// Helper method to read the specified byte array (number of bytes to read is the size of the array).
/// </summary>
/// <param name="inputStream">Input stream.</param>
/// <param name="buffer">The output buffer.</param>
private static void ReadFully(Stream inputStream, byte[] buffer)
{
if (inputStream == null)
{
throw new ArgumentNullException("inputStream");
}
if (buffer == null)
{
throw new ArgumentNullException("buffer");
}
int totalBytesRead = 0;
int bytesLeft = buffer.Length;
if (bytesLeft <= 0)
{
throw new ArgumentException("There is nothing to read for the specified buffer", "buffer");
}
while (totalBytesRead < buffer.Length)
{
var bytesRead = inputStream.Read(buffer, totalBytesRead, bytesLeft);
if (bytesRead > 0)
{
totalBytesRead += bytesRead;
bytesLeft -= bytesRead;
}
else
{
throw new InvalidOperationException("Input stream reaches the end before reading all the bytes");
}
}
}
public void start()
{
...
int size = ns.ReadByte();
byte[] buff = new byte[size];
ReadFully(ns, buff);
using (var memoryStream = new MemoryStream(buff, false))
{
// The StreamReader class is used to extract the UTF-8 string which is encoded with the byte order mark (BOM).
using (var streamReader = new StreamReader(memoryStream, Encoding.UTF8))
{
string message = streamReader.ReadToEnd();
Console.WriteLine("Message from Client: {0}", message);
}
}
...
}
Writer class
First of all, to describe and determine byte the order of the text stream consider including the byte order mark (BOM) for each message (for example).
Also, there is a mistake — wrong "buffer length" value is sent. Let's see the original implementation:
string Message = Console.ReadLine();
byte[] buff = Encoding.UTF8.GetBytes(Message);
// Problem: instead of the length of the string, the size of byte array must be used
// because the UTF-8 encoding is used: generally, string length != "encoded number of bytes".
byte size = (byte)Message.Length;
ns.WriteByte(size);
ns.Write(buff, 0, buff.Length);
ns.Flush();
Corrected version:
// UTF-8 with BOM.
var encoding = new UTF8Encoding(true);
// Buffer encoded as UTF-8 with BOM.
byte[] buff = encoding.GetPreamble()
.Concat(encoding.GetBytes(message))
.ToArray();
// Size of the encoded buffer.
byte size = Convert.ToByte(buff.Length);
ns.WriteByte(size);
ns.Write(buff, 0, buff.Length);
ns.Flush();
Alternative corrected version — the StreamWriter class is used to encode the string as UTF-8 with the byte order mark (BOM):
string message = Console.ReadLine();
using (var memoryStream = new MemoryStream())
{
using (var streamWriter = new StreamWriter(memoryStream, Encoding.UTF8, 1024, true))
{
streamWriter.Write(message);
}
memoryStream.Flush();
byte size = Convert.ToByte(memoryStream.Length);
ns.WriteByte(size);
memoryStream.Seek(0, SeekOrigin.Begin);
memoryStream.CopyTo(ns);
ns.Flush();
}
Java Client
Read class
First, please consider using DataInputStream class because the following statement is not true according to the question:
An ObjectInputStream deserializes primitive data and objects previously written using an ObjectOutputStream.
-- java.io.ObjectInputStream class, Java™ Platform
Standard Ed. 7.
The instantiation of the stream is almost the same:
inStream = new DataInputStream(socket.getInputStream());
Second, there is a mistake — reading the byte array, but ignoring the return value (actual number of bytes read):
String str;
byte size = inStream.readByte();
byte[] buf = new byte[size];
// The InputStream.read() method does not guarantee to read the **whole array** "at once".
// Please use the return value of the method.
inStream.read(buf);
str = new String(buf);
Third, as stated above, the byte order mark (BOM) is included.
Corrected version:
// Note: inStream must be an instance of DataInputStream class.
byte size = inStream.readByte();
byte[] buf = new byte[size];
// The DataInputStream.readFully() method reads the number of bytes required to fill the buffer entirely.
inStream.readFully(buf);
// Create in-memory stream for the byte array and read the UTF-8 string.
try (ByteArrayInputStream inputStream = new ByteArrayInputStream(buf);
// The BOMInputStream class belongs to Apache Commons IO library.
BOMInputStream bomInputStream = new BOMInputStream(inputStream, false)) {
String charsetName = bomInputStream.getBOMCharsetName();
// The IOUtils class belongs to Apache Commons IO library.
String message = IOUtils.toString(bomInputStream, charsetName);
System.out.println("Message form Server : " + message);
}
Writer class
There is a mistake — the encoding is not specified explicitly. Let's see the original implementation:
String str = scanner.nextLine();
byte[] buff = str.getBytes();
Corrected version:
String str = scanner.nextLine();
byte[] byteOrderMarkBytes = ByteOrderMark.UTF_8.getBytes();
byte[] stringBytes = str.getBytes(StandardCharsets.UTF_8);
// The ArrayUtils.addAll() method belongs to Apache Commons Lang library.
byte[] buff = ArrayUtils.addAll(byteOrderMarkBytes, stringBytes);
outStream.writeByte(buff.length);
outStream.write(buff);
outStream.flush();
Alternative corrected version — the ByteArrayOutputStream class is used to concatenate the arrays:
String str = scanner.nextLine();
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
byte[] byteOrderMarkBytes = ByteOrderMark.UTF_8.getBytes();
byteArrayOutputStream.write(byteOrderMarkBytes);
byte[] stringBytes = str.getBytes(StandardCharsets.UTF_8);
byteArrayOutputStream.write(stringBytes);
byteArrayOutputStream.flush();
byte[] buff = byteArrayOutputStream.toByteArray();
outStream.writeByte(buff.length);
outStream.write(buff);
outStream.flush();
Hope this helps!

Blowfish code should be equivalent but is not

We have a class which wraps BouncyCastle (actually SpongyCastle for Android) Blowfish to encrypt data to stream:
public class BlowfishOutputStream extends OutputStream
{
private final OutputStream os;
private final PaddedBufferedBlockCipher bufferedCipher;
Our original code encrypted a whole byte array before writing to the output stream in a single operation
public void write(byte[] raw, int offset, int length) throws IOException
{
byte[] out = new byte[bufferedCipher.getOutputSize(length)];
int result = this.bufferedCipher.processBytes(raw, 0, length, out, 0);
if (result > 0)
{
this.os.write(out, 0, result);
}
}
When sending images (ie large amount of data at once) it results in two copies being retained in memory at once.
The following code is meant to be equivalent but is not, and I do not know why. I can verify that data is being sent (sum of c2 is equivalent to the length) but an intermediate process when it is received on our server discards the image before we get to see what arrives. All I know at this stage is that when the initial code is used, the response is received and the included images can be extracted, when the replacement code is used the response is received (and accepted) but images do not appear to be extracted.
public void write(byte[] raw, int offset, int length) throws IOException
{
// write to the output stream as we encrypt, not all at once.
final byte[] inBuffer = new byte[Constants.ByteBufferSize];
final byte[] outBuffer = new byte[Constants.ByteBufferSize];
ByteArrayInputStream bis = new ByteArrayInputStream(raw);
// read into inBuffer, encrypt into outBuffer and write to output stream
for (int len; (len = bis.read(inBuffer)) != -1;)
{
int c2 = this.bufferedCipher.processBytes(inBuffer, 0, len, outBuffer, 0);
this.os.write(outBuffer, 0, c2);
}
}
Note that the issue is not due to a missing call to doFinal, as this is called when the stream is closed.
public void close() throws IOException
{
byte[] out = new byte[bufferedCipher.getOutputSize(0)];
int result = this.bufferedCipher.doFinal(out, 0);
if (result > 0)
{
this.os.write(out, 0, result);
}
*nb try/catch omitted*
}

Confirmed, although ironically the issue was not with the images but in previous data, but that data was writing the complete raw byte array and not just the range specified. The equivalent code for encrypting the byte array on the fly is:
#Override
public void write(byte[] raw, int offset, int length) throws IOException
{
// write to the stream as we encrypt, not all at once.
final byte[] inBuffer = new byte[Constants.ByteBufferSize];
final byte[] outBuffer = new byte[Constants.ByteBufferSize];
int readStart = offset;
// read into inBuffer, encrypt into outBuffer and write to output stream
while(readStart<length)
{
int readAmount = Math.min(length-readStart, inBuffer.length);
System.arraycopy(raw, readStart, inBuffer, 0, readAmount);
readStart+=readAmount;
int c2 = this.bufferedCipher.processBytes(inBuffer, 0, readAmount, outBuffer, 0);
this.os.write(outBuffer, 0, c2);
}
}