Develop Reference

InputStream to byte array

I have this code:
private static void flow(InputStream is, OutputStream os, byte[] buf)
throws IOException {
int numRead;
while ((numRead = is.read(buf)) >= 0) {
os.write(buf, 0, numRead);
}
}
Which basically streams from is to the OutputStream provided.
My goal is to cache the is when the flow has completed.
As such I have:
cacheService.cache(key, bytes);

The solution to this is to implement a Caching output stream:
public class CachingOutputStream extends OutputStream {
private final OutputStream os;
private final ByteArrayOutputStream baos = new ByteArrayOutputStream();
public CachingOutputStream(OutputStream os) {
this.os = os;
}
public void write(int b) throws IOException {
try {
os.write(b);
baos.write(b);
} catch (Exception e) {
if(e instanceof IOException) {
throw e;
} else {
e.printStackTrace();
}
}
}
public byte[] getCache() {
return baos.toByteArray();
}
public void close() throws IOException {
os.close();
}
public void flush() throws IOException {
os.flush();
}
}
And do this:
final CachingOutputStream cachingOutputStream = new CachingOutputStream(outputStream);
flow(inputStream, cachingOutputStream, buff);
cached = cachingOutputStream.getCache();
if(cached != null) {
cacheService.put(cacheKey, cached);
}

Using org.apache.poi.util.IOUtils,
IOUtils.toByteArray(inputStream);

Transfer a huge file using StreamingOutput without incurring the wrath of heapSpace

I have been trying to stream|transfer a huge file available in the local file-system over restapi using streamingoutput. I keep running into heapSpace error. Can anyone help me figure out what I am doing wrong? As per my understanding, streamingoutput shouldn't keep the file in memory.
Please find the code below:
public Response getBulkBillDownload(#QueryParam("requestID") String requestID,
#QueryParam("zipFileName") String zipFileName) throws RestException {
StreamingOutput stream = null;
try {
File file = null;
Optional<File> document = getCorporatePaymentManager().getBulkBillDownloadResponse(requestID, zipFileName);
if (document.isPresent()) {
file = document.get();
} else {
throw new RestException("File not found");
}
final FileInputStream fStream = new FileInputStream(file);
// register stream to Response and it will callback with server OutputStream
stream = new StreamingOutput() {
#Override
public void write(OutputStream output) throws IOException, WebApplicationException {
pipe(fStream, output);
}
};
} catch (Exception e) {
handleException(e);
}
return Response.status(200).entity(stream).header("Content-Disposition", "attachment; filename=" + zipFileName)
.build();
}
private void pipe(InputStream is, OutputStream os) throws IOException {
byte[] buf=new byte[1024];
int bytesread = 0, bytesBuffered = 0;
while( (bytesread = is.read( buf )) > -1 ) {
os.write( buf, 0, bytesread );
bytesBuffered += bytesread;
if (bytesBuffered > 1024 * 1024) { //flush after 1MB
bytesBuffered = 0;
os.flush();
}
}
os.close();
}

Can I only close fileoutputstream but the channel is living after I send file by Netty?

Few days ago, I struggled with how to access file sent by NettyClient without killing NettyServer. I got solution on StackOverFlow and the detail of question is here. The solution is that the client close channel after sending the file, and the server close the fileoutputstream in channelInactive method. The main code is below.
ClientHandler
public class FileClientHandler extends ChannelInboundHandlerAdapter {
private int readLength = 128;
#Override
public void channelActive(ChannelHandlerContext ctx) throws Exception {
sendFile(ctx.channel());
}
private void sendFile(Channel channel) throws IOException {
File file = new File("C:\\Users\\xxx\\Desktop\\1.png");
FileInputStream fis = new FileInputStream(file);
BufferedInputStream bis = new BufferedInputStream(fis);
ChannelFuture lastFuture = null;
for (;;) {
byte[] bytes = new byte[readLength];
int readNum = bis.read(bytes, 0, readLength);
if (readNum == -1) { // The end of the stream has been reached
bis.close();
fis.close();
lastFuture = sendToServer(bytes, channel, 0);
if(lastFuture == null) { // When our file is 0 bytes long, this is true
channel.close();
} else {
lastFuture.addListener(ChannelFutureListener.CLOSE);
}
return;
}
lastFuture = sendToServer(bytes, channel, readNum);
}
}
private ChannelFuture sendToServer(byte[] bytes, Channel channel, int length)
throws IOException {
return channel.writeAndFlush(Unpooled.copiedBuffer(bytes, 0, length));
}
}
ServerHandler
public class FileServerHandler extends ChannelInboundHandlerAdapter {
private File file = new File("C:\\Users\\xxx\\Desktop\\2.png");
private FileOutputStream fos;
public FileServerHandler() {
try {
if (!file.exists()) {
file.createNewFile();
} else {
file.delete();
file.createNewFile();
}
fos = new FileOutputStream(file);
} catch (IOException e) {
e.printStackTrace();
}
}
#Override
public void channelInactive(ChannelHandlerContext ctx) {
System.out.println("I want to close fileoutputstream!");
try {
fos.close();
} catch (IOException e) {
e.printStackTrace();
}
}
#Override
public void channelRead(ChannelHandlerContext ctx, Object msg)
throws Exception {
ByteBuf buf = (ByteBuf) msg;
try {
buf.readBytes(fos, buf.readableBytes());
} catch (Exception e) {
e.printStackTrace();
} finally {
buf.release(); // Should always be done, even if writing to the file fails
}
}
}
If now I need to send 10 thousands pictures but every picture is small like 1KB. I have to close and then establish channel frequently. It is a thing wasting many resources. How can I only close fileoutputstream but the channel is alive?

This is just an idea, and I have not tested it, but rather than sending each file in its own connection, you could start a stream where you send:
The number of files to be sent (once)
The file info and content (for each file)
The file size
The file name size
The file name
The file content (bytes)
The client would look something like this:
public void sendFiles(Channel channel, File...files) {
ByteBufAllocator allocator = PooledByteBufAllocator.DEFAULT;
int fileCount = files.length;
// Send the file count
channel.write(allocator.buffer(4).writeInt(fileCount));
// For each file
Arrays.stream(files).forEach(f -> {
try {
// Get the file content
byte[] content = Files.readAllBytes(f.toPath());
byte[] fileName = f.getAbsolutePath().getBytes(UTF8);
// Write the content size, filename and the content
channel.write(allocator.buffer(4 + content.length + fileName.length)
.writeInt(content.length)
.writeInt(fileName.length)
.writeBytes(fileName)
.writeBytes(content)
);
} catch (IOException e) {
throw new RuntimeException(e); // perhaps do something better here.
}
});
// Flush the channel
channel.flush();
}
On the server side, you would need a slightly more sophisticated channel handler. I was thinking of a replaying decoder. (Example here)
In that example, the decoder will read all the files and then forward to the next handler which would receive a list of Upload instances, but you could send each upload up the pipeline after each received file so you don't allocate as much memory. But the intent is to send all your files in one stream rather than having to connect/disconnect for each file.

Android: Bluetooth - How to read incoming data

I have successfully paired and connected with a Bluetooth device. I am now interested in receiving all data being transferred between the 2 and seeing whats what.
I am getting the input stream from the socket and attempting to read it. I return this and just log it.
The only way I know of doing this from what I have read is just do read with a byte buffer to return an int. However I should have loads of data coming through. How can I continually read out data being transferred, and also format as bytes rather than an int.
Thanks.
Full code below:
public class ConnectThread {
private BluetoothSocketWrapper bluetoothSocket;
private BluetoothDevice device;
private boolean secure;
private BluetoothAdapter adapter;
private List<UUID> uuidCandidates;
private int candidate;
/**
* #param device the device
* #param secure if connection should be done via a secure socket
* #param adapter the Android BT adapter
* #param uuidCandidates a list of UUIDs. if null or empty, the Serial PP id is used
*/
public ConnectThread(BluetoothDevice device, boolean secure, BluetoothAdapter adapter,
List<UUID> uuidCandidates) {
this.device = device;
this.secure = secure;
this.adapter = adapter;
this.uuidCandidates = uuidCandidates;
if (this.uuidCandidates == null || this.uuidCandidates.isEmpty()) {
this.uuidCandidates = new ArrayList<UUID>();
this.uuidCandidates.add(UUID.fromString("00001101-0000-1000-8000-00805F9B34FB"));
}
}
public BluetoothSocketWrapper connect() throws IOException {
boolean success = false;
while (selectSocket()) {
adapter.cancelDiscovery();
try {
bluetoothSocket.connect();
success = true;
break;
} catch (IOException e) {
//try the fallback
try {
bluetoothSocket = new FallbackBluetoothSocket(bluetoothSocket.getUnderlyingSocket());
Thread.sleep(500);
bluetoothSocket.connect();
success = true;
break;
} catch (FallbackException e1) {
Log.w("BT", "Could not initialize FallbackBluetoothSocket classes.", e);
} catch (InterruptedException e1) {
Log.w("BT", e1.getMessage(), e1);
} catch (IOException e1) {
Log.w("BT", "Fallback failed. Cancelling.", e1);
}
}
}
if (!success) {
throw new IOException("Could not connect to device: "+ device.getAddress());
}
receiveData(bluetoothSocket);
return bluetoothSocket;
}
private boolean selectSocket() throws IOException {
if (candidate >= uuidCandidates.size()) {
return false;
}
BluetoothSocket tmp;
UUID uuid = uuidCandidates.get(candidate++);
Log.i("BT", "Attempting to connect to Protocol: "+ uuid);
if (secure) {
tmp = device.createRfcommSocketToServiceRecord(uuid);
} else {
tmp = device.createInsecureRfcommSocketToServiceRecord(uuid);
}
bluetoothSocket = new NativeBluetoothSocket(tmp);
return true;
}
public static interface BluetoothSocketWrapper {
InputStream getInputStream() throws IOException;
OutputStream getOutputStream() throws IOException;
String getRemoteDeviceName();
void connect() throws IOException;
String getRemoteDeviceAddress();
void close() throws IOException;
BluetoothSocket getUnderlyingSocket();
}
public static class NativeBluetoothSocket implements BluetoothSocketWrapper {
private BluetoothSocket socket;
public NativeBluetoothSocket(BluetoothSocket tmp) {
this.socket = tmp;
}
#Override
public InputStream getInputStream() throws IOException {
return socket.getInputStream();
}
#Override
public OutputStream getOutputStream() throws IOException {
return socket.getOutputStream();
}
#Override
public String getRemoteDeviceName() {
return socket.getRemoteDevice().getName();
}
#Override
public void connect() throws IOException {
socket.connect();
}
#Override
public String getRemoteDeviceAddress() {
return socket.getRemoteDevice().getAddress();
}
#Override
public void close() throws IOException {
socket.close();
}
#Override
public BluetoothSocket getUnderlyingSocket() {
return socket;
}
}
public class FallbackBluetoothSocket extends NativeBluetoothSocket {
private BluetoothSocket fallbackSocket;
public FallbackBluetoothSocket(BluetoothSocket tmp) throws FallbackException {
super(tmp);
try
{
Class<?> clazz = tmp.getRemoteDevice().getClass();
Class<?>[] paramTypes = new Class<?>[] {Integer.TYPE};
Method m = clazz.getMethod("createRfcommSocket", paramTypes);
Object[] params = new Object[] {Integer.valueOf(1)};
fallbackSocket = (BluetoothSocket) m.invoke(tmp.getRemoteDevice(), params);
}
catch (Exception e)
{
throw new FallbackException(e);
}
}
#Override
public InputStream getInputStream() throws IOException {
return fallbackSocket.getInputStream();
}
#Override
public OutputStream getOutputStream() throws IOException {
return fallbackSocket.getOutputStream();
}
#Override
public void connect() throws IOException {
fallbackSocket.connect();
}
#Override
public void close() throws IOException {
fallbackSocket.close();
}
}
public static class FallbackException extends Exception {
/**
*
*/
private static final long serialVersionUID = 1L;
public FallbackException(Exception e) {
super(e);
}
}
public void sendData(BluetoothSocketWrapper socket, int data) throws IOException{
ByteArrayOutputStream output = new ByteArrayOutputStream(4);
output.write(data);
OutputStream outputStream = socket.getOutputStream();
outputStream.write(output.toByteArray());
}
public int receiveData(BluetoothSocketWrapper socket) throws IOException{
byte[] buffer = new byte[256];
ByteArrayInputStream input = new ByteArrayInputStream(buffer);
InputStream inputStream = socket.getInputStream();
inputStream.read(buffer);
return input.read();
}
}

In the first place, stop using ByteArrayInputStream and ByteArrayOutputStream for more control.
If the socket sends/receives text, do this.
TO SEND:
String text = "My message";
socketOutputStream.write(text.getBytes());
TO RECEIVE:
int length = socketInputStream.read(buffer);
String text = new String(buffer, 0, length);
The socketOutputStream should be your bluetoothSocket.getOutputStream().
If the socket sends/receives large loads of data, the key is the while loop to prevent out of memory exceptions. The data will be read by chunks of (for example every 4KB of buffer size), when you choose the buffer size, consider the heap size, if you're live-streaming media, consider latency and quality too.
TO SEND:
int length;
while ((length = largeDataInputStream.read(buffer)) != -1) {
socketOutputStream.write(buffer, 0, length);
}
TO RECEIVE:
int length;
//socketInputStream never returns -1 unless connection is broken
while ((length = socketInputStream.read(buffer)) != -1) {
largeDataOutputStream.write(buffer, 0, length);
if (progress >= dataSize) {
break; //Break loop if progress reaches the limit
}
}
FAQ:
How do I get the size of receiving data? You'll have to make your own implementation to notify remote device to get ready to receive data (including file size), this will require at least a dual-socket connection (2 sockets, 1 device), for example, 1 socket for text fragments and custom commands, and 1 socket for large data, like files or streaming.
What are largeDataInputStream and largeDataOutputStream? These streams can be normal I/O streams, FileInputStream/FileOutputStream or etc.
Why the while loop for BluetoothSocket never finishes? The socket input is continuously receiving data, and the read() methods blocks itself until data is detected. To prevent blocking the code in that line, while loop must be broken.
NOTE: This answer could need an edit. I'm not a native English speaker.

Following the above advice, I am now using this code to retrieve data.
public void receiveData(BluetoothSocketWrapper socket) throws IOException{
InputStream socketInputStream = socket.getInputStream();
byte[] buffer = new byte[256];
int bytes;
// Keep looping to listen for received messages
while (true) {
try {
bytes = socketInputStream.read(buffer); //read bytes from input buffer
String readMessage = new String(buffer, 0, bytes);
// Send the obtained bytes to the UI Activity via handler
Log.i("logging", readMessage + "");
} catch (IOException e) {
break;
}
}
}

Upolading a file from server to client with servlet

I'm trying to write a servlet, that can upload a file from client to server and download a file from server to client from specific location to specific location. But two problems stopped me:
1. When uploading a file from client to server, how to tell to the server where to store the file?
2. (and more important) How to do the downloading from server to client part?
Here is the code so far:
import java.io.FileOutputStream;
import java.io.ObjectInputStream;
import java.io.ObjectOutputStream;
import java.net.ServerSocket;
import java.net.Socket;
public class Server extends Thread {
public static final int PORT = 3333;
public static final int BUFFER_SIZE = 100;
#Override
public void run() {
try {
ServerSocket serverSocket = new ServerSocket(PORT);
while (true) {
Socket s = serverSocket.accept();
saveFile(s);
}
} catch (Exception e) {
e.printStackTrace();
}
}
private void saveFile(Socket socket) throws Exception {
ObjectOutputStream oos = new ObjectOutputStream(socket.getOutputStream());
ObjectInputStream ois = new ObjectInputStream(socket.getInputStream());
FileOutputStream fos = null;
byte [] buffer = new byte[BUFFER_SIZE];
Object o = ois.readObject();
if (o instanceof String) {
fos = new FileOutputStream(o.toString());
} else {
throwException(null);
}
Integer bytesRead = 0;
do {
o = ois.readObject();
if (!(o instanceof Integer)) {
throwException(null);
}
bytesRead = (Integer)o;
o = ois.readObject();
if (!(o instanceof byte[])) {
throwException(null);
}
buffer = (byte[]) o;
fos.write(buffer, 0, bytesRead);
} while (bytesRead == BUFFER_SIZE);
fos.close();
ois.close();
oos.close();
}
public static void throwException(String message) throws Exception {
throw new Exception(message);
}
public static void main(String[] args) {
new Server().start();
}
}
package com.filetransfer.web;
import java.io.*;
import java.net.Socket;
import java.util.Arrays;
import javax.servlet.*;
import javax.servlet.http.*;
public class FileTransfer extends HttpServlet {
private static final long serialVersionUID = 1L;
public static final int PORT = 3333;
public static final int BUFFER_SIZE = 100;
public static final String HOST = "localhost";
public void doGet(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
String action = request.getParameter("option");
System.out.println(action);
if ("upload".equals(action)) {
uploadFile(request);
} else if ("download".equals(action)) {
downloadFile(request, response);
}
}
public void doPost(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
doGet(request, response);
}
public void reportError(HttpServletResponse response, String message) throws IOException {
response.sendError(HttpServletResponse.SC_NOT_FOUND, message);
}
public void uploadFile(HttpServletRequest request) {
String fileLocation = request.getParameter("localfile");
File file = new File(fileLocation);
Socket socket;
try {
socket = new Socket(HOST, PORT);
ObjectInputStream ois = new ObjectInputStream(socket.getInputStream());
ObjectOutputStream oos = new ObjectOutputStream(socket.getOutputStream());
oos.writeObject(file.getName());
FileInputStream fis = new FileInputStream(file);
byte [] buffer = new byte[BUFFER_SIZE];
Integer bytesRead = 0;
while ((bytesRead = fis.read(buffer)) > 0) {
oos.writeObject(bytesRead);
oos.writeObject(Arrays.copyOf(buffer, buffer.length));
}
oos.close();
ois.close();
} catch (Exception e) {
e.printStackTrace();
}
}
public void downloadFile(HttpServletRequest request, HttpServletResponse response) {
File file = new File(request.getParameter("remotefile"));
Socket socket;
InputStream inputStream = null;
OutputStream outputStream = null;
try {
socket = new Socket(HOST, PORT);
response.setContentLength((int)file.length());
outputStream = response.getOutputStream();
inputStream = new BufferedInputStream(new FileInputStream(file));
int nextByte;
while ((nextByte = inputStream.read()) != -1) {
outputStream.write(nextByte);
}
} catch (Exception e) {
e.printStackTrace();
} finally {
if (inputStream != null) {
try {
inputStream.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
}

Since you're using a HTTP servlet, I strongly suggest to use a HTTP client. Don't pass around proprietary and self-invented request/response formats around, but just construct requests and responses according the HTTP protocol. When you adhere a certain transport standard, then there is no doubt that there are several API's and tools available which may ease the job.
In the client side, you can use java.net.URLConnection or Apache HttpClient for this. Sending files over HTTP from client to server (uploading) usually require a multipart/form-data request encoding. Sending files over HTTP from server to client (downloading) usually require just a correct Content-Type header and the entire file as response body.
At the bottom of this answer you can find an example how to upload files by URLConnection (and in this answer an example with Apache HttpClient 4). In this answer you can find an example how to process the uploaded file in servlet. Saving the uploaded file is easy: just write the obtained InputStream to some FileOutputStream. In this article you can find an example how to send the file for download. Saving the downloaded file is also easy, just write URLConnection#getInputStream() to some FileOutputStream.