Develop Reference

Async response streaming with Apache Async Http client

I am using apache async http client to stream objects from azure storage.
I only need to return the HttpResponse object which has the stream associated. My clients will actually have to read from that stream to store the file locally.
So Apache Async clients use a BasicAsyncResponseConsumer which actually buffers the entire file in local memory before calling the completed callback.
I am trying to create my own implementation of AbstractAsyncResponseConsumer so that I can stream the response body instead of actually storing it first but have been unsuccessful to do so till now.
Here is the bare bones cosumer class for reference ->
public class MyConsumer extends` AbstractAsyncResponseConsumer<HttpResponse> {
#Override
protected void onResponseReceived(HttpResponse response) throws HttpException, IOException {
}
#Override
protected void onContentReceived(ContentDecoder decoder, IOControl ioctrl) throws IOException {
}
#Override
protected void onEntityEnclosed(HttpEntity entity, ContentType contentType) throws IOException {
}
#Override
protected HttpResponse buildResult(HttpContext context) throws Exception {
return null;
}
#Override
protected void releaseResources() {
}
}
And here is the code to send the request and return the response ->
public void getFile(HttpRequestBase request) {
MyConsumer myConsumer = new MyConsumer();
HttpAsyncRequestProducer producer =
HttpAsyncMethods.create(request);
CompletableFuture<HttpResponse> future = new CompletableFuture<>();
return Future<HttpResponse> responseFuture =
httpclient.execute(producer,myConsumer,
new FutureCallback<HttpResponse>() {
#Override
public void completed(HttpResponse result) {
//This is called only when all the response body has been read
//future.complete(Result)
}
#Override
public void failed(Exception ex) {
}
#Override
public void cancelled() {
}
});
return future;
}
I will be returning a CompletableFuture of the HttpResponse object to my clients.
They shouldnt be waiting for my http client to read all the response body first in local buffer.
They ideally should start copying directly from the stream provided in the response object.
What should I add inmy implementation of the consumer to get the desired result ?

I don't know if you still have this problem, but if what you want is an InputStream that actually streams data, then you'll want to use the blocking version of Apache HttpClient.
Java's built-in InputStream and OutputStream are inherently blocking, so returning a CompletableFuture of InputStream essentially defeats the purpose. BasicAsyncResponseConsumer buffering the entire response in memory is actually the right thing to do, because that's the only way of making it truly non-blocking.
Another option you can take a look at is HttpAsyncMethods.createZeroCopyConsumer. What it does is that it stores the content to a file in a completely non-blocking way.
Here's an example:
try (CloseableHttpAsyncClient client = HttpAsyncClients.createDefault()) {
client.start();
final CompletableFuture<HttpResponse> cf = new CompletableFuture<>();
client.execute(
HttpAsyncMethods.createGet("https://example.com"),
HttpAsyncMethods.createZeroCopyConsumer(new File("foo.html")),
new FutureCallback<HttpResponse>() {
#Override
public void completed(HttpResponse result) {
cf.complete(result);
}
#Override
public void failed(Exception ex) {
cf.completeExceptionally(ex);
}
#Override
public void cancelled() {
cf.cancel(true);
}
});
// When cf completes, the file will be ready.
// The InputStream inside the HttpResponse will be the FileInputStream of the created file.
}

#FormParameter data becomes null after reading and setting the same data in ContainerRequestContext entityStream

I have implemented filter and I have called getEntityStream of ContainerRequestContext and set the exact value back by using setEntitystream. If i use this filter then #FormParameter data becomes null and if i don't use filter then everything will be fine (as I am not calling getEntityStream) and i have to use filter to capture request data.
Note: I am getting form params from MultivaluedMap formParams but not from #FormParameter.
Environment :- Rest Easy API with Jboss Wildfly 8 server.
#Provider
#Priority(Priorities.LOGGING)
public class CustomLoggingFilter implements ContainerRequestFilter, ContainerResponseFilter{
final static Logger log = Logger.getLogger(CustomLoggingFilter.class);
#Context
private ResourceInfo resourceInfo;
#Override
public void filter(ContainerRequestContext requestContext)
throws IOException {
MDC.put("start-time", String.valueOf(System.currentTimeMillis()));
String entityParameter = readEntityStream(requestContext);
log.info("Entity Parameter :"+entityParameter);
}
private String readEntityStream(ContainerRequestContext requestContext){
ByteArrayOutputStream outStream = new ByteArrayOutputStream();
final InputStream inputStream = requestContext.getEntityStream();
final StringBuilder builder = new StringBuilder();
int read=0;
final byte[] data = new byte[4096];
try {
while ((read = inputStream.read(data)) != -1) {
outStream.write(data, 0, read);
}
} catch (IOException e) {
e.printStackTrace();
}
byte[] requestEntity = outStream.toByteArray();
if (requestEntity.length == 0) {
builder.append("");
} else {
builder.append(new String(requestEntity));
}
requestContext.setEntityStream(new ByteArrayInputStream(requestEntity) );
return builder.toString();
}
return null;
}
}
class customResource
{
//// This code is not working
#POST
#Path("voiceCallBack")
#ApiOperation(value = "Voice call back from Twilio")
public void voiceCallback(#FormParam("param") String param)
{
log.info("param:" + param);
}
// This code is working
#POST
#Path("voiceCallBackMap")
#ApiOperation(value = "Voice call back from Twilio")
public void voiceCallbackMap(final MultivaluedMap<String, String> formParams)
{
String param = formParams.getFirst("param");
}
}
please suggest me solution & Thanks in Advance.

I found during run time that instance of the entity stream (from http request) is of type org.apache.catalina.connector.CoyoteInputStream (I am using jboss-as-7.1.1.Final). But we are setting entity stream with the instance of java.io.ByteArrayInputStream. So Resteasy is unable to bind individual formparmeters.
There are two solutions for this you can use any one of them :
Use this approach How to read JBoss Resteasy's servlet request twice while maintaing #FormParam binding?
Get form parameters like this:
#POST
#Path("voiceCallBackMap")
#ApiOperation(value = "Voice call back from Twilio")
public void voiceCallbackMap(final MultivaluedMap<String, String> formParams)
{
String param = formParams.getFirst("param");
}

How to zip- compress HTTP request with Spring RestTemplate?

How to gzip HTTP request, created by org.springframework.web.client.RestTemplate?
I am using Spring 4.2.6 with Spring Boot 1.3.5 (Java SE, not Android or Javascript in the web browser).
I am making some really big POST requests, and I want request body to be compressed.

I propose two solutions, one simpler without streaming and one that supports streaming.
If you don't require streaming, use a custom ClientHttpRequestInterceptor, a Spring feature.
RestTemplate rt = new RestTemplate();
rt.setInterceptors(Collections.singletonList(interceptor));
Where interceptor could be:
ClientHttpRequestInterceptor interceptor = new ClientHttpRequestInterceptor() {
#Override
public ClientHttpResponse intercept(HttpRequest request, byte[] body, ClientHttpRequestExecution execution)
throws IOException {
request.getHeaders().add("Content-Encoding", "gzip");
byte[] gzipped = getGzip(body);
return execution.execute(request, gzipped);
}
}
getGzip I copied
private byte[] getGzip(byte[] body) throws IOException {
ByteArrayOutputStream byteStream = new ByteArrayOutputStream();
try {
GZIPOutputStream zipStream = new GZIPOutputStream(byteStream);
try {
zipStream.write(body);
} finally {
zipStream.close();
}
} finally {
byteStream.close();
}
byte[] compressedData = byteStream.toByteArray();
return compressedData;
}
After configuring the interceptor all requests will be zipped.
The disadvantage of this approach is that it does not support streaming as the ClientHttpRequestInterceptor receives the content as a byte[]
If you require streaming create a custom ClientHttpRequestFactory, say GZipClientHttpRequestFactory, and use it like this:
SimpleClientHttpRequestFactory requestFactory = new SimpleClientHttpRequestFactory();
requestFactory.setBufferRequestBody(false);
ClientHttpRequestFactory gzipRequestFactory = new GZipClientHttpRequestFactory(requestFactory);
RestTemplate rt = new RestTemplate(gzipRequestFactory);
Where GZipClientHttpRequestFactory is:
public class GZipClientHttpRequestFactory extends AbstractClientHttpRequestFactoryWrapper {
public GZipClientHttpRequestFactory(ClientHttpRequestFactory requestFactory) {
super(requestFactory);
}
#Override
protected ClientHttpRequest createRequest(URI uri, HttpMethod httpMethod, ClientHttpRequestFactory requestFactory)
throws IOException {
ClientHttpRequest delegate = requestFactory.createRequest(uri, httpMethod);
return new ZippedClientHttpRequest(delegate);
}
}
And ZippedClientHttpRequest is:
public class ZippedClientHttpRequest extends WrapperClientHttpRequest
{
private GZIPOutputStream zip;
public ZippedClientHttpRequest(ClientHttpRequest delegate) {
super(delegate);
delegate.getHeaders().add("Content-Encoding", "gzip");
// here or in getBody could add content-length to avoid chunking
// but is it available ?
// delegate.getHeaders().add("Content-Length", "39");
}
#Override
public OutputStream getBody() throws IOException {
final OutputStream body = super.getBody();
zip = new GZIPOutputStream(body);
return zip;
}
#Override
public ClientHttpResponse execute() throws IOException {
if (zip!=null) zip.close();
return super.execute();
}
}
And finally WrapperClientHttpRequest is:
public class WrapperClientHttpRequest implements ClientHttpRequest {
private final ClientHttpRequest delegate;
protected WrapperClientHttpRequest(ClientHttpRequest delegate) {
super();
if (delegate==null)
throw new IllegalArgumentException("null delegate");
this.delegate = delegate;
}
protected final ClientHttpRequest getDelegate() {
return delegate;
}
#Override
public OutputStream getBody() throws IOException {
return delegate.getBody();
}
#Override
public HttpHeaders getHeaders() {
return delegate.getHeaders();
}
#Override
public URI getURI() {
return delegate.getURI();
}
#Override
public HttpMethod getMethod() {
return delegate.getMethod();
}
#Override
public ClientHttpResponse execute() throws IOException {
return delegate.execute();
}
}
This approach creates a request with chunked transfer encoding, this can be changed setting the content length header, if size is known.
The advantage of the ClientHttpRequestInterceptor and/or custom ClientHttpRequestFactory approach is that it works with any method of RestTemplate. An alternate approach, passing a RequestCallback is possible only with execute methods, this because the other methods of RestTemplate internally create their own RequestCallback(s) that produce the content.
BTW it seems that there is little support to decompress gzip request on the server. Also related: Sending gzipped data in WebRequest? that points to the Zip Bomb issue. I think you will have to write some code for it.

Further to the above answer from #TestoTestini, if we take advantage of Java 7+'s 'try-with-resources' syntax (since both ByteArrayOutputStream and GZIPOutputStream implement closeable() ) then we can shrink the getGzip function into the following:
private byte[] getGzip(byte[] body) throws IOException {
try (ByteArrayOutputStream byteStream = new ByteArrayOutputStream();
GZIPOutputStream zipStream = new GZIPOutputStream(byteStream)) {
zipStream.write(body);
byte[] compressedData = byteStream.toByteArray();
return compressedData;
}
}
(I couldn't find a way of commenting on #TestoTestini's original answer and retaining the above code format, hence this Answer).

Since I cannot comment on #roj 's post I'm writing an answer here.
#roj snippet although is neat it actually does not do the same job as #Testo Testini 's snippet.
Testo is closing the streams before:
byteStream.toByteArray();
where in #rog answer, this occurs before the stream.close(), since streams are in the try/resource block.
If you need to use try-with-resources, zipStream should be closed before the byteStream.toByteArray();
The complete snippet should be:
private byte[] getGzip(byte[] body) throws IOException {
try (ByteArrayOutputStream byteStream = new ByteArrayOutputStream();
GZIPOutputStream zipStream = new GZIPOutputStream(byteStream)) {
zipStream.write(body);
zipStream.close();
byte[] compressedData = byteStream.toByteArray();
return compressedData;
}
}
The was getting an error ("Compressed file ended before the end-of-stream marker was reached") and the above fixed the error in my case and I thought that I should share this.

Jersey InputStream is modified in filter. Unable to figure out how to access modified inputStream in Jersey Resource

As discussed in How to use Jersey interceptors to get request body, I am modifying the EntityInputStream in a ContainerRequestFilter.
public filter(ContainerRequest request){
ByteArrayOutputStream out = new ByteArrayOutputStream();
InputStream in = request.getEntityInputStream();
try{
Readerwriter.writeTo(in, out);
byte[] requestEntity = out.toByteArray();
// DO SOMETHING WITH BYTES HERE
request.setEntityInputStream(new ByteArrayInputStream(requestEntity));
}/// error handling code here
}
However, later on I can't figure out how to access the modified InputStream. I can get the ServletContext in the resource, but I can't figure out how to get ahold of the object I actually modified in the filter, the ContainerRequest.
Can I do something like this? Jersey can't start up out when I try this:
#Post
#Path("/test")
public Response test(#Context ContainerRequest cr){
// blah blah
return....
}
Jersey error:
Missing dependecy for method public javax.ws.rs.core.Response example.TestController.test(com.sun.jersey.spi.container.ContainerRequest), annotated with POST of resource, class example.TestController, is not recognized as a valid resource method.
I am stuck on an old version of jersey, 1.8, so I'm not sure if that's part of the problem.

All you need to do is accept an InputStream as the entity body in your resource method. If you want the ByteArrayInputStream just cast it.
#POST
public Response post(InputStream in) {
ByteArrayInputStream bin = (ByteArrayInputStream)in;
}
If you don't already know, how Jersey converts the request stream (for the request body) into Java types (for instance JSON to POJO) is through MessageBodyReaders. You can read more about them at JAX-RS Entity Providers.
Jersey already comes with some standard readers for easily convertible types, for instance String. Most content-types can be converted to String. Likewise, it has a reader to handle InputStream. This is probably the easiest conversion, as the request is already coming in as an InputStream, so really all the reader would need to do is return the original stream, and that's what would get passed to our method.
If we look at the implementation InputStreamProvider, we can see that that's what actually happens. The original stream is simply returned. And since the filter happens before the readers, the reader simply returns the stream that we set.
Here is a complete example using Jersey Test Framework
public class StreamFilterTest extends JerseyTest {
public static class InputStreamFilter implements ContainerRequestFilter {
#Override
public ContainerRequest filter(ContainerRequest request) {
try {
ByteArrayOutputStream out = new ByteArrayOutputStream();
InputStream in = request.getEntityInputStream();
ReaderWriter.writeTo(in, out);
byte[] requestBytes = out.toByteArray();
byte[] worldBytes = " World".getBytes(StandardCharsets.UTF_8);
byte[] newBytes = new byte[requestBytes.length + worldBytes.length];
System.arraycopy(requestBytes, 0, newBytes, 0, requestBytes.length);
System.arraycopy(worldBytes, 0, newBytes, requestBytes.length, worldBytes.length);
request.setEntityInputStream(new ByteArrayInputStream(newBytes));
} catch (IOException ex) {
Logger.getLogger(InputStreamFilter.class.getName()).log(Level.SEVERE, null, ex);
throw new RuntimeException(ex);
}
return request;
}
}
#Path("stream")
public static class StreamResource {
#POST
public String post(InputStream in) throws Exception {
ByteArrayInputStream bin = (ByteArrayInputStream) in;
StringWriter writer = new StringWriter();
ReaderWriter.writeTo(new InputStreamReader(bin), writer);
return writer.toString();
}
}
public static class AppConfig extends DefaultResourceConfig {
public AppConfig() {
super(StreamResource.class);
getContainerRequestFilters().add(new InputStreamFilter());
}
}
#Override
public WebAppDescriptor configure() {
return new WebAppDescriptor.Builder()
.initParam(WebComponent.RESOURCE_CONFIG_CLASS,
AppConfig.class.getName())
.build();
}
#Test
public void should_return_hello_world() {
String response = resource().path("stream").post(String.class, "Hello");
assertEquals("Hello World", response);
}
}
Here's the test dependency
<dependency>
<groupId>com.sun.jersey.jersey-test-framework</groupId>
<artifactId>jersey-test-framework-grizzly2</artifactId>
<version>1.17.1</version>
<scope>test</scope>
</dependency>

Jersey client upload progress

I have a jersey client that need to upload a file big enough to require a progress bar.
The problem is that, for an upload that requires some minutes, i see the bytes transfered to go to 100% as soon as the application has started. Then it takes some minutes to print the "on finished" string. It is as if the bytes were sent to a buffer, and i was reading the transfert-to-the buffer speed instead of the actual upload speed. This makes the progress bar useless.
This is the very simple code:
ClientConfig config = new DefaultClientConfig();
Client client = Client.create(config);
WebResource resource = client.resource("www.myrestserver.com/uploads");
WebResource.Builder builder = resource.type(MediaType.MULTIPART_FORM_DATA_TYPE);
FormDataMultiPart multiPart = new FormDataMultiPart();
FileDataBodyPart fdbp = new FileDataBodyPart("data.zip", new File("data.zip"));
BodyPart bp = multiPart.bodyPart(fdbp);
String response = builder.post(String.class, multiPart);
To get progress state i've added a ContainerListener filter, obviouslt before calling builder.post:
final ContainerListener containerListener = new ContainerListener() {
#Override
public void onSent(long delta, long bytes) {
System.out.println(delta + " : " + long);
}
#Override
public void onFinish() {
super.onFinish();
System.out.println("on finish");
}
};
OnStartConnectionListener connectionListenerFactory = new OnStartConnectionListener() {
#Override
public ContainerListener onStart(ClientRequest cr) {
return containerListener;
}
};
resource.addFilter(new ConnectionListenerFilter(connectionListenerFactory));

In Jersey 2.X, i've used a WriterInterceptor to wrap the output stream with a subclass of Apache Commons IO CountingOutputStream that tracks the writing and notify my upload progress code (not shown).
public class UploadMonitorInterceptor implements WriterInterceptor {
#Override
public void aroundWriteTo(WriterInterceptorContext context) throws IOException, WebApplicationException {
// the original outputstream jersey writes with
final OutputStream os = context.getOutputStream();
// you can use Jersey's target/builder properties or
// special headers to set identifiers of the source of the stream
// and other info needed for progress monitoring
String id = (String) context.getProperty("id");
long fileSize = (long) context.getProperty("fileSize");
// subclass of counting stream which will notify my progress
// indicators.
context.setOutputStream(new MyCountingOutputStream(os, id, fileSize));
// proceed with any other interceptors
context.proceed();
}
}
I then registered this interceptor with the client, or with specific targets where you want to use the interceptor.

it should be enough to provide you own MessageBodyWriter for java.io.File which fires some events or notifies some listeners as progress changes
#Provider()
#Produces(MediaType.APPLICATION_OCTET_STREAM)
public class MyFileProvider implements MessageBodyWriter<File> {
public boolean isWriteable(Class<?> type, Type genericType, Annotation[] annotations, MediaType mediaType) {
return File.class.isAssignableFrom(type);
}
public void writeTo(File t, Class<?> type, Type genericType, Annotation annotations[], MediaType mediaType, MultivaluedMap<String, Object> httpHeaders, OutputStream entityStream) throws IOException {
InputStream in = new FileInputStream(t);
try {
int read;
final byte[] data = new byte[ReaderWriter.BUFFER_SIZE];
while ((read = in.read(data)) != -1) {
entityStream.write(data, 0, read);
// fire some event as progress changes
}
} finally {
in.close();
}
}
#Override
public long getSize(File t, Class<?> type, Type genericType, Annotation[] annotations, MediaType mediaType) {
return t.length();
}
}
and to make your client application uses this new provider simply:
ClientConfig config = new DefaultClientConfig();
config.getClasses().add(MyFileProvider.class);
or
ClientConfig config = new DefaultClientConfig();
MyFileProvider myProvider = new MyFileProvider ();
cc.getSingletons().add(myProvider);
You would have to also include some algorithm to recognize which file is transfered when receiving progress events.
Edited:
I just found that by default HTTPUrlConnection uses buffering. And to disable buffering you could do couple of things:
httpUrlConnection.setChunkedStreamingMode(chunklength) - disables buffering and uses chunked transfer encoding to send request
httpUrlConnection.setFixedLengthStreamingMode(contentLength) - disables buffering and but ads some constraints to streaming: exact number of bytes must be sent
So I suggest the final solution to your problem uses 1st option and would look like this:
ClientConfig config = new DefaultClientConfig();
config.getClasses().add(MyFileProvider.class);
URLConnectionClientHandler clientHandler = new URLConnectionClientHandler(new HttpURLConnectionFactory() {
#Override
public HttpURLConnection getHttpURLConnection(URL url) throws IOException {
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setChunkedStreamingMode(1024);
return connection;
}
});
Client client = new Client(clientHandler, config);

I have successfully used David's answer. However, I would like to extend on it:
The following aroundWriteTo implementation of my WriterInterceptor shows how a panel (or similar) can also be passed to the CountingOutputStream:
#Override
public void aroundWriteTo(WriterInterceptorContext context)
throws IOException, WebApplicationException
{
final OutputStream outputStream = context.getOutputStream();
long fileSize = (long) context.getProperty(FILE_SIZE_PROPERTY_NAME);
context.setOutputStream(new ProgressFileUploadStream(outputStream, fileSize,
(progressPanel) context
.getProperty(PROGRESS_PANEL_PROPERTY_NAME)));
context.proceed();
}
The afterWrite of the CountingOutputStream can then set the progress:
#Override
protected void afterWrite(int n)
{
double percent = ((double) getByteCount() / fileSize);
progressPanel.setValue((int) (percent * 100));
}
The properties can be set on the Invocation.Builder object:
Invocation.Builder invocationBuilder = webTarget.request();
invocationBuilder.property(
UploadMonitorInterceptor.FILE_SIZE_PROPERTY_NAME, newFile.length());
invocationBuilder.property(
UploadMonitorInterceptor.PROGRESS_PANEL_PROPERTY_NAME,
progressPanel);
Perhaps the most important addition to David's answer and the reason why I decided to post my own is the following code:
client.property(ClientProperties.CHUNKED_ENCODING_SIZE, 1024);
client.property(ClientProperties.REQUEST_ENTITY_PROCESSING, "CHUNKED");
The client object is the the javax.ws.rs.client.Client.
It is essential to disable buffering also with the WriterInterceptor approach. Above code is a straightforward way to do this with Jersey 2.x