Develop Reference

Java Vert.x Reverse Proxy automatic Netty Decode

I've a small Vert.x implementation of a Reverse proxy that is functionally working fine.
Downstream requests are done with tryUseCompression(true) (gziped), which our servers can answer accordingly.
To avoid additional decompression / compression we would like that when we receive a request asking for compression, and the downstream answers with the same compression (all in gzip for example) then the HTTP Proxy should not try to decompress the downstream result but instead return as is to the upstream response.
As far as I could follow Vertx code, Netty is being allways called to decode the messages that it receives and we only get the result
See Http1xClientHandler::handleMessage:
HttpObject obj = (HttpObject) msg;
DecoderResult result = obj.decoderResult();
Also found at the HttpChannelConnector::applyHttp1xConnectionOtions
if (client.getOptions().isTryUseCompression()) {
pipeline.addLast("inflater", new HttpContentDecompressor(true));
}
Is it possible to tell Vertx / Netty not to decompress the result of an individual call?
Thanks for the help

No it's not possible. Your best option would be to handle compression/decompression in your code.

How can I more efficently download large files over http?

I'm trying to download large files (<1GB) in Kotlin since I already knew I'm using okhttp and pretty much followed just used the answer from this question. Except that I'm using Kotlin instead of java, so the syntax is slightly diffrent.
val client = OkHttpClient()
val request = Request.Builder().url(urlString).build()
val response = client.newCall(request).execute()
val is = response.body().byteStream()
val input = BufferedInputStream(is)
val output = FileOutputStream(file)
val data = ByteArray(1024)
val total = 0L
val count : Int
do {
count = input.read(data)
total += count
output.write(data, 0, count)
} while (count != -1)
output.flush()
output.close()
input.close()
That works in that it downloads the file without using too much memory but it seems needlessly ineffective in that it constantly tries to write more data without knowing if any new data arrived.
That also seems confirmed with my own tests while running this on a very resource limited VM as it seems to use more CPU while getting a lower download speed then a comparable script in python, and of cause using wget.
What I'm wondering if there is a way where I can give something a callback that gets called if x bytes are available or if it's the end of the file so I don't have to constantly try and get more data without knowing if there is any.
Edit:
If it's not possible with okhttp I don't have a problem using something else, it's just that it was the http library I'm used to.

As of version 11, Java has a built-in HttpClient which implements
asynchronous streams of data with non-blocking back pressure
and that's what you need if you want your code to run only when there's data to process.
If you can afford to upgrade to Java 11, you'll be able to solve your problem out of the box, using the HttpResponse.BodyHandlers.ofFile body handler. You won't have to implement any data transfer logic on your own.
Kotlin example:
fun main(args: Array<String>) {
val client = HttpClient.newHttpClient()
val request = HttpRequest.newBuilder()
.uri(URI.create("https://www.google.com"))
.GET()
.build()
println("Starting download...")
client.send(request, HttpResponse.BodyHandlers.ofFile(Paths.get("google.html")))
println("Done with download.")
}

One could do away with the BufferedInputStream. Or as its default buffer size in Oracle's java is 8192, use a larger ByteArray, say 4096.
However best would be to either use java.nio or try Files.copy:
Files.copy(is, file.toPath());
This removes about 12 lines of code.
An other way is to send the request with a header to deflate gzip compression Accept-Encoding: gzip, so the transmission takes less time. In the response here then possibly wrap is in a new GZipInputStream(is) - when the response header Content-Encoding: gzip is given. Or if feasible store the file compressed with an addition ending .gz; mybiography.md as mybiography.md.gz.

RestEasy : How to send an easy POST request with parameters and api key

I'm a beginner and I'm a little bit lost with Resteasy
I'd like to send a post request with an URL similar to this : http://myurl.com/options?value=3name=picture
String myValue = "3";
String myName = "picture";
String key = "topsecret";
I'm not too sure about what's coming. I've seen several tutorial classes (not very clear to me) and another way similar to this
final MultivaluedMap<String, Object> queryParams = new MultivaluedMapImpl<>();
queryParams.add("value", myValue);
queryParams.add("name", myPicture);
final ResteasyClient client = new ResteasyClientBuilder().build();
final ResteasyWebTarget target = client.target(url).queryParams(queryParams);;
final Builder builder = target.request();
When I write I have loads of warning. Is it the right way do it ? What about the API key ?

First of all, you must check the documentation of the API you want to consume regarding how the API key must be sent to the server. Not all APIs follow the same approach.
For example purposes, let's assume that the API key must be sent in the X-Api-Key header. It's a non standard and I've made it up just to demonstrate how to use the client API.
So you can have the following:
// Create a client
Client client = ClientBuilder.newClient();
// Define a target
WebTarget target = client.target("http://myurl.com/options")
.queryParam("value", "3")
.queryParam("name", "picture");
// Perform a request to the target
Response response = target.request().header("X-Api-Key", "topsecret")
.post(Entity.text(""));
// Process the response
// This part is up to you
// Close the response
response.close();
// Close the client
client.close();
The above code uses the JAX-RS API, which is implemented by RESTEasy. You'd better use Client instead of ResteasyClient whenever possible to ensure portability with other implementations.
The above code also assumes that you want to send an empty text in the request payload. Modify it accordingly.
Response instances that contain an un-consumed entity input stream should be closed. This is typical for scenarios where only the response headers and the status code are processed, ignoring the response entity.
Going beyond the scope of the question, bear in mind that Client instances are heavy-weight objects that manage the underlying client-side communication infrastructure. Hence initialization as well as disposal of a Client instance may be a rather expensive operation.
The documentation advises to create only a small number of Client instances and reuse them when possible. It also states that Client instances must be properly closed before being disposed to avoid leaking resources.

Can Elasticsearch stream the SearchResponse?

I have a rest application that can export some report data from Elasticsearch. It is easy to do with the Java API:
SearchResponse response = getClient()
.prepareSearch("my_index_name")
.setQuery(QueryBuilders.someQuery())
.addAggregation(AggregationBuilders.someAggregation())
.get();
The problem starts with the big responses. Using this code snippet, the response is read to build the SearchResponse object in memory. In my case, the response does not fits in memory.
Paging cannot help because we often need to return the full data and Aggregations do not support paging yet.
I know that I can use the Elasticsearch REST API to read the response as stream, but manually build the request it is cumbersome. I really want something like this:
// my dream API
InputStream response = getClient()
.prepareSearch("my_index_name")
.setQuery(QueryBuilders.someQuery())
.addAggregation(AggregationBuilders.someAggregation())
.getStream();
So, can the Elasticsearch Java API stream the SearchResponse?

A proposal for streaming results does exist but it doesn't seem to have picked up steam so far and was closed (for now).
There's a way to do it with XContentBuilder but that still requires the whole response to be in memory before being sent.
It might not be what you want, but that's the closest thing that I know which could fulfill your need. Worth giving it a try.

I believe there is no way to obtain an InputStream from the Java API (but I might be wrong). I also think there is no way to directly obtain an InputStream in Jest (a REST-based Elasticsearch Java API).
You mention that it is cumbersome to create the search request to the _search endpoint yourself: if you're referring to building the actual json query, I just would like to point out that once you have a SearchSourceBuilder, you can call toString() on it to get a fully working json representation of your query.
SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
sourceBuilder.query(this.getQuery())
.from(this.getFrom())
.size(this.getSize())
.fetchSource(this.getSource(), null);
this.getSort().forEach(sourceBuilder::sort);
sourceBuilder.toString() // the json representation

Reading from response body multiple times in Apache HttpClient 4.x

I'm using the Apache HttpClient 4.2.3 in my application. We store the response of an HTTP call like so:
HttpResponse httpResponse = (DefaultHttpClient)httpClient.execute(httpRequest);
The response body is an InputStream in the 4.x API:
InputStream responseStream = httpResponse.getEntity().getContent();
My problem is I need to read the response body as a string and as a byte[] at various points in the application. But the InputStream used by Apache is an EofSensorInputStream, which means once I reach the stream EOF, it gets closed. Is there anyway I can get the string and byte[] representations multiple times and not close the stream?
I've already tried wrapping the byte array in a new ByteArrayInputStream and setting that as the request body, but it doesn't work since my response body can reach a few gigs. I've also tried this, but I noticed the original response stream still gets closed.
Any pointers would be welcome.
EDIT: On a related note, it would be also be great if I could find the length of the InputStream either without consuming the stream or by reversing the consumption.

1 . I think you have somewhat conflicting requirements:
a)
it doesn't work since my response body can reach a few gigs
b)
Is there anyway I can get the string and byte[] representations multiple times and not close the stream
If you do not have enough memory this is not possible.
Btw, another way to get the response as bytes is EntityUtils.byte[] toByteArray(final HttpEntity entity).
Do you really need N-gigs String? What are you going to do with it?
2 .
it would be also be great if I could find the length of the InputStream
httpResponse.getEntity().getContentLength()
3 . Since the response does not fit into the memory I would suggest to save it into a file (or temp file). Then set up InputStream on that file, and then read it as many times as you need.