Develop Reference

java 11 HttpClient send header then body

I want to use Java 11 HttpClient and send header first, check response and if response is OK then send the body.
How can I send header only?
this is my current code:
HttpClient httpClient = HttpClient.newBuilder()
.version(HttpClient.Version.HTTP_2)
.followRedirects(HttpClient.Redirect.NORMAL)
.connectTimeout(Duration.ofSeconds(10))
.authenticator(Authenticator.getDefault())
.build();
HttpRequest httpRequest = HttpRequest.newBuilder("someEndpoint)
.header(CONTENT_TYPE, MediaType.APPLICATION_JSON_VALUE)
.header(AUTHORIZATION, "someApiKey)
.build();
HttpResponse<String> httpResponse = httpClient.send(httpRequest, HttpResponse.BodyHandlers.ofString());
However with such httpResponse I understand I send the body.

By default, the header comes first in requests.
What you asked is, The first request with header and then with a body are two different requests. A single request can't be broken this way.
If you are talking about, Http HEAD method usage, then
The HEAD method asks for a response identical to that of a GET request, but without the response body.
The HTTP HEAD method requests the headers that would be returned if the HEAD request's URL was instead requested with the HTTP GET method. For example, if a URL might produce a large download, a HEAD request could read its Content-Length header to check the file size without actually downloading the file.
an example to use HEAD method:-
var httpClient: HttpClient = HttpClient.newBuilder()
.connectTimeout(Duration.ofSeconds(10))
.build();
var requestHead = HttpRequest.newBuilder()
.method("HEAD", HttpRequest.BodyPublishers.noBody())
.uri(URI.create("https://www.test.com"))
.build();
val httpResponse = httpClient.send(requestHead, BodyHandlers.discarding());
HttpHeaders headers = response.headers();
headers.map().forEach((key, values) -> {
System.out.printf("%s: %s%n", key, values);
});

Properly streaming of both input and output in HTTP Servlet

I'm trying to write servlet that will handle POST request and stream both input and output. I mean it should read one line of input, do some work on this line, and write one line of output. And it should be able to handle arbitrary long request (so also will produce arbitrary long response) without out of memory exceptions. Here's my first attempt:
protected void doPost(HttpServletRequest request, HttpServletResponse response) {
ServletInputStream input = request.getInputStream();
ServletOutputStream output = response.getOutputStream();
LineIterator lineIt = lineIterator(input, "UTF-8");
while (lineIt.hasNext()) {
String line = lineIt.next();
output.println(line.length());
}
output.flush();
}
Now I tested this servlet using curl and it works, but when I've written client using Apache HttpClient both the client thread and server thread hangs. The client looks like that:
HttpClient client = HttpClientBuilder.create().build();
HttpPost post = new HttpPost(...);
// request
post.setEntity(new FileEntity(new File("some-huge-file.txt")));
HttpResponse response = client.execute(post);
// response
copyInputStreamToFile(response.getEntity().getContent(), new File("results.txt"));
The issue is obvious. Client does it's job sequentially in one thread - first it sends request completely, and only then starts reading the response. But server for every line of input writes one line of output, if client is not reading output (and sequential client isn't) then server is blocked trying to write to the output stream. This in turn blocks client trying to send the input to the server.
I suppose curl works, because it somehow sends input and receives output concurrently (in separate threads?). So the first question is can Apache HttpClient be configured to behave similarly as curl?
The next question is, how to improve the servlet so ill-behaving client won't cause server threads to be hanging? My first attempt is to introduce intermediate buffer, that will gather the output until the client finishes sending input, and only then servlet will start sending output:
ServletInputStream input = request.getInputStream();
ServletOutputStream output = response.getOutputStream();
// prepare intermediate store
int threshold = 100 * 1024; // 100 kB before switching to file store
File file = File.createTempFile("intermediate", "");
DeferredFileOutputStream intermediate = new DeferredFileOutputStream(threshold, file);
// process request to intermediate store
PrintStream intermediateFront = new PrintStream(new BufferedOutputStream(intermediate));
LineIterator lineIt = lineIterator(input, "UTF-8");
while (lineIt.hasNext()) {
String line = lineIt.next();
intermediateFront.println(line.length());
}
intermediateFront.close();
// request fully processed, so now it's time to send response
intermediate.writeTo(output);
file.delete();
This works, and ill-behaving client can use my servlet safely, but on the other hand for these concurrent clients like curl this solution adds an unnecessary latency. The parallel client is reading the response in separate thread, so it will benefit when the response will be produced line by line as the request is consumed.
So I think I need a byte buffer/queue that:
can be written by one thread, and read by another thread
will initially be only in memory
will overflow to disk if necessary (similarly as DeferredFileOutputStream).
In the servlet I'll spawn new thread to read input, process it, and write output to the buffer, and the main servlet thread will read from this buffer and send it to the client.
Do you know any library like to do that? Or maybe my assumptions are wrong and I should do something completely different...

To achieve simultaneously writing and reading you can use Jetty HttpClient
http://www.eclipse.org/jetty/documentation/current/http-client-api.html
I've created pull request to your repo with this code.
HttpClient httpClient = new HttpClient();
httpClient.start();
Request request = httpClient.newRequest("http://localhost:8080/line-lengths");
final OutputStreamContentProvider contentProvider = new OutputStreamContentProvider();
InputStreamResponseListener responseListener = new InputStreamResponseListener();
request.content(contentProvider).method(HttpMethod.POST).send(responseListener); //async request
httpClient.getExecutor().execute(new Runnable() {
public void run() {
try (OutputStream outputStream = contentProvider.getOutputStream()) {
writeRequestBodyTo(outputStream); //writing to stream in another thread
} catch (IOException e) {
e.printStackTrace();
}
}
});
readResponseBodyFrom(responseListener.getInputStream()); //reading response
httpClient.stop();

java httprequest getting the body from the request

I receive a post request from client. This request contains some json data which I want to part on the server side. I have created the server using httpcore. HttpRequestHandler is used for handling the request. Here is the code I thought would work
HttpEntity entity = ((HttpEntityEnclosingRequest)request).getEntity();
InputStream inputStream = entity.getContent();
String str = inputStream.toString();
System.out.println("Post contents: " + str);*/
But I cant seem to find a way to get the body of the request using the HttpRequest object. How can I extract the body from the request object ? Thanks

You should use EntityUtils and it's toString method:
String str = EntityUtils.toString(entity);
getContent returnes stream and you need to read all data from it manually using e.g. BufferedReader. But EntityUtils does it for you.
You can't use toString on stream, because it returns string representation of the object itself not it's data.
One more thing: AFAIK GET requests can't contain body so it seems you get POST request from client.

... and for MultipartEntity use this:
ByteArrayOutputStream baos = new ByteArrayOutputStream();
try {
entity.writeTo(baos);
} catch (IOException e) {
e.printStackTrace();
}
String text = new String(baos.toByteArray());

Google Documents List API file upload 400 response

I am writing my own implementation of the Google Drive client for the Android and I am using the docs list api. Recently I've encountered a following problem:
At first I was using the HttpURLConnection to upload the file but it seems like it writes the data to the socket after a call to getResponseCose(), not when I am writing to the connection's OutputStream, which is a must for me.
Then I've switched to the Apache HttpClient but I'm still getting a 400 response, not sure why. Maybe you will be able to help me. Here is the code used to upload a file.
String putUrl = conn.getHeaderField("Location");//from the previous request
final HttpClient client = new DefaultHttpClient();
final HttpPut put = new HttpPut(putUrl);
MultipartEntity entity = new MultipartEntity(HttpMultipartMode.BROWSER_COMPATIBLE);
put.addHeader("Content-Type", mime==null?"file":mime);
//put.addHeader("Content-Length", String.valueOf(length));
put.addHeader("Content-Range", "bytes 0-"+(length-1)+"/"+length);
put.addHeader("GData-Version", "3.0");
put.addHeader("Authorization", getAuthorizationProperty());
entity.addPart("content", new InputStreamBody(in, name));
put.setEntity(entity);
HttpResponse resp = client.execute(put);
int response = resp.getStatusLine().getStatusCode();
if(response == HttpStatus.SC_CREATED){
lastCreated = parseSingleXMLEntry(resp.getEntity().getContent());
}
Exactly the same headers worked for HttpURLConnection. Maybe the entity is wrong?

Ok, the solution is quite simple, hope it will be useful for someone.
I had to delete all lines which added headers to the request. After that I've added the mime type to the InputStreamBody constructor and overriden the getContentLength() method to provide stream length. Finally it looks like this:
MultipartEntity entity = new MultipartEntity(HttpMultipartMode.BROWSER_COMPATIBLE);
entity.addPart("content", new InputStreamBody(in, ,mime, name){
#Override
public long getContentLength() {
return length;
}
});
put.setEntity(entity);
HttpResponse resp = client.execute(put);
And that's all.

twitter streaming api - set filter in http post request (apache httpcomponents)

I'm trying out the twitter streaming api. I could succesfully filter tweets by using curl, as stated here:
curl -d #tracking http://stream.twitter.com/1/statuses/filter.json -u <user>:<pass>
where tracking is a plain file with the content:
track=Berlin
Now I tried to do the same thing in JavaSE, using Apache's HTTPComponents:
UsernamePasswordCredentials creds = new UsernamePasswordCredentials(<user>, <pass>);
DefaultHttpClient httpClient = new DefaultHttpClient();
httpClient.getCredentialsProvider().setCredentials(AuthScope.ANY, creds);
HttpPost httpPost = new HttpPost("http://stream.twitter.com/1/statuses/filter.json");
HttpParams params = new BasicHttpParams();
params = params.setParameter("track", "Berlin");
httpPost.setParams(params);
try {
HttpResponse httpResponse = httpClient.execute(httpPost);
HttpEntity entity = httpResponse.getEntity();
if (entity != null) {
InputStream instream = entity.getContent();
String t;
BufferedReader br = new BufferedReader(new InputStreamReader(instream));
while(true) {
t = br.readLine();
if(t != null) {
linkedQueue.offer(t);
}
}
}
} catch (IOException ioe) {
System.out.println(ioe.getMessage());
}
finally{
httpClient.getConnectionManager().shutdown();
}
When I run that, I get:
No filter parameters found. Expect at least one parameter: follow track
as a single entry in my linkedQueue. Seems the api wants the parameter in a different form, but cannot find any hint in the documentation. Can somebody share some experiences with the api or see any other problem with the code? Thanks!
EDIT
Putting the filter parameter into the params was a bad idea. As it's post data, it needs to be defined as an Entity before the request is being made:
StringEntity postEntity = new StringEntity("track=Berlin", "UTF-8");
postEntity.setContentType("application/x-www-form-urlencoded");
httpPost.setEntity(postEntity);
That's what I was doing wrong. Thanks Brian!

I suspect you need to post the data as the contents of your HTTP post. The man page for curl -d says:
(HTTP) Sends the specified data in a
POST request to the HTTP server, in
the same way that a browser does when
a user has filled in an HTML form and
presses the submit button. This will
cause curl to pass the data to the
server using the content-type
application/x-www-form-urlencoded.
so I believe you have to set that content type and put the contents of the tracking file in the body of your post.