Develop Reference

How to calculate Content-Length of a request body of a Http Post Request

I am trying to calculate the Content-Length of the request body of a http post request but I keep getting error that indicated wrong Content-Length. Request body looks like following:
Map<String, String> body = {
'grant_type': 'authorization_code',
'client_id': 'clientid',
'code': authCode,
'redirect_uri': 'http://localhost:8080/login/callback',
'code_verifier':
'codeverifier',
};
I tried couple solutions like concatenating content of the body into one string such as following and then convert it into byte array and send the length of it but it didn't work.
String bodyStr = "grant_type:authorization_code" +
"client_id:clientid" +
"code:{$authCode}" +
"redirect_uri:http://localhost:8080/login/callback" +
"code_verifier:codeverifier";
List<int> bytes = utf8.encode(bodyStr);
The post request body is x-www-form-urlencoded format and content length has to be calculated correctly. Any help is appreciated, thanks.

I encapsulated it myself. Generally, I don't need to calculate it. Unless it's a special occasion.
Okhttp3 is recommended

You don't need to make a list of integer...
String bodyStr = "...";
byte[] bytes = bodyStr.getBytes(Charset.forName("UTF-8"));
int len = bytes.length;
response.setContentLength(len);
response.getOutputStream().write(bytes);
This example is using the HttpServletResponse object from a HttpServlet. Not sure if that's what you need.
I have used this a fair amount, it works well.

How to send special characters from javascript at frontend to java at backend?

I have a rest web service whose url is
http://localhost/rest/authenticate?username=username&password=pa+ssw1&rd%
In password parameter have 3 special character.
+ character read as white space
& character remove the all characters. for example - my password like this "passw&rd" and it will read like this "passw"
% character does not read the proper password, its read the null value.
my API like this ...
#Path("/authenticate")
public class AuthenticateService {
private ILiteWebServiceFacade liteWebServiceFacade = ServiceLocator.locateService(ILiteWebServiceFacade.class);
#POST
#Produces(MediaType.APPLICATION_JSON)
public Response authenticate(#FormParam("username") String username,
#FormParam("password") String password)
throws RestException {
AuthenticateResponse res = new AuthenticateResponse();
try {
res = liteWebServiceFacade.mobAuthenticate(username, password);
} catch (RestApplicationException e) {
res.setError(e.getErrorMessage().getErrorCode(), e.getErrorMessage().getErrorMessageKey());
}
return Response.ok(res).build();
}
}
Can you please suggest me how to read all these special character?

Fist of all, don't send passwords in the URL -- it's just really bad security as the query parameters can get logged in Apache or NGINX logs and are often exposed along the way.
Secondly, if you're using HTTP, understand that certain characters need to be escaped.
%2B +
%20 {space}
%26 &
You will have to make sure your code is actually decoding the strings, but depending on your framework it may do that automatically (Spring does, for example) or you may have to add a quick callout to decode.
If you change the behavior of the URI Encoding, you change the HTTP spec, and it's going to make it hard for anyone consuming your API to understand what's going on -- especially if you grab ampersands because any URI encoding will be broken for your API.
See the RFC: Uniform Resource Identifier (URI): Generic Syntax https://www.rfc-editor.org/rfc/rfc3986

Firstly, I recommend you not to send sensitive data like username and password in url. You should send it in request body.
A simple way to do that is Base64 encode on frontend in javascript and decode at backend in java.
FrontEnd: All chrome version, Firefox 1.0 and above, Internet Explorer 10 and above
var string = 'pa+ssw1&rd%';
// Encode the String
var encodedString = btoa(string);
console.log(encodedString); // Outputs: cGErc3N3MSZyZCU=
// Decode the String
var decodedString = atob(encodedString);
console.log(decodedString); // Outputs: pa+ssw1&rd%
Backend:
For Java 8 and above:
import java.util.Base64;
byte[] decoded = Base64.getDecoder().decode(password);
decodedpassword = new String(decoded, "utf-8");
For < Java 8
import org.apache.commons.codec.binary.Base64;
import org.apache.commons.codec.binary.StringUtils;
public String decode1(String s) {
return StringUtils.newStringUtf8(Base64.decodeBase64(s));
}
public String decode2(String s) {
return new String(Base64.decodeBase64(s), "utf-8");
}
Maven / sbt repo: commons-codec, commons-codec, 1.8.

How do I pass json as a query parameter on Rest Post web service in java

How do I pass json as a query parameter on Rest Post web service in java
For example:
https://testvpa.net/WebService/ViALoggingRestWS/ViALoggingService.svc/StartCall?parameter={"machineName":"KK-IVR01","appName":"KKApp","startTime":"2018-02-06T21:38:32","portID":"01","ani":"9189280000","dnis":"8559281111","ctiCallID":"01"}
I am trying something like this:
....
try{
JSONObject obj = new JSONObject();
obj.put("machineName",machineName);
obj.put("appName", appName);
obj.put("startTime", formattedCurrentDate);
obj.put("portID",portID);
obj.put("ani",ani);
obj.put("dnis", dnis);
obj.put("ctiCallID", ctiCallID);
String strobj = obj.toString();
String uri = wsUri+"/StartCall?";
HttpClient client = new HttpClient();
client.getParams().setConnectionManagerTimeout(1300);
client.getParams().setSoTimeout(13000);
PostMethod method = new PostMethod(uri);
method.addRequestHeader("Content-Type", "application/x-www-form-urlencoded");
method.setQueryString("parameter="+strobj );
int statusCode = client.executeMethod(method);
byte[] responseBody = method.getResponseBody();
output = new String(responseBody);
}
....
But I am getting an "Invalid URI" at runtime. It doesn't seem to like the query parameter being a json string. I read somewhere about encoding the json string ... Do I somehow need to encode the json string?

If you are using POST request, you should pass the json object in the body of the request and not in the query params.

You can check this question for more details: Which characters make a URL invalid?
Generally speaking, the accepted characters in a URI are: [A-Z][a-z][0-9]-._~
The following characters are also allowed, but have special meaning in some parts of the URI: :/?#[]#!$&'()*+,;=
Every other character is not allowed and must be percent-encoded. The second set of characters should also be percent-encoded to avoid any parsing problems.
To percent encode a character you take its hex value (e.g. for the space character the hex value is 20) and prefix it with the % character. So John Doe becomes John%20Doe.

Jackson error "Illegal character... only regular white space allowed" when parsing JSON

I am trying to retrieve JSON data from a URL but get the following error:
Illegal character ((CTRL-CHAR, code 31)):
only regular white space (\r, \n,\t) is allowed between tokens
My code:
final URI uri = new URIBuilder(UrlConstants.SEARCH_URL)
.addParameter("keywords", searchTerm)
.addParameter("count", "50")
.build();
node = new ObjectMapper().readTree(new URL(uri.toString())); <<<<< THROWS THE ERROR
The url constructed is i.e https://www.example.org/api/search.json?keywords=iphone&count=50
What is going wrong here? And how can I parse this data successfully?
Imports:
import com.google.appengine.repackaged.org.codehaus.jackson.JsonNode;
import com.google.appengine.repackaged.org.codehaus.jackson.map.ObjectMapper;
import com.google.appengine.repackaged.org.codehaus.jackson.node.ArrayNode;
import org.apache.http.client.utils.URIBuilder;
example response
{
meta: {
indexAllowed: false
},
products: {
products: [
{
id: 1,
name: "Apple iPhone 6 16GB 4G LTE GSM Factory Unlocked"
},
{
id: 2,
name: "Apple iPhone 7 8GB 4G LTE GSM Factory Unlocked"
}
]
}
}

I got this same issue, and I found that it was caused by the Content-Encoding: gzip header. The client application (where the exception was being thrown) was not able to handle this content-encoding. FWIW the client application was using io.github.openfeign:feign-core:9.5.0, and this library appears to have some issues around compression (link).
You might try adding the header Accept-Encoding: identity to your request, however, not all web servers/web applications are configured properly, and some seem to disregard this header. See this question for more details about how to prevent gzipped content.

I had a similar issue. After some research, I found of that restTemplate uses the SimpleClientHttpRequestFactory which does not support gzip encoding. To enable gzip encoding for your response, you will need to set a new request factory for the rest template object - HttpComponentsClientHttpRequestFactory.
restTemplate.setRequestFactory(new HttpComponentsClientHttpRequestFactory());

The message should be pretty self-explanatory:
There is an illegal character (in this case character code 31, i.e. the control code "Unit Separator") in the JSON you are processing.
In other words, the data you are receiving is not proper JSON.
Background:
The JSON spec (RFC 7159) says:
JSON Grammar
A JSON text is a sequence of tokens. The set of tokens includes six
tructural characters, strings, numbers, and three literal names.
[...]
Insignificant whitespace is allowed before or after any of the
six structural characters.
ws = *(
%x20 / ; Space
%x09 / ; Horizontal tab
%x0A / ; Line feed or New line
%x0D ) ; Carriage return
In other words: JSON may contain whitespace between the tokens ("tokens" meaning the part of the JSON, i.e. lists, strings etc.), but "whitespace" is defined to only mean the characters Space, Tab, Line feed and Carriage return.
Your document contains something else (code 31) where only whitespace is allowed, hence is not valid JSON.
To parse this:
Unfortunately, the Jackson library you are using does not offer a way to parse this malformed data. To parse this successfully, you will have to filter the JSON before it is handled by Jackson.
You will probably have to retrieve the (pseudo-)JSON yourself from the REST service, using standard HTTP using, e.g. java.net.HttpUrlConnection. Then suitably filter out "bad" characters, and pass the resulting string to Jackson. How to do this exactly depends on how you use Jackson.
Feel free to ask a separate questions if you are having trouble :-).

I had the same problem. After setting Gzip it was fixed. Please refer my code
public String sendPostRequest(String req) throws Exception {
// Create connection
URL urlObject = new URL(mURL);
HttpURLConnection connection = (HttpURLConnection) urlObject.openConnection();
connection.setRequestMethod("POST");
connection.setRequestProperty("Content-Type", "application/json");
connection.setRequestProperty("Content-Length", Integer.toString(req.getBytes().length));
connection.setRequestProperty("Content-Language", "en-US");
connection.setUseCaches(false);
connection.setDoOutput(true);
// Send request
DataOutputStream wr = new DataOutputStream(connection.getOutputStream());
wr.writeBytes(req);
wr.close();
//Response handling
InputStream responseBody = null;
if (isGzipResponse(connection)) {
responseBody = new GZIPInputStream(connection.getInputStream());
}else{
responseBody = connection.getInputStream();
}
convertStreamToString(responseBody);
return response.toString();
}
protected boolean isGzipResponse(HttpURLConnection con) {
String encodingHeader = con.getHeaderField("Content-Encoding");
return (encodingHeader != null && encodingHeader.toLowerCase().indexOf("gzip") != -1);
}
public void convertStreamToString(InputStream in) throws Exception {
if (in != null) {
ByteArrayOutputStream baos = new ByteArrayOutputStream();
byte[] buffer = new byte[4096];
int length = 0;
while ((length = in.read(buffer)) != -1) {
baos.write(buffer, 0, length);
}
response = new String(baos.toByteArray());
baos.close();
} else {
response = null;
}
}

I had the same issue with zalando logbook in my spring boot application, and after reading the answers in here carefully, I realized, that the response interceptor must be applied after whatever takes care for decompression:
#Configuration
public class RestTemplateConfig {
[....]
#Bean
public RestTemplate restTemplate() {
return new RestTemplateBuilder()
.requestFactory(new MyRequestFactorySupplier())
.build();
}
class MyRequestFactorySupplier implements Supplier<ClientHttpRequestFactory> {
#Override
public ClientHttpRequestFactory get() {
CloseableHttpClient client = HttpClientBuilder.create()
.addInterceptorFirst(logbookHttpRequestInterceptor)
// wrong: .addInterceptorFirst(logbookHttpResponseInterceptor)
.addInterceptorLast(logbookHttpResponseInterceptor)
.build();
HttpComponentsClientHttpRequestFactory clientHttpRequestFactory =
new HttpComponentsClientHttpRequestFactory(client);
return clientHttpRequestFactory;
}
}
}

We had the same issue in out integration tests recently. We have a spring boot application and we use wiremock to mock a integrated microservice server. For one of the test get requests that we had implemented we started getting this error. We had to downgrade wiremock from 2.18.0 to 2.17.0 and it worked fine. Due to some bug the jackson parser and the that particular version of wiremock didn't work together. We didnt have time to figure out what actually the bug was in those libraries.

Those who use FeignClient, please refer to this answer spring-feign-not-compressing-response
Spring is not able to Decode the response on the fly, so you need to define a custom GZip Decoder.
Solved for me.

UTF-8 characters in HTTP POST query form parameters

I am trying to transfer form data from an Android application to a NodeJs server.
My client code is the following (the strings that can contain UTF-8 characters are the values of params):
final HttpPost post = new HttpPost(url);
final MultipartEntityBuilder mpb = MultipartEntityBuilder.create()
.setCharset(Charset.forName("UTF-8")) // tried with or without this line
.setMode(HttpMultipartMode.BROWSER_COMPATIBLE); // tried with or without this line
for (final Entry<String, String> e : params.entrySet()) {
mpb.addTextBody(e.getKey(), e.getValue());
}
post.setEntity(mpb.build());
final HttpClient httpClient = new DefaultHttpClient();
final HttpResponse response = httpClient.execute(request);
And my server code is the following:
app.post('/accesspoint', function(req, res) {
var body = req.body;
var form = new formidable.IncomingForm();
form.encoding = 'utf-8';
form.parse(req, function(err, fields, files) {
console.log(fields);
...
When my input java params has a value containing an UTF-8 character, the log I get server side prints the corresponding value without this character, so it is kind of swallowed at some point. For instance if my input string is "ê", then my server log will print a "" value.
I use a multipart form as I read that it was the best way to send data that can contain non-ASCII characters. Formidable is also apparently the best node package to handle form that can contain UTF-8 characters.
My client side uses Apache HttpClient 4.3.3.
What am I doing wrong?

Ok so I tested with a simple query and the key value ("foo","[ê]") looked at the headers and I saw that my query was still using ISO-8859-1
Content-Disposition: form-data; name="foo"
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit
[]
in contradiction to my builder parameter.
I found the solution via https://stackoverflow.com/a/21020435/592392 and changed my code to:
for (final Entry<String, String> e : params.entrySet()) {
mpb.addTextBody(e.getKey(), e.getValue(), ContentType.create("text/plain", Charset.forName("UTF-8")));
}
And now the server gets the UTF8 chars :)
Anyway, the form builder is quite misleading in my opinion.