I am trying to use the code below to send across a json string as a query param using Spring WebClient.
var client = WebClient.builder()
.baseUrl(baseurl)
.defaultHeader(HttpHeaders.CONTENT_TYPE, MediaType.APPLICATION_JSON_VALUE)
.build()
client.post().uri(uriBuilder -> uriBuilder.path("/api/SomeRequest")
.queryParam("response", responseJsonStr).build()
).retrieve().bodyToMono(ReponseObj.class).block()
I am seeing a error
java.lang.IllegalArgumentException: Not enough variables available to expand questionKey.
From the searching I have done, I can tell the issue is around the the Json String I am supplying as a query parameter(It has '{' and '}'). I think this might have to do with URI encoding. But looking at the details around encoding , I am not sure what the best way to approach this is.
NOTE: I can't make changes to the server that is expecting the Json string as a query param(it later decodes the json after extracting the query param).
Looks like you know that it's a really bad idea to send a json string as query param and not as post body, so I will not go into it any furhter.
You were right. You have to encode the query parameter on the client side to make it work. You can simply encode the param like this:
.queryParam("response", URLEncoder.encode(responseJsonStr, StandardCharsets.UTF_8))
Just make sure that you use the same character set as the server side.
Related
I am trying to complete the Instagram Oauth flow,
I currently have the authorization code which I'm to exchange for the access token. I am to make an x-www-form-urlencoded POST request to this endpoint
"https://api.instagram.com/oauth/access_token?"
This is what I've done so far.
String query = "https://api.instagram.com/oauth/access_token/?client_id=" + clientId +"&client_secret="+ clientSecret+ "&grant_type=authorization_code&redirect_uri="+ redirectUri + "&code=" + code
String response = new URL(query).getText()
A JSON string is expected as response.
Please Keep in mind that I'm a beginner.
I haven't read the Instagram documentation but based on your example code there's a couple of things to keep in mind:
you mentioned that you have to make a POST request, your example makes a GET request
never build a URL with untrusted parameter values. This basically means: always encode parameters, never trust them.
There are dozens of 3rd party HTTP Request libraries that give you flexibility and easier insight into aspects like timeouts and redirects. Java 11 has a built-in HTTP client that might ease this as well. But building on your code provided in your question using basic Java connection primitives this might work:
URL url = new URL("https://api.instagram.com/oauth/access_token/?client_id=${URLEncoder.encode(clientId, 'UTF-8')}&client_secret=${URLEncoder.encode(clientSecret, 'UTF-8')}&grant_type=authorization_code&redirect_uri=${URLEncoder.encode(redirectUri, 'UTF-8')}&code=${URLEncoder.encode(code, 'UTF-8')}")
def jsonString = ((HttpURLConnection) url.openConnection()).with {
setRequestMethod('POST')
setRequestProperty('Accept', 'application/json')
setDoInput(true)
connect()
if (getResponseCode() >= 400)
throw new Exception("Error code = ${getResponseCode()}")
inputStream.text
}
Every URL parameter is encoded so that any non-URL safe characters they contain are made safe, then we tell the connection that it will be a 'POST' and that we expect to get back json as input. inputStream.text is groovy code that takes an inputstream from the connection and reads all of the contents and then closes the stream. Since it is the last line of the with closure it is automatically returned as the value of the closure and assigned to the variable jsonString.
I am creating a REST service in Java ,and have a doubt with regards to params for the GET method .
I have to pass the below params in a GET request
Function
"GET" File status :
Params:
Time Range:(String)
FlowId:(String)
ID_A= or ID_B= or Both (String)
IS_ADD_A= or IS_ADD_B= or both (String)
Regex=(String)
Cookie=XXXXX
So as there are 6 params,so passing it as a query string would not be an efficient way and can't but the same in body(as it is against the HTTP GET specification)
Making this as a POST call would be against the REST principle as I want to get data from the server ,
What would be an efficient way of solving this ,would passing the params as query string is out of question,passing it in body which is against the HTTP spec ,making this as headers which may also be not good ,making this as POST request which will voilate the fielding's REST principle .
Passing data in the body of an HTTP GET call is not only against the spec but causes problems with various server-side technologies which assume you don't need access to the body in a GET call. (Some client side frameworks also have some issues with GET and a query in the body) If you have queried with long parameters I'd go with POST. It's then using POST for getting data but you'd not be the only one having to go this way to support potentially large queries.
If your parameters values aren't very long, using query string is your best option here. 6 params is not a lot, as long you don't exceed the IE limit of characters in the path - 2,048 (http://www.boutell.com/newfaq/misc/urllength.html). For example Google search engine uses many more params then 6. If there is a possibility that the URL path will exceed the limit above, you should use POST instead.
I have a rest application that can export some report data from Elasticsearch. It is easy to do with the Java API:
SearchResponse response = getClient()
.prepareSearch("my_index_name")
.setQuery(QueryBuilders.someQuery())
.addAggregation(AggregationBuilders.someAggregation())
.get();
The problem starts with the big responses. Using this code snippet, the response is read to build the SearchResponse object in memory. In my case, the response does not fits in memory.
Paging cannot help because we often need to return the full data and Aggregations do not support paging yet.
I know that I can use the Elasticsearch REST API to read the response as stream, but manually build the request it is cumbersome. I really want something like this:
// my dream API
InputStream response = getClient()
.prepareSearch("my_index_name")
.setQuery(QueryBuilders.someQuery())
.addAggregation(AggregationBuilders.someAggregation())
.getStream();
So, can the Elasticsearch Java API stream the SearchResponse?
A proposal for streaming results does exist but it doesn't seem to have picked up steam so far and was closed (for now).
There's a way to do it with XContentBuilder but that still requires the whole response to be in memory before being sent.
It might not be what you want, but that's the closest thing that I know which could fulfill your need. Worth giving it a try.
I believe there is no way to obtain an InputStream from the Java API (but I might be wrong). I also think there is no way to directly obtain an InputStream in Jest (a REST-based Elasticsearch Java API).
You mention that it is cumbersome to create the search request to the _search endpoint yourself: if you're referring to building the actual json query, I just would like to point out that once you have a SearchSourceBuilder, you can call toString() on it to get a fully working json representation of your query.
SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
sourceBuilder.query(this.getQuery())
.from(this.getFrom())
.size(this.getSize())
.fetchSource(this.getSource(), null);
this.getSort().forEach(sourceBuilder::sort);
sourceBuilder.toString() // the json representation
I am communicating with a web service that expects a POST parameter and also expect Request body. I have confirmed that such a POST request can be done using a REST Console I have, but I am unable to make such a request in Java using Apache libraries.
In the code below, I am able to POST to the web service, and it correctly receives the contents of the variable raw_body. If I uncomment the first of the two commented lines, the web service receives the "fname" parameter, but it no longer receives the body of the POST.
import org.apache.commons.httpclient.HttpClient;
import org.apache.commons.httpclient.methods.PostMethod;
import org.apache.commons.httpclient.methods.RequestEntity;
...
HttpClient httpClient = new HttpClient();
String urlStr = "http://localhost:8080/MyRestWebService/save";
PostMethod method = new PostMethod(urlStr);
String raw_body = "This is a very long string, much too long to be just another parameter";
RequestEntity re = new StringRequestEntity(raw_body, "text/xml", "UTF-16");
//method.addParameter("fname", "test.txt");
//httpClient.getParams().setParameter("fname", "test.txt");
method.setRequestEntity(re);
How can I transmit both the parameter and the body?
You could use the setQueryString method to add the parameters to the URL that is being POSTed to. From a RESTful perspective I'd argue you should normally not be doing that, however, since a POST should represent a call to a resource and anything that would qualify for a query parameter should be included in the representation that is being transferred in the request body...or it should represent qualification of the resource itself in which case it should be part of the path that is posted to which could then be extracted by the controller using #PathVariable/#PathParam or something similar. So in your case you could also be looking for something like POST /MyRestWebService/files/test.txt or more fittingly a PUT if you're saving the resource and know the URI. The code on the server could pull the filename out from a URL pattern.
You need to make a POST request using multipart-form. Here is the example:
Apache HttpClient making multipart form post
Alternatively, you can make a POST request with the content (parameters and files) encoded using application/x-www-form-urlencoded but it is not recommended when you want to make a POST request with large content, like files.
We have already shipped a client (.NET WinForms) application which sends customer data to Java server. While most of the data sent by client are accepted at server side, some records are truncated because of the presence of & character in it, as client sends raw & and do not URL encode it, we have fixed it by using the below code:
string dataBefore="A & B";
string dataBefore = System.Web.HttpUtility.UrlEncode(dataBefore);
It is impossible for us to update all the client applications(which are already shipped) and we are thinking of a server side fix.
With the help of Fiddler, we have made sure the data has left client in full, but when server reads as below:
//in java
String dataReceied=request.getParameter("data");
it gets truncated if data contains &
Could someone help us suggesting a server side(java) fix for this? Is it possible to access the request stream in java(instead of request.getParameter())?
You can get access to the raw query string using HttpServletRequest.getQueryString() (javadoc), which:
returns a String containing the query string or null if the URL contains no query string. The value is not decoded by the container.
You can them perform manual decoding on that string, instead of using getParameter().
#Wesley's idea of using getParameterMap() may not be useful, because you don't know which order the parameters were supplied in.
I'd suggest implementing this logic as a servlet filter, to decouple the fixing of the broken parameters from your actual servlet logic. This would involve writing a custom subclass of HttpServletRequestWrapper which overrides getParameter() and manuyally decodes the query string. Your servlet would then be able to use the HttpServletrequest API as though everything was tickety boo.
It is cut off because & signifies a new URL parameter in a request like this:
google.com?query=java&page=2. Java converts all these parameters to a Map, so that's where it goes wrong.
Have you tried iterating through request.getParameterMap()? The remaining data is most likely in the name of the next parameter. If that does not work, check out the API of HTTPServletRequest to see if there is another way to get your data.
Good luck!
PS How angry are you guys at the intern that wrote & shipped that client? That sounds messed up!