Develop Reference

Having queryParam with "+" using UriComponentsBuilder

Based on question Spring's UriComponentsBuilder.queryParam issue, what if a query parameter has a plus ("+") in its value? By default, a plus will not encoded. Why and how can I force this?
In my example, I want to use da ISO-coded timestamp and I need to encode the plus, because the other endpoint will interprete the "+" as a space (" ") and this breaks by timestamp.
UriComponentsBuilder builder = UriComponentsBuilder.fromHttpUrl("https://example.com/api/")//
.queryParam("date1", "2020-10-05T08:20:00.000+02:00") //
.query("date2=2020-10-05T08:20:00.000+02:00");
System.out.println(builder.build().toString());
// produces https://example.com/api/?date1=2020-10-05T08:20:00.000+02:00&date2=2020-10-05T08:20:00.000+02:00
System.out.println(builder.build().encode().toString());
// produces https://example.com/api/?date1=2020-10-05T08:20:00.000+02:00&date2=2020-10-05T08:20:00.000+02:00
The expected result should encode the plus sign like 2020-10-05T08%3A20%3A00.000%2B02%3A00 or at least 2020-10-05T08:20:00.000%2B02:00 with %2B.

You can use DefaultUriBuilderFactory to encode the query values as such:
String baseUrl = "https://example.com";
DefaultUriBuilderFactory uriBuilderFactory = new DefaultUriBuilderFactory(baseUrl);
String uri = uriBuilderFactory.uriString("/api/")
.queryParam("date1", "{date1}")
.build("2020-10-05T08:20:00.000+02:00").toString(); // https://example.com/api/?date1=2020-10-05T08%3A20%3A00.000%2B02%3A00

Cannot get '#' symbol in Controller using Spring #RequestParam

I have the following request Url /search?charset=UTF-8&q=C%23C%2B%2B.
My controller looks like
#RequestMapping(method = RequestMethod.GET, params = "q")
public String refineSearch(#RequestParam("q") final String searchQuery,....
and here i have searchQuery = 'CC++'.
'#' is encoded in '%23' and '+' is '%2B'.
Why searchQuery does not contain '#'?
searchQuery in debug

I resolved a similar problem by URL encoding the hash part. We have Spring web server and mix of JS and VueJS client. This fixed my problem:
const location = window.location;
const redirect = location.pathname + encodeURIComponent(location.hash);

The main cause is known as the "fragment identifier". You find more detail for Fragment Identifier right here. It says:
The fragment identifier introduced by a hash mark # is the optional last part of a URL for a document. It is typically used to identify a portion of that document.
When you write # sign, it contains info for clientbase. Put everything only the browser needs here. You can get this problem for all types of URI characters you can look Percent Encoding for this. In my opinion The simple solution is character replacing, you could try replace in serverbase.

Finally i found a problem.In filters chain ServletRequest is wrapped in XSSRequestWrapper with DefaultXSSValueTranslator and here is the method String stripXSS(String value) which iterates through pattern list,in case if value matches with pattern, method will delete it.
Pattern list contains "\u0023" pattern and '#' will be replaced with ""
DefaultXSSValueTranslator.
private String stripXSS(String value) {
Pattern scriptPattern;
if (value != null && value.length() > 0) {
for(Iterator var3 = this.patterns.iterator(); var3.hasNext(); value = scriptPattern.matcher(value).replaceAll("")) {
scriptPattern = (Pattern)var3.next();
}
}
return value;
}

Issue with Special Characters in Retrofit 2 encoding

So I'm looking to make a request to our api to log in a user, however there is a section that gets encoded in the Retrofit 2 method even though its set as encoded = true. The base url is https://testapi.test.ie The parameter I pass as the serverext is mdc.php?action= However even after setting encoded = true the resulting request body is: https://testapi.test.ie/mdc.php%3Faction=login_user&ts=1482924232742 where I require it to be: https://testapi.test.ie/mdc.php?action=login_user&ts=1482924232742 So I can see the issue is the ? symbol. Below is my retrofit method, if anyone can help with this I would appreciate it in order to achieve the correct
#retrofit2.http.POST("/{serverext}login_user&ts={timestamp}")
#retrofit2.http.Multipart
Call<LoginResponseModel> loginUser(#retrofit2.http.Path(value = "serverext", encoded = true) String server,
#retrofit2.http.Part(Constants.USER) String username,
#retrofit2.http.Part(Constants.PASS) String password,
#retrofit2.http.Path("timestamp") Long timestamp);

You use it incorrect. Path is path, Query is query. You need to rewrite your code to use this separately.
#retrofit2.http.POST("{serverext}")
#FormUrlEncoded
Call<LoginResponseModel> loginUser(#retrofit2.http.Path(value = "serverext", encoded = true) String server,
#retrofit2.http.Field(Constants.USER) String username,
#retrofit2.http.Field(Constants.PASS) String password,
#retrofit2.http.Query("timestamp") Long timestamp,
#retrofit2.http.Query("action") String action);
loginUser("mdc.php", username, pass, 42, "login_user")

You need to use #FormUrlEncoded . And you don't need to include package name in all declarations! just import them! its more neat and clean!
#POST("/{serverext}login_user&ts={timestamp}")
#Multipart
#FormUrlEncoded
Call<LoginResponseModel> loginUser(#Path(value = "server", encoded = true) String server,
#Part(SyncStateContract.Constants.USER) String username,
#Part(SyncStateContract.Constants.PASS) String password,
#Path("timestamp") Long timestamp);

Spring's UriComponentsBuilder.queryParam issue

I have recently switched to Spring for consuming REST API calls hosted by ServiceNow.
I am building my URI as below:
UriComponentsBuilder builder = UriComponentsBuilder.fromHttpUrl(baseUrl.toString());
logger.info("URI before Query Param: " + builder.build().encode().toUri());
builder.queryParam("sysparm_limit", "2000000");
builder.queryParam("sysparm_offset", "0");
builder.queryParam("sysparm_exclude_reference_link", "true");
//this line is the issue because there is a = sign here
builder.queryParam("sysparm_query=user_name", snUser.getUser_name());
logger.info("URI after Query Param: " + builder.build().encode().toUri());
The output of this code is:
INFO: URI before Query Param: https://sandbox.service-now.com/api/now/v1/table/sys_user
INFO: URI after Query Param: https://sandbox.service-now.com/api/now/v1/table/sys_user?sysparm_limit=2000000&sysparm_offset=0&sysparm_exclude_reference_link=true&sysparm_query%3Duser_name=AX0011
The problem is with the final builder.queryParam. I am getting output as this:
sysparm_query%3Duser_name=AX0011
but what I want is:
sysparm_query=user_name=AX0011
So that eventually the final URI looks like this:
INFO: URI after Query Param: https://sandbox.service-now.com/api/now/v1/table/sys_user?sysparm_limit=2000000&sysparm_offset=0&sysparm_exclude_reference_link=true&sysparm_query=user_name=Z001NR6
So I tried replacing,
builder.queryParam("sysparm_query=user_name", snUser.getUser_name());
by:
builder.query("sysparm_query=user_name=" + snUser.getUser_name());
which changed the original output from:
INFO: URI after Query Param: https://sandbox.service-now.com/api/now/v1/table/sys_user?sysparm_limit=2000000&sysparm_offset=0&sysparm_exclude_reference_link=true&sysparm_query%3Duser_name=Z001NR6
to:
INFO: URI after Query Param: https://sandbox.service-now.com/api/now/v1/table/sys_user?sysparm_limit=2000000&sysparm_offset=0&sysparm_exclude_reference_link=true&sysparm_query=user_name%3DZ001NR6
Notice how sysparm_query%3Duser_name=Z001NR6 changed to sysparm_query=user_name%3DZ001NR6
Is ther anyway to see a = instead of %3D in the output?

The param looks quite strange - however - you can add it manually using the UriComponentsBuilder#query method:
UriComponentsBuilder builder = UriComponentsBuilder
.fromHttpUrl("https://example.com/api/")
.queryParam("param1", "12345")
.queryParam("param2", "abc")
.query("query=username=JOE");
System.out.println(builder.build().toString());
// produces https://example.com/api/?param1=12345&param2=abc&query=username=JOE
System.out.println(builder.build().encode().toString());
// produces https://example.com/api/?param1=12345&param2=abc&query=username%3DJOE
Manual concatenation:
UriComponentsBuilder builder = UriComponentsBuilder
.fromHttpUrl("https://example.com/api/")
.queryParam("param1", "12345")
.queryParam("param2", "abc");
// the parameter has to be properly url-encoded manually (not shown here)
String uri = builder.build().encode().toString() + "&query=username=JOE";
System.out.println(uri);
// produces: https://example.com/api/?param1=12345&param2=abc&query=username=JOE

The query component of a URL is frequently used to carry information in key=value pairs; you could think of this as a Map<String, String>. In this case, = and & are special characters that delimit these pairs, and they must be encoded when they form part of the key or the value to ensure that anything reading the query string this way is able to parse it properly.
In your case, how you use the builder depends on how you would want to retrieve your data later on. There are two options:
// Building the URL:
builder.queryParam("sysparm_query=user_name", snUser.getUser_name());
// URL contains ...&sysparm_query%3Duser_name=AX0011
// Reading the parsed query map:
Map<String, String> query = ...
String data = query.get("sysparm_query=user_name");
// value is AX0011
Or
// Building the URL:
builder.queryParam("sysparm_query", "user_name=" + snUser.getUser_name());
// URL contains ...&sysparm_query=user_name%3DAX0011
// Reading the parsed query map:
Map<String, String> query = ...
String value = query.get("sysparm_query");
// value is user_name=AX0011
In a correctly encoded URL, one of the = will always be encoded as %3D. Using a UriComponentsBuilder ensures that your URLs will be correctly encoded and that anything reading your URLs will be able to do so properly without data loss.

How can I set pageToken to get item lists from Google Cloud Storage via Java SDK?

I want to set pageToken to get items stored at Google Cloud Storage. I'm using Google API Client Library for Java v1.19.x.
I have no idea to generate pageToken from file path(or file name).
2 files stored in bucket.
my-bucket
/test.csv
/test2.csv
When I tried Google APIs Explorer with following parameters, I could get nextPageToken Cgh0ZXN0LmNzdg==.
And I found out that I can get test.csv string by decoding nextPageToken with base64.
bucket: my-bucket
pageToken:
prefix: test
maxResults: 1
{"kind": "storage#objects", "nextPageToken": "Cgh0ZXN0LmNzdg==", ...}
But How can I get Cgh0ZXN0LmNzdg== from test.csv?
Although I tried Base64 encoding, result didn't match.
import com.google.api.client.repackaged.org.apache.commons.codec.binary.Base64;
String lastFile = "test.csv"
String token = Base64.encodeBase64String(lastFile.getBytes());
String bucket = "my-bucket"
String prefix = "test"
Storage.Objects.List listObjects = client.objects().list(bucket);
listObjects.setPrefix(prefix);
listObjects.setPageToken(token);
long maxResults = 1;
listObjects.setMaxResults(maxResults);
do {
Objects objects = listObjects.execute();
List<StorageObject> items = objects.getItems();
token = objects.getNextPageToken();
listObjects.setPageToken(token);
} while (token != null);

I could get next token from file path string using following codes by myself.
How to get nextToken from path string
String nextToken = base64encode(0x0a + asciiCode + pathString)
asciiCode can be taken between 0x01(SOH) and 0x7f(DEL). It seems to depend on path length.
my-bucket/
a/a(3byte) 0x03
a/ab(4byte) 0x04
test.txt(8byte) 0x08
Notice
If path length is longer than 1024 byte, another rule seems to apply. But I couldn't found out rules.
See also Object Name Requirements
import com.google.common.io.BaseEncoding;
String lastFile = "test.csv"
String token = base64Encode(lastFile);
String bucket = "my-bucket"
String prefix = "test"
Storage.Objects.List listObjects = client.objects().list(bucket);
listObjects.setPrefix(prefix);
listObjects.setPageToken(token);
long maxResults = 1;
listObjects.setMaxResults(maxResults);
do {
Objects objects = listObjects.execute();
List<StorageObject> items = objects.getItems();
token = objects.getNextPageToken();
listObjects.setPageToken(token);
} while (token != null);
private String base64Encode(String path) {
byte[] encoding;
byte[] utf8 = path.getBytes(Charsets.UTF_8);
encoding = new byte[utf8.length + 2];
encoding[0] = 0x0a;
encoding[1] = new Byte(String.valueOf(path.length()));
String s = BaseEncoding.base64().encode(encoding);
return s;
}

I know this question is already answered and is applied to Java, I'd like to mention that this question applies to PHP as well.
With the help of the approved post from sakama above I figured out a PHP version of his solution.
The PHP equivalent for generating the token is as follow:
base64_encode(pack('c', 0x0a) . pack('c', $path_string_length) . pack('a*', $path_string));
The byte pattern seems indeed (as sakama already mentioned) to be:
<line feed><line data length><line data>