Develop Reference

java.net.URISyntaxException: Illegal character in query at index on JMETER

I have an application for which I have recorded Jmeter scripts to conduct load testing. Authorization happens via Azure AD.
I have co-related the auth-related such as access tokens, refresh tokens and id tokens that are generated dynamically and also parameterized them in a request that is responsible for calling the API and which would probably require those tokens to authorize the call to the API.
However, I get an error:-
java.net.URISyntaxException: Illegal character in query at index 98:
at java.base/java.net.URI$Parser.fail(URI.java:2938)
at java.base/java.net.URI$Parser.checkChars(URI.java:3109)
at java.base/java.net.URI$Parser.parseHierarchical(URI.java:3197)
at java.base/java.net.URI$Parser.parse(URI.java:3139)
at java.base/java.net.URI.<init>(URI.java:623)
at java.base/java.net.URL.toURI(URL.java:1063)
at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.sample(HTTPHC4Impl.java:615)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy.sample(HTTPSamplerProxy.java:66)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1281)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1270)
at org.apache.jmeter.threads.JMeterThread.doSampling(JMeterThread.java:630)
at org.apache.jmeter.threads.JMeterThread.executeSamplePackage(JMeterThread.java:558)
at org.apache.jmeter.threads.JMeterThread.processSampler(JMeterThread.java:489)
at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:256)
at java.base/java.lang.Thread.run(Thread.java:832)
I am not sure where I am going wrong. I have co-related 3 dynamic tokens which are generated as part of the auth token request.
Screenshots in the below link:-
Regular Expression Extractor
Setting parameters
Error on Jmeter

I faced this issue. What I did wrong, was that I did not put the "$" sign before the parameters in the URL.
How I resolved the issue:
When I got the exception, I counted the characters in the URL till the index (given in the exception) which made me realize that something is wrong with the given index, then I checked the tutorial which I was following, and Hoorah! the issue has been found.
Example Image

I believe these "tokens" should go into the HTTP Header Manager as placing them as request parameters adds them to the request URL.
If your application expects the tokens to be a part of the URL (which is kind of strange) then you either need to tick the URL Encode? boxes or wrap the values which can contain the characters requiring encoding into __urlencode() function

How to handle curly braces in RequestParam in spring boot

I have a spring boot application with a GET service.
#RequestMapping(value = "/abc/track/{type}", method = RequestMethod.GET)
public void DummFunc(
#RequestParam(value="subs", required = false) String sub,
, HttpServletResponse response) {}
value for subs is an encoded value.
If I pass following as value to parameter subs
{%22endpoint%22:%22https://def.abc.com/tyu/send/eD3vpGNQW28:APA91bHOo8rYrV0xccdQz3okjZJG-QGrJX8LG6ahJnEUpMNfGedqi3hJxQsJx_8BMbH6oDjaSPPEXqzNWchWrGSkhuTkSdemikkys1U22Ipd7MsRw0owWbw89V2fslIVAJ6G5lkyvYuQ%22,%22expirationTime%22:null,%22keys%22:{%22p256dh%22:%22BK0pktn50CMsTQtqwdPlKlJtdFs0LFeXX14T1zgoz6QWnvSTp5dxtChnUP5P1JX0TsjcopbdPKyN31HfABMUyic%22,%22auth%22:%22qbO_z0vGao9wD-E5g8VU-A%22}}
It fails to capture the request and control does not come inside of the function.
If we instead pass as value to parameter subs:
%7B%22endpoint%22:%22https://def.abc.com/tyu/send/dX5q5eV7hFQ:APA91bHib-0QXrMzjatcvTR_uaIeeJK8lf6GmXUC9Jxv0Oxth-BzD4GmWnd4-YpDZv8qSFZ0eSg9mB2YkRvkc5ezdXW5KeaHjuQZfdyDxyBXjJgE-25Xbtlk37pdm8vfLk20k0k_VxW9%22,%22expirationTime%22:null,%22keys%22:%7B%22p256dh%22:%22BCCvcBLpRqp4u0auP688_MUJLsAiwWlQSn5kpyl8YVsEo_J-KpSdnhCmVIE_BhDXBcFYflPK52hqhYf3EaOCyuY%22,%22auth%22:%22iKuW_ESkCZnubWcQu_JK8w%22%7D%7D
It works fine.
Why is this happening? What's wrong with first encoding?
Since server is not able to handle the request, it returns 400. I need to capture such requests and then handle them by encoding them properly. What can be way forward?
I am new to Spring boot/Spring and Java itself. Would be great if I can get some insight.
Also, I can decode both of them online here without any issues- https://www.urldecoder.org/
Edit: Basically, the request that has issue getting through has { and } instead of %7Band %7D.
My question is instead of application failing with 400 bad request,how do I capture such requests in my app, encode them properly and then process them.

This is not related to Java nor the Spring itself but the HTML URL Encoding Reference. URLs can only be sent over the Internet using the ASCII character set.
The unsafe characters are defined in the beginning of RFC-1738 and here is the list:
" < > # % { } | \ ^ ~ [ ] ` including the blank/empty space.
Aside from those, there are also reserved characters where belong the following ones and are used to distinguish the parameters, the key-value representation, the port etc.
; / ? : # = &
The unsafe characters you have used are { and } which are equal to %7B and %7D.
Essentially, you should not be concerned about the data the client sends you in the way you describe. The server must demand the correct form and URL passed. Although, the browsers and REST clients encode those characters automatically, sending them programmatically might cause errors. The only two available solutions in Spring I am aware of is through registering the CharacterEncodingFilter bean (already answered) or the Spring-Boot configuration:
spring.http.encoding.charset=UTF-8
spring.http.encoding.enabled=true
spring.http.encoding.force=true
You need to enable the encoding first and force on HTTP requests and responses.

spring-boot is very much concerned about security. Adding double quote / single quotes or either escaping won't work I guess.
Please go through : https://www.rfc-editor.org/rfc/rfc1738
I think you should try the manual encoding { to %7B and } to %7D
Unsafe:
Characters can be unsafe for a number of reasons. The space
character is unsafe because significant spaces may disappear and
insignificant spaces may be introduced when URLs are transcribed or
typeset or subjected to the treatment of word-processing programs.
The characters "<" and ">" are unsafe because they are used as the
delimiters around URLs in free text; the quote mark (""") is used to
delimit URLs in some systems. The character "#" is unsafe and should
always be encoded because it is used in World Wide Web and in other
systems to delimit a URL from a fragment/anchor identifier that might
follow it. The character "%" is unsafe because it is used for
encodings of other characters. Other characters are unsafe because
gateways and other transport agents are known to sometimes modify
such characters. These characters are "{", "}", "|", "", "^", "~",
"[", "]", and "`".
All unsafe characters must always be encoded within a URL. For
example, the character "#" must be encoded within URLs even in
systems that do not normally deal with fragment or anchor
identifiers, so that if the URL is copied into another system that
does use them, it will not be necessary to change the URL encoding.

Generally this issue comes when the APP server like Undertow, tomcat, Jboss which is hosting your Springboot application does not allow some special characters, which can very from Server to server. This is done to secure URLs and no one can send some special characters to the server to compromise with its functionality.
If still want to allow special characters so that those can reach to controller in Springboot application, you need to allow this configuration in APP server. e.g. this is the configuration required in Undertow server to allow special characters in URL:
#Configuration
public class HttpConfig {
#Bean
public UndertowServletWebServerFactory servletWebServerFactory() {
UndertowServletWebServerFactory factory = new UndertowServletWebServerFactory();
factory.addBuilderCustomizers((UndertowBuilderCustomizer) builder ->
builder.setServerOption(UndertowOptions.ALLOW_UNESCAPED_CHARACTERS_IN_URL, Boolean.TRUE));
return factory;
}
}
In order to make springboot work well with UTF-8 encoding/decoding of rest URLs, add this configuration in application.properties file, for springboot version 2.6:
server.servlet.encoding.charset=UTF-8
server.servlet.encoding.enabled=true
server.servlet.encoding.force=true
server.servlet.encoding.force-request=true
server.servlet.encoding.force-response=true
Best approach is to encode special characters in the client application which is calling this rest URL to comply with security principles

Due to Tomcat upgraded their security in URL param.
Please find possible solution
https://github.com/bohnman/squiggly/issues/42#issuecomment-386658525

Getting unescaped servlet path from a HttpRequest object

I have a custom proxy servlet that has to deal with URL-s that contain special characters (e.g. ; , . / in their) in their path. This is because it is a RESTful application that has ugly path params by design. (Don't comment it as it is not mine.)
My client, (actually wget, because browsers tend to show unescaped the URL) send a request to this URL:
http://localhost:8080/MyApplication/proxy/foo/ugly%3Apart%2Fcomes%3Bhere/children
//note: %2F = '/', %3A = ':', %3B = ';'
In my servlet (mapped to /proxy/*) when I try to forward the GET request, I am unable to reconstruct it because HttpRequest.getPathInfo() returns me the URL unescaped:
http://localhost:8080/MyApplication/proxy/foo/ugly:part/comes;here/children
And therefore the information of which /s and ;s were originally escaped or unescaped is lost. And that makes a difference for me, for example ; makes my URL a so called matrix URL, see http://www.w3.org/DesignIssues/MatrixURIs.html, or all the REST path parameters get shifted by slashes.
Actually I found this issue on a Glassfish server, so I'm not sure if different application servers treat this differently or not. I found only this in the Servlet API:
getPathInfo() Returns any extra path information associated with the
URL the client sent when it made this request.
How could I get the original, unescaped request URL that was sent by the client?

Have a look at HttpServletRequest's getRequestURI() and getRequestURL() methods.
If you need to remove context and servlet mappings, look at getContextPath() and getServletPath().

Is there a way to intentionaly ignore RFC-2396 in constructing a URI

I'm trying to construct a java.net.URI with illegal characters as the server is a piece of old hardware, not being able to decode RFC-2396 complients URIs. This server demands a http-get with a URI like this
http://192.168.5.10/cgi-bin/Read.exe?+0+ABC/info.xml+<g>Template</g>
?+0+ABC/info.xml+<g>Template</g> is the illegal part of the URI and I'll get an java.net.MalformedURLException Exception.
Decoding this part of the URI to %2B0%2BABC%2Finfo.xml%2B%3Cg%3ETemplate%3C%2Fg%3E get's me a server exception.
I'm wondering if there is a way to get non RFC-2396 complient java.net.URIs on the wire?

The only way to do this would be to not use URLConnection at all, and just write the HTTP request by hand.
If the hardware you're connecting to is determined, so you're pretty sure on what headers to expect, this should not be that difficult; HTTP is a relatively easy protocol.
I did this once myself :)

Safe Data serialization for Plain HTTP GET & POST communication

I'm using the client's browser to submit HTTP request.
For report generation the securityToken is submitted as POST, for report download the same token needs to be submitted by the user browser, this time using GET.
What encoding would you recommend for the securityToken which actually represents encrypted data.
I've tried BASE64 but this fails because the standard can include the "+" character which gets translated in HTTP GET to ' ' (blank space).
Then I tried URL Encoding, but this fails because for HTTP POST stuff such as %3d are transmitted without translation but when browser does HTTP GET with the data %3d is converted to '='.
What encoding would you recommend, to allow safe transmission over HTTP POST & GET without data being misinterpreted.
The environment is Java, Tomcat.
Thank you,
Maxim.

Hex string.
Apache commons-codec has a Hex class that provides this functionality.
It will look like this:
http://youraddress.com/context/servlet?param=ac7432be432b21

Well, you can keep the Base64 and use this solution:
Code for decoding/encoding a modified base64 URL