Develop Reference

How to include a ? in Java 11 HTTP Client URL? [duplicate]

Say I have a URL
http://example.com/query?q=
and I have a query entered by the user such as:
random word £500 bank $
I want the result to be a properly encoded URL:
http://example.com/query?q=random%20word%20%A3500%20bank%20%24
What's the best way to achieve this? I tried URLEncoder and creating URI/URL objects but none of them come out quite right.

URLEncoder is the way to go. You only need to keep in mind to encode only the individual query string parameter name and/or value, not the entire URL, for sure not the query string parameter separator character & nor the parameter name-value separator character =.
String q = "random word £500 bank $";
String url = "https://example.com?q=" + URLEncoder.encode(q, StandardCharsets.UTF_8);
When you're still not on Java 10 or newer, then use StandardCharsets.UTF_8.toString() as charset argument, or when you're still not on Java 7 or newer, then use "UTF-8".
Note that spaces in query parameters are represented by +, not %20, which is legitimately valid. The %20 is usually to be used to represent spaces in URI itself (the part before the URI-query string separator character ?), not in query string (the part after ?).
Also note that there are three encode() methods. One without Charset as second argument and another with String as second argument which throws a checked exception. The one without Charset argument is deprecated. Never use it and always specify the Charset argument. The javadoc even explicitly recommends to use the UTF-8 encoding, as mandated by RFC3986 and W3C.
All other characters are unsafe and are first converted into one or more bytes using some encoding scheme. Then each byte is represented by the 3-character string "%xy", where xy is the two-digit hexadecimal representation of the byte. The recommended encoding scheme to use is UTF-8. However, for compatibility reasons, if an encoding is not specified, then the default encoding of the platform is used.
See also:
What every web developer must know about URL encoding

I would not use URLEncoder. Besides being incorrectly named (URLEncoder has nothing to do with URLs), inefficient (it uses a StringBuffer instead of Builder and does a couple of other things that are slow) Its also way too easy to screw it up.
Instead I would use URIBuilder or Spring's org.springframework.web.util.UriUtils.encodeQuery or Commons Apache HttpClient.
The reason being you have to escape the query parameters name (ie BalusC's answer q) differently than the parameter value.
The only downside to the above (that I found out painfully) is that URL's are not a true subset of URI's.
Sample code:
import org.apache.http.client.utils.URIBuilder;
URIBuilder ub = new URIBuilder("http://example.com/query");
ub.addParameter("q", "random word £500 bank \$");
String url = ub.toString();
// Result: http://example.com/query?q=random+word+%C2%A3500+bank+%24

You need to first create a URI like:
String urlStr = "http://www.example.com/CEREC® Materials & Accessories/IPS Empress® CAD.pdf"
URL url = new URL(urlStr);
URI uri = new URI(url.getProtocol(), url.getUserInfo(), url.getHost(), url.getPort(), url.getPath(), url.getQuery(), url.getRef());
Then convert that URI to an ASCII string:
urlStr = uri.toASCIIString();
Now your URL string is completely encoded. First we did simple URL encoding and then we converted it to an ASCII string to make sure no character outside US-ASCII remained in the string. This is exactly how browsers do it.

Guava 15 has now added a set of straightforward URL escapers.

The code
URL url = new URL("http://example.com/query?q=random word £500 bank $");
URI uri = new URI(url.getProtocol(), url.getUserInfo(), IDN.toASCII(url.getHost()), url.getPort(), url.getPath(), url.getQuery(), url.getRef());
String correctEncodedURL = uri.toASCIIString();
System.out.println(correctEncodedURL);
Prints
http://example.com/query?q=random%20word%20%C2%A3500%20bank%20$
What is happening here?
1. Split URL into structural parts. Use java.net.URL for it.
2. Encode each structural part properly!
3. Use IDN.toASCII(putDomainNameHere) to Punycode encode the hostname!
4. Use java.net.URI.toASCIIString() to percent-encode, NFC encoded Unicode - (better would be NFKC!). For more information, see: How to encode properly this URL
In some cases it is advisable to check if the URL is already encoded. Also replace '+' encoded spaces with '%20' encoded spaces.
Here are some examples that will also work properly
{
"in" : "http://نامه‌ای.com/",
"out" : "http://xn--mgba3gch31f.com/"
},{
"in" : "http://www.example.com/‥/foo",
"out" : "http://www.example.com/%E2%80%A5/foo"
},{
"in" : "http://search.barnesandnoble.com/booksearch/first book.pdf",
"out" : "http://search.barnesandnoble.com/booksearch/first%20book.pdf"
}, {
"in" : "http://example.com/query?q=random word £500 bank $",
"out" : "http://example.com/query?q=random%20word%20%C2%A3500%20bank%20$"
}
The solution passes around 100 of the test cases provided by Web Platform Tests.

Using Spring's UriComponentsBuilder:
UriComponentsBuilder
.fromUriString(url)
.build()
.encode()
.toUri()

The Apache HttpComponents library provides a neat option for building and encoding query parameters.
With HttpComponents 4.x use:
URLEncodedUtils
For HttpClient 3.x use:
EncodingUtil

Here's a method you can use in your code to convert a URL string and map of parameters to a valid encoded URL string containing the query parameters.
String addQueryStringToUrlString(String url, final Map<Object, Object> parameters) throws UnsupportedEncodingException {
if (parameters == null) {
return url;
}
for (Map.Entry<Object, Object> parameter : parameters.entrySet()) {
final String encodedKey = URLEncoder.encode(parameter.getKey().toString(), "UTF-8");
final String encodedValue = URLEncoder.encode(parameter.getValue().toString(), "UTF-8");
if (!url.contains("?")) {
url += "?" + encodedKey + "=" + encodedValue;
} else {
url += "&" + encodedKey + "=" + encodedValue;
}
}
return url;
}

In Android, I would use this code:
Uri myUI = Uri.parse("http://example.com/query").buildUpon().appendQueryParameter("q", "random word A3500 bank 24").build();
Where Uri is a android.net.Uri

In my case I just needed to pass the whole URL and encode only the value of each parameters.
I didn't find common code to do that, so (!!) so I created this small method to do the job:
public static String encodeUrl(String url) throws Exception {
if (url == null || !url.contains("?")) {
return url;
}
List<String> list = new ArrayList<>();
String rootUrl = url.split("\\?")[0] + "?";
String paramsUrl = url.replace(rootUrl, "");
List<String> paramsUrlList = Arrays.asList(paramsUrl.split("&"));
for (String param : paramsUrlList) {
if (param.contains("=")) {
String key = param.split("=")[0];
String value = param.replace(key + "=", "");
list.add(key + "=" + URLEncoder.encode(value, "UTF-8"));
}
else {
list.add(param);
}
}
return rootUrl + StringUtils.join(list, "&");
}
public static String decodeUrl(String url) throws Exception {
return URLDecoder.decode(url, "UTF-8");
}
It uses Apache Commons' org.apache.commons.lang3.StringUtils.

Use this：
URLEncoder.encode(query, StandardCharsets.UTF_8.displayName());
or this:
URLEncoder.encode(query, "UTF-8");
You can use the following code.
String encodedUrl1 = UriUtils.encodeQuery(query, "UTF-8"); // No change
String encodedUrl2 = URLEncoder.encode(query, "UTF-8"); // Changed
String encodedUrl3 = URLEncoder.encode(query, StandardCharsets.UTF_8.displayName()); // Changed
System.out.println("url1 " + encodedUrl1 + "\n" + "url2=" + encodedUrl2 + "\n" + "url3=" + encodedUrl3);

How to create this url in java

I have this url to create in java. The parameters base-url, US, and test will come from config file, while part1 and part2 will be variables in the program. So the skeleton is something like.
https://base-url/search/part1/part2?region=US&env=test
Currently i am using an inefficient way to create this url using strings concatenations
AppConfig.findString(baseurlstring) + "/search/" + part1 + "/"
+ part2 + "?region=" + AppConfig.findString(regionString)
+ "&env=" + AppConfig.findString(envString);
Is there a more efficient to do it using the URL class, and then convert it to a string?

StringBuilder is the way to code in this case.
For example:
StringBuilder builder = new StringBuilder()
Any time you want to build or add to the URL,
For example, use: builder.append(AppConfig.findString(regionString))
In the end you reach the URL you want to build by calling append() repeatedly.
Now the reason to do this, is because StringBuilder is faster than StringBuffer but as a caveat it is non-synchronous (not thread safe).
Read the StringBuilder documentation.

StringBuilder is one way but it still results in rather verbose code.
StringBuilder builder = new StringBuilder();
builder.append(AppConfig.findString(baseurlstring))
.append("/search/")
.append(part1)
.append("/")
.append(part2)
.append("?region=")
.append(AppConfig.findString(regionString))
.append("&env=")
.append(AppConfig.findString(envString));
Another way is to use a java.lang.String::format() as in
String fmt = "https://%s/search/%s/%s?region=%s&env=%s");
...
String url = String.format(fmt,
AppConfig.findString(baseurlstring),
part1,
part2
AppConfig.findString(regionString),
AppConfig.findString(envString));
This method is known to be slower than using StringBuilder but unless you're doing hundreds of thousands of these the time difference will be on the order of milliseconds.

Csv: search for String and replace with another string

I have a .csv file that contains:
scenario, custom, master_data
1, ${CUSTOM}, A_1
I have a string:
a, b, c
and I want to replace 'custom' with 'a, b, c'. How can I do that and save to the existing .csv file?

Probably the easiest way is to read in one file and output to another file as you go, modifying it on a per-line basis
You could try something with tokenizers, this may not be completely correct for your output/input, but you can adapt it to your CSV file formatting
BufferedReader reader = new BufferedReader(new FileReader("input.csv"));
BufferedWriter writer = new BufferedWriter(new FileWriter("output.csv"));
String custom = "custom";
String replace = "a, b, c";
for(String line = reader.readLine(); line != null; line = reader.readLine())
{
String output = "";
StringTokenizer tokenizer = new StringTokenizer(line, ",");
for(String token = tokenizer.nextToken(); tokenizer.hasMoreTokens(); token = tokenizer.nextToken())
if(token.equals(custom)
output = "," + replace;
else
output = "," + token;
}
readInventory.close();
If this is for a one off thing, it also has the benefit of not having to research regular expressions (which are quite powerful and useful, good to know, but maybe for a later date?)

Have a look at Can you recommend a Java library for reading (and possibly writing) CSV files?
And once the values have been read, search for strings / value that start with ${ and end with }. Use Java Regular Expressions like \$\{(\w)\}. Then use some map for looking up the found key, and the related value. Java Properties would be a good candidate.
Then write a new csv file.

Since your replacement string is quite unique you can do it quickly without complicated parsing by just reading your file into a buffer, and then converting that buffer into a string. Replace all occurrences of the text you wish to replace with your target text. Then convert the string to a buffer and write that back to the file...
Pattern.quote is required because your string is a regular expression. If you don't quote it you may run into unexpected results.
Also it's generally not smart to overwrite your source file. Best is to create a new file then delete the old and rename the new to the old. Any error halfway will then not delete all your data.
final Path yourPath = Paths.get("Your path");
byte[] buff = Files.readAllBytes(yourPath);
String s = new String(buff, Charset.defaultCharset());
s = s.replaceAll(Pattern.quote("${CUSTOM}"), "a, b, c");
Files.write(yourPath, s.getBytes());

Java Replacing Help Needed

Hey guy's so am trying to replace all characters and numbers to get the /hello/what/ only without the REMOVEThis4.PNG i don't want to use string.replace("REMOVEThis4.PNG", ""); cause i wanna use it on other strings not only that
Any help is great my code
String sFile = "/hello/what/REMOVEThis4.PNG";
if (sFile.contains("/")){
String Replaced = sFile.replaceAll("(?s)", "");
System.out.println(Replaced);
}
I want the the output to be
/hello/what/
Only thanks alot!

If you are trying to parse a path, I recommend to find the last index of /, and get the substring to this index plus one. So
string = string.substring(0, string.lastIndexOf("/") + 1);

No need to use regular expressions in your case:
String sFile = "/hello/what/REMOVEThis4.PNG";
// TODO check actual last index of "/" against -1
System.out.println(sFile.substring(0, sFile.lastIndexOf("/") + 1));
Output
/hello/what/
Note
In case you are dealing with actual files, you can probably spare yourself the String manipulation and use File.getParent() instead:
File file = new File("/hello/what/REMOVEThis4.PNG");
System.out.println(file.getParent());
Output (may change depending on your system)
\hello\what

Use Java's File API:
String example = "/hello/what/REMOVEThis4.PNG";
File file = new File(example);
System.out.println(example);
String absolutePath = file.getAbsolutePath();
String filePath = absolutePath.substring(0, absolutePath.lastIndexOf(File.separator));
System.out.println(filePath);

Parsing A Text File, So That Every Line Is Stored As An Array Value

Basically, I want to parse, line by line, a Text file so that every line is in it's own array value.
E.g.
Hi There,
My Name's Aiden,
Not Really.
Array[0] = "Hi There"
Array[1] = "My Name's Aiden"
Array[2] = "Not Really"
But all the examples I have read already just confuse me and lead me to get frustrated. Maybe it's the way I approach it.
I don't know how to go about it, a point in the right direction would be most satisfying.

My suggestion is to use List<String> instead of String[] as arrays have fixed size, and that size is unknown before reading. Afterward one could make an array out of it, but to no real purpose.
For reading one has to know the encoding of the file.
Path path = Paths.get("C:/Users/Me/list.txt");
//Charset encoding = StandardCharsets.UTF_8;
Charset encoding = Charset.defaultCharset();
List<String> lines = Files.readAllLines(path, encoding);
for (String line : lines) {
...
}
for (int i = 0; i < lines.size(); ++i) {
String line = lines.get(i);
lines.set(i, "-- " + line;
}