Develop Reference

Decode an escaped string from VBScript in Java

I tried to decode the following string,
String str = "AT%26amp%3BT%20Network%20Client%20%u2013%20IBM";
System.out.println(StringEscapeUtils.unescapeHtml(str));
try {
System.out.println("res:"+java.net.URLDecoder.decode(str, "UTF-8"));
} catch (UnsupportedEncodingException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
Both methods fail as below,
AT%26amp%3BT%20Network%20Client%20%u2013%20IBM
Exception in thread "main" java.lang.IllegalArgumentException: URLDecoder: Illegal hex characters in escape (%) pattern - For input string: "u2"
at java.net.URLDecoder.decode(URLDecoder.java:173)
at decrypt.DecryptHtml.main(DecryptHtml.java:19)
The source of the string is a VBS script that uses the Escape function. How can I decode this string?

Unfortunately, from reading the documentation, it appears that Microsoft Has Done It Again (tm): "non standard xxx", where here "xxx" is "escaping format".
Specifically, in the documentation of the VBScript function, it is said that:
[...]Unicode characters that have a value greater than 255 are stored using the %uxxxx format.
(Hey, MS: there is no such thing as "Unicode characters"; those are called code points)
Great. So you need your own decoding function.
Fortunately, we use Java. And since this proprietary escape sequence only covers Unicode code points in the Basic Multilingual Plane (U+0000 to U+FFFF), and since char is a UTF-16 code unit, and since there is a 1 to 1 mapping between BMP and UTF-16, this makes our job a little easier.
Here is the code:
public final class MSUnescaper
{
private static final char PERCENT = '%';
private static final char NONSTANDARD_PCT_ESCAPE = 'u';
private MSUnescaper()
{
}
public static String unescape(final String input)
{
final StringBuilder sb = new StringBuilder(input.length());
final CharBuffer buf = CharBuffer.wrap(input);
char c;
while (buf.hasRemaining()) {
c = buf.get();
if (c != PERCENT) {
sb.append(c);
continue;
}
if (!buf.hasRemaining())
throw new IllegalArgumentException();
c = buf.get();
sb.append(c == NONSTANDARD_PCT_ESCAPE
? msEscape(buf) : standardEscape(buf, c));
}
return sb.toString();
}
private static char standardEscape(final CharBuffer buf, final char c)
{
if (!buf.hasRemaining())
throw new IllegalArgumentException();
final char[] array = { c, buf.get() };
return (char) Integer.parseInt(new String(array), 16);
}
private static char msEscape(final CharBuffer buf)
{
if (buf.remaining() < 4)
throw new IllegalArgumentException();
final char[] array = new char[4];
buf.get(array);
return (char) Integer.parseInt(new String(array), 16);
}
public static void main(final String... args)
{
final String input = "AT%26amp%3BT%20Network%20Client%20%u2013%20IBM";
System.out.println(unescape(input));
}
}
Output:
AT&T Network Client – IBM

String str = "AT%26amp%3BT%20Network%20Client%20%[here]u[here]2013%20IBM"
I think this string is invalid. %u20 is not valid charecter.
If you remove u from your string you can encode it.
For reference: w3schools html url encodeing

Splitting a String without spaces

I have the following string which is generated by an external program (OpenVAS) and returned to my program successfully as a string.
<create_target_response id="b4c8de55-94d8-4e08-b20e-955f97a714f1" status_text="OK, resource created" status="201"></create_target_response>
I am trying to split the string to give me the "b4c8d....14f1" without the inverted commas. I have tried all sorts of escape methods and keep getting the else method "String does not contain a Target ID". I have tried removing the IF statement checking for the string, but continue to have the same issue. The goal is to get my id string into jTextField6. String Lob contains the full string as above.
if (Lob.contains("id=\"")){
// put the split here
String[] parts = Lob.split("id=\"");
String cut1 = parts[1];
String[] part2 = cut1.split("\"");
String TaskFinal = part2[0];
jTextField6.setText(TaskFinal);
}
else {
throw new IllegalArgumentException("String does not contain a Target ID");
}
} catch (IOException e) {
e.printStackTrace();
}
It seems I only need to escape the " and not the = (Java kicks up an error if i do)
Thanks in advance
EDIT: Code as it stands now using jSoup lib - The 'id' string won't display. Any ideas?
Thanks
private void jButton1ActionPerformed(java.awt.event.ActionEvent evt) {
// TODO add your handling code here:
String TargIP = jTextField1.getText(); // Get IP Address
String TargName = jTextField5.getText(); // Get Target Name
String Vag = "8d32ad99-ac84-4fdc-b196-2b379f861def";
String Lob = "";
final String dosCommand = "cmd /c omp -u admin -w admin --xml=\"<create_target><name>" + TargName + "</name><hosts>" + TargIP + "</hosts></create_target>\"";
3</comment><config id='daba56c8-73ec-11df-a475-002264764cea'/><target id='" + Vag + "'/></create_task>\"";
final String location = "C:\\";
try {
final Process process = Runtime.getRuntime().exec(
dosCommand + " " + location);
final InputStream in = process.getInputStream();
int ch;
while((ch = in.read()) != -1) {
System.out.print((char)ch);
Lob = String.valueOf((char)ch);
jTextArea2.append(Lob);
}
} catch (IOException e) {
e.printStackTrace();
}
String id = Jsoup.parse(Lob).getAllElements().attr("id");
System.out.println(id); // This doesn't output?
}

Split on the basis of ". You can get all the key values.
String str = "<create_target_response id=\"b4c8de55-94d8-4e08-b20e-955f97a714f1\" status_text=\"OK, resource created\" status=\"201\"></create_target_response>";
String[] tokens = str.split("\\\"");
System.out.println(tokens[1]);
System.out.println(tokens[5]);
output:
b4c8de55-94d8-4e08-b20e-955f97a714f1
201

This will get you your job id more easily:
int idStart = Lob.indexOf("id=")+("id=\"").length();
System.out.println(Lob.substring(idStart,Lob.indexOf("\"",idStart)));

Everyone's telling you to use an XML parser (and they're right) but noone's showing you how.
Here goes:
String lob = ...
Using Jsoup from http://jsoup.org, actually an HTML parser but also handles XML neatly:
String id = Jsoup.parse(lob).getAllElements().attr("id");
// b4c8de55-94d8-4e08-b20e-955f97a714f1
With built-in Java XML APIs, less concise but no addtional libraries:
Document dom = DocumentBuilderFactory.newInstance().newDocumentBuilder()
.parse(new InputSource(new StringReader(lob)));
String id = dom.getDocumentElement().getAttribute("id");
// b4c8de55-94d8-4e08-b20e-955f97a714f1

This is a lot simpler than you're making it, to my mind. First, split on space, then check if an = is present. If it is, split on the =, and finally remove the " from the second token.
The tricky bit is the spaces inside of the "". This will require some regular expressions, which you can work out from this question.
Example
String input; // Assume this contains the whole string.
String pattern; // Have fun working out the regex.
String[] values = input.split(pattern);
for(String value : values)
{
if(value.contains("=")) {
String[] pair = value.split("=");
String key = pair[0];
String value = pair[1].replaceAll("\"");
// Do something with the values.
}
}
Advantage of my approach
Is that provided the input follows the format of key="value" key="value", you can parse anything that comes through, rather than hard coding the name of the attributes.
And if this is XML..
Then use an XML parser. There is a good (awesome) answer that explains why you shouldn't be using Stringmanipulation to parse XML/HTML. Here is the answer.

You can use a regex to extract what is needed; what is more, it looks like the value of id is a UUID. Therefore:
private static final Pattern PATTERN
= Pattern.compile("\\bid=\"([^\"]+)\"");
// In code...
public String getId(final String input)
{
final Matcher m = PATTERN.matcher(input);
if (!m.find())
throw new IllegalArgumentException("String does not contain a Target ID");
final String uuid = m.group(1);
try {
UUID.fromString(uuid);
} catch (IllegalArgumentException ignored) {
throw new IllegalArgumentException("String does not contain a Target ID");
}
return uuid;
}

Decode a string in Java

How do I properly decode the following string in Java
http%3A//www.google.ru/search%3Fhl%3Dru%26q%3Dla+mer+powder%26btnG%3D%u0420%A0%u0421%u045F%u0420%A0%u0421%u2022%u0420%A0%u0421%u2018%u0420%u040E%u0420%u0453%u0420%A0%u0421%u201D+%u0420%A0%u0420%u2020+Google%26lr%3D%26rlz%3D1I7SKPT_ru
When I use URLDecoder.decode() I the following error
java.lang.IllegalArgumentException: URLDecoder: Illegal hex characters in escape (%) pattern - For input string: "u0"
Thanks,
Dave

According to Wikipedia, "there exist a non-standard encoding for Unicode characters: %uxxxx, where xxxx is a Unicode value".
Continuing: "This behavior is not specified by any RFC and has been rejected by the W3C".
Your URL contains such tokens, and the Java URLDecoder implementation doesn't support those.

%uXXXX encoding is non-standard, and was actually rejected by W3C, so it's natural, that URLDecoder does not understand it.
You can make small function, which will fix it by replacing each occurrence of %uXXYY with %XX%YY in your encoded string. Then you can procede and decode the fixed string normally.

we started with Vartec's solution but found out additional issues. This solution works for UTF-16, but it can be changed to return UTF-8. The replace all is left for clarity reasons and you can read more at http://www.cogniteam.com/wiki/index.php?title=DecodeEncodeJavaScript
static public String unescape(String escaped) throws UnsupportedEncodingException
{
// This code is needed so that the UTF-16 won't be malformed
String str = escaped.replaceAll("%0", "%u000");
str = str.replaceAll("%1", "%u001");
str = str.replaceAll("%2", "%u002");
str = str.replaceAll("%3", "%u003");
str = str.replaceAll("%4", "%u004");
str = str.replaceAll("%5", "%u005");
str = str.replaceAll("%6", "%u006");
str = str.replaceAll("%7", "%u007");
str = str.replaceAll("%8", "%u008");
str = str.replaceAll("%9", "%u009");
str = str.replaceAll("%A", "%u00A");
str = str.replaceAll("%B", "%u00B");
str = str.replaceAll("%C", "%u00C");
str = str.replaceAll("%D", "%u00D");
str = str.replaceAll("%E", "%u00E");
str = str.replaceAll("%F", "%u00F");
// Here we split the 4 byte to 2 byte, so that decode won't fail
String [] arr = str.split("%u");
Vector<String> vec = new Vector<String>();
if(!arr[0].isEmpty())
{
vec.add(arr[0]);
}
for (int i = 1 ; i < arr.length ; i++) {
if(!arr[i].isEmpty())
{
vec.add("%"+arr[i].substring(0, 2));
vec.add("%"+arr[i].substring(2));
}
}
str = "";
for (String string : vec) {
str += string;
}
// Here we return the decoded string
return URLDecoder.decode(str,"UTF-16");
}

After having had a good look at the solution presented by #ariy I created a Java based solution that is also resilient against encoded characters that have been chopped into two parts (i.e. half of the encoded character is missing). This happens in my usecase where I need to decode long urls that are sometimes chopped at 2000 chars length. See What is the maximum length of a URL in different browsers?
public class Utils {
private static Pattern validStandard = Pattern.compile("%([0-9A-Fa-f]{2})");
private static Pattern choppedStandard = Pattern.compile("%[0-9A-Fa-f]{0,1}$");
private static Pattern validNonStandard = Pattern.compile("%u([0-9A-Fa-f][0-9A-Fa-f])([0-9A-Fa-f][0-9A-Fa-f])");
private static Pattern choppedNonStandard = Pattern.compile("%u[0-9A-Fa-f]{0,3}$");
public static String resilientUrlDecode(String input) {
String cookedInput = input;
if (cookedInput.indexOf('%') > -1) {
// Transform all existing UTF-8 standard into UTF-16 standard.
cookedInput = validStandard.matcher(cookedInput).replaceAll("%00%$1");
// Discard chopped encoded char at the end of the line (there is no way to know what it was)
cookedInput = choppedStandard.matcher(cookedInput).replaceAll("");
// Handle non standard (rejected by W3C) encoding that is used anyway by some
// See: https://stackoverflow.com/a/5408655/114196
if (cookedInput.contains("%u")) {
// Transform all existing non standard into UTF-16 standard.
cookedInput = validNonStandard.matcher(cookedInput).replaceAll("%$1%$2");
// Discard chopped encoded char at the end of the line
cookedInput = choppedNonStandard.matcher(cookedInput).replaceAll("");
}
}
try {
return URLDecoder.decode(cookedInput,"UTF-16");
} catch (UnsupportedEncodingException e) {
// Will never happen because the encoding is hardcoded
return null;
}
}
}

Java UTF-8 to ASCII conversion with supplements

we are accepting all sorts of national characters in UTF-8 string on the input, and we need to convert them to ASCII string on the output for some legacy use. (we don't accept Chinese and Japanese chars, only European languages)
We have a small utility to get rid of all the diacritics:
public static final String toBaseCharacters(final String sText) {
if (sText == null || sText.length() == 0)
return sText;
final char[] chars = sText.toCharArray();
final int iSize = chars.length;
final StringBuilder sb = new StringBuilder(iSize);
for (int i = 0; i < iSize; i++) {
String sLetter = new String(new char[] { chars[i] });
sLetter = Normalizer.normalize(sLetter, Normalizer.Form.NFC);
try {
byte[] bLetter = sLetter.getBytes("UTF-8");
sb.append((char) bLetter[0]);
} catch (UnsupportedEncodingException e) {
}
}
return sb.toString();
}
The question is how to replace all the german sharp s (ß, Đ, đ) and other characters that get through the above normalization method, with their supplements (in case of ß, supplement would probably be "ss" and in case od Đ supplement would be either "D" or "Dj").
Is there some simple way to do it, without million of .replaceAll() calls?
So for example: Đonardan = Djonardan, Blaß = Blass and so on.
We can replace all "problematic" chars with empty space, but would like to avoid this to make the output as similar to the input as possible.
Thank you for your answers,
Bozo

You want to use ICU4J. It includes the com.ibm.icu.text.Transliterator class, which apparently can do what you are looking for.

Here's my converter which uses lucene...
private final KeywordTokenizer keywordTokenizer = new KeywordTokenizer(new StringReader(""));
private final ASCIIFoldingFilter asciiFoldingFilter = new ASCIIFoldingFilter(keywordTokenizer);
private final TermAttribute termAttribute = (TermAttribute) asciiFoldingFilter.getAttribute(TermAttribute.class);
public String process(String line)
{
if (line != null)
{
try
{
keywordTokenizer.reset(new StringReader(line));
if (asciiFoldingFilter.incrementToken())
{
return termAttribute.term();
}
}
catch (IOException e)
{
logger.warn("Failed to parse: " + line, e);
}
}
return null;
}

I'm using something like this:
Transliterator transliterator = Transliterator.getInstance("Any-Latin; Upper; Lower; NFD; [:Nonspacing Mark:] Remove; NFC", Transliterator.FORWARD);

Is there some simple way to do it, without million of .replaceAll() calls?
If you just support European, Latin-based languages, around 100 should be enough; that's definitely doable: Grab the Unicode charts for Latin-1 Supplement and Latin Extended-A and get the String.replace party started. :-)

How do I trim a file extension from a String in Java?

What's the most efficient way to trim the suffix in Java, like this:
title part1.txt
title part2.html
=>
title part1
title part2

This is the sort of code that we shouldn't be doing ourselves. Use libraries for the mundane stuff, save your brain for the hard stuff.
In this case, I recommend using FilenameUtils.removeExtension() from Apache Commons IO

str.substring(0, str.lastIndexOf('.'))

As using the String.substring and String.lastIndex in a one-liner is good, there are some issues in terms of being able to cope with certain file paths.
Take for example the following path:
a.b/c
Using the one-liner will result in:
a
That's incorrect.
The result should have been c, but since the file lacked an extension, but the path had a directory with a . in the name, the one-liner method was tricked into giving part of the path as the filename, which is not correct.
Need for checks
Inspired by skaffman's answer, I took a look at the FilenameUtils.removeExtension method of the Apache Commons IO.
In order to recreate its behavior, I wrote a few tests the new method should fulfill, which are the following:
Path Filename
-------------- --------
a/b/c c
a/b/c.jpg c
a/b/c.jpg.jpg c.jpg
a.b/c c
a.b/c.jpg c
a.b/c.jpg.jpg c.jpg
c c
c.jpg c
c.jpg.jpg c.jpg
(And that's all I've checked for -- there probably are other checks that should be in place that I've overlooked.)
The implementation
The following is my implementation for the removeExtension method:
public static String removeExtension(String s) {
String separator = System.getProperty("file.separator");
String filename;
// Remove the path upto the filename.
int lastSeparatorIndex = s.lastIndexOf(separator);
if (lastSeparatorIndex == -1) {
filename = s;
} else {
filename = s.substring(lastSeparatorIndex + 1);
}
// Remove the extension.
int extensionIndex = filename.lastIndexOf(".");
if (extensionIndex == -1)
return filename;
return filename.substring(0, extensionIndex);
}
Running this removeExtension method with the above tests yield the results listed above.
The method was tested with the following code. As this was run on Windows, the path separator is a \ which must be escaped with a \ when used as part of a String literal.
System.out.println(removeExtension("a\\b\\c"));
System.out.println(removeExtension("a\\b\\c.jpg"));
System.out.println(removeExtension("a\\b\\c.jpg.jpg"));
System.out.println(removeExtension("a.b\\c"));
System.out.println(removeExtension("a.b\\c.jpg"));
System.out.println(removeExtension("a.b\\c.jpg.jpg"));
System.out.println(removeExtension("c"));
System.out.println(removeExtension("c.jpg"));
System.out.println(removeExtension("c.jpg.jpg"));
The results were:
c
c
c.jpg
c
c
c.jpg
c
c
c.jpg
The results are the desired results outlined in the test the method should fulfill.

String foo = "title part1.txt";
foo = foo.substring(0, foo.lastIndexOf('.'));

BTW, in my case, when I wanted a quick solution to remove a specific extension, this is approximately what I did:
if (filename.endsWith(ext))
return filename.substring(0,filename.length() - ext.length());
else
return filename;

Use a method in com.google.common.io.Files class if your project is already dependent on Google core library. The method you need is getNameWithoutExtension.

you can try this function , very basic
public String getWithoutExtension(String fileFullPath){
return fileFullPath.substring(0, fileFullPath.lastIndexOf('.'));
}

String fileName="foo.bar";
int dotIndex=fileName.lastIndexOf('.');
if(dotIndex>=0) { // to prevent exception if there is no dot
fileName=fileName.substring(0,dotIndex);
}
Is this a trick question? :p
I can't think of a faster way atm.

I found coolbird's answer particularly useful.
But I changed the last result statements to:
if (extensionIndex == -1)
return s;
return s.substring(0, lastSeparatorIndex+1)
+ filename.substring(0, extensionIndex);
as I wanted the full path name to be returned.
So "C:\Users\mroh004.COM\Documents\Test\Test.xml" becomes
"C:\Users\mroh004.COM\Documents\Test\Test" and not
"Test"

filename.substring(filename.lastIndexOf('.'), filename.length()).toLowerCase();

Use a regex. This one replaces the last dot, and everything after it.
String baseName = fileName.replaceAll("\\.[^.]*$", "");
You can also create a Pattern object if you want to precompile the regex.

If you use Spring you could use
org.springframework.util.StringUtils.stripFilenameExtension(String path)
Strip the filename extension from the given Java resource path, e.g.
"mypath/myfile.txt" -> "mypath/myfile".
Params: path – the file path
Returns: the path with stripped filename extension

private String trimFileExtension(String fileName)
{
String[] splits = fileName.split( "\\." );
return StringUtils.remove( fileName, "." + splits[splits.length - 1] );
}

String[] splitted = fileName.split(".");
String fileNameWithoutExtension = fileName.replace("." + splitted[splitted.length - 1], "");

create a new file with string image path
String imagePath;
File test = new File(imagePath);
test.getName();
test.getPath();
getExtension(test.getName());
public static String getExtension(String uri) {
if (uri == null) {
return null;
}
int dot = uri.lastIndexOf(".");
if (dot >= 0) {
return uri.substring(dot);
} else {
// No extension.
return "";
}
}

org.apache.commons.io.FilenameUtils version 2.4 gives the following answer
public static String removeExtension(String filename) {
if (filename == null) {
return null;
}
int index = indexOfExtension(filename);
if (index == -1) {
return filename;
} else {
return filename.substring(0, index);
}
}
public static int indexOfExtension(String filename) {
if (filename == null) {
return -1;
}
int extensionPos = filename.lastIndexOf(EXTENSION_SEPARATOR);
int lastSeparator = indexOfLastSeparator(filename);
return lastSeparator > extensionPos ? -1 : extensionPos;
}
public static int indexOfLastSeparator(String filename) {
if (filename == null) {
return -1;
}
int lastUnixPos = filename.lastIndexOf(UNIX_SEPARATOR);
int lastWindowsPos = filename.lastIndexOf(WINDOWS_SEPARATOR);
return Math.max(lastUnixPos, lastWindowsPos);
}
public static final char EXTENSION_SEPARATOR = '.';
private static final char UNIX_SEPARATOR = '/';
private static final char WINDOWS_SEPARATOR = '\\';

The best what I can write trying to stick to the Path class:
Path removeExtension(Path path) {
return path.resolveSibling(path.getFileName().toString().replaceFirst("\\.[^.]*$", ""));
}

dont do stress on mind guys. i did already many times. just copy paste this public static method in your staticUtils library for future uses ;-)
static String removeExtension(String path){
String filename;
String foldrpath;
String filenameWithoutExtension;
if(path.equals("")){return "";}
if(path.contains("\\")){ // direct substring method give wrong result for "a.b.c.d\e.f.g\supersu"
filename = path.substring(path.lastIndexOf("\\"));
foldrpath = path.substring(0, path.lastIndexOf('\\'));;
if(filename.contains(".")){
filenameWithoutExtension = filename.substring(0, filename.lastIndexOf('.'));
}else{
filenameWithoutExtension = filename;
}
return foldrpath + filenameWithoutExtension;
}else{
return path.substring(0, path.lastIndexOf('.'));
}
}

I would do like this:
String title_part = "title part1.txt";
int i;
for(i=title_part.length()-1 ; i>=0 && title_part.charAt(i)!='.' ; i--);
title_part = title_part.substring(0,i);
Starting to the end till the '.' then call substring.
Edit:
Might not be a golf but it's effective :)

Keeping in mind the scenarios where there is no file extension or there is more than one file extension
example Filename : file | file.txt | file.tar.bz2
/**
*
* #param fileName
* #return file extension
* example file.fastq.gz => fastq.gz
*/
private String extractFileExtension(String fileName) {
String type = "undefined";
if (FilenameUtils.indexOfExtension(fileName) != -1) {
String fileBaseName = FilenameUtils.getBaseName(fileName);
int indexOfExtension = -1;
while (fileBaseName.contains(".")) {
indexOfExtension = FilenameUtils.indexOfExtension(fileBaseName);
fileBaseName = FilenameUtils.getBaseName(fileBaseName);
}
type = fileName.substring(indexOfExtension + 1, fileName.length());
}
return type;
}

String img = "example.jpg";
// String imgLink = "http://www.example.com/example.jpg";
URI uri = null;
try {
uri = new URI(img);
String[] segments = uri.getPath().split("/");
System.out.println(segments[segments.length-1].split("\\.")[0]);
} catch (Exception e) {
e.printStackTrace();
}
This will output example for both img and imgLink

private String trimFileName(String fileName)
{
String[] ext;
ext = fileName.split("\\.");
return fileName.replace(ext[ext.length - 1], "");
}
This code will spilt the file name into parts where ever it has " . ", For eg. If the file name is file-name.hello.txt then it will be spilted into string array as , { "file-name", "hello", "txt" }. So anyhow the last element in this string array will be the file extension of that particular file , so we can simply find the last element of any arrays with arrayname.length - 1, so after we get to know the last element, we can just replace the file extension with an empty string in that file name. Finally this will return file-name.hello. , if you want to remove also the last period then you can add the string with only period to the last element of string array in the return line. Which should look like,
return fileName.replace("." + ext[ext.length - 1], "");

public static String removeExtension(String file) {
if(file != null && file.length() > 0) {
while(file.contains(".")) {
file = file.substring(0, file.lastIndexOf('.'));
}
}
return file;
}