Develop Reference

Regex \P{IsHan} does not work well in Java 8

I want to remove all non-Chinese characters in a String, and retain Chinese Characters.
Here is an example:
input -> 勇𣌀hi你好👋（）【】「」{}[]()
output -> 勇𣌀你好
First of all, I try to extract all Chinese characters and append each to a StringBuilder using the following code:
public static String extractAllChineseCharacters(String input){
Matcher matcher = Pattern.compile("\\p{IsHan}").matcher(input);
StringBuilder output = new StringBuilder();
while(matcher.find()){
output.append(input.substring(matcher.start(),matcher.end()));
}
return output.toString();
}
The code works well, but i want a more concise code.
Then i try to replace all non-Chinese character with "" using the following code
public static String replaceCharacters(String input){
Matcher matcher = Pattern.compile("\\P{IsHan}").matcher(input);
return matcher.replaceAll("");
}
but the code doesn't work as i expect.
When the input is 勇𣌀hi你好👋（）【】「」{}[](),
the output is ??你好
i enter the debug mode, and find out the output's charSequences is \uD87E\uD84e你好
i know that java adopt UTF-16 to store a codeunit in a char variable, so 勇(U+2F825)is presents as a surrogate pair \uD87E\uDC25 in charSequence,and 𣌀(U+23300)is presents as \uD84C\uDF00
My question is:
the regex pattern "\p{IsHan}"can match \uD87E\uDC25 two chars as a Chinese character "勇",but the regex pattern "\P{IsHan}" can‘t precisely match non-Chinese character.Why is that so?
Can anyone give some help? Thanks in advance!

The use of \p{IsHan} works in Java 8 - see below. Perhaps the input is encoded is a way that makes the matching fail?
Regarding "The code works well, but I want a more concise code", rather than match what you want, match what you don't want and remove it:
public static String extractAllChineseCharacters(String input) {
return input.replaceAll("\\P{IsHan}", "");
}
See live demo running Java 8.
FYI:
\p{IsHan}" matches any Chinese character
\P{IsHan}" matches any non Chinese character.
Test:
String input = "\uD87E\uDC25\uD84C\uDF00hi你好\uD83D\uDC4B（）【】「」{}[]()";
System.out.println(input);
System.out.println(input.replaceAll("\\P{IsHan}", ""));
Output:
勇𣌀hi你好👋（）【】「」{}[]()
勇𣌀你好

Java Regex is including new line in match

I'm trying to match a regular expression to textbook definitions that I get from a website.
The definition always has the word with a new line followed by the definition. For example:
Zither
Definition: An instrument of music used in Austria and Germany It has from thirty to forty wires strung across a shallow sounding board which lies horizontally on a table before the performer who uses both hands in playing on it Not to be confounded with the old lute shaped cittern or cithern
In my attempts to get just the word (in this case "Zither") I keep getting the newline character.
I tried both ^(\w+)\s and ^(\S+)\s without much luck. I thought that maybe ^(\S+)$ would work, but that doesn't seem to successfully match the word at all. I've been testing with rubular, http://rubular.com/r/LPEHCnS0ri; which seems to successfully match all my attempts the way I want, despite the fact that Java doesn't.
Here's my snippet
String str = ...; //Here the string is assigned a word and definition taken from the internet like given in the example above.
Pattern rgx = Pattern.compile("^(\\S+)$");
Matcher mtch = rgx.matcher(str);
if (mtch.find()) {
String result = mtch.group();
terms.add(new SearchTerm(result, System.nanoTime()));
}
This is easily solved by triming the resulting string, but that seems like it should be unnecessary if I'm already using a regular expression.
All help is greatly appreciated. Thanks in advance!

Try using the Pattern.MULTILINE option
Pattern rgx = Pattern.compile("^(\\S+)$", Pattern.MULTILINE);
This causes the regex to recognise line delimiters in your string, otherwise ^ and $ just match the start and end of the string.
Although it makes no difference for this pattern, the Matcher.group() method returns the entire match, whereas the Matcher.group(int) method returns the match of the particular capture group (...) based on the number you specify. Your pattern specifies one capture group which is what you want captured. If you'd included \s in your Pattern as you wrote you tried, then Matcher.group() would have included that whitespace in its return value.

With regular expressions the first group is always the complete matching string. In your case you want group 1, not group 0.
So changing mtch.group() to mtch.group(1) should do the trick:
String str = ...; //Here the string is assigned a word and definition taken from the internet like given in the example above.
Pattern rgx = Pattern.compile("^(\\w+)\s");
Matcher mtch = rgx.matcher(str);
if (mtch.find()) {
String result = mtch.group(1);
terms.add(new SearchTerm(result, System.nanoTime()));
}

A late response, but if you are not using Pattern and Matcher, you can use this alternative of DOTALL in your regex string
(?s)[Your Expression]
Basically (?s) also tells dot to match all characters, including line breaks
Detailed information: http://www.vogella.com/tutorials/JavaRegularExpressions/article.html

Just replace:
String result = mtch.group();
By:
String result = mtch.group(1);
This will limit your output to the contents of the capturing group (e.g. (\\w+)) .

Try the next:
/* The regex pattern: ^(\w+)\r?\n(.*)$ */
private static final REGEX_PATTERN =
Pattern.compile("^(\\w+)\\r?\\n(.*)$");
public static void main(String[] args) {
String input = "Zither\n Definition: An instrument of music";
System.out.println(
REGEX_PATTERN.matcher(input).matches()
); // prints "true"
System.out.println(
REGEX_PATTERN.matcher(input).replaceFirst("$1 = $2")
); // prints "Zither = Definition: An instrument of music"
System.out.println(
REGEX_PATTERN.matcher(input).replaceFirst("$1")
); // prints "Zither"
}

Now, how can I screen-scrape such a html line (using java)?

I am trying to screen-scrape a html page so I can extract desired valuable data from it and into a text file. So far it's going well until I came across this within the html page:
<td> <b>In inventory</b>: 0.3 kg<br /><b>Equipped</b>: -4.5 kg
The above line in the html code for the page often varies. So it need to figure about a way to scan the line (regardless of what it contains) for the weight (in this case would be 0.3 and -4.5) and store this data into 2 seperate doubles as of such:
double inventoryWeight = 0.3 double equippedWeight = -4.5
I would like this to be done using pure java; if need be, do not hesitate to notify me of any third-party programs which can be executed within my java application to achieve this (but please vividly explain if so).
Thank you a bunch!

RegEx is usually a good solution for scraping text. Parentheses denote "capturing groups", which are stored and can then be accessed using Matcher.group(). [-.\d]+ matches anything consisting of one or more digits (0-9), periods, and hyphens. .* matches anything (but sometimes not newline characters). Here it's just used to essentially "throw away" everything you don't care about.
import java.util.regex.*;
public class Foo {
public static void main(String[] args) {
String regex = ".*inventory<\\/b>: ([-.\\d]+).*Equipped<\\/b>: ([-.\\d]+).*";
String text = "<td> <b>In inventory</b>: 0.3 kg<br /><b>Equipped</b>: -4.5 kg";
// Look for a match
Pattern pattern = Pattern.compile(regex);
Matcher matcher = pattern.matcher(text);
// Get the matched text
if (matcher.matches()) {
String inventoryWeight = matcher.group(1);
String equippedWeight = matcher.group(2);
System.out.println("Inventory weight: " + inventoryWeight);
System.out.println("Equipped weight: " + equippedWeight);
} else {
System.out.println("No match!");
}
}
}

Do you have this piece of html as String? If so, just search for <b>Equipped</b>. Then get <b>Equipped</b> end char position plus one. And then build new string by appending char by char until it's not a number or dot.
When you have those numbers in String variables you simply convert them to Doubles by using double aDouble = Double.parseDouble(aString)

How to remove special characters from a string?

I want to remove special characters like:
- + ^ . : ,
from an String using Java.

That depends on what you define as special characters, but try replaceAll(...):
String result = yourString.replaceAll("[-+.^:,]","");
Note that the ^ character must not be the first one in the list, since you'd then either have to escape it or it would mean "any but these characters".
Another note: the - character needs to be the first or last one on the list, otherwise you'd have to escape it or it would define a range ( e.g. :-, would mean "all characters in the range : to ,).
So, in order to keep consistency and not depend on character positioning, you might want to escape all those characters that have a special meaning in regular expressions (the following list is not complete, so be aware of other characters like (, {, $ etc.):
String result = yourString.replaceAll("[\\-\\+\\.\\^:,]","");
If you want to get rid of all punctuation and symbols, try this regex: \p{P}\p{S} (keep in mind that in Java strings you'd have to escape back slashes: "\\p{P}\\p{S}").
A third way could be something like this, if you can exactly define what should be left in your string:
String result = yourString.replaceAll("[^\\w\\s]","");
This means: replace everything that is not a word character (a-z in any case, 0-9 or _) or whitespace.
Edit: please note that there are a couple of other patterns that might prove helpful. However, I can't explain them all, so have a look at the reference section of regular-expressions.info.
Here's less restrictive alternative to the "define allowed characters" approach, as suggested by Ray:
String result = yourString.replaceAll("[^\\p{L}\\p{Z}]","");
The regex matches everything that is not a letter in any language and not a separator (whitespace, linebreak etc.). Note that you can't use [\P{L}\P{Z}] (upper case P means not having that property), since that would mean "everything that is not a letter or not whitespace", which almost matches everything, since letters are not whitespace and vice versa.
Additional information on Unicode
Some unicode characters seem to cause problems due to different possible ways to encode them (as a single code point or a combination of code points). Please refer to regular-expressions.info for more information.

This will replace all the characters except alphanumeric
replaceAll("[^A-Za-z0-9]","");

As described here
http://developer.android.com/reference/java/util/regex/Pattern.html
Patterns are compiled regular expressions. In many cases, convenience methods such as String.matches, String.replaceAll and String.split will be preferable, but if you need to do a lot of work with the same regular expression, it may be more efficient to compile it once and reuse it. The Pattern class and its companion, Matcher, also offer more functionality than the small amount exposed by String.
public class RegularExpressionTest {
public static void main(String[] args) {
System.out.println("String is = "+getOnlyStrings("!&(*^*(^(+one(&(^()(*)(*&^%$##!#$%^&*()("));
System.out.println("Number is = "+getOnlyDigits("&(*^*(^(+91-&*9hi-639-0097(&(^("));
}
public static String getOnlyDigits(String s) {
Pattern pattern = Pattern.compile("[^0-9]");
Matcher matcher = pattern.matcher(s);
String number = matcher.replaceAll("");
return number;
}
public static String getOnlyStrings(String s) {
Pattern pattern = Pattern.compile("[^a-z A-Z]");
Matcher matcher = pattern.matcher(s);
String number = matcher.replaceAll("");
return number;
}
}
Result
String is = one
Number is = 9196390097

Try replaceAll() method of the String class.
BTW here is the method, return type and parameters.
public String replaceAll(String regex,
String replacement)
Example:
String str = "Hello +-^ my + - friends ^ ^^-- ^^^ +!";
str = str.replaceAll("[-+^]*", "");
It should remove all the {'^', '+', '-'} chars that you wanted to remove!

To Remove Special character
String t2 = "!##$%^&*()-';,./?><+abdd";
t2 = t2.replaceAll("\\W+","");
Output will be : abdd.
This works perfectly.

Use the String.replaceAll() method in Java.
replaceAll should be good enough for your problem.

You can remove single char as follows:
String str="+919595354336";
String result = str.replaceAll("\\\\+","");
System.out.println(result);
OUTPUT:
919595354336

If you just want to do a literal replace in java, use Pattern.quote(string) to escape any string to a literal.
myString.replaceAll(Pattern.quote(matchingStr), replacementStr)

How to create article spinner regex in Java?

Say for example I want to take this phrase:
{{Hello|What's Up|Howdy} {world|planet} |
{Goodbye|Later}
{people|citizens|inhabitants}}
and randomly make it into one of the following:
Hello world
Goodbye people
What's Up word
What's Up planet
Later citizens
etc.
The basic idea is that enclosed within every pair of braces will be an unlimited number of choices separated by "|". The program needs to go through and randomly choose one choice for each set of braces. Keep in mind that braces can be nested endlessly within each other. I found a thread about this and tried to convert it to Java, but it did not work. Here is the python code that supposedly worked:
import re
from random import randint
def select(m):
choices = m.group(1).split('|')
return choices[randint(0, len(choices)-1)]
def spinner(s):
r = re.compile('{([^{}]*)}')
while True:
s, n = r.subn(select, s)
if n == 0: break
return s.strip()
Here is my attempt to convert that Python code to Java.
public String generateSpun(String text){
String spun = new String(text);
Pattern reg = Pattern.compile("{([^{}]*)}");
Matcher matcher = reg.matcher(spun);
while (matcher.find()){
spun = matcher.replaceFirst(select(matcher.group()));
}
return spun;
}
private String select(String m){
String[] choices = m.split("|");
Random random = new Random();
int index = random.nextInt(choices.length - 1);
return choices[index];
}
Unfortunately, when I try to test this by calling
generateAd("{{Hello|What's Up|Howdy} {world|planet} | {Goodbye|Later} {people|citizens|inhabitants}}");
In the main of my program, it gives me an error in the line in generateSpun where Pattern reg is declared, giving me a PatternSyntaxException.
java.util.regex.PatternSyntaxException: Illegal repetition
{([^{}]*)}
Can someone try to create a Java method that will do what I am trying to do?

Here are some of the problems with your current code:
You should reuse your compiled Pattern, instead of Pattern.compile every time
You should reuse your Random, instead of new Random every time
Be aware that String.split is regex-based, so you must split("\\|")
Be aware that curly braces in Java regex must be escaped to match literally, so Pattern.compile("\\{([^{}]*)\\}");
You should query group(1), not group() which defaults to group 0
You're using replaceFirst wrong, look up Matcher.appendReplacement/Tail instead
Random.nextInt(int n) has exclusive upper bound (like many such methods in Java)
The algorithm itself actually does not handle arbitrarily nested braces properly
Note that escaping is done by preceding with \, and as a Java string literal it needs to be doubled (i.e. "\\" contains a single character, the backslash).
Attachment
Source code and output with above fix but no major change to algorithm

To fix the regex, add backslashes before the outer { and }. These are meta-characters in Java regexes. However, I don't think that will result in a working program. You are modifying the variable spun after it has been bound to the regex, and I do not think the returned Matcher will reflect the updated value.
I also don't think the python code will work for nested choices. Have you actually tried the python code? You say it "supposedly works", but it would be wise to verify that before you spend a lot of time porting it to Java.

Well , I just created one in PHP & Python , demo here http://spin.developerscrib.com , its at a very early stage so might not work to expectation , the source code is on github : https://github.com/razzbee/razzy-spinner

Use this, will work... I did, and working great
Pattern p = Pattern.compile("cat");
Matcher m = p.matcher("one cat two cats in the yard");
StringBuffer sb = new StringBuffer();
while (m.find()) {
m.appendReplacement(sb, "dog");
}
m.appendTail(sb);
System.out.println(sb.toString());
and here
private String select(String m){
String[] choices = m.split("|");
Random random = new Random();
int index = random.nextInt(choices.length - 1);
return choices[index];
}
m.split("|") use m.split("\\|")
Other wise it splits each an every character
and use Pattern.compile("\\{([^{}]*)\\}");