I want to extract names from the following input using regular expression.
Student Names:
Name1
Name2
Name3
Parent Names:
Name1
Name2
Name3
I am using the following method to match the data and I am not supposed to modify the method. I have to come up with regular expression that works with this method.
public void parseName(String patternRegX){
Pattern patternDomainStatus = Pattern.compile(patternRegX);
Matcher matcherName = patternName.matcher(inputString);
List<String> tmp=new ArrayList<String>();
while (matcherName.find()){
if (!matcherName.group(2).isEmpty())
tmp.add(matcherName.group(2));
}
}
I came up with a regular expression that could get me the desired result, but the problem I found was that grouping doesn't work inside square brackets([]).
private String studentRegX="(Student Names:\n[ +(\S+)\n]+\n)";
I am using the following regular expression now, but that is getting me only the last name in each set.
private String studentRegX="Student Names:\\n( +(\\S+)\\n)+\\n";
private String parentRegX="Parent Names:\\n( +(\\S+)\\n)+\\n";
Thank you in advance for the help.
First of all, I hope you can change the parseName method a little bit, because it doesn't compile. patternDomainStatus and patternName are probably supposed to refer to the same object:
Pattern pattern = Pattern.compile(patternRegX);
Matcher matcherName = pattern.matcher(inputString);
Secondly, you need to think about your regex a little differently.
Right now, your regexes are trying to match entire chunks with multiple names in them. But matcherName.find() finds "the next subsequence of the input sequence that matches the pattern" (per the javadoc).
So what you want is a regex that matches a single name. matcherName.find() will loop through each part of your string that matches that regex.
Because regex has little to do with algorithmic prowess, here an answer:
On Windows the line break is unfortunately "\r\n".
I check that a newline preceded and that there is at least some white space before the name.
The name may have a space.
With look-behind I check that "Parent Names" follows.
Then
Pattern.compile("(?s)(?<=\n)[ \t]+([^\r\n]*)\r?\n(?=.*Parent Names)");
// ~~~~ '.' also matches newline
// ~~~~~~~ look-behind must be newline
// ~~~~~~ whitespace (spaces/tabs)
// ~~~~~~~~~~ group 1, name
// ~~~~~~~~~~~~~~~~~~~~ look-ahead
Without say, a bit different algorithm would be more solid and understandable.
To make it group(2) instead of the above group(1), you could introduce extra braces before: ([ \t]+)
It can be done using the \G anchor all in a single regex.
This opens it up for a little regex algorithmic prowess.
Each match will be either:
Group 1 is not NULL/empty - New student group, group 3 will contain first student name.
Group 2 is not NULL/empty - New parent group, group 3 will contain first parent name.
Group 3 is never NULL/empty - The first/next either student or parent name depending on which
group 1 or 2 last matched.
In all cases, group 3 will contain a name that has been trimmed and ready to put into an array.
# "~(?mi-)(?:(?!\\A)\\G|^(?:(Student)|(Parent))[ ]Names:)\\s*^(?!(?:Student|Parent)[ ]Names:)[^\\S\\r\\n]*(.+?)[^\\S\\r\\n]*$~"
(?xmi-) # Inline 'Expanded, multiline, case insensitive' modifiers
(?:
(?! \A ) # Here, matched before, give Name a first chance
\G # to match again.
|
^ # BOL
(?:
( Student ) # (1), New 'Student' group
| ( Parent ) # (2), New 'Parent' group
)
[ ] Names:
)
# Name section
\s* # Consume all whitespace up until the start of a Name line
^ # BOL
(?!
(?: Student | Parent ) # Names only, Not the start of Student/Parent group here
[ ] Names:
)
[^\S\r\n]* # Trim leading whitespace ( can use \h if supported )
( .+? ) # (3), the Name
[^\S\r\n]* # Trim trailing whitespace ( can use \h if supported )
$ # EOL
If you're not already familiar with the difference between repeating a capturing group and capturing a repeating group, that's worth reading up on. One resource for that is http://www.regular-expressions.info/captureall.html, but others would be fine too.
If you already knew about that difference and were trying to capture a repeating group already with what you've written above, then please edit your post to explain what you're trying to do (a letter-by-letter explanation would be ideal, so we see what you understand and what you don't, so we can help you with whatever you're stuck on).
I see what I believe is the solution, but since this is clearly homework, I'm not willing to simply give it to you. But I'd be happy to help you figure it out.
--- Edit: ---
You're only getting one match because the regex requires "Student Names:" or "Parent Names:" to be in each match, so you can only match once. For your regex to match multiple times in a row (as required by the while (matcherName.find())), you need to get the "Student Names:" and "Parent Names:" out of the regex, so the regex can match repeatedly.
It's easy to get all of the names (both students and parents), with just a regex that looks for newlines followed by one or more spaces and then text. The challenge is to differentiate the student names (which come before the "Parent Names:" line) from the parent names (which come after the "Parent Names:" line). The key concept for differentiating between them is lookaheads, which can be positive or negative. Take a look at them and see if you can figure out how to implement this using lookaheads.
Also, you may find that group #2 isn't the group you really want to use. It's unfortunate that the group number is hard-coded, but since it is, you can tweak your regex to make groups non-capturing with (?:stuff) syntax. That will let you reduce the number of groups and ensure that the group you actually want is #2.
I am reading an Oracle tutorial on regular expressions. I am on the topic Capturing groups. Though the reference is excellent, but except that a parenthesis represents a group, I am finding many difficulties in understanding the topic. Here are my confusions.
What is the significance of counting groups in an expression?
What are non-capturing groups?
Elaborating with examples would be nice.
One usually doesn't count groups other than to know which group has which number. E.g. ([abc])([def](\d+)) has three groups, so I know to refer to them as \1, \2 and \3. Note that group 3 is inside 2. They are numbered from the left by where they begin.
When searching with regex to find something in a string, as opposed to matching when you make sure the whole string matches the subject, group 0 will give you just the matched string, but not the stuff that was before or after it. Imagine if you will a pair of brackets around your whole regex. It's not part of the total count because it's not really considered a group.
Groups can be used for other things than capturing. E.g. (foo|bar) will match "foo" or "bar". If you're not interested in the contents of a group, you can make it non-capturing (e.g: (?:foo|bar) (varies by dialect)), so as not to "use up" the numbers assigned to groups. But you don't have to, it's just convenient sometimes.
Say I want to find a word that begins and ends in the same letter: \b([a-z])[a-z]*\1\b The \1 will then be the same as whatever the first group captured. Of course it can be used for much more powerful stuff, but I think you'll get the idea.
(Coming up with relevant examples is certainly the hardest part.)
Edit: I answered when the questions were:
What is the significance of counting groups in an expression?
There is a special group, called as group-0, which means the entire expression. It is not reported by groupCount() method. Why is that?
I don't understand what are non-capturing groups?
Why we need back-references? What is the significance of back-references?
Say you have a string, abcabc, and you want to figure out whether the first part of the string matches the second part. You can do this with a single regex by using capturing groups and backreferences. Here is the regex I would use:
(.+)\1
The way this works is .+ matches any sequence of characters. Because it is in parentheses, it is caught in a group. \1 is a backreference to the 1st capturing group, so it is the equivalent of the text caught by the capturing group. After a bit of backtracking, the capturing group matches the first part of the string, abc. The backreference \1 is now the equivalent of abc, so it matches the second half of the string. The entire string is now matched, so it is confirmed that the first half of the string matches the second half.
Another use of backreferences is in replacing. Say you want to replace all {...} with [...], if the text inside { and } is only digits. You can easily do this with capturing groups and backreferences, using the regex
{(\d+)}
And replacing with that with [\1].
The regex matches {123} in the string abc {123} 456, and captures 123 in the first capturing group. The backreference \1 is now the equivalent of 123, so replacing {(\d+)} in abc {123} 456 with [\1] results in abc [123] 456.
The reason non-capturing groups exist is because groups in general have more uses that just capturing. The regex (xyz)+ matches a string that consists entirely of the group, xyz, repeated, such as xyzxyzxyz. A group is needed because xyz+ only matches xy and then z repeated, i.e. xyzzzzz. The problem with using capturing groups is that they are slightly less efficient compared to non-capturing groups, and they take up an index. If you have a complicated regex with a lot of groups in it, but you only need to reference a single one somewhere in the middle, it's a lot better to just reference \1 rather than trying to count all the groups up to the one you want.
I hope this helps!
Can't think of an appropriate example at the moment, but I'm assuming someone might need to know the number of sub matches in the RegEx.
Group 0 is always the entire base match. I'm assuming groupCount() just lets you know how many capture groups you've specified in the expression.
A non-capturing group (?:) would be used to, well, not capture a group. Ex. if you need to test if a string contains one of several words and don't want to capture the word in a new group: (?:hello|hi there) world !== hello|hi there world. The first matches "hello world" or "hi there world" but the second matches "hello" or "hi there world".
They can be used as a part of a multitude of powerful reasons, such as testing whether or not a number is prime or composite. :) Or you could simply test to ensure a search parameter isn't repeated, ie. ^(\d)(?!.*\1)\d+$ would ensure the first digit is unique in a string.
Straightforward question, it's just difficult to google regex syntax...
I'm going through the HortonWorks Hive tutorials (Hive uses same regex as Java), and the following SELECT statement uses regex to pull from what's probably JSON data...
INSERT OVERWRITE TABLE batting
SELECT
regexp_extract(col_value,'^(?:([^,]*)\.?){1}',1) player_id,
regexp_extract(col_value,'^(?:([^,]*)\.?){2}',1) year,
regexp_extract(col_value,'^(?:([^,]*)\.?){9}',1) run
FROM temp_batting;
The data looks like this:
PlayerID,yearID,stint,teamID,lgID,G,G_batting,AB,R,H,2B,3B,HR,RBI,SB,CS,BB,SO,IBB,HBP,SH,SF,GIDP,G_old
aardsda01,2004,1,SFN,NL,11,11,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,11
aardsda01,2006,1,CHN,NL,45,43,2,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,45
aardsda01,2007,1,CHA,AL,25,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2
And so PlayerID is in column1, year is column2, R (runs) is column 9. How is regexp_extract successfully pulling this data?
I'm new to non-capturing groups, but it looks to me like the entire thing is a non-capturing group. Also, I'm used to seeing {1}, {2}, or {9} in the form [0-9]{9} meaning it matches a 9-digit number. In this case it looks like it's pointing to the 9th match of something, what is this syntax called?
First break apart the regex:
^(?:([^,]*)\.?){n}
^ is the start of a String
(?:...){n} is a non-capturing group repeated n times
([^,]*) is a negated character class, it matches "not ," zero or more times
\.? is an optional (literal) .
So, how does this work?
The non-capturing group is solely there for the numeric quantifier, i.e. it makes the entire pattern in the group repeat n times.
The actual pattern being captured is in the capturing group ([^,]*). I'm not sure why the optional . is there and I don't see any inputs ending with a . in your sample data but I assume there are some.
What happens is the the group is captured n times but only the last capture is stored and this is stored in the first group, i.e. group 1. This is the default in the regexp_extract.
So when the pattern repeats once in the first case we capture the first element on the comma separated array. When the pattern repeats twice in the second example we capture the second element. When the pattern repeats nine times then the ninth element is captured.
The pattern itself is actually pretty horrible as it allows for a zero length pattern to be repeated, this means that the regex engine can backtrack a lot if there is a non-matching pattern. I imagine this isn't an issue for you but it is generally bad practice.
It would be best to either make the [^,]* possessive by adding a +:
^(?:([^,]*+)\.?){n}
Or make the entire non-capturing group atomic:
^(?>([^,]*)\.?){n}
I believe that a good way to practice and learn regex is on this site: http://www.regexr.com/
Just paste your expression on there, and delete/replace parts of it. It'll all make a bit more sense than trying to decipher a regex by sight.
Another way to do this without using regex is to use split function
select split('aardsda01,2006,1,CHN,NL,45,43,2,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,45',',')[0] as player_id,
split('aardsda01,2006,1,CHN,NL,45,43,2,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,45',',')[1] as year,
split('aardsda01,2006,1,CHN,NL,45,43,2,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,45',',')[1] as runs
I'm trying to get record1, record2, record3 from text:
"Record1 ANY TEXT 123 4 5 Record2 ANOTHER TEXT 90-8098 Record3 MORE TEXT ASD 123"
Each record appears ONE or ZERO times.
I use pattern:
(Record1.*)?(Record2.*)?(Record3.*)?
If each record appears,
matcher.group(1) == "Record1 ANY TEXT 123 4 5 Record2 ANOTHER TEXT 90-8098 Record3 MORE TEXT ASD 123"
matcher.group(2) == null
matcher.group(3) == null
If I use pattern:
(Record1.*)(Record2.*)(Record3.*)
matcher.group(1) == "Record1 ANY TEXT 123 4 5 "
matcher.group(2) == "Record2 ANOTHER TEXT 90-8098 "
matcher.group(3) == "Record3 MORE TEXT ASD 123"
It's exatly what I want, but each record can appear zero time and this regexp not suitable
What pattern should I use?
You want to make your quantifiers non-greedy, and you want to use anchors:
^.*?(Record1.*?)?(Record2.*?)?(Record3.*?)?$
In your original expression, your .* was basically consuming everything to the end of the string, because that's how regular expressions behave, by default (called greedy matching). Since the second and third groups were optional, there was no reason for the engine not to simply match everything with that first .*—it was the most efficient match.
By adding a ? after any quantifier, e.g. *? or +? or ?? or {m,n}?, you instruct the engine to match as little as possible, i.e. invoke non-greedy matching.
So, why the anchors? Well, if you invoke non-greedy matching, the engine's going to try to match as little as possible. So, it'd match nothing, since all your groups are optional! By forcing the whole expression to match the beginning, ^, as well as the end, $, you force to regular expression to find some way to match as few characters as possible via .*?, but still match as much as needed to get all the details.
If your text is tightly packed and is composed of just Record, why not use split
(if Java calls it split).
split regex:
# "(?:(?!Record)[\\S\\s])*(Record[\\S\\s]*?)(?=Record|$(?!\\n))"
(?:
(?! Record )
[\S\s]
)*
( Record [\S\s]*? )
(?=
Record
| $ (?! \n )
)
Directly from this java API (ctrl + f) + "Group name":
The captured input associated with a group is always the subsequence
that the group most recently matched. If a group is evaluated a second
time because of quantification then its previously-captured value, if
any, will be retained if the second evaluation fails. Matching the
string "aba" against the expression (a(b)?)+, for example, leaves
group two set to "b". All captured input is discarded at the beginning
of each match.
I know how capturing groups work and how they work with backreference.
However I have not got the point of the API bit I above quoted. Is somebody able to put it down in other words?
Thanks in advance.
That quote says that:
If you have used a quantifier - +, *, ? or {m,n}, on your capture group, and your group is matched more than once, then only the last match will be associated with the capture group, and all the previous matches will be overridden.
For e.g.: If you match (a)+ against the string - "aaaaaa", your capture group 1 will refer to the last a.
Now consider the case, where you have a nested capture group as in the example shown in your quote:
`(a(b)?)+`
matching this regex with the string - "aba", you get the following 2 matches:
"ab" - Capture Group 1 = "ab" (due to outer parenthesis), Capture Group 2 = "b"(due to inner parenthesis)
"a" - Capture Group 1 = "a", Capture Group 2 = None. (This is because second capture group (b)? is optional. So, it successfully matches the last a.
So, finally your Capture group 1 will contain "a",which overrides earlier captured group "ab", and Capture group 2 will contain "b", which is not overridden.
Named captures or not is irrelevant in this case.
Consider this input text:
foo-bar-baz
and this regex:
[a-z]+(-[a-z]+)*
Now the question is what is captured by group 1?
As the regex progresses through the text, it first matches -bar which is then the contents of group 1; but then it goes on in the text and recognizes -baz which is now the new content of group 1.
Therefore, -bar is "lost": the regex engine has discarded it because further text in the input matched the capturing group. This is what is meant by this:
[t]he captured input associated with a group is always the subsequence that the group most recently matched [emphasis mine]