I am new to working with ANTLR parser.
Here is my grammar:
grammar Commands;
file_ : expression EOF;
expression : Command WhiteSpace Shape ;
WhiteSpace : [\t]+ -> skip;
NewLine : ('\r'?'\n'|'\r') -> skip;
Shape : ('square'|'triangle'|'circle'|'hexagon'|'line');
Command : ('fill'|'draw'|'delete');
I am trying to parse a list of sentences such as:
draw circle;
draw triangle;
delete circle;
I'm getting
token recognition error at:' '
Can anyone tell me what is the problem?
PS: I'm working in java 15
UPDATE
file_ : expressions EOF;
expressions
: expressions expression
| expression
;
expression : Command WhiteSpace Shape NewLine ;
WhiteSpace : [\t]+ -> skip;
NewLine : ('\r'?'\n'|'\r') -> skip;
Shape : ('square'|'triangle'|'circle'|'hexagon'|'line');
Command : ('fill'|'draw'|'delete');
Added support for multiple expressions.
I'm getting the same error.
UPDATE
grammar Commands;
file_ : expressions EOF;
expressions
: expressions expression
| expression
;
expression : Command Shape;
WhiteSpace : [\t]+ -> skip;
NewLine : ('\r'?'\n'|'\r') -> skip;
Shape : ('square'|'triangle'|'circle'|'hexagon'|'line');
Command : ('fill'|'draw'|'delete');
Even if I don't include WhiteSpace, I get the same token recognition error.
OK, the errors:
line 3:6 token recognition error at: ' '
line 3:13 token recognition error at: ';'
mean that the lexer encountered a white space char (or semi colon), but there is no lexer rule that matches any of these characters. You must include them in your grammar. Let's say you add them like this (note: still incorrect!):
Semi : ';';
WhiteSpace : [ \t]+ -> skip;
When trying with the rules above, you'd get the error:
line 1:5 missing WhiteSpace at 'circle'
This means the parser cannot match the rule expression : Command WhiteSpace Shape ; to the input draw circle;. This is because inside the lexer, you're skipping all white space characters. This means these tokens will not be available inside a parser rule. Remove them from your parser.
You'll also see the error:
line 1:11 mismatched input ';' expecting <EOF>
which means the input contains a Semi token, and the parser did not expect that. Include the Semi token in your expression rule:
grammar Commands;
file_ : expression EOF;
expression : Command Shape Semi;
Semi : ';';
WhiteSpace : [ \t]+ -> skip;
NewLine : ('\r'?'\n'|'\r') -> skip;
Shape : ('square'|'triangle'|'circle'|'hexagon'|'line');
Command : ('fill'|'draw'|'delete');
The grammar above will work for single expressions. If you want to match multiple expressions, you could do:
expressions
: expressions expression
| expression
;
but given that ANTLR generates LL parsers (not LR as the name ANTLR suggests), it is easier (and makes the parse tree easier to traverse later on) to do this:
expressions
: expression+
;
If you're going to skip all white space chars, you might as well remove the NewLine rule and do this:
WhiteSpace : [ \t\r\n]+ -> skip;
One more thing, the lexer now creates Shape and Command tokens which all have the same type. I'd do something like this instead:
shape : Square | Triangle | ...;
Square : 'square';
Triangle : 'triangle';
...
which will make your life easier while traversing the parse tree when you want to evaluate the input (if that is what you're going to do).
I'd go for something like this:
grammar Commands;
file_ : expressions EOF;
expressions : expression+;
expression : command shape Semi;
shape : Square | Traingle | Circle | Hexagon | Line;
command : Fill | Draw | Delete;
Semi : ';';
WhiteSpace : [ \t\r\n]+ -> skip;
Square : 'square';
Traingle : 'triangle';
Circle : 'circle';
Hexagon : 'hexagon';
Line : 'line';
Fill : 'fill';
Draw : 'draw';
Delete : 'delete';
Your whitespace token rule WhiteSpace only allows for tabs. add a space to it.
WhiteSpace : [ \t]+ -> skip;
(usually, there's more to a whitespace rule than that, but it should solve your immediate problem.
You also haven't accounted for the ';' in your input. Either add it to a rule, or remove from your test input temporarily.
expression : Command Shape ';' ;
This would fix it, but seems like it might not be what you really need.
Related
i compiled the :
grammar Hello; // Define a grammar called Hello
r : 'hello' ID ; // match keyword hello followed by an identifier
ID : [a-z]+ ; // match lower-case identifiers
WS : [ \t\r\n]+ -> skip ; // skip spaces, tabs, newlines, \r (Windows)r
the command to generate the class files (notice im creating it with -package hellogrammer ) :
java -jar antlr-4.9.2-complete.jar -package hellogrammer -o c:\Dev\my\java\ANTLR\test_project\core\src\main\java\hellogrammer c:\Dev\my\java\ANTLR\test_project\core\src\main\java\Hello.g4
and it creates the files just fine , then i compile the files and it looks like this :
c:\Dev\my\java\ANTLR\test_project\core\target\classes\hellogrammer>ls -1
HelloBaseListener.class
HelloLexer.class
HelloListener.class
'HelloParser$RContext.class'
HelloParser.class
now when I try to execute the TestRig command im getting no response from the command line :
c:\Dev\my\java\ANTLR\test_project\core\target\classes>java -cp ".;C:/Dev/my/java/ANTLR/antlr-4.9.2-complete.jar" org.antlr.v4.gui.TestRig hellogrammer.Hello -tree
it just stacks with no error or any response ...
The TestRig requires two separate parameters, first the grammar name and the the start rule name. This TestRig then begins parsing input from the input stream, so, you can either type input (with a Ctrl-D for signal EOF), or you can redirect your input to stdin with <
Try:
c:\Dev\my\java\ANTLR\test_project\core\target\classes>java -cp ".;C:/Dev/my/java/ANTLR/antlr-4.9.2-complete.jar" org.antlr.v4.gui.TestRig hellogrammar.Hello r -tree < “your input file”
am very new to this ANTLR tool, Need help on writing grammar rules in ANTRL, for converting/parsing plaint text to equivalent .xml file, using java.
please any one help me on this.
i tried as below as per my understanding, and it works for single line(parser) not for full configList(parser)
ANTLR grammar rules below is my grammar .g4
grammar MyTest;
acl : 'acl number' INT configList ('#' configList)* ;
configList : config ('\n' config)*;
config : line ('\n' line)* ;
line : line WORD INT (WORD)+ ((SOURCE_LOW_IP)* |(WORD)* |(SOURCE_LOW_IP)*)+
|WORD INT (WORD)+
;
fragment
DIGIT : ('0'..'9');
INT : [0-9]+ ; // Define token INT as one or more digits
//WORD : [A-Za-z][A-Za-z_\-]* ;
WORD : [A-Za-z][A-Za-z_\-]* ;
NEWLINE:'\r'? '\n' ; // return newlines to parser (is end-statement signal)
WS : [ \t\r\n]+ -> skip ; // toss out whitespace
SOURCE_LOW_IP : INT '.' INT '.' INT '.' INT ; // match IPs in parser
sample input of config list:
acl number 3001
rule 0 permit ip source any
rule 1 permit ip source 172.16.10.1
#
rule 2 permit ip source 172.16.10.2 0.0.0.255
rule 3 deny destination any
rule 4 deny destination 172.16.10.4
rule 5 deny destination 172.16.10.5 0.0.0.255
#
rule 6 permit ip source any destination 172.16.10.6 0.0.0.255
rule 7 permit ip source 172.16.10.7 0.0.0.255 destination 172.16.11.7
#
expected for output format as below( this will be taken care using java once antlr generates .java and other files)
<filterRuleLists>
<filterRuleList id='3001'>
<filterRule action='ALLOW' protocol='ANY'>
<sourceIPRange low='0.0.0.0' high='255.255.255.255' />
<destinationIPRange low='0.0.0.0' high='255.255.255.255' />
<fileLine file='config' startLine='4' stopLine='4' />
</filterRule>
<filterRule action='ALLOW' protocol='ANY'>
<sourceIPRange low='172.16.10.1' high='172.16.10.1' />
<destinationIPRange low='0.0.0.0' high='255.255.255.255' />
<fileLine file='config' startLine='5' stopLine='5' />
</filterRule>
</filterRuleList>
</filterRuleLists>
I am familiar with parser generators, but not ANTLR4 specifically, so this is a best-guess: I strongly suspect the grammar rules
configList : config ('\n' config)*;
config : line ('\n' line)* ;
should be rewritten as
configList : config (NEWLINE config)*;
config : line (NEWLINE line)* ;
as the fragment rule
NEWLINE:'\r'? '\n' ; // return newlines to parser (is end-statement signal)
will cause any '\n' characters to be processed into NEWLINE tokens.
The situation:
rule : block+ ;
block : '[' String ']' ;
String : ([a-z] | '[' | '\\]')+ ;
Trick is String can contain [ without backslash escape and ] with backslasash escape, so in this example:
[hello\]world][hello[[world]
First block can be parsed correctly, but the second one... parser is trying find ] for every [. Is there way to say antlr parser to ignore this standalone [? I can't change format, but i need to find some workaround with antlr.
PS: Without antlr there is algorythm to avoid this, something like: collect [ in queue before we will find first ] and use only head of queue. But I really need antlr =_=
You can use Lexer modes.
Lexical modes allow us to split a single lexer grammar into multiple
sublexers. The lexer can only return tokens matched by rules from the
current mode.
You can read more about lexer rules in antlr documentation here.
First you will need to divide you grammar into separate lexer and parser. Than just use another mode after you see open bracket.
Parser grammar:
parser grammar TestParser;
options { tokenVocab=TestLexer; }
rul : block+ ;
block : LBR STRING RBR ;
Lexer grammar:
lexer grammar TestLexer;
LBR: '[' -> pushMode(InString);
mode InString;
STRING : ([a-z] | '\\]' | '[')+ ;
RBR: ']' -> popMode;
Working example is here.
You can read the documentation on lexer modes
I'm looking for some advice on my school project. I am supposed to create a program that takes a logical expression and outputs a truth table for it. The actually creating of the truth table for me is not difficult at all and I've already wrote the methods in Java for it. I would like to know if there are any classes in java that I could use to parse the expression for me and put it into a stack. If not I'm looking for help on parsing the expression. It's the parentheses that get me whenever I try and think it through. Also if this would be easier in any other language I would be open to doing it in that. Perl is probably my next best language.
Some examples
(P && Q) -> R
(P || Q || R) && ((P -> R) -> Q)
If you're allowed to use a parser generator tool like ANTLR, here's how you could get started. The grammar for a simple logic-language could look like this:
grammar Logic;
parse
: expression EOF
;
expression
: implication
;
implication
: or ('->' or)*
;
or
: and ('||' and)*
;
and
: not ('&&' not)*
;
not
: '~' atom
| atom
;
atom
: ID
| '(' expression ')'
;
ID : ('a'..'z' | 'A'..'Z')+;
Space : (' ' | '\t' | '\r' | '\n')+ {$channel=HIDDEN;};
However, if you'd parse input like (P || Q || R) && ((P -> R) -> Q) with a parser generated from the grammar above, the parse tree would contain the parenthesis (something you're not interested in after parsing the expression) and the operators would not be the root of each sub-trees, which doesn't make your life any easier if you're interested in evaluating the expression.
You'll need to tell ANTLR to omit certain tokens from the AST (this can be done by placing a ! after the token/rule) and make certain tokens/rules the root of their (sub) tree (this can be done by placing a ^ after it). Finally, you need to indicate in the options section of your grammar that you want a proper AST to be created instead of a simple parse tree.
So, the grammar above would look like this:
// save it in a file called Logic.g
grammar Logic;
options {
output=AST;
}
// parser/production rules start with a lower case letter
parse
: expression EOF! // omit the EOF token
;
expression
: implication
;
implication
: or ('->'^ or)* // make `->` the root
;
or
: and ('||'^ and)* // make `||` the root
;
and
: not ('&&'^ not)* // make `&&` the root
;
not
: '~'^ atom // make `~` the root
| atom
;
atom
: ID
| '('! expression ')'! // omit both `(` and `)`
;
// lexer/terminal rules start with an upper case letter
ID : ('a'..'z' | 'A'..'Z')+;
Space : (' ' | '\t' | '\r' | '\n')+ {$channel=HIDDEN;};
You can test the parser with the following class:
import org.antlr.runtime.*;
import org.antlr.runtime.tree.*;
import org.antlr.stringtemplate.*;
public class Main {
public static void main(String[] args) throws Exception {
// the expression
String src = "(P || Q || R) && ((P -> R) -> Q)";
// create a lexer & parser
LogicLexer lexer = new LogicLexer(new ANTLRStringStream(src));
LogicParser parser = new LogicParser(new CommonTokenStream(lexer));
// invoke the entry point of the parser (the parse() method) and get the AST
CommonTree tree = (CommonTree)parser.parse().getTree();
// print the DOT representation of the AST
DOTTreeGenerator gen = new DOTTreeGenerator();
StringTemplate st = gen.toDOT(tree);
System.out.println(st);
}
}
Now to run the Main class, do:
*nix/MacOS
java -cp antlr-3.3.jar org.antlr.Tool Logic.g
javac -cp antlr-3.3.jar *.java
java -cp .:antlr-3.3.jar Main
Windows
java -cp antlr-3.3.jar org.antlr.Tool Logic.g
javac -cp antlr-3.3.jar *.java
java -cp .;antlr-3.3.jar Main
which will print a DOT source of the following AST:
(image produced with graphviz-dev.appspot.com)
Now all you need to do is evaluate this AST! :)
In Perl you can use Regexp::Grammars to do the parsing. It may be a little on the "grenade to kill an ant" side, but it should work.
Edit: Here is a (very quick) example which might get you going.
#!/usr/bin/env perl
use strict;
use warnings;
use Regexp::Grammars;
use Data::Dumper;
my $parser = qr/
<nocontext:>
<Logic>
<rule: Logic> <[Element]>*
<rule: Element> <Group> | <Operator> | <Item>
<rule: Group> \( <[Element]>* \)
<rule: Operator> (?:&&) | (?:\|\|) | (?:\-\>)
<rule: Item> \w+
/xms; #/ #Fix Syntax Highlight
my $text = '(P && Q) -> R';
print Dumper \%/ if $text =~ $parser; #/ #Fix Syntax Highlight
Look into JavaCC or ANTLR.
Regexps won't work.
You can probably also run your own parser using StreamTokenizer.
Building an expression parser is easy. Attaching actions to compute a value as you parse it is easy, too.
I assume you can write a BNF for your expression language.
This answer shows you how to build a parser easily, if you have a BNF.
Is there an alternative for flex/bison that is usable on 8-bit embedded systems?
If you want to write your own parser, use the Shunting-yard algorithm to get rid of parentheses by converting the expression from infix into postfix notation or directly into a tree.
Another parser generator for Java is CUP.
I'm looking for some advice on my school project. I am supposed to create a program that takes a logical expression and outputs a truth table for it. The actually creating of the truth table for me is not difficult at all and I've already wrote the methods in Java for it. I would like to know if there are any classes in java that I could use to parse the expression for me and put it into a stack. If not I'm looking for help on parsing the expression. It's the parentheses that get me whenever I try and think it through. Also if this would be easier in any other language I would be open to doing it in that. Perl is probably my next best language.
Some examples
(P && Q) -> R
(P || Q || R) && ((P -> R) -> Q)
If you're allowed to use a parser generator tool like ANTLR, here's how you could get started. The grammar for a simple logic-language could look like this:
grammar Logic;
parse
: expression EOF
;
expression
: implication
;
implication
: or ('->' or)*
;
or
: and ('||' and)*
;
and
: not ('&&' not)*
;
not
: '~' atom
| atom
;
atom
: ID
| '(' expression ')'
;
ID : ('a'..'z' | 'A'..'Z')+;
Space : (' ' | '\t' | '\r' | '\n')+ {$channel=HIDDEN;};
However, if you'd parse input like (P || Q || R) && ((P -> R) -> Q) with a parser generated from the grammar above, the parse tree would contain the parenthesis (something you're not interested in after parsing the expression) and the operators would not be the root of each sub-trees, which doesn't make your life any easier if you're interested in evaluating the expression.
You'll need to tell ANTLR to omit certain tokens from the AST (this can be done by placing a ! after the token/rule) and make certain tokens/rules the root of their (sub) tree (this can be done by placing a ^ after it). Finally, you need to indicate in the options section of your grammar that you want a proper AST to be created instead of a simple parse tree.
So, the grammar above would look like this:
// save it in a file called Logic.g
grammar Logic;
options {
output=AST;
}
// parser/production rules start with a lower case letter
parse
: expression EOF! // omit the EOF token
;
expression
: implication
;
implication
: or ('->'^ or)* // make `->` the root
;
or
: and ('||'^ and)* // make `||` the root
;
and
: not ('&&'^ not)* // make `&&` the root
;
not
: '~'^ atom // make `~` the root
| atom
;
atom
: ID
| '('! expression ')'! // omit both `(` and `)`
;
// lexer/terminal rules start with an upper case letter
ID : ('a'..'z' | 'A'..'Z')+;
Space : (' ' | '\t' | '\r' | '\n')+ {$channel=HIDDEN;};
You can test the parser with the following class:
import org.antlr.runtime.*;
import org.antlr.runtime.tree.*;
import org.antlr.stringtemplate.*;
public class Main {
public static void main(String[] args) throws Exception {
// the expression
String src = "(P || Q || R) && ((P -> R) -> Q)";
// create a lexer & parser
LogicLexer lexer = new LogicLexer(new ANTLRStringStream(src));
LogicParser parser = new LogicParser(new CommonTokenStream(lexer));
// invoke the entry point of the parser (the parse() method) and get the AST
CommonTree tree = (CommonTree)parser.parse().getTree();
// print the DOT representation of the AST
DOTTreeGenerator gen = new DOTTreeGenerator();
StringTemplate st = gen.toDOT(tree);
System.out.println(st);
}
}
Now to run the Main class, do:
*nix/MacOS
java -cp antlr-3.3.jar org.antlr.Tool Logic.g
javac -cp antlr-3.3.jar *.java
java -cp .:antlr-3.3.jar Main
Windows
java -cp antlr-3.3.jar org.antlr.Tool Logic.g
javac -cp antlr-3.3.jar *.java
java -cp .;antlr-3.3.jar Main
which will print a DOT source of the following AST:
(image produced with graphviz-dev.appspot.com)
Now all you need to do is evaluate this AST! :)
In Perl you can use Regexp::Grammars to do the parsing. It may be a little on the "grenade to kill an ant" side, but it should work.
Edit: Here is a (very quick) example which might get you going.
#!/usr/bin/env perl
use strict;
use warnings;
use Regexp::Grammars;
use Data::Dumper;
my $parser = qr/
<nocontext:>
<Logic>
<rule: Logic> <[Element]>*
<rule: Element> <Group> | <Operator> | <Item>
<rule: Group> \( <[Element]>* \)
<rule: Operator> (?:&&) | (?:\|\|) | (?:\-\>)
<rule: Item> \w+
/xms; #/ #Fix Syntax Highlight
my $text = '(P && Q) -> R';
print Dumper \%/ if $text =~ $parser; #/ #Fix Syntax Highlight
Look into JavaCC or ANTLR.
Regexps won't work.
You can probably also run your own parser using StreamTokenizer.
Building an expression parser is easy. Attaching actions to compute a value as you parse it is easy, too.
I assume you can write a BNF for your expression language.
This answer shows you how to build a parser easily, if you have a BNF.
Is there an alternative for flex/bison that is usable on 8-bit embedded systems?
If you want to write your own parser, use the Shunting-yard algorithm to get rid of parentheses by converting the expression from infix into postfix notation or directly into a tree.
Another parser generator for Java is CUP.