Antlr4 has a new class ParseTreeWalker. But how do I use it? I am looking for a minimal working example. My grammar file is 'gram.g4' and I want to parse a file 'program.txt'
Here is my code so far. (This assumes ANTLR has run my grammar file and created all of the gramBaseListener, gramLexer, etc etc):
import org.antlr.v4.runtime.*;
import org.antlr.v4.runtime.tree.*;
import static org.antlr.v4.runtime.CharStreams.fromFileName;
public class launch{
public static void main(String[] args) {
CharStream cs = fromFileName("gram.g4"); //load the file
gramLexer lexer = new gramLexer(cs); //instantiate a lexer
CommonTokenStream tokens = new CommonTokenStream(lexer); //scan stream for tokens
gramParser parser = new gramParser(tokens); //parse the tokens
// Now what?? How do I connect the above with the below?
ParseTreeWalker walker = new ParseTreeWalker(); // how do I use this to parse program.txt??
}}
I am using java but I assume it is similar in other languages.
The ANTLR documentation (http://www.antlr.org/api/Java/index.html) is short on examples. There are many tutorials on the internet but they are mostly for ANTLR version 3. The few using version 4 don't work or are outdated (for example, there is no parser.init() function, and classes like ANTLRInputStream are depreciated)
Thanks in advance for anyone who can help.
For each of your parser rules in your grammar the generated parser will have a corresponding method with that name. Calling that method will start parsing at that rule.
Therefore if your "root-rule" is named start then you'd start parsing via gramParser.start() which returns a ParseTree. This tree can then be fed into the ParseTreeWalker alongside with the listener you want to be using.
All in all it could look something like this (EDITED BY OP):
import org.antlr.v4.runtime.*;
import org.antlr.v4.runtime.tree.*;
import static org.antlr.v4.runtime.CharStreams.fromFileName;
public class launch{
public static void main(String[] args) {
CharStream cs = fromFileName("program.txt"); //load the file
gramLexer lexer = new gramLexer(cs); //instantiate a lexer
CommonTokenStream tokens = new CommonTokenStream(lexer); //scan stream for tokens
gramParser parser = new gramParser(tokens); //parse the tokens
ParseTree tree = parser.start(); // parse the content and get the tree
Mylistener listener = new Mylistener();
ParseTreeWalker walker = new ParseTreeWalker();
walker.walk(listener,tree);
}}
************ NEW FILE Mylistener.java ************
public class Mylistener extends gramBaseListener {
#Override public void enterEveryRule(ParserRuleContext ctx) { //see gramBaseListener for allowed functions
System.out.println("rule entered: " + ctx.getText()); //code that executes per rule
}
}
Of course you have to replace <listener> with your implementation of BaseListener
And just one small sidenode: In Java it is convention to start classnames with capital letters and I'd advise you to stick to that in order for making the code more readable for other people.
This example should work with ANTLR 4.8.
Below the example you can find references to setup your Java env, API and Listeners.
public class Launch {
public static void main(String[] args) {
InputStream inputStream = null;
MyprogramLexer programLexer = null;
try {
File file = new File("/program.txt");
inputStream = new FileInputStream(file);
programLexer = new MyprogramLexer(CharStreams.fromStream(inputStream)); // read your program input and create lexer instance
} finally {
if (inputStream != null) {
inputStream.close();
}
}
/* assuming a basic grammar:
myProgramStart: TOKEN1 otherRule TOKEN2 ';' | TOKENX finalRule ';'
...
*/
CommonTokenStream tokens = new CommonTokenStream(programLexer); // get tokens
MyParser parser = new MyParser(tokens);
MyProgramListener listener = new MyProgramListener(); // your custom extension from BaseListener
parser.addParseListener(listener);
parser.myProgramStart().enterRule(listener); // myProgramStart is your grammar rule to parse
// what we had built?
MyProgram myProgramInstance = listener.getMyProgram(); // in your listener implementation populate a MyProgram instance
System.out.println(myProgramInstance.toString());
}
}
References:
https://www.antlr.org/api/Java/
https://tomassetti.me/antlr-mega-tutorial/#java-setup
https://riptutorial.com/antlr/example/16571/listener-events-using-labels
Related
Compiling this using cmd : javac Test.java. However compilation fails, saying it cant find symbol parser.prog(). Any ideas?
import org.antlr.runtime.*;
public class TestT {
public static void main(String[] args) throws Exception {
// Create an TLexer that feeds from that stream
//TLexer lexer = new TLexer(new ANTLRInputStream(System.in));
TLexer lexer = new TLexer(new ANTLRFileStream("input.txt"));
// Create a stream of tokens fed by the lexer
CommonTokenStream tokens = new CommonTokenStream(lexer);
// Create a parser that feeds off the token stream
TParser parser = new TParser(tokens);
// Begin parsing at rule prog
parser.prog();
}
}
In your T.g4 grammar (or T.g), you must also have a parser rule named prog:
grammar T;
prog
: ...
;
...
Looking at your generated parser, I see you have a parser rule like this:
filter
: expression EOF
;
Use that instead:
// Begin parsing at rule prog
parser.filter();
I have an antlr4 based project with a Main class containing this code:
package com.progur.langtutorial;
import java.io.FileInputStream;
import java.io.IOException;
import org.antlr.v4.runtime.CharStream;
import org.antlr.v4.runtime.CharStreams;
import org.antlr.v4.runtime.CommonTokenStream;
public class Main {
#SuppressWarnings("deprecation")
public static void main(String[] args) {
try {
ANTLRInputStream input = new ANTLRInputStream(
new FileInputStream(args[0]));
GYOOLexer lexer = new GYOOLexer(input);
GYOOParser parser = new GYOOParser(new CommonTokenStream(lexer));
parser.addParseListener(new MyListener());
// Start parsing
parser.program();
} catch (IOException e) {
e.printStackTrace();
}
}
}
However, since ANTLRInputStream is deprecated, I need to use CharStream instead.
But, when I tried to use CharStream, I cannot move further than,
CharStream input = new ANTLRInputStream(
new FileInputStream(args[0]));
This is because I do not know how to replace the part of the statement after the '=' sign. I tried CharStreams.fromFileName(new FileInputStream(args[0])); but then eclipse states this error "CharStreams.fromFileName cannot be resolved to a type". I also tried CharStreams.fromFileName(args[0]); with the same result.
I even tried CharStream input = new CharStreams.fromFileName("test"); where "test" is the program written to test the language parser that I've written. It was also the same.
I am also having another error in parser.addParseListener(new MyListener()); where it says MyListener cannot be resolved to a type. What could that mean? In every tutorial I looked there was a random name for where 'MyListener()' is.
What should be the correct statement for this?
Thanks!
Like this:
CharStream charStream = CharStreams.fromString("test");
where "test" is the input itself to be parsed.
Or when the input is in a file, do this:
CharStream charStream = CharStreams.fromFileName("/path/to/file.ext");
If that doesn't work, you need to inspect the exception that is thrown (most likely the file is not where ANTLR is looking for it: try an absolute path).
I am using antlr v4 for extracting parse tree of java programs for other purposes. I have started from this sample: ANTLR v4 visitor sample
And I have tested the steps on given link to check if it works and everything gone right:
java Run
a = 1+2
b = a^2
c = a+b*(a-1)
a+b+c
^Z
Result: 33.0
And then I wrote my own to parse java programs as Structure below:
|_Java.g4
|_Java.tokens
|_JavaBaseVisitor.java
|_JavaLexer.java
|_JavaLexer.tokens
|_JavaParser.java
|_JavaTreeExtractorVisitor.java
|_JavaVisitor.java
|_Run.java
And the Run.java is as below:
import org.antlr.v4.runtime.*;
import org.antlr.v4.runtime.tree.*;
public class Run {
public static void main(String[] args) throws Exception {
CharStream input = CharStreams.fromFileName("F:\\Projects\\Java\\Netbeans\\ASTProj\\JavaTreeExtractor\\prog.java");
JavaLexer lexer = new JavaLexer(input);
CommonTokenStream tokens = new CommonTokenStream(lexer);
JavaParser parser = new JavaParser(tokens);
ParseTree tree = parser.getContext();
JavaTreeExtractorVisitor calcVisitor = new JavaTreeExtractorVisitor();
String result = calcVisitor.visit(tree);
System.out.println("Result: " + result);
}
}
But at the statement ParseTree tree = parser.getContext(); the tree object gets null.
As I am new to antlr, any suggestions for me to check or any solution?
(If more info is required, just notify me).
TG.
Assuming you're using the grammar here, you want the starting point for parsing a Java file to be
ParseTree tree = parser.compilationUnit();
(For anyone not using that grammar, you want whatever you named your top-level parser rule.)
Shouldn't you be doing:
ParseTree tree = parser.input();
as in the calculator example?
I use the grammar Java.g from the ANTLR wiki produces a lexer and parser for Java source files.Then use the following code to generate an abstract syntax tree (AST).
ANTLRInputStream input = new ANTLRInputStream(new FileInputStream(fileName));
JavaLexer lexer = new JavaLexer(input); // create lexer
// create a buffer of tokens pulled from the lexer
CommonTokenStream tokens = new CommonTokenStream(lexer);
JavaParser parser = new JavaParser(tokens); // create parser
JavaParser.javaSource_return r = parser.javaSource(); // parse rule 'javaSource'
/*RuleReturnScope result = parser.compilationUnit();
CommonTree t = (CommonTree) result.getTree();*/
// WALK TREE
// get the tree from the return structure for rule prog
CommonTree t = (CommonTree)r.getTree();
Then modify the AST. For example,replace "File file = new File(filepath, fileType);" to
"S3Object _file = new S3Object(_fileName);" by modify the AST node. After this,I want to translate this AST to java source code.I modify the JavaTreeParser.g and write a stringtemplate and use the following method to get the java source code:
FileReader groupFileR = new FileReader("src/com/googlecode/zcg/templates/JavaTemplate.stg");
StringTemplateGroup templates = new StringTemplateGroup(groupFileR);
groupFileR.close();
// create a stream of tree nodes from AST built by parser
CommonTreeNodeStream nodes = new CommonTreeNodeStream(t);
// tell it where it can find the token objects
nodes.setTokenStream(tokens);
JavaTreeParser walker = new JavaTreeParser(nodes); // create the tree Walker
walker.setTemplateLib(templates); // where to find templates
// invoke rule prog, passing in information from parser
JavaTreeParser.javaSource_return r2 = walker.javaSource();
// EMIT BYTE CODES
// get template from return values struct
StringTemplate output = (StringTemplate)r2.getTemplate();
System.out.println(output.toString()); // render full template
If I don't modify the AST,it will get the java source code correctly,but after I modify the AST,it doesn't get the right java source code(the AST was modified correctly).For example,if I input the following souce code,and translate to AST,then modify "File file = new File(filepath, fileType);" to "S3Object _file = new S3Object(_fileName);":
public void methodname(String address){
String filepath = "file";
int fileType = 3;
File file = new File(filepath, fileType);
}
the result will be the following:
public void methodname( String address)
{
String filepath="file";
int fileType=3;
methodname (Stringaddress){Stringfilepath;//it's not what I wanted
}
Am I doing it wrong? Is there a more proper way for me to solve this problem?
unfortunately I cannot recommend doing source to source translation by rewriting the abstract syntax trees; try using the parse trees. If I remember ANTLR 3 can also generate those easily.
Ter
I have generated and compiled a grammar with ANTLR4. VIA the command line I am able to see if there is an error, but I am having issues integrating this parser into a java program successfully. I am able to use ANTLR4 methods as I've added the JAR's to my library in Eclipse, however I am completely unable to retrieve token text or find out if an error is being generated in any sort of meaningful manner. Any help would be appreciated. If I'm being ambiguous by any means, please let me know and I'll delve into more detail.
Looking at previous versions, an equivalent method to something like compilationUnit() might be what I want.
Something like this should work (assuming you generated GeneratedLexer and GeneratedParser from your grammar):
import java.io.FileInputStream;
import java.io.InputStream;
import org.antlr.v4.runtime.ANTLRInputStream;
import org.antlr.v4.runtime.CommonTokenStream;
import org.antlr.v4.runtime.tree.ParseTree;
import test.GeneratedLexer;
import test.GeneratedParser;
public class Main {
public static void main(String[] args) throws Exception {
String inputFile = null;
if (args.length > 0) {
inputFile = args[0];
}
InputStream is = System.in;
if (inputFile != null) {
is = new FileInputStream(inputFile);
}
ANTLRInputStream input = new ANTLRInputStream(is);
GeneratedLexer lexer = new GeneratedLexer(input);
CommonTokenStream tokens = new CommonTokenStream(lexer);
GeneratedParser parser = new GeneratedParser(tokens);
ParseTree tree = parser.startRule();
// Do something useful with the tree (e.g. use a visitor if you generated one)
System.out.println(tree.toStringTree(parser));
}
}
You could also use a parser and lexer interpreter if you don't want to pregenerate them from your grammar (or you have a dynamic grammar).