I have generated and compiled a grammar with ANTLR4. VIA the command line I am able to see if there is an error, but I am having issues integrating this parser into a java program successfully. I am able to use ANTLR4 methods as I've added the JAR's to my library in Eclipse, however I am completely unable to retrieve token text or find out if an error is being generated in any sort of meaningful manner. Any help would be appreciated. If I'm being ambiguous by any means, please let me know and I'll delve into more detail.
Looking at previous versions, an equivalent method to something like compilationUnit() might be what I want.
Something like this should work (assuming you generated GeneratedLexer and GeneratedParser from your grammar):
import java.io.FileInputStream;
import java.io.InputStream;
import org.antlr.v4.runtime.ANTLRInputStream;
import org.antlr.v4.runtime.CommonTokenStream;
import org.antlr.v4.runtime.tree.ParseTree;
import test.GeneratedLexer;
import test.GeneratedParser;
public class Main {
public static void main(String[] args) throws Exception {
String inputFile = null;
if (args.length > 0) {
inputFile = args[0];
}
InputStream is = System.in;
if (inputFile != null) {
is = new FileInputStream(inputFile);
}
ANTLRInputStream input = new ANTLRInputStream(is);
GeneratedLexer lexer = new GeneratedLexer(input);
CommonTokenStream tokens = new CommonTokenStream(lexer);
GeneratedParser parser = new GeneratedParser(tokens);
ParseTree tree = parser.startRule();
// Do something useful with the tree (e.g. use a visitor if you generated one)
System.out.println(tree.toStringTree(parser));
}
}
You could also use a parser and lexer interpreter if you don't want to pregenerate them from your grammar (or you have a dynamic grammar).
Related
This question already has answers here:
Java interpreter? [closed]
(10 answers)
Closed 9 years ago.
For debug reasons, I want to be able to run code that is typed in through the console. For example:
BufferedReader br = new BufferedReader(new InputStreamReader(System.in));
while(true){
String str = br.readLine(); //This can return 'a = 5;','b = "Text";' or 'pckg.example.MyClass.run(5);'
if(str == null)
return;
runCode(str); //How would I do this?
}
PLEASE DON'T ACTUALLY USE THIS
I was under the assumption you wanted to evaluate a string as Java code, not some scripting engine like Javascript, so
I created this on a whim after reading this, using the compiler API mark mentioned. It's probably very bad practice but it (somewhat) works like you wanted it to. I doubt it'll be much use in debugging since it runs the code in the context of a new class. Sample usage is included at the bottom.
import javax.tools.JavaCompiler;
import javax.tools.StandardJavaFileManager;
import javax.tools.ToolProvider;
import java.io.BufferedReader;
import java.io.File;
import java.io.InputStreamReader;
import java.io.PrintWriter;
import java.net.URL;
import java.net.URLClassLoader;
import java.util.Arrays;
public class main {
public static void runCode(String s) throws Exception{
JavaCompiler jc = ToolProvider.getSystemJavaCompiler();
StandardJavaFileManager sjfm = jc.getStandardFileManager(null, null, null);
File jf = new File("test.java"); //create file in current working directory
PrintWriter pw = new PrintWriter(jf);
pw.println("public class test {public static void main(){"+s+"}}");
pw.close();
Iterable fO = sjfm.getJavaFileObjects(jf);
if(!jc.getTask(null,sjfm,null,null,null,fO).call()) { //compile the code
throw new Exception("compilation failed");
}
URL[] urls = new URL[]{new File("").toURI().toURL()}; //use current working directory
URLClassLoader ucl = new URLClassLoader(urls);
Object o= ucl.loadClass("test").newInstance();
o.getClass().getMethod("main").invoke(o);
}
public static void main(String[] args) {
BufferedReader br = new BufferedReader(new InputStreamReader(System.in));
while(true){
try {
String str = br.readLine(); //This can return 'a = 5;','b = "Text";' or 'pckg.example.MyClass.run(5);'
if(str == null)
return;
runCode(str); //How would I do this?
} catch(Exception e) {
e.printStackTrace();
}
}
}
}
//command line
> System.out.println("hello");
hello
> System.out.println(3+2+3+4+5+2);
19
> for(int i = 0; i < 10; i++) {System.out.println(i);}
0
1
2
3
4
5
6
7
8
9
With the SimpleJavaFileObject you could actually avoid using a file, as shown here, but the syntax seems a bit cumbersome so I just opted for a file in the current working directory.
EDIT: Convert String to Code offers a similar approach but it's not fully fleshed out
If the code is in JavaScript then you can run it with JavaScript engine:
Object res = new ScriptEngineManager().getEngineByName("js").eval(str);
JavaScript engine is part of Java SE since 1.6. See this guide http://download.java.net/jdk8/docs/technotes/guides/scripting/programmer_guide/index.html for details
You can use the Java scripting API which is located in the Package javax.script. There you can include several scripting languages like bsh for example.
You can find a programmer's guide on the web page of Oracle.
Rhino, which is some kind of JavaScript is already included with the Oracle JVM.
For this you may want to look into Java Compiler API. I haven't studied much as to how this works, but it allows you to load a java file, compile and load the class in an already running system. Maybe it can be repurposed into accepting input from console.
For a general compiler you could use Janino which will allow you to compile and run Java code. The expression evaluator may help with your example.
If you are just looking to evaluate expressions while debugging then Eclispe has the Display view which allows you to execute expressions. See this question.
I have an antlr4 based project with a Main class containing this code:
package com.progur.langtutorial;
import java.io.FileInputStream;
import java.io.IOException;
import org.antlr.v4.runtime.CharStream;
import org.antlr.v4.runtime.CharStreams;
import org.antlr.v4.runtime.CommonTokenStream;
public class Main {
#SuppressWarnings("deprecation")
public static void main(String[] args) {
try {
ANTLRInputStream input = new ANTLRInputStream(
new FileInputStream(args[0]));
GYOOLexer lexer = new GYOOLexer(input);
GYOOParser parser = new GYOOParser(new CommonTokenStream(lexer));
parser.addParseListener(new MyListener());
// Start parsing
parser.program();
} catch (IOException e) {
e.printStackTrace();
}
}
}
However, since ANTLRInputStream is deprecated, I need to use CharStream instead.
But, when I tried to use CharStream, I cannot move further than,
CharStream input = new ANTLRInputStream(
new FileInputStream(args[0]));
This is because I do not know how to replace the part of the statement after the '=' sign. I tried CharStreams.fromFileName(new FileInputStream(args[0])); but then eclipse states this error "CharStreams.fromFileName cannot be resolved to a type". I also tried CharStreams.fromFileName(args[0]); with the same result.
I even tried CharStream input = new CharStreams.fromFileName("test"); where "test" is the program written to test the language parser that I've written. It was also the same.
I am also having another error in parser.addParseListener(new MyListener()); where it says MyListener cannot be resolved to a type. What could that mean? In every tutorial I looked there was a random name for where 'MyListener()' is.
What should be the correct statement for this?
Thanks!
Like this:
CharStream charStream = CharStreams.fromString("test");
where "test" is the input itself to be parsed.
Or when the input is in a file, do this:
CharStream charStream = CharStreams.fromFileName("/path/to/file.ext");
If that doesn't work, you need to inspect the exception that is thrown (most likely the file is not where ANTLR is looking for it: try an absolute path).
Antlr4 has a new class ParseTreeWalker. But how do I use it? I am looking for a minimal working example. My grammar file is 'gram.g4' and I want to parse a file 'program.txt'
Here is my code so far. (This assumes ANTLR has run my grammar file and created all of the gramBaseListener, gramLexer, etc etc):
import org.antlr.v4.runtime.*;
import org.antlr.v4.runtime.tree.*;
import static org.antlr.v4.runtime.CharStreams.fromFileName;
public class launch{
public static void main(String[] args) {
CharStream cs = fromFileName("gram.g4"); //load the file
gramLexer lexer = new gramLexer(cs); //instantiate a lexer
CommonTokenStream tokens = new CommonTokenStream(lexer); //scan stream for tokens
gramParser parser = new gramParser(tokens); //parse the tokens
// Now what?? How do I connect the above with the below?
ParseTreeWalker walker = new ParseTreeWalker(); // how do I use this to parse program.txt??
}}
I am using java but I assume it is similar in other languages.
The ANTLR documentation (http://www.antlr.org/api/Java/index.html) is short on examples. There are many tutorials on the internet but they are mostly for ANTLR version 3. The few using version 4 don't work or are outdated (for example, there is no parser.init() function, and classes like ANTLRInputStream are depreciated)
Thanks in advance for anyone who can help.
For each of your parser rules in your grammar the generated parser will have a corresponding method with that name. Calling that method will start parsing at that rule.
Therefore if your "root-rule" is named start then you'd start parsing via gramParser.start() which returns a ParseTree. This tree can then be fed into the ParseTreeWalker alongside with the listener you want to be using.
All in all it could look something like this (EDITED BY OP):
import org.antlr.v4.runtime.*;
import org.antlr.v4.runtime.tree.*;
import static org.antlr.v4.runtime.CharStreams.fromFileName;
public class launch{
public static void main(String[] args) {
CharStream cs = fromFileName("program.txt"); //load the file
gramLexer lexer = new gramLexer(cs); //instantiate a lexer
CommonTokenStream tokens = new CommonTokenStream(lexer); //scan stream for tokens
gramParser parser = new gramParser(tokens); //parse the tokens
ParseTree tree = parser.start(); // parse the content and get the tree
Mylistener listener = new Mylistener();
ParseTreeWalker walker = new ParseTreeWalker();
walker.walk(listener,tree);
}}
************ NEW FILE Mylistener.java ************
public class Mylistener extends gramBaseListener {
#Override public void enterEveryRule(ParserRuleContext ctx) { //see gramBaseListener for allowed functions
System.out.println("rule entered: " + ctx.getText()); //code that executes per rule
}
}
Of course you have to replace <listener> with your implementation of BaseListener
And just one small sidenode: In Java it is convention to start classnames with capital letters and I'd advise you to stick to that in order for making the code more readable for other people.
This example should work with ANTLR 4.8.
Below the example you can find references to setup your Java env, API and Listeners.
public class Launch {
public static void main(String[] args) {
InputStream inputStream = null;
MyprogramLexer programLexer = null;
try {
File file = new File("/program.txt");
inputStream = new FileInputStream(file);
programLexer = new MyprogramLexer(CharStreams.fromStream(inputStream)); // read your program input and create lexer instance
} finally {
if (inputStream != null) {
inputStream.close();
}
}
/* assuming a basic grammar:
myProgramStart: TOKEN1 otherRule TOKEN2 ';' | TOKENX finalRule ';'
...
*/
CommonTokenStream tokens = new CommonTokenStream(programLexer); // get tokens
MyParser parser = new MyParser(tokens);
MyProgramListener listener = new MyProgramListener(); // your custom extension from BaseListener
parser.addParseListener(listener);
parser.myProgramStart().enterRule(listener); // myProgramStart is your grammar rule to parse
// what we had built?
MyProgram myProgramInstance = listener.getMyProgram(); // in your listener implementation populate a MyProgram instance
System.out.println(myProgramInstance.toString());
}
}
References:
https://www.antlr.org/api/Java/
https://tomassetti.me/antlr-mega-tutorial/#java-setup
https://riptutorial.com/antlr/example/16571/listener-events-using-labels
This question already has answers here:
Java interpreter? [closed]
(10 answers)
Closed 9 years ago.
For debug reasons, I want to be able to run code that is typed in through the console. For example:
BufferedReader br = new BufferedReader(new InputStreamReader(System.in));
while(true){
String str = br.readLine(); //This can return 'a = 5;','b = "Text";' or 'pckg.example.MyClass.run(5);'
if(str == null)
return;
runCode(str); //How would I do this?
}
PLEASE DON'T ACTUALLY USE THIS
I was under the assumption you wanted to evaluate a string as Java code, not some scripting engine like Javascript, so
I created this on a whim after reading this, using the compiler API mark mentioned. It's probably very bad practice but it (somewhat) works like you wanted it to. I doubt it'll be much use in debugging since it runs the code in the context of a new class. Sample usage is included at the bottom.
import javax.tools.JavaCompiler;
import javax.tools.StandardJavaFileManager;
import javax.tools.ToolProvider;
import java.io.BufferedReader;
import java.io.File;
import java.io.InputStreamReader;
import java.io.PrintWriter;
import java.net.URL;
import java.net.URLClassLoader;
import java.util.Arrays;
public class main {
public static void runCode(String s) throws Exception{
JavaCompiler jc = ToolProvider.getSystemJavaCompiler();
StandardJavaFileManager sjfm = jc.getStandardFileManager(null, null, null);
File jf = new File("test.java"); //create file in current working directory
PrintWriter pw = new PrintWriter(jf);
pw.println("public class test {public static void main(){"+s+"}}");
pw.close();
Iterable fO = sjfm.getJavaFileObjects(jf);
if(!jc.getTask(null,sjfm,null,null,null,fO).call()) { //compile the code
throw new Exception("compilation failed");
}
URL[] urls = new URL[]{new File("").toURI().toURL()}; //use current working directory
URLClassLoader ucl = new URLClassLoader(urls);
Object o= ucl.loadClass("test").newInstance();
o.getClass().getMethod("main").invoke(o);
}
public static void main(String[] args) {
BufferedReader br = new BufferedReader(new InputStreamReader(System.in));
while(true){
try {
String str = br.readLine(); //This can return 'a = 5;','b = "Text";' or 'pckg.example.MyClass.run(5);'
if(str == null)
return;
runCode(str); //How would I do this?
} catch(Exception e) {
e.printStackTrace();
}
}
}
}
//command line
> System.out.println("hello");
hello
> System.out.println(3+2+3+4+5+2);
19
> for(int i = 0; i < 10; i++) {System.out.println(i);}
0
1
2
3
4
5
6
7
8
9
With the SimpleJavaFileObject you could actually avoid using a file, as shown here, but the syntax seems a bit cumbersome so I just opted for a file in the current working directory.
EDIT: Convert String to Code offers a similar approach but it's not fully fleshed out
If the code is in JavaScript then you can run it with JavaScript engine:
Object res = new ScriptEngineManager().getEngineByName("js").eval(str);
JavaScript engine is part of Java SE since 1.6. See this guide http://download.java.net/jdk8/docs/technotes/guides/scripting/programmer_guide/index.html for details
You can use the Java scripting API which is located in the Package javax.script. There you can include several scripting languages like bsh for example.
You can find a programmer's guide on the web page of Oracle.
Rhino, which is some kind of JavaScript is already included with the Oracle JVM.
For this you may want to look into Java Compiler API. I haven't studied much as to how this works, but it allows you to load a java file, compile and load the class in an already running system. Maybe it can be repurposed into accepting input from console.
For a general compiler you could use Janino which will allow you to compile and run Java code. The expression evaluator may help with your example.
If you are just looking to evaluate expressions while debugging then Eclispe has the Display view which allows you to execute expressions. See this question.
I'd like to get all the methods from a file (.text or .java), but I don't know the file's name yet (the user can choose it with jFileChooser). So I don't know the class's name. I have this code:
import java.io.BufferedReader;
import java.io.File;
import java.io.FileReader;
import java.lang.reflect.Method;
import java.net.URL;
import java.net.URLClassLoader;
import javax.swing.JFileChooser;
public class Test {
public static void main(String[] args) throws Throwable {
JFileChooser fc = new JFileChooser();
File f = null;
if (fc.showOpenDialog(null) == JFileChooser.APPROVE_OPTION) {
f = fc.getSelectedFile();
}
int errorCode = com.sun.tools.javac.Main.compile(new String[]{
"-classpath", "bin",
"-d", "../Tmp",
f.getAbsolutePath()});
System.out.println("errorCode:" + errorCode);
File classesDir = new File("../Tmp");
ClassLoader parentLoader = Test.class.getClassLoader();
URLClassLoader loader1 = new URLClassLoader(
new URL[]{classesDir.toURL()}, parentLoader);
BufferedReader br = new BufferedReader(new FileReader(f));
String load = "";
while ((load = br.readLine()) != null) {
if (load.startsWith("package")) {
load = load.replaceAll("package", "") + "." + f.getName().substring(0, f.getName().indexOf("."));
load = load.replace(";", "").trim();
break;
}
}
Class cls1 = loader1.loadClass(load);
Method[] methods = cls1.getDeclaredMethods();
for (Method m : methods) {
System.out.println(m.getName());
}
}
}
It works, if the class doesn't contains "extends", or uses another class's methods, but if it do, I get errors.
What should I do to fix these problems? I think it has to do something with "classpath" and "bin"
It does have to do with the -classpath bin option. The compiler needs to have access to all of the classes that the target class depends on. If you want to keep using this approach, you'll have to give the user some way to define their own classpath, to include something other than "bin".
It's not clear what your goal is, but other options include:
Working with classes the user has already compiled
Simply parsing the file, rather than compiling it into a Java class
Elaborating on the second option, you could use a Java parser to analyze the text. Usually the parser will create a tree structure, an abstract syntax tree, which the compiler traverses, often several times doing different compilation steps. However, parsing doesn't have to be followed by compilation; you can do whatever analysis you like on the AST.
I have used the ANTLR Java grammar. It produces an AST, and the ANTLR toolkit provides a grammar that you can use to write a "tree parser" that performs the actions you write when it finds certain structures in the AST. This "tree parser" concept is unique to ANTLR; most grammars will just stop with the AST.