Develop Reference

Am I reinventing the wheel with this token replacing code?

I have use case where I have a line of text containing nesting tokens (like { and }), and I wish to transform certain substrings nested at specific depths.
Example, capitalize the word moo at depth 1:
moo [moo [moo moo]] moo ->
moo [MOO [moo moo]] moo
Achieved by:
replaceTokens(input, 1, "[", "]", "moo", String::toUpperCase);
Or real world example, supply "--options" not already colored with the color sequence cyan:
#|blue --ignoreLog|# works, but --ignoreOutput silences everything. ->
#|blue --ignoreLog|# works, but #|cyan --ignoreOutput|# silences everything.
Achieved by:
replaceTokens(input, 0, "#|", "|#", "--\\w*", s -> format("#|cyan %s|#", s));
I have implemented this logic and though I feel pretty good about it (except performance probably), I also feel I reinvented the wheel. Here's how I implemented it:
set currentPos to zero
while (input line not fully consumed) {
take the remaining line
if the open token is matched, add to output, increase counter and advance pos accordingly
else if the close token is matched, add to output, decrease counter and advance pos accordingly
else if the counter matches provided depth and given regex matches, invoke replacer function and advance pos accordingly
else just record the next character and advance pos by 1
}
Here's the actual implementation:
public static String replaceNestedTokens(String lineWithTokens, int nestingDepth, String tokenOpen, String tokenClose, String tokenRegexToReplace, Function<String, String> tokenReplacer) {
final Pattern startsWithOpen = compile(quote(tokenOpen));
final Pattern startsWithClose = compile(quote(tokenClose));
final Pattern startsWithTokenToReplace = compile(format("(?<token>%s)", tokenRegexToReplace));
final StringBuilder lineWithTokensReplaced = new StringBuilder();
int countOpenTokens = 0;
int pos = 0;
while (pos < lineWithTokens.length()) {
final String remainingLine = lineWithTokens.substring(pos);
if (startsWithOpen.matcher(remainingLine).lookingAt()) {
countOpenTokens++;
lineWithTokensReplaced.append(tokenOpen);
pos += tokenOpen.length();
} else if (startsWithClose.matcher(remainingLine).lookingAt()) {
countOpenTokens--;
lineWithTokensReplaced.append(tokenClose);
pos += tokenClose.length();
} else if (countOpenTokens == nestingDepth) {
Matcher startsWithTokenMatcher = startsWithTokenToReplace.matcher(remainingLine);
if (startsWithTokenMatcher.lookingAt()) {
String matchedToken = startsWithTokenMatcher.group("token");
lineWithTokensReplaced.append(tokenReplacer.apply(matchedToken));
pos += matchedToken.length();
} else {
lineWithTokensReplaced.append(lineWithTokens.charAt(pos++));
}
} else {
lineWithTokensReplaced.append(lineWithTokens.charAt(pos++));
}
assumeTrue(countOpenTokens >= 0, "Unbalanced token sets: closed token without open token\n\t" + lineWithTokens);
}
assumeTrue(countOpenTokens == 0, "Unbalanced token sets: open token without closed token\n\t" + lineWithTokens);
return lineWithTokensReplaced.toString();
}
I couldn't make it work with a regex like this or this (or Scanner) solution, but I feel I'm reinventing the wheel and could solve this with (vanilla Java) out-of-the-box classes with less code. Also, I'm pretty sure this is a performance nightmare with all the inline patterns/matcher instances and substrings.
Suggestions?

You could be using a parser like ANTLR to create a grammar to describe your language or syntax. Then use a listener or visitor to make an interpreter of tokens.
A sample of the grammar would be like this (what I can infer from your code):
grammar Expr;
prog: (expr NEWLINE)* ;
expr: id '[' expr ']'
| '#|' expr '|#'
| '--ignoreLog' expr
| '--ignoreOutput' expr
| string
;
string: [a-zA-Z0-9];
NEWLINE : [\r\n]+ ;

ANTLR4 AST Creation - How to create an AstVistor

With the help of this SO question How to create AST with ANTLR4? I was able to create the AST Nodes, but I'm stuck at coding the BuildAstVisitor as depicted in the accepted answer's example.
I have a grammar that starts like this:
mini: (constDecl | varDef | funcDecl | funcDef)* ;
And I can neither assign a label to the block (antlr4 says label X assigned to a block which is not a set), and I have no idea how to visit the next node.
public Expr visitMini(MiniCppParser.MiniContext ctx) {
return visitConstDecl(ctx.constDecl());
}
I have the following problems with the code above: I don't know how to decide whether it's a constDecl, varDef or any other option and ctx.constDecl() returns a List<ConstDeclContext> whereas I only need one element for the visitConstDecl function.
edit:
More grammar rules:
mini: (constDecl | varDef | funcDecl | funcDef)* ;
//--------------------------------------------------
constDecl: 'const' type ident=ID init ';' ;
init: '=' ( value=BOOLEAN | sign=('+' | '-')? value=NUMBER ) ;
// ...
//--------------------------------------------------
OP_ADD: '+';
OP_SUB: '-';
OP_MUL: '*';
OP_DIV: '/';
OP_MOD: '%';
BOOLEAN : 'true' | 'false' ;
NUMBER : '-'? INT ;
fragment INT : '0' | [1-9] [0-9]* ;
ID : [a-zA-Z]+ ;
// ...
I'm still not entirely sure on how to implement the BuildAstVisitor. I now have something along the lines of the following, but it certainly doesn't look right to me...
#Override
public Expr visitMini(MiniCppParser.MiniContext ctx) {
for (MiniCppParser.ConstDeclContext constDeclCtx : ctx.constDecl()) {
visit(constDeclCtx);
}
return null;
}
#Override
public Expr visitConstDecl(MiniCppParser.ConstDeclContext ctx) {
visit(ctx.type());
return visit(ctx.init());
}

If you want to get the individual subrules then implement the visitXXX functions for them (visitConstDecl(), visitVarDef() etc.) instead of the visitMini() function. They will only be called if there's really a match for them in the input. Hence you don't need to do any checks for occurences.

ANTLR4 -> Check a literal is STRING_LITERAL or a NUMBER_LITERAL

Currently, I'm using this ANTLR4 grammar (part of) in order to get strings and numbers:
Go figure this summarized grammar:
gramm
: expr SCOL
;
expr
: literal #LiteralExpression
;
literal
: NUMERIC_LITERAL
| STRING_LITERAL
;
NUMERIC_LITERAL
: DIGIT+ ( '.' DIGIT* )? ( E [-+]? DIGIT+ )?
| '.' DIGIT+ ( E [-+]? DIGIT+ )?
;
STRING_LITERAL
: '\'' ( ~'\'' | '\'\'' )* '\''
;
SPACES
: [ \u000B\t\r\n] -> channel(HIDDEN)
;
fragment DIGIT : [0-9];
So, I'm implementing an GrammBaseVisitor<Void>.
I'm not quite to figure out how to check whether a literal is a NUMERIC_LITERAL or a STRING_LITERAL.
As far I've been able to get, I've override visitLiteral() and visitLiteralExpression():
#Override
public Void visitLiteral(LiteralContext ctx) {
// TODO What should I do here in order to check whether
// ctx contains an STRING_LITERAL or a NUMBER_LITERAL?
return super.visitLiteral(ctx);
}
#Override
public Void visitLiteralExpression(LiteralExpressionContext ctx) {
return super.visitLiteralExpression(ctx);
}
What's the difference between visitLiteral and visitLiteralExpression()?

Your literal production consists of two possible terminals, the numeric and string literal. Whether parsed input contains one or the other, you can determine with null checks inside visitLiteral.
#Override
public Object visitLiteral(LiteralContext ctx) {
TerminalNode numeric = ctx.NUMERIC_LITERAL();
TerminalNode string = ctx.STRING_LITERAL();
if (numeric != null) {
System.out.println(numeric.getSymbol().getType());
} else if (string != null) {
System.out.println(string.getSymbol().getType());
}
return super.visitLiteral(ctx);
}
You can visit all terminals by overriding visitTerminal.
#Override
public Object visitTerminal(TerminalNode node) {
int type = node.getSymbol().getType(); // matches a constant in your parser
switch (type) {
case GrammParser.NUMERIC_LITERAL:
System.out.println("numeric literal");
break;
case GrammParser.STRING_LITERAL:
System.out.println("string literal");
break;
}
System.out.println(node.getSymbol().getText());
return super.visitTerminal(node);
}
What's the difference between visitLiteral and visitLiteralExpression()?
The former represents your literal production and the latter represents your expr production. Note that # symbol has special meaning in ANTLR 4 syntax, representing a label - a name for alternatives inside productions. It is not a comment. Since your expr only has one alternative, it becomes visitLiteralExpression. Try commenting it out (//) and see your generated code change.

Split by comma pattern matching

What pattern can I use to split a string like this:
f.id AS id, CONCAT(a1.id, a2.id, a3.id) AS cnp, SUM(A3.nr) AS sum
in such a way that the result is an array of 3 groups like this:
f.id AS id
CONCAT(a1.id, a2.id, a3.id) AS cnp
SUM(A3.nr) AS sum
Can I match a comma that is not enclosed by parentheses?

The pattern appears to always take the format ... AS ... and you can just use a regular expression to match that:
Pattern p = Pattern.compile("(.*? as .*?)(,|$)", Pattern.CASE_INSENSITIVE );
String query = "f.id AS id, CONCAT(a1.id, a2.id, a3.id) AS cnp, SUM(A3.nr) AS sum";
Matcher m = p.matcher( query );
while ( m.find() ){
System.out.println( m.group(1) );
}
IDEONE
So long as you are not expecting any correlated sub-queries to be nested in your select values (or other edge cases such as strings containing ' as error,' AS id, ...) then this ought to work for inputs similar to your format.

Probably there is a killer regular expression for this, but what would be more maintanable could be to:
Temporarily set placeholders on blocks between parentheses
Split the result on the desired separator
Replace the placeholders with their original values
To make step 1 more general, you should insert placeholders at sections where the separator should not function. As long as you are able to accurately determine what those sections are, you could apply this recipe.

Using an actual SQL Parser, as suggested by #KevinEsche, is probably the most robust choice.
However, if you don't require parsing of all SQL expressions, I would just use plain old char matching: go through the string a character at a time, counting how deeply nested in the brackets you are:
List<String> parts = new ArrayList<>();
int i = 0;
int depth = 0;
while (i < str.length()) {
int start = i;
while (i < str.length()) {
char ch = str.charAt(i);
if (ch == '(') {
depth++;
} else if (ch == ')') {
depth--;
} else if (ch == ',' && depth == 0) {
break;
}
i++;
}
// Maybe check that depth == 0 here.
parts.add(str.substring(start, i));
i++; // To skip the comma.
}

Thank you for your answers. I tried to vote but I can't yet.
I used look ahead pattern to solve the problem:
String pattern = ",(?!([^(]*\\)))";
String str = "f.id AS id, CONCAT(a1.id, a2.id, a3.id) AS cnp, SUM(A3.nr) AS sum";
String strg [] = str.split(pattern);
for(int i=0;i<strg.length;i++) {
System.err.println("Group "+i+" is "+strg[i]);
}
And the result is:
Group 0 is f.id AS id
Group 1 is CONCAT(a1.id, a2.id, a3.id) AS cnp
Group 2 is SUM(A3.nr) AS sum

In the end is too complicated to write a SQL Parser so I decided to use ANTLR4.
I used the example from here and works fine.
https://github.com/bkiers/sqlite-parser
But I don't know how to extract only some parts of the query(select, joins, order...) and I can't find any examples online. Can someoane show how this is done?
Thank you.

Simple ANTLR error

I'm starting with ANTLR, but I get some errors and I really don't understand why.
Here you have my really simple grammar
grammar Expr;
options {backtrack=true;}
#header {}
#members {}
expr returns [String s]
: (LETTER SPACE DIGIT | TKDC) {$s = $DIGIT.text + $TKDC.text;}
;
// TOKENS
SPACE : ' ' ;
LETTER : 'd' ;
DIGIT : '0'..'9' ;
TKDC returns [String s] : 'd' SPACE 'C' {$s = "d C";} ;
This is the JAVA source, where I only ask for the "expr" result:
import org.antlr.runtime.*;
class Testantlr {
public static void main(String[] args) throws Exception {
ExprLexer lex = new ExprLexer(new ANTLRFileStream(args[0]));
CommonTokenStream tokens = new CommonTokenStream(lex);
ExprParser parser = new ExprParser(tokens);
try {
System.out.println(parser.expr());
} catch (RecognitionException e) {
e.printStackTrace();
}
}
}
The problem comes when my input file has the following content d 9.
I get the following error:
x line 1:2 mismatched character '9' expecting 'C'
x line 1:3 no viable alternative at input '<EOF>'
Does anyone knwos the problem here?

There are a few things wrong with your grammar:
lexer rules can only return Tokens, so returns [String s] is ignored after TKDC;
backtrack=true in your options section does not apply to lexer rules, that is why you get mismatched character '9' expecting 'C' (no backtracking there!);
the contents of your expr rule: (LETTER SPACE DIGIT | TKDC) {$s = $DIGIT.text + $TKDC.text;} doesn't make much sense (to me). You either want to match LETTER SPACE DIGIT or TKDC, yet you're trying to grab the text of both choices: $DIGIT.text and $TKDC.text.
It looks to me TKDC needs to be "promoted" to a parser rule instead.
I think you dumbed down your example a bit too much to illustrate the problem you were facing. Perhaps it's a better idea to explain your actual problem instead: what are you trying to parse exactly?