Error converting Flink CEP table in Dataset<Row> object - java

I've created an BatchTableEnviroment in Apache Flink and created a Table object where I loaded data in. Now I want to search for some patterns. I'm doing this with the Detecting patterns in Tables CEP library. The task of the query below is to find the longest period of a mGroup for which the avgResult did not go below a certain threshold. Where a mGroup is a Integer value like 100, 200, 300 etc. Avgresult is double value. When I compile the query part I dont get any error. I get the error when I convert the Table to a DataSet<row>. Below the query u can see the error message.
ExecutionEnvironment fbEnv = ExecutionEnvironment.getExecutionEnvironment();
BatchTableEnvironment tableEnv = BatchTableEnvironment.create(fbEnv);
Table trendTable = tableEnv.sqlQuery(
" SELECT * " +
" FROM tableAvg " +
" MATCH_RECOGNIZE(" +
" PARTITION BY mType " +
" ORDER BY mGroup " +
" MEASURES " +
" FIRST(A.mGroup) as startGr, " +
" LAST(A.mGroup) as endGr, " +
" A.avgResult as avgRes" +
" ONE ROW PER MATCH " +
" AFTER MATCH SKIP PAST LAST ROW " +
" PATTERN (A+ B) " +
" DEFINE " +
" A AS A.avgResult < 50 " +
") "
);
tableEnv.registerTable("TrendTable", trendTable);
DataSet<Row> result = tableEnv.toDataSet(trendTable, Row.class);
/////////////////////ERROR MESSAGE BELOW
Exception in thread "main" org.apache.flink.table.api.TableException: Cannot generate a valid execution plan for the given query:
FlinkLogicalMatch(partition=[[$1]], order=[[2]], outputFields=[[mType, startGr, endGr, avgRes]], allRows=[false], after=[FLAG(SKIP PAST LAST ROW)], pattern=[(PATTERN_QUANTIFIER(_UTF-16LE'A', 1, -1, false), _UTF-16LE'B')], isStrictStarts=[false], isStrictEnds=[false], subsets=[[]], patternDefinitions=[[<(PREV(A.$0, 0), 50)]], inputFields=[[sumResult, mType, EXPR$2]])
FlinkLogicalSort(sort0=[$2], dir0=[ASC])
FlinkLogicalCalc(expr#0..5=[{inputs}], expr#6=[/($t2, $t3)], expr#7=[99], expr#8=[>($t5, $t7)], sumResult=[$t6], mType=[$t1], EXPR$2=[$t4], $condition=[$t8])
FlinkLogicalAggregate(group=[{0, 1}], agg#0=[SUM($2)], agg#1=[SUM($3)], agg#2=[MAX($4)], agg#3=[COUNT()])
FlinkLogicalCalc(expr#0..2=[{inputs}], expr#3=[1], expr#4=[-($t0, $t3)], expr#5=[100], expr#6=[/($t4, $t5)], expr#7=[1.0:DECIMAL(2, 1)], $f0=[$t6], mType=[$t1], mValue=[$t2], $f3=[$t7], mID=[$t0])
FlinkLogicalTableSourceScan(table=[[default_catalog, default_database, H]], fields=[mID, dateTime, mValue, unixDateTime, mType], source=[CsvTableSource(read fields: mID, mType, mValue)])
This exception indicates that the query uses an unsupported SQL feature.
Please check the documentation for the set of currently supported SQL features.
at org.apache.flink.table.plan.Optimizer.runVolcanoPlanner(Optimizer.scala:245)
at org.apache.flink.table.plan.Optimizer.optimizePhysicalPlan(Optimizer.scala:170)
at org.apache.flink.table.plan.BatchOptimizer.optimize(BatchOptimizer.scala:57)
at org.apache.flink.table.api.internal.BatchTableEnvImpl.translate(BatchTableEnvImpl.scala:280)
at org.apache.flink.table.api.java.internal.BatchTableEnvironmentImpl.toDataSet(BatchTableEnvironmentImpl.scala:71)
at StreamTableEnv.main(StreamTableEnv.java:169)

The CEP library and MATCH_RECOGNIZE only work on top of the streaming API (and not batch), meaning that you need to use a StreamTableEnvironment rather than a BatchTableEnviroment.

Related

Spring Neo4j Pass Node Property as Parameter

I am trying to pass a property as a parameter.
I have tried this:
String get_interest="MATCH(user:User{id:{id}})-[watched:WATCHED]->(movie:Movie{title:{title}}) " +
"MATCH(movie)-[:BELONGS_TO]->(category:Category) " +
"MATCH(category)<-[:BELONGS_TO]-(similarMovie:Movie) " +
"WHERE NOT EXISTS((user) -[:WATCHED]->(similarMovie))" +
"RETURN similarMovie Limit 20";
But it fails with this error:
"error": "Internal Server Error",
"message": "Cypher execution failed with code 'Neo.ClientError.Statement.SyntaxError': Invalid input '{': expected \"+\" or \"-\" (line 1, column 24 (offset: 23))
I have tried this and it worked:
String get_interest="MATCH(user:User{id:\"02331\"})-[watched:WATCHED]->(movie:Movie{title:\"The Mask\"}) " +
"MATCH(movie)-[:BELONGS_TO]->(category:Category) " +
"MATCH(category)<-[:BELONGS_TO]-(similarMovie:Movie) " +
"WHERE NOT EXISTS((user) -[:WATCHED]->(similarMovie))" +
"RETURN similarMovie Limit 20";
But I want to pass other User ids and titles.
This is my Repository
#Query(get_interest)
Collection<Movie> getMovieByInterest(#Param("id") String id,
#Param("title") String title);
Params are bound as variables. Variables are interpolated with dollar:
"MATCH(user:User{id:$id})-[watched:WATCHED]->(movie:Movie{title:$title})"

Rollover api is not creating indices automatically - Java rest high level client

I'm using Java High level rest client in my project and i want to limit my indices based on document count. I'm using rollover api but its not creating indices automatically.
The code will be given below.
I'm creating index pattern so that my custom analyzer will be applied to all other indices that follow the respective pattern.
PutIndexTemplateRequest request = new PutIndexTemplateRequest("testingtemplate");
request.source("{\n" +
" \"index_patterns\":[\n" +
" \"test_log-*\"\n" +
" ],\n" +
" \"settings\": {\n" +
" \"analysis\": {\n" +
" \"analyzer\": { \n" +
" \"my_analyzer\": {\n" +
" \"type\": \"custom\",\n" +
" \"tokenizer\": \"whitespace\",\n" +
" \"filter\": []\n" +
" }\n" +
" }\n" +
" }\n" +
" },\n" +
" \"mappings\": {\n" +
" \"properties\": {\n" +
" \"fullLog\": {\n" +
" \"type\": \"text\",\n" +
" \"analyzer\": \"my_analyzer\"\n" +
" }\n" +
" }\n" +
" }\n" +
" }",XContentType.JSON);
return client.indices().putTemplate(request,RequestOptions.DEFAULT).isAcknowledged();
My Rollover code. Here i want to rollover one when index gets one or document.
final RestHighLevelClient client = new RestHighLevelClient(
RestClient.builder(new HttpHost("localhost", 9200, "http")));
boolean isIndexTemplateCreated = createIndexTemplate(client);
System.out.println(isIndexTemplateCreated);
CreateIndexRequest request = new CreateIndexRequest("test_log-1");
request.alias(new Alias("temp_alias_new"));
CreateIndexResponse createIndexResponse = client.indices().create(request, RequestOptions.DEFAULT);
RolloverRequest roll_req = new RolloverRequest("temp_alias_new",null);
roll_req.addMaxIndexAgeCondition(new TimeValue(7, TimeUnit.DAYS));
roll_req.addMaxIndexDocsCondition(1);
roll_req.addMaxIndexSizeCondition(new ByteSizeValue(5, ByteSizeUnit.GB));
RolloverResponse rolloverResponse = client.indices().rollover(roll_req, RequestOptions.DEFAULT);
Map<String,Object> map = new HashMap<>();
map.put("fullLog","Hi");
client.index(new IndexRequest("temp_alias_new").source(map), RequestOptions.DEFAULT);
map.put("fullLog","hello");
client.index(new IndexRequest("temp_alias_new").source(map), RequestOptions.DEFAULT);
But the code is not working and the rollover api is not creating indices automatically. All the 2 documents are stored only in test_log-1 index.
Is there any mistake in my code?
Thanks
Note: Rollover will not happen automatically.
Elasticsearch tries to rollover the index on getting a rollover request.
For example consider the following sequences:
You have a new index test_log-1 which is empty and is being pointed by the alias temp_alias_new.
If you try to rollover now, none of the conditions mentioned along with rollover request holds good as the index is new and empty. So this time rollover fails.
Add some documents to the index.
Now try the rollover with the condition maxIndexDocsCondition(1), it will rollover. Because the condition holds good.
Update
With the latest release of elastic search, you can use ILM (Index Life cycle Management) to automate the rollover.
Here is doc link for more info: https://www.elastic.co/guide/en/elasticsearch/reference/7.x/getting-started-index-lifecycle-management.html

Spring Data MongoDB $where is not allowed in this context

I'm trying to make a Mongo criteria where the search value must be equal to FIELD1 + "-" + FIELD2:
What I've tried:
Criteria.where("$where").is("this." + SOFTWARE_NAME + " + '-' + this." + SOFTWARE_VERSION " == '" + value + "'");
But when I run a query with that criteria then I'm getting this exception:
org.springframework.data.mongodb.UncategorizedMongoDbException:
Command failed with error 2 (BadValue): '$where is not allowed in this
context' on server hostname:17091. The full
response is {"operationTime": {"$timestamp": {"t": 1614185267, "i":
1}}, "ok": 0.0, "errmsg": "$where is not allowed in this context",
"code": 2, "codeName": "BadValue"
Even if I change the criteria to "this." + SOFTWARE_NAME + " == this." + SOFTWARE_NAME still the same exception is returned. I've seen many answers with a criteria that uses where("$where") so why it doesn't work for me?
When I run that query in Mongo directly:
{"$where" : "this.softwareName + '-' + this.softwareVersion == 'software-1.0.2'"}
then everything works perfectly fine.
EDIT
The query in the logs looks like this:
"loggerName":"org.springframework.data.mongodb.core.MongoTemplate","message":"Executing
count: { "$where" : "this.softwareName + '-' + this.softwareVersion
== 'software-1.0.2'"} in collection ...

Parse html content for a value

I receive a Http response after a call as Html String and I would like to scrape certain value stored inside the ReportViewer1 variable.
<html>
....................
...........
<script type="text/javascript">
var ReportViewer1 = new ReportViewer('ReportViewer1', 'ReportViewer1_ReportToolbar', 'ReportViewer1_ReportArea_WaitControl', 'ReportViewer1_ReportArea_ReportCell', 'ReportViewer1_ReportArea_PreviewFrame', 'ReportViewer1_ParametersAreaCell', 'ReportViewer1_ReportArea_ErrorControl', 'ReportViewer1_ReportArea_ErrorLabel', 'ReportViewer1_CP', '/app/Telerik.ReportViewer.axd', 'a90a0d41efa6429eadfefa42fc529de1', 'Percent', '100', '', 'ReportViewer1_EditorPlaceholder', 'ReportViewer1_CalendarFrame', 'ReportViewer1_ReportArea_DocumentMapCell', {
CurrentPageToolTip: 'STR_TELERIK_MSG_CUR_PAGE_TOOL_TIP',
ExportButtonText: 'Export',
ExportToolTip: 'Export',
ExportSelectFormatText: 'Export to the selected format',
FirstPageToolTip: 'First page',
LabelOf: 'of',
LastPageToolTip: 'Last Page',
ProcessingReportMessage: 'Generating report...',
NoPageToDisplay: 'No page to display.',
NextPageToolTip: 'Next page',
ParametersToolTip: 'Click to close parameters area|Click to open parameters area',
DocumentMapToolTip: 'Hide document map|Show document map',
PreviousPageToolTip: 'Previous page',
TogglePageLayoutToolTip: 'Switch to interactive view|Switch to print preview',
SessionHasExpiredError: 'Session has expired.',
SessionHasExpiredMessage: 'Please, refresh the page.',
PrintToolTip: 'Print',
RefreshToolTip: 'Refresh',
NavigateBackToolTip: 'Navigate back',
NavigateForwardToolTip: 'Navigate forward',
ReportParametersSelectAllText: '<select all>',
ReportParametersSelectAValueText: '<select a value>',
ReportParametersInvalidValueText: 'Invalid value.',
ReportParametersNoValueText: 'Value required.',
ReportParametersNullText: 'NULL',
ReportParametersPreviewButtonText: 'Preview',
ReportParametersFalseValueLabel: 'False',
ReportParametersInputDataError: 'Missing or invalid parameter value. Please input valid data for all parameters.',
ReportParametersTrueValueLabel: 'True',
MissingReportSource: 'The source of the report definition has not been specified.',
ZoomToPageWidth: 'Page Width',
ZoomToWholePage: 'Full Page'
}, 'ReportViewer1_ReportArea_ReportArea', 'ReportViewer1_ReportArea_SplitterCell', 'ReportViewer1_ReportArea_DocumentMapCell', true, true, 'PDF', 'ReportViewer1_RSID', true);
</script>
...................
...................
</html>
The value is a90a0d41efa6429eadfefa42fc529de1 and this is in the middle of this content:
'/app/Telerik.ReportViewer.axd', 'a90a0d41efa6429eadfefa42fc529de1', 'Percent', '100',
Whats the best way I can parse this value using Java?
Parse the HTML with String class
public class HtmlParser {
public static void main(String args[]){
String result = getValuesProp(html);
System.out.println("Result: "+ result);
}
static String PIVOT = "Telerik.ReportViewer.axd";
public static String getValuesProp(String json) {
String subString;
int i = json.indexOf(PIVOT);
i+= PIVOT.length();
//', chars
i+=2;
subString = json.substring(i);
i = subString.indexOf("'");
i++;
subString = subString.substring(i);
i = subString.indexOf("'");
subString = subString.substring(0,i);
return subString;
}
static String html ="<html>\n" +
"\n" +
"<script type=\"text/javascript\">\n" +
" var ReportViewer1 = new ReportViewer('ReportViewer1', 'ReportViewer1_ReportToolbar', 'ReportViewer1_ReportArea_WaitControl', 'ReportViewer1_ReportArea_ReportCell', 'ReportViewer1_ReportArea_PreviewFrame', 'ReportViewer1_ParametersAreaCell', 'ReportViewer1_ReportArea_ErrorControl', 'ReportViewer1_ReportArea_ErrorLabel', 'ReportViewer1_CP', '/app/Telerik.ReportViewer.axd', 'a90a0d41efa6429eadfefa42fc529de1', 'Percent', '100', '', 'ReportViewer1_EditorPlaceholder', 'ReportViewer1_CalendarFrame', 'ReportViewer1_ReportArea_DocumentMapCell', {\n" +
" CurrentPageToolTip: 'STR_TELERIK_MSG_CUR_PAGE_TOOL_TIP',\n" +
" ExportButtonText: 'Export',\n" +
" ExportToolTip: 'Export',\n" +
" ExportSelectFormatText: 'Export to the selected format',\n" +
" FirstPageToolTip: 'First page',\n" +
" LabelOf: 'of',\n" +
" LastPageToolTip: 'Last Page',\n" +
" ProcessingReportMessage: 'Generating report...',\n" +
" NoPageToDisplay: 'No page to display.',\n" +
" NextPageToolTip: 'Next page',\n" +
" ParametersToolTip: 'Click to close parameters area|Click to open parameters area',\n" +
" DocumentMapToolTip: 'Hide document map|Show document map',\n" +
" PreviousPageToolTip: 'Previous page',\n" +
" TogglePageLayoutToolTip: 'Switch to interactive view|Switch to print preview',\n" +
" SessionHasExpiredError: 'Session has expired.',\n" +
" SessionHasExpiredMessage: 'Please, refresh the page.',\n" +
" PrintToolTip: 'Print',\n" +
" RefreshToolTip: 'Refresh',\n" +
" NavigateBackToolTip: 'Navigate back',\n" +
" NavigateForwardToolTip: 'Navigate forward',\n" +
" ReportParametersSelectAllText: '<select all>',\n" +
" ReportParametersSelectAValueText: '<select a value>',\n" +
" ReportParametersInvalidValueText: 'Invalid value.',\n" +
" ReportParametersNoValueText: 'Value required.',\n" +
" ReportParametersNullText: 'NULL',\n" +
" ReportParametersPreviewButtonText: 'Preview',\n" +
" ReportParametersFalseValueLabel: 'False',\n" +
" ReportParametersInputDataError: 'Missing or invalid parameter value. Please input valid data for all parameters.',\n" +
" ReportParametersTrueValueLabel: 'True',\n" +
" MissingReportSource: 'The source of the report definition has not been specified.',\n" +
" ZoomToPageWidth: 'Page Width',\n" +
" ZoomToWholePage: 'Full Page'\n" +
" }, 'ReportViewer1_ReportArea_ReportArea', 'ReportViewer1_ReportArea_SplitterCell', 'ReportViewer1_ReportArea_DocumentMapCell', true, true, 'PDF', 'ReportViewer1_RSID', true);\n" +
" </script>\n" +
"\n" +
"</html>";
}
I would read the text a line at a time like how most files are read. Because the format will always be the same, you look for a line that begins with the characters "var ReportViewer1." Then you know you have found the line you want. You may need to strip some white space, although it will always be formatted with the same whitespace too (up to you really.)
When you have the line, use the String .split() method to split that line into an array. There are nice delimiters there to split on ... "," or " " or ", " ... again, see what works best for you.
Test the split up line parts for '/app/Telerik.ReportViewer.axd' ... the next member of your split array will be the value you are looking for.
Again, the formatting will always be the same, so you can rely on that to find your variable. Of course, study the html text to make sure it does always follow the same format within the line you are investigating, but looking at it, I assume it probably does.
Again, find your line ... split it on a delimiter ... and use some logic to find the element you are after in the split up line parts.

parameters in lucene queries in neo4j repositories with SDN?

I need to be able to search for substrings in a text field, via a parameterised repository method, in neo4j 1.9.5
Ideally I want to be able to call
getInteractionsByTermAndDateRange(String term,
long startMillis, long endMillis
and get back every WRInteraction where the 'content' field contains 'term', with the pubMillis value within the specified range( 'content' is specified as a FULLTEXT index in the WRInteraction object declaration)
First attempt:
#Query("START n=node:WRInteraction('content:*{0}*') " + " WHERE "
+ " n.pubMillis >= {1} AND n.pubMillis <= {2}" + " RETURN "
+ " n")
Iterable<WRInteraction> getInteractionsByTermAndDateRange(String term,
long startMillis, long endMillis);
This throws
Caused by: org.apache.lucene.queryParser.ParseException: Cannot parse 'content:*{0}*':
Encountered " "}" "} "" at line 1, column 11.
Was expecting one of:
"TO" ...
<RANGEEX_QUOTED> ...
<RANGEEX_GOOP> ...
at org.apache.lucene.queryParser.QueryParser.parse(QueryParser.java:211) ~[lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
at org.neo4j.index.impl.lucene.IndexType.query(IndexType.java:300) ~[neo4j-lucene-index-1.9.5.jar:1.9.5]
Second try -- pass the whole lucene query via the parameter:
#Query("START n=node:WRInteraction('content:{0}') " + " WHERE "
+ " n.pubMillis >= {1} AND n.pubMillis <= {2}" + " RETURN "
+ " n")
doesn't fare any better... what's the pattern I should be using here? The key requirement is to be able to pass a substring to a repository method as a parameter, and return any WRInteractions where that substring is present int he 'content' field. Should be easy, right?
Thanks
You need to specify the lucene query as a whole.
#Query("START n=node:WRInteraction({0}) WHERE n.pubMillis >= {1} AND n.pubMillis <= {2} RETURN n")
Iterable<WRInteraction> getInteractionsByTermAndDateRange(String query,
long startMillis, long endMillis);
Call this method like:
repository.getInteractionsByTermAndDateRange("content:*" + term + "*", 0, 0);

Categories

Resources