Develop Reference

Spark streaming throws java.util.ConcurrentModificationException while adding JsonArray

I got an error during the following process. I'm aware that it seems like this error is thrown because it tried to read the whole records in the partition (rec) but trying to assign it to string (Str=jsonArray.toJSONString();) at the same time I'm using 5-sec batch interval in spark streaming configuration. Any suggestions for this code? Please kindly help. Thanks
The error is in this line :
Str=jsonArray.toJSONString();
Below is my full function :
MapRowRDD.foreachRDD(rdd ->{
rdd.foreachPartition(
rec-> {
while(rec.hasNext()) {
JSONObject record = rec.next();
i=i+1;
if(TimeUnit.MINUTES.convert(new SimpleDateFormat("yyyy-MM-dd HH:mm:ss")
.parse((String) record.get("DATE_TRANSACTION"))
.getTime()-DateUtils.addMinutes(new Date(), -5)
.getTime(),TimeUnit.MILLISECONDS)>=0 || Integer.valueOf((String) record.get("EVENT_TYPE"))<0) {
jsonArray.add(record);
if(i % v_BATCH_WINDOW == 0)
{
try {
Str=jsonArray.toJSONString();
HttpResponse<String> Response = ui.post(v_REST_API_ENDPOINT).body(Str).asString();
out_JSON=Response.getBody();
log.warn("Response : " + out_JSON.toString());
}
catch(UnirestConfigException e){
System.out.println("UnirestConfigException occured "+ e.toString());
e.printStackTrace();
}
jsonArray.clear();
i=0;
}
}
publishToKafka(record.toString(), outputTopic, props);
}
Str=jsonArray.toJSONString();
if (!Str.equals("[]") && Str!=null && !Str.isEmpty()) {
HttpResponse<String> Response = ui.post(v_REST_API_ENDPOINT).body(Str).asString();
}
jsonArray.clear();
i=0;
}
);
});

As you know this exception occurs when you modify and iterate the same collection at the same time via different threads. jsonArray is not thread-safe replace that with some thread-safe collections like Vector and see this works

How to set the custom field "startDate" in Jira with a Java API

Here is the code:
public String createDeploymentTask(String project_key, String summary, String description) throws ResponseException { //ADD FIELDS: DATE...
IssueRestClient issueClient = restClient.getIssueClient();
IssueInputBuilder iib = new IssueInputBuilder();
iib.setProjectKey(project_key);
iib.setSummary(summary);
iib.setIssueTypeId(new Long(10800));
iib.setDescription(description);
iib.setFieldValue("customfield_15031", new Timestamp(System.currentTimeMillis()));
IssueInput issue = iib.build();
BasicIssue issueObj = null;
try {
issueObj = issueClient.createIssue(issue).claim();
} catch (RestClientException e) {
throw new ResponseException(400, "Input is invalid (e.g. missing required fields, invalid field values, and so forth)\n" + e.getMessage());
}
System.out.println("Issue " + issueObj.getKey() + " created successfully");
return issueObj.getKey();
}
I get the exception:
com.atlassian.jira.rest.client.api.domain.input.CannotTransformValueException: Any of available transformers was able to transform given value. Value is: java.sql.Timestamp: 2018-07-20 17:20:06.65
What format should have the second parameter in
iib.setFieldValue("customfield_15031", new Timestamp(System.currentTimeMillis()));
?

BaseValueTransformer that is used by IssueInputBuilder doesn't accept any time or date related value. You will have to format date on your own and pass as a string value. The format is 2018-07-25 so you will have to use format yyyy-MMM-dd.

Parsin error causing blank screen

enter image description here
Hello, I am with a problem that my code does not worked, where the cause of error is(Problem parsing the earthquake JSON results) with this do with my app when executed stay with blank screen. My code be reffers to udacity course about network and the all code can be find on the link below. Please I am begginner in progamming i don't know more what do.
(The code what I used)
public class QueryUtil {
/**
* Sample JSON response for a USGS query
*/
private static final String SAMPLE_JSON_RESPONSE = "{\"type\":\"FeatureCollection\",\"metadata\":{\"generated\":1462295443000,\"url\":\"http://earthquake.usgs.gov/fdsnws/event/1/query?format=geojson&starttime=2016-01-01&endtime=2016-01-31&minmag=6&limit=10\",\"title\":\"USGS Earthquakes\",\"status\":200,\"api\":\"1.5.2\",\"limit\":10,\"offset\":1,\"count\":10},\"features\":[{\"type\":\"Feature\",\"properties\":{\"mag\":7.2,\"place\":\"88km N of Yelizovo, Russia\",\"time\":1454124312220,\"updated\":1460674294040,\"tz\":720,\"url\":\"http://earthquake.usgs.gov/earthquakes/eventpage/us20004vvx\",\"detail\":\"http://earthquake.usgs.gov/fdsnws/event/1/query?eventid=us20004vvx&format=geojson\",\"felt\":2,\"cdi\":3.4,\"mmi\":5.82,\"alert\":\"green\",\"status\":\"reviewed\",\"tsunami\":1,\"sig\":798,\"net\":\"us\",\"code\":\"20004vvx\",\"ids\":\",at00o1qxho,pt16030050,us20004vvx,gcmt20160130032510,\",\"sources\":\",at,pt,us,gcmt,\",\"types\":\",cap,dyfi,finite-fault,general-link,general-text,geoserve,impact-link,impact-text,losspager,moment-tensor,nearby-cities,origin,phase-data,shakemap,tectonic-summary,\",\"nst\":null,\"dmin\":0.958,\"rms\":1.19,\"gap\":17,\"magType\":\"mww\",\"type\":\"earthquake\",\"title\":\"M 7.2 - 88km N of Yelizovo, Russia\"},\"geometry\":{\"type\":\"Point\",\"coordinates\":[158.5463,53.9776,177]},\"id\":\"us20004vvx\"},\n" +
"{\"type\":\"Feature\",\"properties\":{\"mag\":6.1,\"place\":\"94km SSE of Taron, Papua New Guinea\",\"time\":1453777820750,\"updated\":1460156775040,\"tz\":600,\"url\":\"http://earthquake.usgs.gov/earthquakes/eventpage/us20004uks\",\"detail\":\"http://earthquake.usgs.gov/fdsnws/event/1/query?eventid=us20004uks&format=geojson\",\"felt\":null,\"cdi\":null,\"mmi\":4.1,\"alert\":\"green\",\"status\":\"reviewed\",\"tsunami\":1,\"sig\":572,\"net\":\"us\",\"code\":\"20004uks\",\"ids\":\",us20004uks,gcmt20160126031023,\",\"sources\":\",us,gcmt,\",\"types\":\",cap,geoserve,losspager,moment-tensor,nearby-cities,origin,phase-data,shakemap,tectonic-summary,\",\"nst\":null,\"dmin\":1.537,\"rms\":0.74,\"gap\":25,\"magType\":\"mww\",\"type\":\"earthquake\",\"title\":\"M 6.1 - 94km SSE of Taron, Papua New Guinea\"},\"geometry\":{\"type\":\"Point\",\"coordinates\":[153.2454,-5.2952,26]},\"id\":\"us20004uks\"},\n" +
"{\"type\":\"Feature\",\"properties\":{\"mag\":6.3,\"place\":\"50km NNE of Al Hoceima, Morocco\",\"time\":1453695722730,\"updated\":1460156773040,\"tz\":0,\"url\":\"http://earthquake.usgs.gov/earthquakes/eventpage/us10004gy9\",\"detail\":\"http://earthquake.usgs.gov/fdsnws/event/1/query?eventid=us10004gy9&format=geojson\",\"felt\":117,\"cdi\":7.2,\"mmi\":5.28,\"alert\":\"green\",\"status\":\"reviewed\",\"tsunami\":0,\"sig\":695,\"net\":\"us\",\"code\":\"10004gy9\",\"ids\":\",us10004gy9,gcmt20160125042203,\",\"sources\":\",us,gcmt,\",\"types\":\",cap,dyfi,geoserve,impact-text,losspager,moment-tensor,nearby-cities,origin,phase-data,shakemap,tectonic-summary,\",\"nst\":null,\"dmin\":2.201,\"rms\":0.92,\"gap\":20,\"magType\":\"mww\",\"type\":\"earthquake\",\"title\":\"M 6.3 - 50km NNE of Al Hoceima, Morocco\"},\"geometry\":{\"type\":\"Point\",\"coordinates\":[-3.6818,35.6493,12]},\"id\":\"us10004gy9\"},\n" +
"{\"type\":\"Feature\",\"properties\":{\"mag\":7.1,\"place\":\"86km E of Old Iliamna, Alaska\",\"time\":1453631430230,\"updated\":1460156770040,\"tz\":-540,\"url\":\"http://earthquake.usgs.gov/earthquakes/eventpage/us10004gqp\",\"detail\":\"http://earthquake.usgs.gov/fdsnws/event/1/query?eventid=us10004gqp&format=geojson\",\"felt\":1816,\"cdi\":7.2,\"mmi\":6.6,\"alert\":\"green\",\"status\":\"reviewed\",\"tsunami\":1,\"sig\":1496,\"net\":\"us\",\"code\":\"10004gqp\",\"ids\":\",at00o1gd6r,us10004gqp,ak12496371,gcmt20160124103030,\",\"sources\":\",at,us,ak,gcmt,\",\"types\":\",cap,dyfi,finite-fault,general-link,general-text,geoserve,impact-link,impact-text,losspager,moment-tensor,nearby-cities,origin,phase-data,shakemap,tectonic-summary,trump-origin,\",\"nst\":null,\"dmin\":0.72,\"rms\":2.11,\"gap\":19,\"magType\":\"mww\",\"type\":\"earthquake\",\"title\":\"M 7.1 - 86km E of Old Iliamna, Alaska\"},\"geometry\":{\"type\":\"Point\",\"coordinates\":[-153.4051,59.6363,129]},\"id\":\"us10004gqp\"},\n" +
"{\"type\":\"Feature\",\"properties\":{\"mag\":6.6,\"place\":\"215km SW of Tomatlan, Mexico\",\"time\":1453399617650,\"updated\":1459963829040,\"tz\":-420,\"url\":\"http://earthquake.usgs.gov/earthquakes/eventpage/us10004g4l\",\"detail\":\"http://earthquake.usgs.gov/fdsnws/event/1/query?eventid=us10004g4l&format=geojson\",\"felt\":11,\"cdi\":2.7,\"mmi\":3.92,\"alert\":\"green\",\"status\":\"reviewed\",\"tsunami\":1,\"sig\":673,\"net\":\"us\",\"code\":\"10004g4l\",\"ids\":\",at00o1bebo,pt16021050,us10004g4l,gcmt20160121180659,\",\"sources\":\",at,pt,us,gcmt,\",\"types\":\",cap,dyfi,geoserve,impact-link,impact-text,losspager,moment-tensor,nearby-cities,origin,phase-data,shakemap,tectonic-summary,\",\"nst\":null,\"dmin\":2.413,\"rms\":0.98,\"gap\":74,\"magType\":\"mww\",\"type\":\"earthquake\",\"title\":\"M 6.6 - 215km SW of Tomatlan, Mexico\"},\"geometry\":{\"type\":\"Point\",\"coordinates\":[-106.9337,18.8239,10]},\"id\":\"us10004g4l\"},\n" +
"{\"type\":\"Feature\",\"properties\":{\"mag\":6.7,\"place\":\"52km SE of Shizunai, Japan\",\"time\":1452741933640,\"updated\":1459304879040,\"tz\":540,\"url\":\"http://earthquake.usgs.gov/earthquakes/eventpage/us10004ebx\",\"detail\":\"http://earthquake.usgs.gov/fdsnws/event/1/query?eventid=us10004ebx&format=geojson\",\"felt\":51,\"cdi\":5.8,\"mmi\":6.45,\"alert\":\"green\",\"status\":\"reviewed\",\"tsunami\":1,\"sig\":720,\"net\":\"us\",\"code\":\"10004ebx\",\"ids\":\",us10004ebx,pt16014050,at00o0xauk,gcmt20160114032534,\",\"sources\":\",us,pt,at,gcmt,\",\"types\":\",associate,cap,dyfi,geoserve,impact-link,impact-text,losspager,moment-tensor,nearby-cities,origin,phase-data,shakemap,\",\"nst\":null,\"dmin\":0.281,\"rms\":0.98,\"gap\":22,\"magType\":\"mww\",\"type\":\"earthquake\",\"title\":\"M 6.7 - 52km SE of Shizunai, Japan\"},\"geometry\":{\"type\":\"Point\",\"coordinates\":[142.781,41.9723,46]},\"id\":\"us10004ebx\"},\n" +
"{\"type\":\"Feature\",\"properties\":{\"mag\":6.1,\"place\":\"12km WNW of Charagua, Bolivia\",\"time\":1452741928270,\"updated\":1459304879040,\"tz\":-240,\"url\":\"http://earthquake.usgs.gov/earthquakes/eventpage/us10004ebw\",\"detail\":\"http://earthquake.usgs.gov/fdsnws/event/1/query?eventid=us10004ebw&format=geojson\",\"felt\":3,\"cdi\":2.2,\"mmi\":2.21,\"alert\":\"green\",\"status\":\"reviewed\",\"tsunami\":0,\"sig\":573,\"net\":\"us\",\"code\":\"10004ebw\",\"ids\":\",us10004ebw,gcmt20160114032528,\",\"sources\":\",us,gcmt,\",\"types\":\",cap,dyfi,geoserve,impact-text,losspager,moment-tensor,nearby-cities,origin,phase-data,shakemap,tectonic-summary,\",\"nst\":null,\"dmin\":5.492,\"rms\":1.04,\"gap\":16,\"magType\":\"mww\",\"type\":\"earthquake\",\"title\":\"M 6.1 - 12km WNW of Charagua, Bolivia\"},\"geometry\":{\"type\":\"Point\",\"coordinates\":[-63.3288,-19.7597,582.56]},\"id\":\"us10004ebw\"},\n" +
"{\"type\":\"Feature\",\"properties\":{\"mag\":6.2,\"place\":\"74km NW of Rumoi, Japan\",\"time\":1452532083920,\"updated\":1459304875040,\"tz\":540,\"url\":\"http://earthquake.usgs.gov/earthquakes/eventpage/us10004djn\",\"detail\":\"http://earthquake.usgs.gov/fdsnws/event/1/query?eventid=us10004djn&format=geojson\",\"felt\":8,\"cdi\":3.4,\"mmi\":3.74,\"alert\":\"green\",\"status\":\"reviewed\",\"tsunami\":0,\"sig\":594,\"net\":\"us\",\"code\":\"10004djn\",\"ids\":\",us10004djn,gcmt20160111170803,\",\"sources\":\",us,gcmt,\",\"types\":\",cap,dyfi,geoserve,impact-text,losspager,moment-tensor,nearby-cities,origin,phase-data,shakemap,tectonic-summary,\",\"nst\":null,\"dmin\":1.139,\"rms\":0.96,\"gap\":33,\"magType\":\"mww\",\"type\":\"earthquake\",\"title\":\"M 6.2 - 74km NW of Rumoi, Japan\"},\"geometry\":{\"type\":\"Point\",\"coordinates\":[141.0867,44.4761,238.81]},\"id\":\"us10004djn\"},\n" +
"{\"type\":\"Feature\",\"properties\":{\"mag\":6.5,\"place\":\"227km SE of Sarangani, Philippines\",\"time\":1452530285900,\"updated\":1459304874040,\"tz\":480,\"url\":\"http://earthquake.usgs.gov/earthquakes/eventpage/us10004dj5\",\"detail\":\"http://earthquake.usgs.gov/fdsnws/event/1/query?eventid=us10004dj5&format=geojson\",\"felt\":1,\"cdi\":2.7,\"mmi\":7.5,\"alert\":\"green\",\"status\":\"reviewed\",\"tsunami\":1,\"sig\":650,\"net\":\"us\",\"code\":\"10004dj5\",\"ids\":\",at00o0srjp,pt16011050,us10004dj5,gcmt20160111163807,\",\"sources\":\",at,pt,us,gcmt,\",\"types\":\",cap,dyfi,geoserve,impact-link,impact-text,losspager,moment-tensor,nearby-cities,origin,phase-data,shakemap,tectonic-summary,\",\"nst\":null,\"dmin\":3.144,\"rms\":0.72,\"gap\":22,\"magType\":\"mww\",\"type\":\"earthquake\",\"title\":\"M 6.5 - 227km SE of Sarangani, Philippines\"},\"geometry\":{\"type\":\"Point\",\"coordinates\":[126.8621,3.8965,13]},\"id\":\"us10004dj5\"},\n" +
"{\"type\":\"Feature\",\"properties\":{\"mag\":6,\"place\":\"Pacific-Antarctic Ridge\",\"time\":1451986454620,\"updated\":1459202978040,\"tz\":-540,\"url\":\"http://earthquake.usgs.gov/earthquakes/eventpage/us10004bgk\",\"detail\":\"http://earthquake.usgs.gov/fdsnws/event/1/query?eventid=us10004bgk&format=geojson\",\"felt\":0,\"cdi\":1,\"mmi\":0,\"alert\":\"green\",\"status\":\"reviewed\",\"tsunami\":0,\"sig\":554,\"net\":\"us\",\"code\":\"10004bgk\",\"ids\":\",us10004bgk,gcmt20160105093415,\",\"sources\":\",us,gcmt,\",\"types\":\",cap,dyfi,geoserve,losspager,moment-tensor,nearby-cities,origin,phase-data,shakemap,\",\"nst\":null,\"dmin\":30.75,\"rms\":0.67,\"gap\":71,\"magType\":\"mww\",\"type\":\"earthquake\",\"title\":\"M 6.0 - Pacific-Antarctic Ridge\"},\"geometry\":{\"type\":\"Point\",\"coordinates\":[-136.2603,-54.2906,10]},\"id\":\"us10004bgk\"}],\"bbox\":[-153.4051,-54.2906,10,158.5463,59.6363,582.56]}";
/**
* Create a private constructor because no one should ever create a {#link QueryUtil} object.
* This class is only meant to hold static variables and methods, which can be accessed
* directly from the class name QueryUtils (and an object instance of QueryUtils is not needed).
*/
private QueryUtil() {
}
/**
* Return a list of {#link InfosQuake} objects that has been built up from
* parsing a JSON response.
*/
public static ArrayList<InfosQuake> extractEarthquakes() {
// Create an empty ArrayList that we can start adding earthquakes to
ArrayList<InfosQuake> earthquakes = new ArrayList<>();
// Try to parse the SAMPLE_JSON_RESPONSE. If there's a problem with the way the JSON
// is formatted, a JSONException exception object will be thrown.
// Catch the exception so the app doesn't crash, and print the error message to the logs.
try {
// TODO: Parse the response given by the SAMPLE_JSON_RESPONSE string and
// build up a list of Earthquake objects with the corresponding data.
JSONObject root = new JSONObject(SAMPLE_JSON_RESPONSE);
// Getting json array code.
JSONArray earthQuakeArrays = root.getJSONArray("Feature");
// Looping through all infosNodes that we want.
int i;
for (i = 0; i < earthQuakeArrays.length(); i += 1);
JSONObject currentEarthQuake = earthQuakeArrays.getJSONObject(i);
JSONObject properties = currentEarthQuake.getJSONObject("properties");
String magnitude = properties.getString("mag");
String location = properties.getString("place");
String time = properties.getString("time");
InfosQuake earthquake = new InfosQuake(magnitude, location, time);
earthquakes.add(earthquake);
} catch (JSONException e) {
// If an error is thrown when executing any of the above statements in the "try" block,
// catch the exception here, so the app doesn't crash. Print a log message
// with the message from the exception.
Log.e("QueryUtils", "Problem parsing the earthquake JSON results", e);
}
// Return the list of earthquakes
return earthquakes;
}
}
(The error that is causing)
Problem parsing the earthquake JSON results
org.json.JSONException: No value for Feature
at org.json.JSONObject.get(JSONObject.java:389)
at org.json.JSONObject.getJSONArray(JSONObject.java:584)
at com.example.android.earthquakeapp.QueryUtil.extractEarthquakes(QueryUtil.java:60)
at com.example.android.earthquakeapp.EarthquakeActivity.onCreate(EarthquakeActivity.java:20)
at android.app.Activity.performCreate(Activity.java:6575)
at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1134)
at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:3143)
at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:3300)
at android.app.ActivityThread.access$1000(ActivityThread.java:211)
at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1705)
at android.os.Handler.dispatchMessage(Handler.java:102)
at android.os.Looper.loop(Looper.java:145)
at android.app.ActivityThread.main(ActivityThread.java:6946)
at java.lang.reflect.Method.invoke(Native Method)
at java.lang.reflect.Method.invoke(Method.java:372)
at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:1404)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1199)
https://classroom.udacity.com/courses/ud843/lessons/1335cf7d-bb4f-48c6-8503-f14b127d2abc/concepts/0eca5789-48d7-4662-8e87-d71490519bd4

You can parse like this below
yours did mistake in getting the array using the key features and your didnt loop correctly.
public static ArrayList<InfosQuake> extractEarthquakes() {
ArrayList<InfosQuake> earthquakes = new ArrayList<>();
try {
// TODO: Parse the response given by the SAMPLE_JSON_RESPONSE string and
// build up a list of Earthquake objects with the corresponding data.
JSONObject root = new JSONObject(SAMPLE_JSON_RESPONSE);
// Getting json array code.
JSONArray earthQuakeArrays = root.optJSONArray("features");
if (earthQuakeArrays == null) {
return earthquakes;
}
// Looping through all infosNodes that we want.
for (int i = 0; i < earthQuakeArrays.length(); i++) {
JSONObject currentEarthQuake = earthQuakeArrays.optJSONObject(i);
if (currentEarthQuake == null) {
continue;
}
JSONObject properties = currentEarthQuake.optJSONObject("properties");
if (properties == null) { // In your case we dont have any other things to get so continue
continue;
}
String magnitude = properties.optString("mag");
String location = properties.optString("place");
String time = properties.optString("time");
InfosQuake earthquake = new InfosQuake(magnitude, location, time);
earthquakes.add(earthquake);
}
} catch (JSONException e) {
// If an error is thrown when executing any of the above statements in the "try" block,
// catch the exception here, so the app doesn't crash. Print a log message
// with the message from the exception.
Log.e("QueryUtils", "Problem parsing the earthquake JSON results", e);
}
// Return the list of earthquakes
return earthquakes;
}

SQLiteException: near "null": syntax error (code 1): , while compiling: create table

Please help me, i get this errors when i run my app use Idea
here is screen of error
http://prntscr.com/en3lcu
here is part of code where create table
private String getTableDDL(final Class<? extends GlassContract.Table> table) {
return getTableDDL(table, GlassContract.getTableName(table));
}
private String getTableDDL(final Class<? extends GlassContract.Table> table, String tableName) {
final StringBuilder sql = new StringBuilder(128);
sql.append("create table ").append(tableName).append(" (");
for (final Field field : table.getFields()) {
if (field.getName().startsWith("_") || field.isAnnotationPresent(Deprecated.class))
continue;
try {
sql.append(field.get(null));
} catch (Exception ignore) {
}
try {
final Field type = table.getDeclaredField("_SQL_" + field.getName() + "_TYPE");
sql.append(' ').append(type.get(null));
} catch (Exception ignore) {
sql.append(" TEXT");
}
sql.append(',');
}
try {
final Field type = table.getDeclaredField("_PK_COMPOSITE");
sql.append("PRIMARY KEY(").append(type.get(null)).append(")");
sql.append(',');
} catch (Exception ignore) {
// ignore
}
try {
final Field type = table.getDeclaredField("_UNIQUE_COMPOSITE");
sql.append("UNIQUE(").append(type.get(null)).append(")");
sql.append(',');
} catch (Exception ignore) {
// ignore
}
sql.setLength(sql.length() - 1); // chop off last comma
sql.append(')');
Log.v(TAG, "DDL for " + table.getSimpleName() + ": " + sql);
return sql.toString();
}
I please help me, because I break my head))

Are you really trying to create text fields in your table that are named null?
Even if this works (and I am not sure it does), you are duplicating this and creating two identically named fields called null

The first field in that CREATE TABLE statement doesn't have a proper name (it is "null"). That's why it blows up.
There's more fields with illegal names as well.

Integrating Kafka with Apache Calcite

I'm trying to integrate calcite with Kafka, I refrenced CsvStreamableTable.
Each ConsumerRecord is convert to Object[] using the fowlloing code:
static class ArrayRowConverter extends RowConverter<Object[]> {
private List<Schema.Field> fields;
public ArrayRowConverter(List<Schema.Field> fields) {
this.fields = fields;
}
#Override
Object[] convertRow(ConsumerRecord<String, GenericRecord> consumerRecord) {
Object[] objects = new Object[fields.size()+1];
int i = 0 ;
objects[i++] = consumerRecord.timestamp();
for(Schema.Field field : this.fields) {
Object obj = consumerRecord.value().get(field.name());
if( obj instanceof Utf8 ){
objects[i ++] = obj.toString();
}else {
objects[i ++] = obj;
}
}
return objects;
}
}
Enumerator is implemented as following,one thread is constantly polling records from kafka and put them into a queue, getRecord() method poll from that queue:
public E current() {
return current;
}
public boolean moveNext() {
for(;;) {
if(cancelFlag.get()) {
return false;
}
ConsumerRecord<String, GenericRecord> record = getRecord();
if(record == null) {
try {
Thread.sleep(200L);
} catch (InterruptedException e) {
e.printStackTrace();
}
continue;
}
current = rowConvert.convertRow(record);
return true;
}
}
I tested SELECT STREAM * FROM Kafka.clicks, it works fine.
rowtime is the first column explicitly added,and the value is record Timestamp of Kafka.
But when I tried
SELECT STREAM FLOOR(rowtime TO HOUR)
AS rowtime,ip,COUNT(*) AS c FROM KAFKA.clicks GROUP BY FLOOR(rowtime TO HOUR), ip
It threw exception
java.sql.SQLException: Error while executing SQL "SELECT STREAM FLOOR(rowtime TO HOUR) AS rowtime,ip,COUNT(*) AS c FROM KAFKA.clicks GROUP BY FLOOR(rowtime TO HOUR), ip": From line 1, column 85 to line 1, column 119: Streaming aggregation requires at least one monotonic expression in GROUP BY clause
at org.apache.calcite.avatica.Helper.createException(Helper.java:56)
at org.apache.calcite.avatica.Helper.createException(Helper.java:41)

You need to declare that the "ROWTIME" column is monotonic. In MockCatalogReader, note how "ROWTIME" is declared monotonic in the "ORDERS" and "SHIPMENTS" streams. That’s why some queries in SqlValidatorTest.testStreamGroupBy() are valid and others are not. The key method relied up by the validator is SqlValidatorTable.getMonotonicity(String columnName).