I am trying to read cloud SQL table in java beam using JdbcIO.Read. I want to convert each row in Resultset into GenericData.Record using .withRowMapper(Resultset resultSet) method. Is there a way I can pass JSON Schema String as input in .withRowMapper method like ParDo accepts sideInputs as PCollectionView
I have tried doing both reads operations (read from information_schema.columns and My Table in same JdbcIO.Read transform). However, I would like to have Schema PCollection generated first and then read table using JdbcIO.Read
I am generating Avro schema of table on the fly like this :
PCollection<String> avroSchema= pipeline.apply(JdbcIO.<String>read()
.withDataSourceConfiguration(config)
.withCoder(StringUtf8Coder.of())
.withQuery("SELECT DISTINCT column_name, data_type \n" +
"FROM information_schema.columns\n" +
"WHERE table_name = " + "'" + tableName + "'")
.withRowMapper((JdbcIO.RowMapper<String>) resultSet -> {
// code here to generate avro schema string
// this works fine for me
}))
Creating PCollectionView which will hold my json schema for each table.
PCollectionView<String> s = avroSchema.apply(View.<String>asSingleton());
// I want to access this view as side input in next JdbcIO.Read operation
// something like this ;
pipeline.apply(JdbcIO.<String>read()
.withDataSourceConfiguration(config)
.withCoder(StringUtf8Coder.of())
.withQuery(queryString)
.withRowMapper(new JdbcIO.RowMapper<String>() {
#Override
public String mapRow(ResultSet resultSet) throws Exception {
// access schema here and use it to parse and create
//GenericData.Record from ResultSet fields as per schema
return null;
}
})).
withSideInputs(My PCollectionView here); // this option is not there right now.
Is there any better way to approach this problem?
At this point IOs API do not accept SideInputs.
It should be feasible to add ParDo right after read and do mapping there. That ParDo can accept side inputs.
Related
I have created a delta table and now I'm trying to insert data to that table using foreachBatch(). I've followed this example. The only difference is that I'm using Java and not in a notebook, but I suppose that should not make any difference?
My code looks as follows:
spark.sql("CREATE TABLE IF NOT EXISTS example_src_table(id int, load_date timestamp) USING DELTA LOCATION '/mnt/delta/events/example_src_table'");
Dataset<Row> exampleDF = spark.sql("SELECT e.id as id, e.load_date as load_date FROM example e");
try {
exampleDF
.writeStream()
.format("delta")
.foreachBatch((dataset, batchId) -> {
dataset.persist();
// Set the dataframe to view name
dataset.createOrReplaceTempView("updates");
// Use the view name to apply MERGE
// NOTE: You have to use the SparkSession that has been used to define the `updates` dataframe
dataset.sparkSession().sql("MERGE INTO example_src_table e" +
" USING updates u" +
" ON e.id = u.id" +
" WHEN NOT MATCHED THEN INSERT (e.id, e.load_date) VALUES (u.id, u.load_date)");
})
.outputMode("update")
.option("checkpointLocation", "/mnt/delta/events/_checkpoints/example_src_table")
.start();
} catch (TimeoutException e) {
e.printStackTrace();
}
This code runs without any problems, but there is no data written to the delta table with url '/mnt/delta/events/example_src_table'. Anyone know what I'm doing wrong?
I'm using Spark 3.0 and Java 8.
EDIT
Tested on a Databricks Notebook using Scala, and then it worked just fine.
try to follow a syntax like the following one in case you want to update the data with the new data
WHEN NOT MATCHED THEN
UPDATE SET e.load_date = u.load_date AND e.id = u.id
If you only want to add the data it occupies something like this
WHEN NOT MATCHED THEN INSERT *
I have a column in my postgres DB called metadata which stores a JSON string, and the type is TEXT.
I'm trying to run a query to update a field named myCount inside the JSON. I'm using Spring Boot and JDBC.
String query = "UPDATE " + mTableName + " SET metadata = jsonb_set(metadata::jsonb, '{myCount}', ?)::text" +
" WHERE scope = ?";
PreparedStatement preparedStmt = jdbcTemplate.getDataSource().getConnection().prepareStatement(query);
preparedStmt.setInt (1, myCount);
preparedStmt.setString(2, scope);
// execute the java preparedstatement
return preparedStmt.executeUpdate();
I got the following error: ERROR: function jsonb_set(jsonb, unknown, integer) does not
Any ide ahow I can run a query that updates the myCount column inside the JSON?
function jsonb_set(jsonb, unknown, integer) does not
Tells you that you are trying to call the function with an integer value as the last parameter. But the function is defined as jsonb_set(jsonb, text[], jsonb) so you will need to convert the integer value to a JSONB value:
SET metadata = jsonb_set(metadata::jsonb, '{myCount}'::text[], to_jsonb(?))::text"
I'm trying to use EsperIO to load some information from database and use it in other queries with different conditions. To do it I'm using the following code:
ConfigurationDBRef dbConfig = new ConfigurationDBRef();
dbConfig.setDriverManagerConnection("org.postgresql.Driver",
"jdbc:postgresql://localhost:5432/myDatabase",
"myUser", "myPassword");
Configuration engineConfig = new Configuration();
engineConfig.addDatabaseReference("myDatabase", dbConfig);
// Custom class
engineConfig.addEventType("UserFromDB", UserDB.class);
EPServiceProvider esperEngine = EPServiceProviderManager.getDefaultProvider(engineConfig);
String statement = "insert into UserFromDB "
+ " select * from sql:myDatabase ['SELECT * from data.user']";
//Install this query in the engine
EPStatement queryEngineObject = esperEngine.getEPAdministrator().createEPL(statement);
// 1. At this point I can iterate over queryEngineObject without problems getting the information sent by database
// This query is only a 'dummy example', the 'final queries' are more complex
statement = "select * from UserFromDB";
EPStatement queryEngineObject2 = esperEngine.getEPAdministrator().createEPL(statement);
// 2. If I try to iterate over queryEngineObject2 I receive no data
How can I reuse UserFromDB stored information in other queries? (in the above example, in queryEngineObject2)
You don't have a stream since the database doesn't provide a stream. The database query provides rows only when its being iterated/pulled.
One option is to loop over each row and send it into the engine using "sendEvent":
// create other EPL statements before iterating
Iterator<EventBean> it = statement.iterator();
while(it.hasNext()) {
epService.getEPRuntime().sendEvent(event);
}
I am trying to make a simple program to fetch data from a table.
I am following http://www.avaje.org/ebean/getstarted_props.html#iud but am unable to get data. I have created new Entity Class from Database from Netbeans (which creates classes from relations). Here is what I am using:
ebean.properties
ebean.ddl.generate=true
ebean.ddl.run=true
ebean.debug.sql=true
ebean.debug.lazyload=false
ebean.logging=all
ebean.logging.logfilesharing=all
ebean.logging.directory=D:\\logs
ebean.logging.iud=sql
ebean.logging.query=sql
ebean.logging.sqlquery=sql
ebean.logging.txnCommit=none
datasource.default=h2
datasource.h2.username=sa
datasource.h2.password=
datasource.h2.databaseUrl=jdbc:h2:mem:tests;DB_CLOSE_DELAY=-1
datasource.h2.databaseDriver=org.h2.Driver
datasource.h2.minConnections=1
datasource.h2.maxConnections=25
datasource.h2.heartbeatsql=select 1
datasource.h2.isolationlevel=read_committed
datasource.mysql.username=root
datasource.mysql.password=kalsym#123
datasource.mysql.databaseUrl=jdbc:mysql://127.0.0.1:3306/wsp
datasource.mysql.databaseDriver=com.mysql.jdbc.Driver
datasource.mysql.minConnections=1
datasource.mysql.maxConnections=25
datasource.mysql.isolationlevel=read_committed
Table Data
Insert into routing_algo_type (name, description) values ('LCR', 'Least Cost Routing');
Code to fetch data
RoutingAlgoType routingObj = new RoutingAlgoType();
routingObj.setName("LCR");
RoutingAlgoType routingObj2 = Ebean.find(RoutingAlgoType.class, routingObj);
System.out.println("Got "+routingObj2.getDescription());
Now find returns null, which means it cant find the data?
I used following code to test connection
String sql = "select count(*) as count from dual";
SqlRow row = Ebean.createSqlQuery(sql).findUnique();
Integer i = row.getInteger("count");
System.out.println("Got " + i + " - DataSource good.");
Result from above code is
Got 1 - DataSource good.
Is there any way to check the connection?
Hi basically im doing this to retrieve data from database to array
for(i=0;i<numfilas;i++){
HashMap<Object, Object> rowdata = new HashMap<Object, Object>(cur.getNextRow());
for(j=0;j<numcolumnas;j++){
datos[posicion]=rowdata.get(nombrecolumnas[j]).toString();
posicion++;
}
}
then I pass the data to EditTexts so the user can edit it and after that I update the array, the problem is how do I take this data and send it back to the database?
Am I getting in trouble with datatypes? because the array is String type and the database had int Type, String Type, long type .....
thanks in advance
Am I getting in trouble with datatypes? because the array is String type and the database had int Type, String Type, long type .....
You could be, if any of the fields you are trying to update are Date/Time fields in Access. Jackcess is able to implicitly cast strings to numbers (in many cases, anyway), but it is not able to do that when it comes to dates.
For the sample data in a table named [Members]
MemberID MemberName SponsorID DateJoined FeePaid
-------- ---------- --------- ---------- -------
1 Gord 2014-01-16 0
the following code works fine
try (Database db = DatabaseBuilder.open(new File("C:/Users/Public/mdbTest.mdb"))) {
Table table = db.getTable("Members");
Row row = CursorBuilder.findRow(table, Collections.singletonMap("MemberID", 1));
if (row != null) {
row.put("SponsorID", "0"); // "Long Integer" in Access
row.put("FeePaid", "130"); // "Currency" in Access
table.updateRow(row);
}
else {
System.out.println("row not found.");
}
} catch (Exception e) {
e.printStackTrace(System.out);
}
However, this will not work
row.put("DateJoined", "2014-01-23"); // "Date/Time" in Access
because Jackcess cannot implicitly cast the string value to its internal (numeric) date value, Instead you need to do something like this
org.joda.time.DateTime dt = new org.joda.time.DateTime("2014-01-23");
row.put("DateJoined", dt.getMillis());
As an alternative, you may want to investigate UCanAccess. It is a pure-Java JDBC driver that uses Jackcess to perform reads and writes on the Access database but lets you do it using more "normal" SQL methods like this:
Connection conn=DriverManager.getConnection(
"jdbc:ucanaccess://C:/Users/Public/mdbTest.mdb");
PreparedStatement ps = conn.prepareStatement(
"UPDATE Members SET " +
"SponsorID=?, " +
"DateJoined=?, " +
"FeePaid=? " +
"WHERE MemberID=1");
ps.setInt(1, Integer.parseInt("0"));
org.joda.time.DateTime dt = new org.joda.time.DateTime("2014-01-23");
ps.setTimestamp(2, new Timestamp(dt.getMillis()));
ps.setBigDecimal(3, new BigDecimal("130"));
ps.executeUpdate();