Develop Reference

Push Data to Existing Table in BigQuery

I am using Java and SQL to push data to a Timestamp partitioned table in BigQuery. In my code, I specify the destination table:
.setDestinationTable(TableId.of("MyDataset", "MyTable"))
When I run it, it creates a table perfectly. However, when I attempt to insert new data, it throws a BigQueryException claiming the table already exists:
Exception in thread "main" com.google.cloud.bigquery.BigQueryException:
Already Exists: Table MyProject:MyDataset.MyTable

After some documentation digging, I found a solution that works:
.setWriteDisposition(WriteDisposition.WRITE_APPEND)
Adding the above appends any data (even if it's duplicate). I'm not sure why the default setting for .setDestinationTable() is the equivalent of WRITE_EMPTY, which returns a duplicate error. The google docs for .setDestinationTable() are:
Describes the table where the query results should be stored. If not
present, a new table will be created to store the results.
The docs should probably clarify the default value.

Dash in Schema Name - Groovy SQL query error

A Quick groovy syntax question here:
I'm working with Groovy SQL capabilities (Groovy 2.4)
My Schema contains dashes, like "SAMPLE-SCHEMA" and my table is called "SAMPLE_TABLE"
When I run the following I'm getting an exception that the relation does not exist.
I'm running against Postgres 9.6 with the correct driver.
def sql = Sql.newInstance(...)
sql.eachRow('SELECT SAMPLE_COLUMN FROM \"SAMPLE-SCHEMA\".SAMPLE_TABLE') {
row -> // do something with row here
}
If I query another schema without Dashes it works flowlessly.
The exception message is:
Caught: org.postgresql.util.PSQLException: ERROR: relation "SAMPLE-SCHEMA.SAMPLE_TABLE" does not exist
How can I adjust my query to make it work? Thanks

Ok, I've found the answer, the schema in postgresql is case sensitive, so I by mistake called "SAMPLE-SCHEMA" and it should have been "sample-schema" instead.
I'm not deleting the question because it might help someone

Error while using Lucene with H2 Database

I want to implement a small fullText search in my project that uses H2 Database (embedded). As I know I have to use Lucene for fullText engine for find relevance results (not only containing results).
But I can't use it. This block is Lucene initiation:
FullTextLucene.init(connection);
FullTextLucene.createIndex(connection, "PUBLIC", Tables.COURSES_DETAIL, Columns.NAME);
Also I used this way:
stmt.execute(
"create alias if not exists FTL_INIT for \"org.h2.fulltext.FullTextLucene.init\"");
stmt.execute("call FTL_INIT()");
stmt.execute(
String.format("CALL FTL_CREATE_INDEX('PUBLIC','%s',%s)", Tables.COURSES_DETAIL, "NULL"));
But this error happens at runtime:
Error creating or initializing trigger "FTL_COURSES_DETAIL" object, class "org.h2.fulltext.FullTextLucene$FullTextTrigger", cause: "org.h2.message.DbException: Class ""org.h2.fulltext.FullTextLucene$FullTextTrigger"" not found [90086-197]"; see root cause for details; SQL statement:
CREATE TRIGGER IF NOT EXISTS "PUBLIC"."FTL_COURSES_DETAIL" AFTER INSERT, UPDATE, DELETE, ROLLBACK ON "PUBLIC"."COURSES_DETAIL" FOR EACH ROW CALL "org.h2.fulltext.FullTextLucene$FullTextTrigger"
After I downgraded H2 library to latest 'stable' version (1.4.196) the error has been changed:
Caused by: java.lang.NoSuchMethodError: org.apache.lucene.store.FSDirectory.open(Ljava/io/File;)Lorg/apache/lucene/store/FSDirectory;
and sometimes this error:
Exception calling user-defined function: "init(conn1: url=jdbc:default:connection user=INFC): org.apache.lucene.store.FSDirectory.open(Ljava/io/File;)Lorg/apache/lucene/store/FSDirectory;"; SQL statement:
call FTL_INIT()

I found a solution. But I know this isn't best one.
I downgraded Lucene lib to 3.6.2 and used plain queries instead of FullTextLucene functions.

How to fix the Exception: Saving data in the Hive serde table Please use the insertInto() API as an alternative. Spark:2.1.0

We are trying to save Dataframe to a Hive Table using the saveAsTable() method. But, We are getting the below exception. We are trying to store the data as TextInputFormat.
Exception in thread "main" org.apache.spark.sql.AnalysisException: Saving data in the Hive serde table `cdx_network`.`inv_devices_incr` is not supported yet. Please use the insertInto() API as an alternative..;
reducedFN.write().mode(SaveMode.Append).saveAsTable("cdx_network.alert_pas_incr");
I tried insertInto() and also enableHiveSupport() and it works. But, I want to use saveAsTable() .
I want to understand why the saveAsTable() does not work. I tried going through the documentation and also the code. Did not get much understanding. It supposed to be working. I have seen issues raised by people who are using Parquet format but for TextFileInputFormat i did not see any issues.
Table definition
CREATE TABLE `cdx_network.alert_pas_incr`(
`alertid` string,
`alerttype` string,
`alert_pas_documentid` string)
ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION
'maprfs:/apps/cdx-dev/alert_pas_incr'
TBLPROPERTIES (
'COLUMN_STATS_ACCURATE'='{\"BASIC_STATS\":\"true\"}',
'numFiles'='0',
'numRows'='0',
'rawDataSize'='0',
'totalSize'='0',
'transient_lastDdlTime'='1524121971')

Looks like this is bug. I made a little research and found this issue SPARK-19152. Fixed version is 2.2.0. Unfortunately I can’t verify it, cause my company’s cluster uses version 2.1.0

empire db code-gen double parse failure in reading "0.-127" from table meta data

I'm trying to reverse engineer using empire db code-gen on my project tables in Oracle.
For a primary key column (ID), the meta-data returned has invalid values ( COLUMN_SIZE is 0 and for field DECIMAL_DIGITS is -127 ), which is the cause of Exception : For input string: "0.-127"
Can anyone enligthen me on why DECIMAL_DIGITS is -127.
Exception in thread "main" java.lang.NumberFormatException: For input string: "0.-127"
at sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1241)
at java.lang.Double.parseDouble(Double.java:540)
at org.apache.empire.db.codegen.CodeGenParser.addColumn(CodeGenParser.java:368)
at org.apache.empire.db.codegen.CodeGenParser.populateTable(CodeGenParser.java:300)
at org.apache.empire.db.codegen.CodeGenParser.populateDatabase(CodeGenParser.java:168)
at org.apache.empire.db.codegen.CodeGenParser.loadDbModel(CodeGenParser.java:96)
at org.apache.empire.db.codegen.CodeGenerator.generate(CodeGenerator.java:57)
at org.apache.empire.db.codegen.CodeGenerator.generate(CodeGenerator.java:72)
at org.apache.empire.db.codegen.CodeGenerator.main(CodeGenerator.java:45)
I invoke the CodeGenerator using mvn generate-sources
PS: I've tried both ojdbc6 and ojdbc14 jars, it didn't work.

It seems to be related to the issue reported here: DatabaseMetaData.getColumns() returns unexpected values in COLUMN_SIZE and DECIMAL_DIGITS for INTEGER columns (the -127 is coming from "int decimalDig = rs.getInt("DECIMAL_DIGITS");")
And running the JVM with the option "-Doracle.jdbc.J2EE13Compliant=true" seems to fix it. Could you try?

Looks like this is a bug in Oracle JDBC:
https://community.oracle.com/thread/3677876
As workaround you could change line 368 in CodeGenParser.java if rs.getInt("DECIMAL_DIGITS") is -127.