Develop Reference

Google Cloud Search Database Connector Issues

I am working with Google Cloud search (https://developers.google.com/cloud-search/docs/guides/?_ga=2.124920714.-122300216.1578247736) and I am attempting to index a Cloud SQL instance. Presently I am using the guide as shown here (https://developers.google.com/cloud-search/docs/guides/database-connector#important-considerations). I have registered source in G-Suite. I have a Cloud Search Service Account, I tested that I connect to the Cloud SQL instance from my Compute Engine instance which I can.
My config file is as follows with the necessary information replaced with XXXX:
#
# data source access
api.sourceId=xxxxxxxxxxx
api.identitySourceId=xxxxxxxxxxxxxxxx
api.serviceAccountPrivateKeyFile=./private-key.json
#
# database access
db.url=jdbc:mysql:///<database>?cloudSqlInstance=<cloud_sql_instance>&socketFactory=mysql-socket-factory-connector-j-8&useSSL=false&user=xxxxxxxxx&password=xxxxxxxxx
#
# traversal SQL statements
db.allRecordsSql=select field_1, field_2, field_3 from table;
#
# schedule traversals
schedule.traversalIntervalSecs=36000
schedule.performTraversalOnStart=true
schedule.incrementalTraversalIntervalSecs=3600
#
# column definitions
db.allColumns= field1, field2, field3
db.uniqueKeyColumns=field1
url.columns=field1
#
# content fields
contentTemplate.db.title=field1
db.contentColumns=field1, field2, field3
#
# setting ACLs to "entire domain accessible"
defaultAcl.mode=fallback
defaultAcl.public=true
with the jdbc connection string being based off https://github.com/GoogleCloudPlatform/cloud-sql-jdbc-socket-factory. I'm at the stage where I run:
java \
-cp "google-cloudsearch-database-connector-v1-0.0.3.jar:mysql-connector-java-5.1.41-bin.jar" \
com.google.enterprise.cloudsearch.database.DatabaseFullTraversalConnector \
[-Dconfig=mysql.config]
but i get the error Failed to initialize connector. The full stacktrace is:
Jan 05, 2020 6:29:38 PM com.google.enterprise.cloudsearch.sdk.indexing.IndexingApplication startUp
SEVERE: Failed to initialize connector
com.google.enterprise.cloudsearch.sdk.StartupException: Failed to initialize connector
at com.google.enterprise.cloudsearch.sdk.Application.startConnector(Application.java:150)
at com.google.enterprise.cloudsearch.sdk.indexing.IndexingApplication.startUp(IndexingApplication.java:96)
at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62)
at com.google.common.util.concurrent.Callables$4.run(Callables.java:122)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:871)
at com.google.common.io.BaseEncoding$StandardBaseEncoding.trimTrailingPadding(BaseEncoding.java:672)
at com.google.common.io.BaseEncoding.decodeChecked(BaseEncoding.java:226)
at com.google.common.io.BaseEncoding.decode(BaseEncoding.java:212)
at com.google.api.client.util.Base64.decodeBase64(Base64.java:93)
at com.google.api.services.cloudsearch.v1.model.Item.decodeVersion(Item.java:329)
at com.google.enterprise.cloudsearch.sdk.indexing.IndexingServiceImpl.indexItem(IndexingServiceImpl.java:678)
at com.google.enterprise.cloudsearch.sdk.indexing.DefaultAcl.<init>(DefaultAcl.java:203)
at com.google.enterprise.cloudsearch.sdk.indexing.DefaultAcl.<init>(DefaultAcl.java:93)
at com.google.enterprise.cloudsearch.sdk.indexing.DefaultAcl$Builder.build(DefaultAcl.java:466)
at com.google.enterprise.cloudsearch.sdk.indexing.DefaultAcl.fromConfiguration(DefaultAcl.java:266)
at com.google.enterprise.cloudsearch.sdk.indexing.template.FullTraversalConnector.init(FullTraversalConnector.java:182)
at com.google.enterprise.cloudsearch.sdk.indexing.template.FullTraversalConnector.init(FullTraversalConnector.java:97)
at com.google.enterprise.cloudsearch.sdk.Application.startConnector(Application.java:142)
... 4 more
Jan 05, 2020 6:29:38 PM com.google.enterprise.cloudsearch.sdk.BatchRequestService shutDown
INFO: Shutting down batching service. flush on shutdown: true
Exception in thread "main" java.lang.IllegalStateException: Expected the service IndexingApplication [FAILED] to be RUNNING, but the service has FAILED
at com.google.common.util.concurrent.AbstractService.checkCurrentState(AbstractService.java:344)
at com.google.common.util.concurrent.AbstractService.awaitRunning(AbstractService.java:280)
at com.google.common.util.concurrent.AbstractIdleService.awaitRunning(AbstractIdleService.java:175)
at com.google.enterprise.cloudsearch.sdk.Application.start(Application.java:122)
at com.google.enterprise.cloudsearch.database.DatabaseFullTraversalConnector.main(DatabaseFullTraversalConnector.java:30)
Caused by: com.google.enterprise.cloudsearch.sdk.StartupException: Failed to initialize connector
at com.google.enterprise.cloudsearch.sdk.Application.startConnector(Application.java:150)
at com.google.enterprise.cloudsearch.sdk.indexing.IndexingApplication.startUp(IndexingApplication.java:96)
at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62)
at com.google.common.util.concurrent.Callables$4.run(Callables.java:122)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:871)
at com.google.common.io.BaseEncoding$StandardBaseEncoding.trimTrailingPadding(BaseEncoding.java:672)
at com.google.common.io.BaseEncoding.decodeChecked(BaseEncoding.java:226)
at com.google.common.io.BaseEncoding.decode(BaseEncoding.java:212)
at com.google.api.client.util.Base64.decodeBase64(Base64.java:93)
at com.google.api.services.cloudsearch.v1.model.Item.decodeVersion(Item.java:329)
at com.google.enterprise.cloudsearch.sdk.indexing.IndexingServiceImpl.indexItem(IndexingServiceImpl.java:678)
at com.google.enterprise.cloudsearch.sdk.indexing.DefaultAcl.<init>(DefaultAcl.java:203)
at com.google.enterprise.cloudsearch.sdk.indexing.DefaultAcl.<init>(DefaultAcl.java:93)
at com.google.enterprise.cloudsearch.sdk.indexing.DefaultAcl$Builder.build(DefaultAcl.java:466)
at com.google.enterprise.cloudsearch.sdk.indexing.DefaultAcl.fromConfiguration(DefaultAcl.java:266)
at com.google.enterprise.cloudsearch.sdk.indexing.template.FullTraversalConnector.init(FullTraversalConnector.java:182)
at com.google.enterprise.cloudsearch.sdk.indexing.template.FullTraversalConnector.init(FullTraversalConnector.java:97)

This issue is "Caused by: java.lang.NullPointerException".
You should verify the line referred in the stack trace to find the culprit variable, which should be revealed by expanding the view of the "4 more" error lines in the stack.
Here's information about this exception.
Here's information about how to read a java stack trace

If I had to guess you're not establishing a connection to your database. Your configuration file looks good to me except db.url. It should look something like this from my experience jdbc:mysql://localhost:1433;DatabaseName=ExampleDatabase. The errors you typically get for other properties tell you exactly what the issue is (example field_1 does not exist in table) , but in this case it's not establishing a connection. Try rewriting the db.url.

Failed to initialize connector --> This exception occurs when there is something wrong with your config file. But usually it tells you what's wrong along with the message.
Did you try running Cloud SQL proxy on the Compute Engine VM and connecting through that?

H2: JdbcSQLException: This CLOB or BLOB reference timed out

Upon using h2 in java (calling getBytes function) I'm getting the following error. Google didn't get me anywhere. I wonder if anyone has clues what's going on.
Caused by: org.h2.jdbc.JdbcSQLException: This CLOB or BLOB reference timed out: "533668/-3" [90039-190]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:345)
at org.h2.message.DbException.get(DbException.java:179)
at org.h2.message.DbException.get(DbException.java:155)
at org.h2.store.LobStorageMap.getInputStream(LobStorageMap.java:287)
at org.h2.value.ValueLobDb.getInputStream(ValueLobDb.java:384)
at org.h2.value.ValueLobDb.getBytesNoCopy(ValueLobDb.java:318)
at org.h2.value.ValueLobDb.getBytes(ValueLobDb.java:304)
at org.h2.jdbc.JdbcResultSet.getBytes(JdbcResultSet.java:1059)
at edu.illinois.cs.cogcomp.srl.caches.SentenceDBHandler$1.next(SentenceDBHandler.java:214)

This error message can occur if you have a long running transaction for instance. You can increase the timeout by setting the LOB_TIMEOUT property on the database connection.
Database setting LOB_TIMEOUT (default: 300000, which means 5 minutes).
The number of milliseconds a temporary LOB reference is kept until it times out. After the timeout, the LOB is no longer accessible using this reference.
https://www.h2database.com/javadoc/org/h2/engine/DbSettings.html#LOB_TIMEOUT

Error Creating Table java.sql.SQLException: JDBC LLR, table verify failed for table 'QQE4D.WL_LLR_I0_LTAPP107', failed to create table

I have an issue with weblogic and jdbc configuration. We are using weblogic 12c and jdbc to connect to oracle.
I am getting an error while creating a table with name WL_LLR_ADMINSERVER.
Below is the error
Failing reason: javax.transaction.SystemException: Failed to call registerLoggingResourceTransactions() weblogic.transaction.loggingresource.LoggingResourceException: weblogic.transaction.loggingresource.LoggingResourceException: java.sql.SQLException: JDBC LLR, table verify failed for table 'QQE4D.WL_LLR_I0_LTAPP107', failed to create table javax.transaction.SystemException: Failed to call registerLoggingResourceTransactions() weblogic.transaction.loggingresource.LoggingResourceException: weblogic.transaction.loggingresource.LoggingResourceException: java.sql.SQLException: JDBC LLR, table verify failed for table 'QQE4D.WL_LLR_I0_LTAPP107', failed to create table.
The query being fired to create the table is
JDBC loggging last resource (LLR) connection pool KernelDataSource did not find its table and is creating a new one using the following SQL:
CREATE TABLE QQE4D.WL_LLR_I0_LTAPP107 (XIDSTR VARCHAR(40) NOT NULL PRIMARY KEY, POOLNAMESTR VARCHAR(64), RECORDSTR VARCHAR(1000))>
The code congigured for the datasource in xml file is
<jdbc-system-resource>
<name>KernelDataSource</name>
<target>cluster</target>
<descriptor-file-name>jdbc/KernelDataSource-int-jdbc.xml</descriptor-file-name>
The datasource xml contains the connection url, driver name, username and the encrypted password details.
I have seen some errors like below but for my case i am not able to create the table itself so i could find the exact solution for my issue.
weblogic.transaction.loggingresource.LoggingResourceException: java.sql.SQLException: JDBC LLR, table verify failed for table 'DEV_SOAINFRA.WL_LLR_ADMINSERVER', row 'JDBC LLR Domain//Server' record had unexpected value 'osb_domain//AdminServer' expected 'osb_cluster_domain//AdminServer' ONLY the original domain and server that creates an LLR table may access it.
Even i don't see these datasources linked to this server in weblogic console as well under monitoring tab. I guess we can see once this issue gets resolved.
I have seen that the atable name should not exceed 18 characters and my table WL_LLR_SERVERNAME is WL_LLR_I0_LTAPP107 which is exactly 18 characters and this is also not an issue.
Can someone please help me in this regard.

org.h2.jdbc.JdbcSQLException: General error: "java.lang.NullPointerException" [50000-178]

I have a small 30 MB h2 database file. Driver version is 1.4.178.
Everything worked fine but recently the DB stop to work with
exception:
org.h2.jdbc.JdbcSQLException: General error: "java.lang.NullPointerException" [50000-178]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:344)
at org.h2.message.DbException.get(DbException.java:167)
at org.h2.message.DbException.convert(DbException.java:294)
at org.h2.engine.Database.openDatabase(Database.java:293)
at org.h2.engine.Database.<init>(Database.java:256)
at org.h2.engine.Engine.openSession(Engine.java:57)
at org.h2.engine.Engine.openSession(Engine.java:164)
at org.h2.engine.Engine.createSessionAndValidate(Engine.java:142)
at org.h2.engine.Engine.createSession(Engine.java:125)
at org.h2.server.TcpServerThread.run(TcpServerThread.java:150)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.NullPointerException
at org.h2.mvstore.DataUtils.parseMap(DataUtils.java:630)
at org.h2.mvstore.MVStore.openMap(MVStore.java:411)
at org.h2.mvstore.db.TransactionStore.<init>(TransactionStore.java:96)
at org.h2.mvstore.db.MVTableEngine$Store.<init>(MVTableEngine.java:161)
at org.h2.mvstore.db.MVTableEngine.init(MVTableEngine.java:94)
at org.h2.engine.Database.getPageStore(Database.java:2355)
at org.h2.engine.Database.open(Database.java:659)
at org.h2.engine.Database.openDatabase(Database.java:262)
... 7 more
at org.h2.engine.SessionRemote.done(SessionRemote.java:610)
at org.h2.engine.SessionRemote.initTransfer(SessionRemote.java:129)
at org.h2.engine.SessionRemote.connectServer(SessionRemote.java:434)
at org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:315)
at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:107)
at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:91)
at org.h2.Driver.connect(Driver.java:74)
at org.h2.server.web.WebServer.getConnection(WebServer.java:684)
at org.h2.server.web.WebApp.test(WebApp.java:896)
at org.h2.server.web.WebApp.process(WebApp.java:222)
at org.h2.server.web.WebApp.processRequest(WebApp.java:171)
at org.h2.server.web.WebThread.process(WebThread.java:138)
at org.h2.server.web.WebThread.run(WebThread.java:94)
at java.lang.Thread.run(Thread.java:744)
The problem occurs in my application and using H2 web frontend.

It's unclear if you're trying to fix the cause of this NPE or if you're trying to retrieve the content of the database.
I suggest the following:
retrieve your data using procedure described at http://h2database.com/html/tutorial.html#upgrade_backup_restore
upgrade your stack to move to the last version of 1.3 (176) or the latest 1.4 (188).
restore the data with this new stack
run and test your server
If the NPE problem still occurs, you'll need to provide more details in this forum.

Cassandra NoHostAvailableException: All host(s) tried for query failed in Production

We have 10 Cassandra nodes in production running Cassandra-2.1.8. We recently upgraded to 2.1.8 version. Previously we were using only 3 nodes running Cassandra-2.1.2. First we upgraded the initial 3 nodes from 2.1.2 to 2.1.8 (following the procedure as described in Upgrading Cassandra). Then we added 7 more nodes running Cassandra-2.1.8 in cluster. Then we started our client programs. For first few hours everything worked fine, but after few hours, we saw some errors in client program logs like
Thread-0 [29/07/15 17:41:23.356] ERROR com.cleartrail.entityprofiling.engine.InterpretationWriter - Error:com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: [/172.50.33.161:9041, /172.50.33.162:9041, /172.50.33.95:9041, /172.50.33.96:9041, /172.50.33.165:9041, /172.50.33.166:9041, /172.50.33.163:9041, /172.50.33.164:9041, /172.50.33.42:9041, /172.50.33.167:9041] - use getErrors() for details)
at com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:65)
at com.datastax.driver.core.DefaultResultSetFuture.extractCauseFromExecutionException(DefaultResultSetFuture.java:259)
at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:175)
at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:52)
at com.cleartrail.entityprofiling.engine.InterpretationWriter.WriteInterpretation(InterpretationWriter.java:430)
at com.cleartrail.entityprofiling.engine.Profiler.buildProfile(Profiler.java:1042)
at com.cleartrail.messageconsumer.consumer.KafkaConsumer.run(KafkaConsumer.java:336)
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: [/172.50.33.161:9041, /172.50.33.162:9041, /172.50.33.95:9041, /172.50.33.96:9041, /172.50.33.165:9041, /172.50.33.166:9041, /172.50.33.163:9041, /172.50.33.164:9041, /172.50.33.42:9041, /172.50.33.167:9041] - use getErrors() for details)
at com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:102)
at com.datastax.driver.core.RequestHandler$1.run(RequestHandler.java:176)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Now, I double checked the Firewall (as suggested in few posts), ports, timeouts in client as well as nodes and they all are correct.
I am also not closing the connection anywhere in between. I am using batch queries with batch size of 1000 and the queries are update queries updating counters in my table with three columns
entity , twfwv , cvalue
where entity and twfwv columns are text and primary key and cvalue is counter column.
I even restarted all my nodes (because this trick helped me in my dev environment when I faced the same exception) but its not helping. Please suggest what can be the probable problem here.

My issue was resolved by checking the errors collection of NoHostAvailableException as advised by Olivier Michallat in the comments. For me it was the protocol version on the cluster configuration. Mine was null, setting it to 3 fixed the problem.

My issue was resolved by removing/using a property to set or unset the custom load balancing TokenAwarePolicy my connection was using, and relying on the default.
Specifically, I was trying to get a local spring boot app talking to a single dockerized Cassandra instance.
Cluster.Builder builder = Cluster.builder()
.addContactPoints(cassandraProperties.getHosts())
.withPort(cassandraProperties.getPort())
.withProtocolVersion(ProtocolVersion.V4)
.withRetryPolicy(new LoggingRetryPolicy(DefaultRetryPolicy.INSTANCE))
.withCredentials(cassandraProperties.getUsername(), cassandraProperties.getPassword())
.withCodecRegistry(codecRegistry);
if (loadBalanced) {
builder.withLoadBalancingPolicy(
new TokenAwarePolicy(DCAwareRoundRobinPolicy.builder().withLocalDc(localDc).build()));
}