Develop Reference

Google Cloud Search Database Connector Issues

I am working with Google Cloud search (https://developers.google.com/cloud-search/docs/guides/?_ga=2.124920714.-122300216.1578247736) and I am attempting to index a Cloud SQL instance. Presently I am using the guide as shown here (https://developers.google.com/cloud-search/docs/guides/database-connector#important-considerations). I have registered source in G-Suite. I have a Cloud Search Service Account, I tested that I connect to the Cloud SQL instance from my Compute Engine instance which I can.
My config file is as follows with the necessary information replaced with XXXX:
#
# data source access
api.sourceId=xxxxxxxxxxx
api.identitySourceId=xxxxxxxxxxxxxxxx
api.serviceAccountPrivateKeyFile=./private-key.json
#
# database access
db.url=jdbc:mysql:///<database>?cloudSqlInstance=<cloud_sql_instance>&socketFactory=mysql-socket-factory-connector-j-8&useSSL=false&user=xxxxxxxxx&password=xxxxxxxxx
#
# traversal SQL statements
db.allRecordsSql=select field_1, field_2, field_3 from table;
#
# schedule traversals
schedule.traversalIntervalSecs=36000
schedule.performTraversalOnStart=true
schedule.incrementalTraversalIntervalSecs=3600
#
# column definitions
db.allColumns= field1, field2, field3
db.uniqueKeyColumns=field1
url.columns=field1
#
# content fields
contentTemplate.db.title=field1
db.contentColumns=field1, field2, field3
#
# setting ACLs to "entire domain accessible"
defaultAcl.mode=fallback
defaultAcl.public=true
with the jdbc connection string being based off https://github.com/GoogleCloudPlatform/cloud-sql-jdbc-socket-factory. I'm at the stage where I run:
java \
-cp "google-cloudsearch-database-connector-v1-0.0.3.jar:mysql-connector-java-5.1.41-bin.jar" \
com.google.enterprise.cloudsearch.database.DatabaseFullTraversalConnector \
[-Dconfig=mysql.config]
but i get the error Failed to initialize connector. The full stacktrace is:
Jan 05, 2020 6:29:38 PM com.google.enterprise.cloudsearch.sdk.indexing.IndexingApplication startUp
SEVERE: Failed to initialize connector
com.google.enterprise.cloudsearch.sdk.StartupException: Failed to initialize connector
at com.google.enterprise.cloudsearch.sdk.Application.startConnector(Application.java:150)
at com.google.enterprise.cloudsearch.sdk.indexing.IndexingApplication.startUp(IndexingApplication.java:96)
at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62)
at com.google.common.util.concurrent.Callables$4.run(Callables.java:122)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:871)
at com.google.common.io.BaseEncoding$StandardBaseEncoding.trimTrailingPadding(BaseEncoding.java:672)
at com.google.common.io.BaseEncoding.decodeChecked(BaseEncoding.java:226)
at com.google.common.io.BaseEncoding.decode(BaseEncoding.java:212)
at com.google.api.client.util.Base64.decodeBase64(Base64.java:93)
at com.google.api.services.cloudsearch.v1.model.Item.decodeVersion(Item.java:329)
at com.google.enterprise.cloudsearch.sdk.indexing.IndexingServiceImpl.indexItem(IndexingServiceImpl.java:678)
at com.google.enterprise.cloudsearch.sdk.indexing.DefaultAcl.<init>(DefaultAcl.java:203)
at com.google.enterprise.cloudsearch.sdk.indexing.DefaultAcl.<init>(DefaultAcl.java:93)
at com.google.enterprise.cloudsearch.sdk.indexing.DefaultAcl$Builder.build(DefaultAcl.java:466)
at com.google.enterprise.cloudsearch.sdk.indexing.DefaultAcl.fromConfiguration(DefaultAcl.java:266)
at com.google.enterprise.cloudsearch.sdk.indexing.template.FullTraversalConnector.init(FullTraversalConnector.java:182)
at com.google.enterprise.cloudsearch.sdk.indexing.template.FullTraversalConnector.init(FullTraversalConnector.java:97)
at com.google.enterprise.cloudsearch.sdk.Application.startConnector(Application.java:142)
... 4 more
Jan 05, 2020 6:29:38 PM com.google.enterprise.cloudsearch.sdk.BatchRequestService shutDown
INFO: Shutting down batching service. flush on shutdown: true
Exception in thread "main" java.lang.IllegalStateException: Expected the service IndexingApplication [FAILED] to be RUNNING, but the service has FAILED
at com.google.common.util.concurrent.AbstractService.checkCurrentState(AbstractService.java:344)
at com.google.common.util.concurrent.AbstractService.awaitRunning(AbstractService.java:280)
at com.google.common.util.concurrent.AbstractIdleService.awaitRunning(AbstractIdleService.java:175)
at com.google.enterprise.cloudsearch.sdk.Application.start(Application.java:122)
at com.google.enterprise.cloudsearch.database.DatabaseFullTraversalConnector.main(DatabaseFullTraversalConnector.java:30)
Caused by: com.google.enterprise.cloudsearch.sdk.StartupException: Failed to initialize connector
at com.google.enterprise.cloudsearch.sdk.Application.startConnector(Application.java:150)
at com.google.enterprise.cloudsearch.sdk.indexing.IndexingApplication.startUp(IndexingApplication.java:96)
at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62)
at com.google.common.util.concurrent.Callables$4.run(Callables.java:122)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:871)
at com.google.common.io.BaseEncoding$StandardBaseEncoding.trimTrailingPadding(BaseEncoding.java:672)
at com.google.common.io.BaseEncoding.decodeChecked(BaseEncoding.java:226)
at com.google.common.io.BaseEncoding.decode(BaseEncoding.java:212)
at com.google.api.client.util.Base64.decodeBase64(Base64.java:93)
at com.google.api.services.cloudsearch.v1.model.Item.decodeVersion(Item.java:329)
at com.google.enterprise.cloudsearch.sdk.indexing.IndexingServiceImpl.indexItem(IndexingServiceImpl.java:678)
at com.google.enterprise.cloudsearch.sdk.indexing.DefaultAcl.<init>(DefaultAcl.java:203)
at com.google.enterprise.cloudsearch.sdk.indexing.DefaultAcl.<init>(DefaultAcl.java:93)
at com.google.enterprise.cloudsearch.sdk.indexing.DefaultAcl$Builder.build(DefaultAcl.java:466)
at com.google.enterprise.cloudsearch.sdk.indexing.DefaultAcl.fromConfiguration(DefaultAcl.java:266)
at com.google.enterprise.cloudsearch.sdk.indexing.template.FullTraversalConnector.init(FullTraversalConnector.java:182)
at com.google.enterprise.cloudsearch.sdk.indexing.template.FullTraversalConnector.init(FullTraversalConnector.java:97)

This issue is "Caused by: java.lang.NullPointerException".
You should verify the line referred in the stack trace to find the culprit variable, which should be revealed by expanding the view of the "4 more" error lines in the stack.
Here's information about this exception.
Here's information about how to read a java stack trace

If I had to guess you're not establishing a connection to your database. Your configuration file looks good to me except db.url. It should look something like this from my experience jdbc:mysql://localhost:1433;DatabaseName=ExampleDatabase. The errors you typically get for other properties tell you exactly what the issue is (example field_1 does not exist in table) , but in this case it's not establishing a connection. Try rewriting the db.url.

Failed to initialize connector --> This exception occurs when there is something wrong with your config file. But usually it tells you what's wrong along with the message.
Did you try running Cloud SQL proxy on the Compute Engine VM and connecting through that?

Oracle 11g dbms_java.start_jmx_agent throws an exception

I'm trying to profile a Java stored proc inside an Oracle DB. My user has been granted role JMXSERVER, but when I run call dbms_java.start_jmx_agent('22222', 'false', 'false'); I get:
ORA-29532: Java call terminated by uncaught Java exception: java.lang.RuntimeException: java.lang.RuntimeException: Management agent class failed
ORA-06512: at "SYS.DBMS_JAVA", line 803
ORA-06512: at "SYS.DBMS_JAVA", line 812
ORA-06512: at line 1
I've traced the error to this line in JDK: https://github.com/frohoff/jdk8u-dev-jdk/blob/master/src/share/classes/sun/management/Agent.java#L483, and the exception details are printed to System.err, but where does Oracle write its stderr to?

You have to redirect the output like this:
select DBMS_JAVA.SET_OUTPUT_TO_SQL('1', 'begin dbms_output.put_line(:1); end;','TEXT') from dual;
Then the error will be printed to the SQL session output. In my case the culprit was a missing file javavm//lib/management/management.properties, so I went to server admins.

You can also redirect the Java System.out and System.err to DBMS_OUTPUT:
dbms_java.set_output(100);
By the way, I tried the DBMS_JAVA.SET_OUTPUT_TO_SQL function as provided by Alexey but it failed with "Parameter 2 has invalid SQL" error. Did it work for you?

"heros.solver.CountingThreadPoolExecutor - Worker thread execution failed: null Exceptions" when running nightly build soot-trunk

I was converting apk file to jimple files and then converting they back to the .dex file immediately. But I cannot do the second the step successfully.
Command line used:
java -Xmx4g -jar soot-trunk.jar soot.Main -p cg.spark verbose:true,on-fly-cg:true -w -allow-phantom-refs -force-android-jar /home/xia/Downloads/android-platforms-master/android-17 -src-prec apk -f jimple -process-dir com.halfbrick.fruitninjafree.apk
In the beginning, it keeps on throwing this exception:
java.lang.RuntimeException: Error parsing class com.google.android.gms.plus.model.moments.MomentBuffer [22,40] expecting: quoted name, identifier
at soot.JimpleClassSource.resolve(JimpleClassSource.java:61)
at soot.SootResolver.bringToHierarchy(SootResolver.java:239)
at soot.SootResolver.bringToSignatures(SootResolver.java:266)
at soot.SootResolver.bringToBodies(SootResolver.java:304)
at soot.SootResolver.processResolveWorklist(SootResolver.java:163)
at soot.SootResolver.resolveClass(SootResolver.java:131)
at soot.Scene.loadClass(Scene.java:725)
at soot.Scene.loadClassAndSupport(Scene.java:710)
at soot.Scene.loadNecessaryClasses(Scene.java:1448)
at soot.Main.run(Main.java:243)
at soot.Main.main(Main.java:147)
Caused by: soot.jimple.parser.parser.ParserException: [22,40] expecting: quoted name, identifier
at soot.jimple.parser.parser.Parser.parse(Parser.java:1454)
at soot.jimple.parser.JimpleAST.(JimpleAST.java:57)
at soot.JimpleClassSource.resolve(JimpleClassSource.java:42)
... 10 more
I found that there are some syntax mistakes (might be) in the converted jimple file. For example, in the jimple files, there are some classes named like this:
com.google.android.gms.internal.'if'
java.lang.'annotation'.Annotation
Then I fixed the mistakes manually (by remove single quote and replace 'if' with other variable name like iff).
After I fixed the above mistakes, it shows another exception:
Warning: android.content.OperationApplicationException is a phantom class!
Warning: android.database.MatrixCursor$RowBuilder is a phantom class!
Warning: android.content.ContentProviderResult is a phantom class!
[Call Graph] For information on where the call graph may be incomplete, use the verbose option to the cg phase.
[Thread-1] ERROR heros.solver.CountingThreadPoolExecutor - Worker thread execution failed: null
[Thread-8] ERROR heros.solver.CountingThreadPoolExecutor - Worker thread execution failed: null
[Thread-7] ERROR heros.solver.CountingThreadPoolExecutor - Worker thread execution failed: null
java.lang.NullPointerException
at soot.toolkits.graph.UnitGraph.(UnitGraph.java:76)
at soot.toolkits.graph.ExceptionalUnitGraph.(ExceptionalUnitGraph.java:158)
at soot.jimple.toolkits.scalar.UnreachableCodeEliminator.internalTransform(UnreachableCodeEliminator.java:79)
at soot.BodyTransformer.transform(BodyTransformer.java:51)
at soot.Transform.apply(Transform.java:105)
at soot.JimpleBodyPack.applyPhaseOptions(JimpleBodyPack.java:61)
at soot.JimpleBodyPack.internalApply(JimpleBodyPack.java:95)
at soot.Pack.apply(Pack.java:125)
at soot.jimple.JimpleMethodSource.getBody(JimpleMethodSource.java:49)
at soot.SootMethod.getBodyFromMethodSource(SootMethod.java:91)
at soot.SootMethod.retrieveActiveBody(SootMethod.java:322)
at soot.PackManager$3.run(PackManager.java:1223)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[Thread-6] ERROR heros.solver.CountingThreadPoolExecutor - Worker thread execution failed: null
java.lang.NullPointerException
at soot.toolkits.graph.UnitGraph.(UnitGraph.java:76)
at soot.toolkits.graph.ExceptionalUnitGraph.(ExceptionalUnitGraph.java:158)
at soot.jimple.toolkits.scalar.UnreachableCodeEliminator.internalTransform(UnreachableCodeEliminator.java:79)
at soot.BodyTransformer.transform(BodyTransformer.java:51)
at soot.Transform.apply(Transform.java:105)
at soot.JimpleBodyPack.applyPhaseOptions(JimpleBodyPack.java:61)
...
At first I think it's the bug of the tool, but the answer of other question says that the bug has been fixed. But I still get this problem. Does any one know where the problem is? Thank you very much.

Value annotation can't find a value if it's a number

Application.yaml:
request.delay.ms: 200
A spring component:
#Value("${request.delay.ms}")
private long requestDelay;
Now, as strange as it sounds, it looks like spring can successfully resolve the property if it's value is a string, i.e. if I change 200 to 200a. It does throw an error
Caused by: java.lang.NumberFormatException: For input string: "200a" but that's expected.
If I change the value back to 200 I get
Caused by: org.springframework.beans.TypeMismatchException: Failed to convert value of type 'java.lang.String' to required type 'long'; nested exception is java.lang.NumberFormatException: For input string: "${request.delay.ms}"
And if I run tomcat in debug mode and step through the spring code, it works fine with no errors.
String #Value(s) are resolved fine in the same component all the time.

If you are using YAML instead of .properties you should write it in the following manner:
request:
delay:
ms: 200
which will be transformed to properties:
request.delay.ms=200 but you are mixing both files
See more here

Getting Exceptions from logs on *nix using sed/awk/whatever

I want to get exceptions from log files created by tomcat.
Yes, I did some research, but because I don't have any experience with sed or awk - adjusting what I've found to what I need is kinda difficult.
A sample file is shown below to have something to work on:
at org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:86)
Caused by: org.apache.cxf.transport.http.HTTPException: HTTP response '503: Service Temporarily Unavailable' when communicating with http://66.66.66.66:1234/aaa/bbb/ccc
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:1546)
at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62)
... 39 more
2014-10-24 11:40:01,558 ERROR [aaa.bbb.ccc.ddd.SomeClass] - some exception on parsing '2007/11/45' bla bla
javax.xml.ws.WebServiceException: Could not send Message.
at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:145)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:525)
Caused by: org.apache.cxf.transport.http.HTTPException: HTTP response '503: Service Temporarily Unavailable' when communicating with http://66.66.66.66:1234/aaa/bbb/ccc
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:1546)
at org.apache.cxf.frontend.ClientProxy.invokeSync(ClientProxy.java:88)
at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:134)
... 32 more
2014-10-24 11:40:01,561 ERROR [aaa.bbb.ccc.ddd] - some error with id = 1214
java.lang.NullPointerException
at sun.reflect.GeneratedMethodAccessor181.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:525)
2014-10-24 11:44:48,253 INFO [org.springframework.orm.hibernate3.annotation.AnnotationSessionFactoryBean] - Closing Hibernate SessionFactory
2014-10-24 11:44:48,253 INFO [org.hibernate.impl.SessionFactoryImpl] - closing
2014-10-24 11:44:48 org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
2014-10-24 11:44:50 org.apache.coyote.http11.Http11Protocol destroy
INFO: Stopping Coyote HTTP/1.1 on http-8096
As we can see there are: 2 FULL EXCEPTIONS (we want them), 1 PARTIAL EXCEPTION(we dont want it). To make this example short I deleted some important stuff, like log4j:ERRORs, which we dont want.
so far I tried:
AWK (its my 1st day with AWK, please dont laugh :D). its pretty straight forward.
it finds "/t" (tab) + at + " " (empty space) in the beginning of every line. if 2 lines before it match given conditions (Exception and date) it prints them too. it works pretty well, but it also prints partial exception, which we DO NOT want.
BEGIN {
preprevious = "";
previous = "";
}
/^\tat / {
if( previous != "" ) {
if(preprevious ~ /20[0-9][0-9]-[0-9][0-9]-[0-9][0-9]/){
print preprevious;
preprevious = "";
}
if(previous ~ /.*Exception/) {
print previous;
previous = "";
}
}
print;
next;
}
{ preprevious = previous;
previous = $0; }
which i run like this:
awk -f awkScript testFileExceptions.txt
and SED in a bash script (I prefer that)
#!/bin/sh
if [ "$#" -eq "2" ]
then
tail -n $2 $1 | sed -n "/ ERROR \[/,/20[0-9][0-9]-[0-9][0-9]-[0-9][0-9]/p"
else
echo "usage: scriptName filePath amountOfLastLinesInFile"
fi
it matches '" "+ERROR+" "' (with empty spaces on both sides of ERROR, so log4j:ERROR will not be matched) to date. it kind of works...
disadvantages:
1)it will work IF there are some additional lines between exceptions - like this:
at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:134)
... 32 more
2014-10-24 11:40:01,561 INFO [aaa.bbb.ccc.ddd] - AAAAAAAAAAAAAAAAAAAAAAAA
2014-10-24 11:40:01,561 ERROR [aaa.bbb.ccc.ddd] - some error with id = 1214
java.lang.NullPointerException
at sun.reflect.GeneratedMethodAccessor181.invoke(Unknown Source)
if not - then the 2nd exception will not be shown
2)it will also print out the last matched line (which is the one with the date match)
so to sum up, what I want on outcome is:
2014-10-24 11:40:01,558 ERROR [aaa.bbb.ccc.ddd.SomeClass] - some exception on parsing '2007/11/45' bla bla javax.xml.ws.WebServiceException: Could not send Message.
at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:145)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:525)
Caused by: org.apache.cxf.transport.http.HTTPException: HTTP response '503: Service Temporarily Unavailable' when communicating with http://66.66.66.66:1234/aaa/bbb/ccc
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:1546)
at org.apache.cxf.frontend.ClientProxy.invokeSync(ClientProxy.java:88)
at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:134)
... 32 more
2014-10-24 11:40:01,561 ERROR [aaa.bbb.ccc.ddd] - some error with id = 1214 java.lang.NullPointerException
at sun.reflect.GeneratedMethodAccessor181.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:525)
I also want to save different exceptions to different files (lets say exceptionOutputNNN.txt, for example exceptionOutput001.txt), but I'll figure it out later, shouldn't be that hard after I figured out how to do that for XMLs... :P
well.. that's it. I hope someone can help me:)
cheers
edit: please note, that Exceptions can end with "... NN more" and with simple "\tat org.*"

Still not sure exactly what you want
This should work for what output you want though
awk '/^[0-9]+/{x=0}/ERROR/{x=1}x' file
output
2014-10-24 11:40:01,558 ERROR [aaa.bbb.ccc.ddd.SomeClass] - some exception on parsing '2007/11/45' bla bla
javax.xml.ws.WebServiceException: Could not send Message.
at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:145)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:525)
Caused by: org.apache.cxf.transport.http.HTTPException: HTTP response '503: Service Temporarily Unavailable' when communicating with http://66.66.66.66:1234/aaa/bbb/ccc
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:1546)
at org.apache.cxf.frontend.ClientProxy.invokeSync(ClientProxy.java:88)
at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:134)
... 32 more
2014-10-24 11:40:01,561 ERROR [aaa.bbb.ccc.ddd] - some error with id = 1214
java.lang.NullPointerException
at sun.reflect.GeneratedMethodAccessor181.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:525)
Edit: For your original file
awk 'a=/^[0-9]+/{x=0}a&&/ERROR/{x=1}x' file
or
awk '(/^[0-9]/&&x=/ERROR/)||x' file

for any people viewing this post in the future, below is the script which uses Jidders solution and saves output to files (1 file perexception)
#!/bin/sh
if [ "$#" -eq "2" ]
then
rm output/errorOutputFile* 2>/dev/null
tail -n $2 $1 | awk '(/^[0-9]/&&x=/ERROR/)||x' | awk '/^[0-9]/{g++} { print $0 > "errorOutputFile"g".txt"}'
else
echo "usage: scriptName filePath amountOfLastLinesInFile"
fi
cheers