Develop Reference

Getting java.sql.SQLException: Routine (cdc_endcapture) can not be resolved while initializing IfxCDCEngine builder on Informix Change Stream Client

Tried out this code for Informix Change Streams API for Java . from https://informix.hcldoc.com/14.10/help/index.jsp?topic=%2Fcom.ibm.cdc.doc%2Fids_cdc_streamapi.htm
I Keep on getting this error .Is there any fix for this or solution??
16:03:57.054 [main] DEBUG com.informix.stream.cdc.IfxCDCEngine - Closing down CDC engine
Exception in thread "main" java.sql.SQLException: Routine (cdc_opensess) can not be resolved.
at com.informix.util.IfxErrMsg.buildExceptionWithMessage(IfxErrMsg.java:422)
at com.informix.util.IfxErrMsg.buildIsamException(IfxErrMsg.java:401)
at com.informix.jdbc.IfxSqli.addException(IfxSqli.java:3022)
at com.informix.jdbc.IfxSqli.receiveError(IfxSqli.java:3273)
at com.informix.jdbc.IfxSqli.dispatchMsg(IfxSqli.java:2269)
at com.informix.jdbc.IfxSqli.receiveMessage(IfxSqli.java:2194)
at com.informix.jdbc.IfxSqli.executePrepare(IfxSqli.java:1194)
at com.informix.jdbc.IfxPreparedStatement.setupExecutePrepare(IfxPreparedStatement.java:245)
at com.informix.jdbc.IfxCallableStatement.<init>(IfxCallableStatement.java:143)
at com.informix.jdbc.IfxSqliConnect.prepareCall(IfxSqliConnect.java:5924)
at com.informix.jdbc.IfxSqliConnect.prepareCall(IfxSqliConnect.java:2499)
at com.informix.stream.cdc.IfxCDCEngine.init(IfxCDCEngine.java:177)
at com.example.informixchangestream.demo.CDCExDemoApplicationample.main(CDCExDemoApplicationample.java:33)
Suppressed: com.informix.stream.impl.IfxStreamException: Unable to end cdc capture
at com.informix.stream.cdc.IfxCDCEngine.endCapture(IfxCDCEngine.java:295)
at com.informix.stream.cdc.IfxCDCEngine.unwatchTable(IfxCDCEngine.java:275)
at com.informix.stream.cdc.IfxCDCEngine.close(IfxCDCEngine.java:343)
at com.example.informixchangestream.demo.CDCExDemoApplicationample.main(CDCExDemoApplicationample.java:52)
Caused by: java.sql.SQLException: Routine (cdc_endcapture) can not be resolved.
at com.informix.util.IfxErrMsg.buildExceptionWithMessage(IfxErrMsg.java:422)
at com.informix.util.IfxErrMsg.buildIsamException(IfxErrMsg.java:401)
at com.informix.jdbc.IfxSqli.addException(IfxSqli.java:3022)
at com.informix.jdbc.IfxSqli.receiveError(IfxSqli.java:3273)
at com.informix.jdbc.IfxSqli.dispatchMsg(IfxSqli.java:2269)
at com.informix.jdbc.IfxSqli.receiveMessage(IfxSqli.java:2194)
at com.informix.jdbc.IfxSqli.executePrepare(IfxSqli.java:1194)
at com.informix.jdbc.IfxPreparedStatement.setupExecutePrepare(IfxPreparedStatement.java:245)
at com.informix.jdbc.IfxCallableStatement.<init>(IfxCallableStatement.java:143)
at com.informix.jdbc.IfxSqliConnect.prepareCall(IfxSqliConnect.java:5924)
at com.informix.jdbc.IfxSqliConnect.prepareCall(IfxSqliConnect.java:2499)
at com.informix.stream.cdc.IfxCDCEngine.endCapture(IfxCDCEngine.java:282)
... 3 more
My code looks like this :
try (IfxCDCEngine engine = builder.build()) {
// initialize the engine (creates the connections and begins listening for
// changes)
engine.init();
IfmxStreamRecord record = null;
// This loop is where you can inject logic that compiles
// transactions, look for commits, throw away rollbacks
// The data here is all Java typed, so it can be easily then
// sent to MQTT, other JDBC drivers, streaming engines, or anything
// else you can think of.
while ((record = engine.getRecord()) != null) {
// Print out the basic record information
System.out.println("record changed");
System.out.println(record);
// If it is an insert/update/delete, print the column data
if (record.hasOperationData()) {
System.out.println("operation data");
System.out.println(((IfxCDCOperationRecord) record).getData());
}
}
}

This error is happens when the CDC routines are not found. Make sure that you first run the CDC setup script with the server to create the syscdcv1 database.
Then make sure your DataSource connects to the syscdcv1 database when you try CDC operations. It's not intuitive, but you 'connect' to the syscdcv1 database, then add watchers on databases and tables you are interested in getting the changes for.
Documentation on the CDC setup on the server is here, you need step #2 at least.
https://informix.hcldoc.com/14.10/help/index.jsp?topic=%2Fcom.ibm.cdc.doc%2Fids_cdc_streamapi.htm
A working example can be found here with a README and java code that you can use that might be easier to follow than the pages of the official docs
https://github.com/informix/informix-db-examples/tree/master/streaming/cdc

Cannot connect MongoDB Atlas from a newly created Spring Boot app with a connection string [duplicate]

I have deployed this Python app on Heroku and i want it to connect to a MongoDB Atlas cluster. I used my string to connect to the cluster, but for some reason i keep getting raise OperationFailure(msg % errmsg, code, response)
pymongo.errors.OperationFailure: bad auth Authentication failed. I checked twice and both the user and the password are correct. Any idea on why this is happening?
from pymongo import MongoClient
import time
import random
import time
import datetime
client = MongoClient('mongodb+srv://USER:<MYPASSWORD>#test-2liju.mongodb.net/test?retryWrites=true')
db = client.one
mycol = client["tst"]
while True:
test = int(random.randrange(-99999990,90000000,1))
dic = {"num": test}
result = db.tst.insert_one(dic)
print(test)
time.sleep(5)

Stupid error, i had to type MYPASSWORD instead of <MYPASSWORD>, without the <>

Don't use any special char in password, like '+' or '='.
I use OpenSSL to generate a password like u4wY9AOwnOLMY+h9EQ==. Came across bad auth Authentication failed.
After using MongoDB Compass it told me don't use special char, so I remove those and use like 'u4wY9AOwnOLMYh9EQ'.
Then it works.

check the compatibility of the version of the Python driver you choose from the Mongodb Atlas Connections. versions above 3.4 are not supported by mongoengine flask

Couchbase Client Configuration for enhanced durability

I am trying to implement ATPlus scan consistency within my couchbase java application. I have updated my queries to include the consistentWith(mutationState):
RawJsonDocument courseJsonDocument = toRawJsonDocument(course, true);
RawJsonDocument insertedJsonDocument = bucket.insert(courseJsonDocument);
MutationState insertMutationState = MutationState.from(insertedJsonDocument);
.....
N1qlQuery.simple(GET_COURSE_BY_ID_QUERY, N1qlParams.build().consistentWith(mutationState));
I'm trying to achieve read-your-own-write, but when I run the query immediately after inserting the document, nothing is found, so I must be doing something wrong. I think what I am missing is actually enabling enhanced durability on the client configuration.
I see examples of how to do it in .NET, but I can't figure out how to 'enable enhanced durability' in JAVA. Here is my cluster configuration:
Cluster cluster = CouchbaseCluster.create(DefaultCouchbaseEnvironment.builder()
.queryServiceConfig(QueryServiceConfig.create(1, 100))
.mutationTokensEnabled(true)
.observeIntervalDelay(Delay.fixed(100, TimeUnit.MICROSECONDS))
.connectTimeout(timeout)
.build(),
clusterHost);

Hive JDBC connection problems

I am trying to connect to Hive2 server via JDBC with kerberos authentication. After numerous attempts to make it work, I can't get it to work with the Cloudera driver.
If someone can help me to solve the problem, I can greatly appreciate it.
I have this method:
private Connection establishConnection() {
final String driverPropertyClassName = "driver";
final String urlProperty = "url";
Properties hiveProperties = config.getMatchingProperties("hive.jdbc");
String driverClassName = (String) hiveProperties.remove(driverPropertyClassName);
String url = (String) hiveProperties.remove(urlProperty);
Configuration hadoopConfig = new Configuration();
hadoopConfig.set("hadoop.security.authentication", "Kerberos");
String p = config.getProperty("hadoop.core.site.path");
Path path = new Path(p);
hadoopConfig.addResource(path);
UserGroupInformation.setConfiguration(hadoopConfig);
Connection conn = null;
if (driverClassName != null) {
try {
UserGroupInformation.loginUserFromKeytab(config.getProperty("login.user"), config.getProperty("keytab.file"));
Driver driver = (Driver) Class.forName(driverClassName).newInstance();
DriverManager.registerDriver(driver);
conn = DriverManager.getConnection(url, hiveProperties);
} catch (Throwable e) {
LOG.error("Failed to establish Hive connection", e);
}
}
return conn;
}
URL for the server, that I am getting from the properties in the format described in Cloudera documentation
I am getting an exception:
2018-05-05 18:26:49 ERROR HiveReader:147 - Failed to establish Hive connection
java.sql.SQLException: [Cloudera][HiveJDBCDriver](500164) Error initialized or created transport for authentication: Peer indicated failure: Unsupported mechanism type PLAIN.
at com.cloudera.hiveserver2.hivecommon.api.HiveServer2ClientFactory.createTransport(Unknown Source)
at com.cloudera.hiveserver2.hivecommon.api.ZooKeeperEnabledExtendedHS2Factory.createClient(Unknown Source)
...
I thought, that it is missing AuthMech attribute and added AuthMech=1 to the URL. Now I am getting:
java.sql.SQLNonTransientConnectionException: [Cloudera][JDBC](10100) Connection Refused: [Cloudera][JDBC](11640) Required Connection Key(s): KrbHostFQDN, KrbServiceName; [Cloudera][JDBC](11480) Optional Connection Key(s): AsyncExecPollInterval, AutomaticColumnRename, CatalogSchemaSwitch, DecimalColumnScale, DefaultStringColumnLength, DelegationToken, DelegationUID, krbAuthType, KrbRealm, PreparedMetaLimitZero, RowsFetchedPerBlock, SocketTimeOut, ssl, StripCatalogName, transportMode, UseCustomTypeCoercionMap, UseNativeQuery, zk
at com.cloudera.hiveserver2.exceptions.ExceptionConverter.toSQLException(Unknown Source)
at com.cloudera.hiveserver2.jdbc.common.BaseConnectionFactory.checkResponseMap(Unknown Source)
...
But KrbHostFQDN is already specified in the principal property as required in the documentation.
Am I missing something or is this documentation wrong?

Below is the one of the similar kind of problem statement in Impala (just JDBC engine changes others are same) that is resolved by setting "KrbHostFQDN" related properties in JDBC connection string itself.
Try to use the URL below. Hopefully works for u.
String jdbcConnStr = "jdbc:impala://myserver.mycompany.corp:21050/default;SSL=1;AuthMech=1;KrbHostFQDN=myserver.mycompany.corp;KrbRealm=MYCOMPANY.CORP;KrbServiceName=impala"
I suppose that if you are not using SSL=1 but only Kerberos, you just drop that part from the connection string and don't worry about setting up SSL certificates in the java key store, which is yet another hassle.
However in order to get Kerberos to work properly we did the following:
Install MIT Kerberos 4.0.1, which is a kerberos ticket manager. (This is for Windows)
This ticket manager asks you for authentication every time you initiate a connection, creates a ticket and stores it in a kerberos_ticket.dat binary file, whose location can be configured somehow but I do not recall exactly how.
Finally, before launching your JAVA app you have to set an environment variable KRB5CCNAME=C:/path/to/kerberos_ticket.dat. In your java app, you can check that the variable was correctly set by doing System.out.println( "KRB5CCNAME = " + System.getenv( "KRB5CCNAME" ) ). If you are working with eclipse or other IDE you might even have to close the IDE,set up the environment variable and start the IDE again.
NOTE: this last bit is very important, I have observed that if this variable is not properly set up, the connection wont be established...
In Linux, instead MIT Kerberos 4.0.1, there is a program called kinit which does the same thing, although without a graphical interface, which is even more convenient for automation.

I wanted to put it in the comment but it was too long for the comment, therefore I am placing it here:
I tried your suggestion and got another exception:
java.sql.SQLException: [Cloudera]HiveJDBCDriver Error
creating login context using ticket cache: Unable to obtain Principal
Name for authentication .
May be my problem is, that I do not have environment variable KRB5CCNAME set.
I, honestly, never heard about it before.
What is supposed to be in that ticket file.
I do have, however, following line in my main method:
System.setProperty("java.security.krb5.conf", "path/to/krb5.conf");
Which is supposed to be used by
UserGroupInformation.loginUserFromKeytab(config.getProperty("login.user"), config.getProperty("keytab.file"));
to obtain the kerberos ticket.

To solve this issue update Java Cryptography Extension for the Java version that you use in your system.
Here's the link when you can download JCE for Java 1.7
Uncompress and overwrite those files in $JDK_HOME/jre/lib/security
Restart your computer.

Vertx HttpClient getNow not working

I have problem with vertx HttpClient.
Here's code which shows that tests GET using vertx and plain java.
Vertx vertx = Vertx.vertx();
HttpClientOptions options = new HttpClientOptions()
.setTrustAll(true)
.setSsl(false)
.setDefaultPort(80)
.setProtocolVersion(HttpVersion.HTTP_1_1)
.setLogActivity(true);
HttpClient client = vertx.createHttpClient(options);
client.getNow("google.com", "/", response -> {
System.out.println("Received response with status code " + response.statusCode());
});
System.out.println(getHTML("http://google.com"));
Where getHTML() is from here: How do I do a HTTP GET in Java?
This is my output:
<!doctype html><html... etc <- correct output from plain java
Feb 08, 2017 11:31:21 AM io.vertx.core.http.impl.HttpClientRequestImpl
SEVERE: java.net.UnknownHostException: failed to resolve 'google.com'. Exceeded max queries per resolve 3
But vertx can't connect. What's wrong here? I'm not using any proxy.

For reference: a solution, as described in this question and in tsegismont's comment here, is to set the flag vertx.disableDnsResolver to true:
-Dvertx.disableDnsResolver=true
in order to fall back to the JVM DNS resolver as explained here:
sometimes it can be desirable to use the JVM built-in resolver, the JVM system property -Dvertx.disableDnsResolver=true activates this behavior
I observed this DNS resolution issue with a redis client in a kubernetes environment.

I had this issue, what caused it for me was stale DNS servers being picked up by the Java runtime, i.e. servers registered for a network the machine was no longer connected to. The issue is first in the Sun JNDI implementation, it also exists in Netty which uses JNDI to bootstrap its list of name servers on most platforms, then finally shows up in VertX.
I think a good place to fix this would be in the Netty layer where the set of default DNS servers is bootstrapped. I have raised a ticket with the Netty project so we'll see if they agree with me! Here is the Netty ticket
In the mean time a fairly basic workaround is to filter the default DNS servers detected by Netty, based on whether they are reachable or not. Here is a code Sample in Kotlin to apply before constructing the main VertX instance.
// The default set of name servers provided by JNDI can contain stale entries
// This default set is picked up by Netty and in turn by VertX
// To work around this, we filter for only reachable name servers on startup
val nameServers = DefaultDnsServerAddressStreamProvider.defaultAddressList()
val reachableNameServers = nameServers.stream()
.filter {ns -> ns.address.isReachable(NS_REACHABLE_TIMEOUT)}
.map {ns -> ns.address.hostAddress}
.collect(Collectors.toList())
if (reachableNameServers.size == 0)
throw StartupException("There are no reachable name servers available")
val opts = VertxOptions()
opts.addressResolverOptions.servers = reachableNameServers
// The primary Vertx instance
val vertx = Vertx.vertx(opts)
A little more detail in case it is helpful. I have a company machine, which at some point was connected to the company network by a physical cable. Details of the company's internal name servers were set up by DHCP on the physical interface. Using the wireless interface at home, DNS for the wireless interface gets set to my home DNS while the config for the physical interface is not updated. This is fine since that device is not active, ipconfig /all does not show the internal company DNS servers. However, looking in the registry they are still there:
Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\Interfaces
They get picked up by the JNDI mechanism, which feeds Netty and in turn VertX. Since they are not reachable from my home location, DNS resolution fails. I can imagine this home/office situation is not unique to me! I don't know whether something similar could occur with multiple virtual interfaces on containers or VMs, it could be worth looking at if you are having problems.

Here is the sample code which works for me.
public class TemplVerticle extends HttpVerticle {
public static void main(String[] args) {
Vertx vertx = Vertx.vertx();
// Create the web client and enable SSL/TLS with a trust store
WebClient client = WebClient.create(vertx,
new WebClientOptions()
.setSsl(true)
.setTrustAll(true)
.setDefaultPort(443)
.setKeepAlive(true)
.setDefaultHost("www.w3schools.com")
);
client.get("www.w3schools.com")
.as(BodyCodec.string())
.send(ar -> {
if (ar.succeeded()) {
HttpResponse<String> response = ar.result();
System.out.println("Got HTTP response body");
System.out.println(response.body().toString());
} else {
ar.cause().printStackTrace();
}
});
}
}

Try using web client instead of httpclient, here you have an example (with rx):
private val client: WebClient = WebClient.create(vertx, WebClientOptions()
.setSsl(true)
.setTrustAll(true)
.setDefaultPort(443)
.setKeepAlive(true)
)
open fun <T> get(uri: String, marshaller: Class<T>): Single<T> {
return client.getAbs(host + uri).rxSend()
.map { extractJson(it, uri, marshaller) }
}
Another option is to use getAbs.