Develop Reference

Why I'm not able to connect to HBase running as Docker container?

I have my Java Spring app that deals with HBase.
Here is my configuration:
#Configuration
public class HbaseConfiguration {
#Bean
public HbaseTemplate hbaseTemplate(#Value("${hadoop.home.dir}") final String hadoopHome,
#Value("${hbase.zookeeper.quorum}") final String quorum,
#Value("${hbase.zookeeper.property.clientPort}") final String port)
throws IOException, ServiceException {
System.setProperty("hadoop.home.dir", hadoopHome);
org.apache.hadoop.conf.Configuration configuration = HBaseConfiguration.create();
configuration.set("hbase.zookeeper.quorum", quorum);
configuration.set("hbase.zookeeper.property.clientPort", port);
HBaseAdmin.checkHBaseAvailable(configuration);
return new HbaseTemplate(configuration);
}
}
#HBASE
hbase.zookeeper.quorum = localhost
hbase.zookeeper.property.clientPort = 2181
hadoop.home.dir = C:/hadoop
Before asking the question I tried to figure out the problem on my own and found this link https://github.com/sel-fish/hbase.docker
But still, I get an error
org.apache.hadoop.net.ConnectTimeoutException: 10000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=myhbase/192.168.99.100:60000]
Could I ask you to help me and clarify how can I connect my local Java app with HBase running in Docker?

Accessing AWS RDS using IAM Authentication and Spring JDBC (DataSource and JdbcTemplace)

I am not able to figure out how to implement this. Any help and/or pointers will be greatly appreciated.
Currently, my Java/Spring application backend is deployed on EC2 and accessing MySQL on RDS successfully using the regular Spring JDBC setup. That is, storing database info in application.properties and configuring DataSource and JdbcTemplate in #Configuration class. Everything works fine.
Now, I need to access MySQL on RDS securely. RDS instance has IAM Authentication enabled. I have also successfully created IAM role and applied inline policy. Then, following the AWS RDS documentation and Java example on this link, I am able to access the database from a standalone Java class successfully using Authentication Token and the user I created instead of regular db username and password. This standalone Java class is dealing with "Connection" object directly.
The place I am stuck is how I translate this to Spring JDBC configuration. That is, setting up DataSource and JdbcTemplate beans for this in my #Configuration class.
What would be a correct/right approach to implement this?
----- EDIT - Start -----
I am trying to implement this as a library that can be used for multiple projects. That is, it will be used as a JAR and declared as a dependency in a project's POM file. This library is going to include configurable AWS Services like this RDS access using general DB username and password, RDS access using IAM Authentication, KMS (CMK/data keys) for data encryption, etc.
Idea is to use this library on any web/app server depending on the project.
Hope this clarifies my need more.
----- EDIT - End -----
DataSource internally has getConnection() so I can basically create my own DataSource implementation to achieve what I want. But is this a good approach?
Something like:
public class MyDataSource implements DataSource {
#Override
public Connection getConnection() throws SQLException {
Connection conn = null;
// get a connection using IAM Authentication Token for accessing AWS RDS, etc. as in the AWS docs
return conn;
}
#Override
public Connection getConnection(String username, String password) throws SQLException {
return getConnection();
}
//other methods
}

You can use the following snippet as a replacement for the default connection-pool provided by SpringBoot/Tomcat. It will refresh the token password every 10 minutes, since the token is valid for 15 minutes. Also, it assumes the region can be extracted from the DNS hostname. If this is not the case, you'll need to specify the region to use.
public class RdsIamAuthDataSource extends org.apache.tomcat.jdbc.pool.DataSource {
private static final Logger LOG = LoggerFactory.getLogger(RdsIamAuthDataSource.class);
/**
* The Java KeyStore (JKS) file that contains the Amazon root CAs
*/
public static final String RDS_CACERTS = "/rds-cacerts";
/**
* Password for the ca-certs file.
*/
public static final String PASSWORD = "changeit";
public static final int DEFAULT_PORT = 3306;
#Override
public ConnectionPool createPool() throws SQLException {
return pool != null ? pool : createPoolImpl();
}
protected synchronized ConnectionPool createPoolImpl() throws SQLException {
return pool = new RdsIamAuthConnectionPool(poolProperties);
}
public static class RdsIamAuthConnectionPool extends ConnectionPool implements Runnable {
private RdsIamAuthTokenGenerator rdsIamAuthTokenGenerator;
private String host;
private String region;
private int port;
private String username;
private Thread tokenThread;
public RdsIamAuthConnectionPool(PoolConfiguration prop) throws SQLException {
super(prop);
}
#Override
protected void init(PoolConfiguration prop) throws SQLException {
try {
URI uri = new URI(prop.getUrl().substring(5));
this.host = uri.getHost();
this.port = uri.getPort();
if (this.port < 0) {
this.port = DEFAULT_PORT;
}
this.region = StringUtils.split(this.host,'.')[2]; // extract region from rds hostname
this.username = prop.getUsername();
this.rdsIamAuthTokenGenerator = RdsIamAuthTokenGenerator.builder().credentials(new DefaultAWSCredentialsProviderChain()).region(this.region).build();
updatePassword(prop);
final Properties props = prop.getDbProperties();
props.setProperty("useSSL","true");
props.setProperty("requireSSL","true");
props.setProperty("trustCertificateKeyStoreUrl",getClass().getResource(RDS_CACERTS).toString());
props.setProperty("trustCertificateKeyStorePassword", PASSWORD);
super.init(prop);
this.tokenThread = new Thread(this, "RdsIamAuthDataSourceTokenThread");
this.tokenThread.setDaemon(true);
this.tokenThread.start();
} catch (URISyntaxException e) {
throw new RuntimeException(e.getMessage());
}
}
#Override
public void run() {
try {
while (this.tokenThread != null) {
Thread.sleep(10 * 60 * 1000); // wait for 10 minutes, then recreate the token
updatePassword(getPoolProperties());
}
} catch (InterruptedException e) {
LOG.debug("Background token thread interrupted");
}
}
#Override
protected void close(boolean force) {
super.close(force);
Thread t = tokenThread;
tokenThread = null;
if (t != null) {
t.interrupt();
}
}
private void updatePassword(PoolConfiguration props) {
String token = rdsIamAuthTokenGenerator.getAuthToken(GetIamAuthTokenRequest.builder().hostname(host).port(port).userName(this.username).build());
LOG.debug("Updated IAM token for connection pool");
props.setPassword(token);
}
}
}
Please note that you'll need to import Amazon's root/intermediate certificates to establish a trusted connection. The example code above assumes that the certificates have been imported into a file called 'rds-cacert' and is available on the classpath. Alternatively, you can also import them into the JVM 'cacerts' file.
To use this data-source, you can use the following properties for Spring:
datasource:
url: jdbc:mysql://dbhost.xyz123abc.us-east-1.rds.amazonaws.com/dbname
username: iam_app_user
driver-class-name: com.mysql.cj.jdbc.Driver
type: com.mydomain.jdbc.RdsIamAuthDataSource
Using Spring Java config:
#Bean public DataSource dataSource() {
PoolConfiguration props = new PoolProperties();
props.setUrl("jdbc:mysql://dbname.abc123xyz.us-east-1.rds.amazonaws.com/dbschema");
props.setUsername("iam_dbuser_app");
props.setDriverClassName("com.mysql.jdbc.Driver");
return new RdsIamAuthDataSource(props);
}
UPDATE: When using MySQL, you can also decide to use the MariaDB JDBC driver, which has builtin support for IAM authentication:
spring:
datasource:
host: dbhost.cluster-xxx.eu-west-1.rds.amazonaws.com
url: jdbc:mariadb:aurora//${spring.datasource.host}/db?user=xxx&credentialType=AWS-IAM&useSsl&serverSslCert=classpath:rds-combined-ca-bundle.pem
type: org.mariadb.jdbc.MariaDbPoolDataSource
The above requires MariaDB and AWS SDK libraries, and needs the CA-bundle in the classpath

I know this is an older question, but after a some searching I found a pretty easy way you can now do this using the MariaDB driver. In version 2.5 they added an AWS IAM credential plugin to the driver. It will handle generating, caching and refreshing the token automatically.
I've tested using Spring Boot 2.3 with the default HikariCP connection pool and it is working fine for me with these settings:
spring.datasource.url=jdbc:mariadb://host/db?credentialType=AWS-IAM&useSsl&serverSslCert=classpath:rds-combined-ca-bundle.pem
spring.datasource.driver-class-name=org.mariadb.jdbc.Driver
spring.datasource.username=iam_username
#spring.datasource.password=dont-need-this
spring.datasource.hikari.maxLifetime=600000
Download rds-combined-ca-bundle.pem and put it in src/main/resources so you can connect via SSL.
You will need these dependencies on the classpath as well:
runtime 'org.mariadb.jdbc:mariadb-java-client'
runtime 'com.amazonaws:aws-java-sdk-rds:1.11.880'
The driver uses the standard DefaultAWSCredentialsProviderChain so make sure you have credentials with policy allowing IAM DB access available wherever you are running your app.
Hope this helps someone else - most examples I found online involved custom code, background threads, etc - but using the new driver feature is much easier!

There is a library that can make this easy. Effectively you just override the getPassword() method in the HikariDataSource. You use STS to assume the role and send a "password" for that role.
<dependency>
<groupId>io.volcanolabs</groupId>
<artifactId>rds-iam-hikari-datasource</artifactId>
<version>1.0.4</version>
</dependency>

Creating mongodb database user in Java using spring data mongodb

I need to create a mongodb database user in a Spring boot application using spring data mongodb. I will be creating this user as part of application startup.
I could not find any reference for doing this using spring data mongodb.
Is that possible by using Spring data mongodb?

I had the same issue in the past and I end up by creating the user before the context load, like this:
#Configuration
#EnableAutoConfiguration
#ComponentScan
public class Application extends SpringBootServletInitializer {
#SuppressWarnings("resource")
public static void main(final String[] args) {
createMongoDbUser();
ConfigurableApplicationContext context = SpringApplication.run(Application.class, args);
}
private void createMongoDbUser() {
MongoClient mongo = new MongoClient(HOST, PORT);
MongoDatabase db = mongo.getDatabase(DB);
Map<String, Object> commandArguments = new BasicDBObject();
commandArguments.put("createUser", USER_NAME);
commandArguments.put("pwd", USER_PWD);
String[] roles = { "readWrite" };
commandArguments.put("roles", roles);
BasicDBObject command = new BasicDBObject(commandArguments);
db.runCommand(command);
}
}

Spring-data-mongodb will create the db all by itself if it can't find it, when declaring your mongo-db factory.
For instance, I declare my db-factory in xml using the following:
<mongo:db-factory id="mongofactory" dbname="dbNameHere" mongo-ref="mongo" />
I did not have to create it myself, it was created by spring-data-mongodb upon firing may app the first time.

Connecting to Mongo database through SSH tunnel in Java

FIXED (edited code to reflect changes I made)
I'm trying to connect to a Mongo database through an SSH tunnel using Java.
I'm using the Mongo driver 3.0.2 and jcraft (JSch) to create an SSH tunnel.
The idea is that I:
connect to the machine hosting the MongoDB installation through SSH
set up port forwarding from a local port to the remote MongoDB port
connect to MongoDB remotely
My code looks like this:
// forwarding ports
private static final String LOCAL_HOST = "localhost";
private static final String REMOTE_HOST = "127.0.0.1";
private static final Integer LOCAL_PORT = 8988;
private static final Integer REMOTE_PORT = 27017; // Default mongodb port
// ssh connection info
private static final String SSH_USER = "<username>";
private static final String SSH_PASSWORD = "<password>";
private static final String SSH_HOST = "<remote host>";
private static final Integer SSH_PORT = 22;
private static Session sshSession;
public static void main(String[] args) {
try {
java.util.Properties config = new java.util.Properties();
config.put("StrictHostKeyChecking", "no");
JSch jsch = new JSch();
sshSession = null;
sshSession = jsch.getSession(SSH_USER, SSH_HOST, SSH_PORT);
sshSession.setPassword(SSH_PASSWORD);
sshSession.setConfig(config);
sshSession.connect();
sshSession.setPortForwardingL(LOCAL_PORT, REMOTE_HOST, REMOTE_PORT);
MongoClient mongoClient = new MongoClient(LOCAL_HOST, LOCAL_PORT);
mongoClient.setReadPreference(ReadPreference.nearest());
MongoCursor<String> dbNames = mongoClient.listDatabaseNames().iterator();
while (dbNames.hasNext()) {
System.out.println(dbNames.next());
}
} catch (Exception e) {
e.printStackTrace();
} finally {
sshSession.delPortForwardingL(LOCAL_PORT);
sshSession.disconnect();
}
}
This code, when run, doesn't EDIT: does work. Connecting to the SSH server works just fine, but connecting to the Mongo database behind it doesn't work and returns this error:
INFO: Exception in monitor thread while connecting to server localhost:8988
com.mongodb.MongoSocketReadException: Prematurely reached end of stream
at com.mongodb.connection.SocketStream.read(SocketStream.java:88)
at com.mongodb.connection.InternalStreamConnection.receiveResponseBuffers(InternalStreamConnection.java:491)
at com.mongodb.connection.InternalStreamConnection.receiveMessage(InternalStreamConnection.java:221)
at com.mongodb.connection.CommandHelper.receiveReply(CommandHelper.java:134)
at com.mongodb.connection.CommandHelper.receiveCommandResult(CommandHelper.java:121)
at com.mongodb.connection.CommandHelper.executeCommand(CommandHelper.java:32)
at com.mongodb.connection.InternalStreamConnectionInitializer.initializeConnectionDescription(InternalStreamConnectionInitializer.java:83)
at com.mongodb.connection.InternalStreamConnectionInitializer.initialize(InternalStreamConnectionInitializer.java:43)
at com.mongodb.connection.InternalStreamConnection.open(InternalStreamConnection.java:115)
at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:127)
at java.lang.Thread.run(Unknown Source)
I've tried doing this through command line as follows:
$ ssh <user>#<host> -p 22 -X -C
$ <enter requested password>
<user>#<host>$ mongo
<user>#<host>$ MongoDB shell version: 2.6.10
<user>#<host>$ connecting to: test
So this seems to work. I'm at a loss as to why the Java code (which should be doing roughly the same thing) doesn't work.

I managed to make it work (tried to forward port to "localhost" rather than "127.0.0.1", changing it fixed it) edit: I guess the server was listening specifically on localhost rather than 127.0.0.1

This code is run successfully, but the main problem is your mongo db is stopped. Please check the instance of the mongo is running or not.
sudo systemctl status mongod
if it is not running
sudo systemctl start mongod

Spring Boot application with H2 file database

I'm trying to have a H2 database setup on spring boot application startup. I have configured the database in application.properties:
spring.datasource.url = jdbc:h2:file:~/testdb
spring.datasource.username = sa
spring.datasource.password = sa
spring.datasource.driverClassName = org.h2.Driver
The Application.java file:
#Configuration
#ComponentScan
#EnableAutoConfiguration
public class Application {
public static void main(String[] args) {
SpringApplication.run(Application.class, args);
CreateH2Database createH2Database = new CreateH2Database();
createH2Database.create();
}
}
CreateH2Database.java:
public class CreateH2Database {
private Logger log = Logger.getLogger(CreateH2Database.class);
#Autowired
protected JdbcTemplate jdbcTemplate;
public void create() {
log.info("Creating H2 Database");
createUsers();
}
private void createUsers() {
log.info("Creating users table");
jdbcTemplate.execute("create table if not exists users (id serial, first_name varchar(255), last_name varchar(255))");
String[] names = "John Woo;Jeff Dean;Josh Bloch;Josh Long".split(";");
for (String fullname : names) {
String[] name = fullname.split(" ");
log.info("Inserting user record for " + name[0] + " " + name[1] + "\n");
jdbcTemplate.update(
"INSERT INTO users(first_name,last_name) values(?,?)",
name[0], name[1]);
}
}
}
Once the application is started it should create the Users table if it doesn't already exist, and insert the users into the table. If the table does already exist, I don't want it to be modified.
I get a NullPointerException on jdbcTemplate.execute. How can I get the jdbcTemplate injected? All the example I've seen require the datasource to be created manually, and then the JdbcTemplate is created. However, the datasource in this example seems to be created based on the application.properties values.
Is this the correct approach to setup the database (i.e. calling a CreateH2Database after starting the SpringApplication)? Will this approach work if I want to run the application as a WAR on another application server?

Since you are using Spring Boot, you should take advantage of it's database initialization features. There is no need to roll out your own implementation.
All you have to do is have the files schema.sql and data.sql on the root of the classpath (most likely under /resources). Spring Boot will auto-detect these and run the first one in order to create the database and the second one to populate it.
Check out this part of the Spring Boot documentation
If you need to perform the initialization conditionally (perhaps only when running integration tests), you can advantage of Spring profiles. What you would do in that case is have the properties file for the test profile contain
spring.datasource.initialize=true
while the properties file for the other profiles would contain
spring.datasource.initialize=false