Develop Reference

MongoURI #value variable is getting nulled

In my code below, mongoURI initially pulls the correct URI string from application.properties, and connects to the database successfully. However, once I make a call to getUserByAuth0ID, I'm getting a "java.net.UnknownHostException: null: Name or service not known" error and debug statements show that mongoURI is now set to null.
What's going on? Nowhere in my code do I touch the value of mongoURI. My previous version of the code has mongoURI hardcoded as a variable and it runs with no issues.
#Service
public class DBConnectService {
private static MongoCollection<User> users;
private static Logger logger = LogManager.getLogger(DBConnectService.class);
#Value("${package.mongoURI}")
private String mongoURI;
/** Opens a connection to mongodb for the length of the program operation */
#PostConstruct
public void init() {
logger.info("Connecting to MongoDB");
try {
System.out.println (mongoURI); // URI prints out correctly here
CodecRegistry pojoCodecRegistry = fromProviders(PojoCodecProvider.builder().automatic(true).build());
CodecRegistry codecRegistry = fromRegistries(MongoClientSettings.getDefaultCodecRegistry(), pojoCodecRegistry);
MongoClientSettings clientSettings = MongoClientSettings.builder()
.applyConnectionString(new ConnectionString(mongoURI))
.codecRegistry(codecRegistry)
.build();
MongoClient mongoClient = MongoClients.create(clientSettings);
MongoDatabase database = mongoClient.getDatabase("db");
users = database.getCollection("users", User.class);
} catch(Exception e) {
logger.error("MongoDB connection failure:\n" + e);
}
}
public User getUserByAuth0ID (String authID) {
System.out.println (mongoURI); // URI prints out here as null
User user = getUser(authID, "auth0ID");
if (user == null) {
user = createUserAccount(authID);
}
return user;
}
public static User getUser (String info, String field) {
User user = users.find(eq(field, info)).first();
return user;
}
public static User createUserAccount (String authID) {
JsonObject newUserInfo = Auth0Service.getUserInfo(authID);
if (newUserInfo.get("email_verified").getAsBoolean()) {
User newUser = new User()
.setEmail(newUserInfo.get("email").getAsString())
.setName(newUserInfo.get("name").getAsString())
.setAuth0ID(authID);
users.insertOne(newUser);
return newUser;
} else {
logger.info ("Email NOT verified");
return null;
}
}
Application.properties line:
# --- MongoDB ---
package.mongoURI = mongodb+srv://admin:secretURL/?retryWrites=true&w=majority

Your #Value annotation has incorrect details of mongoURI.Either use #Value("${nornir.mongoURI}") or change to package.mongoURI inside application.properties.
Edit:
It is more likely you are calling getUserByAuth0ID manually something like --
DBConnectService service = new DBConnectService();
service.getUserByAuth0ID();
Because if mongoURI is coming as null, it means, this method getUserByAuth0ID is not getting called via spring way, i.e. by autowiring DBConnectService & then accessing this method; but rather it is getting called manually i.e. by manually creating object of DBConnectService.
If this is case, then it is obvious that your normal java object don't know about #Value annotation & hence it is not setting any value.
#PostConstruct will always works as it will get executed at startup when bean is getting created & hence #Value is working properly there.
To fix this issue, make sure you have spring bean of DBConnectService & you are accessing getUserByAuth0ID via that bean.
Edit 2 : --
Basic pseudo about how to use this in your calling class :
#Autowired
DBConnectService service;
public void yourdbMethod(){
service.getUserByAuth0ID();
}

How to create and configure MariaDBDataSource manually

I'm having problems with creating unpooled datasource with MariaDBDataSource class.
MariaDbDataSource mysqlDs = new MariaDbDataSource(connectionUrl);
mysqlDs.setPassword(password);
mysqlDs.setUser(username);
return wrapWithPool(mysqlDs);
wrapWithPool simply wraps the given datasource with a pooled one (c3p0 pool).
But I fail to checkout a connection from the pool. Whenever I do
datasource.getConnection()
I get
org.mariadb.jdbc.internal.util.dao.QueryException: Could not connect: Access denied for user 'someuser'#'somehost' (using password: NO)
Not sure why? I do set non empty password. Is there anything else to set on the MariaDbDatasource class to make it use the password?
edit:
Ok, so it seems that when I do not wrap the MariaDbDataSource all works ok.
So c3p0 is breaking up the connection, and from debug I see it fails to get the password...
The wrap method is quite simple
private static DataSource wrapWithPool(DataSource unpooled) throws SQLException {
unpooled.setLoginTimeout(HOST_REACH_TIMEOUT.getValue());
Map<String, Object> poolOverrideProps = new HashMap<>();
poolOverrideProps.put("maxPoolSize", CONNECTION_POOL_SIZE.getValue());
poolOverrideProps.put("minPoolSize", 1);
poolOverrideProps.put("checkoutTimeout", HOST_REACH_TIMEOUT.getValue() * 2);
return DataSources.pooledDataSource(unpooled, poolOverrideProps);
}
And it works perfecly fine with other drivers (oracle, jtds). Why not with mariaDb?

Ok, so I discovered the problem. For some reason, the c3p0 when creating the pool, wraps the given DataSource class within own WrapperConnectionPoolDataSourceBase class. Then it tries to detect the authentication parameters from it using reflection. Since MariaDBDataSource does not provide the getPassword method, the discovered value is null, and thus the error message about not using the password.
So as a workaround I did a simple wrapper
private static class MariaDbDExtender extends MariaDbDataSource {
private String password;
public MariaDbDExtender(String connectionUrl) throws SQLException {
super(connectionUrl);
}
#Override
public void setPassword(String pass) {
this.password = pass;
super.setPassword(pass);
}
//this method is required to allow c3p0 magically use reflection to get correct password for connection
public String getPassword() {
return password;
}
}
and later on
MariaDbDExtender mysqlDs = new MariaDbDExtender(connectionUrl);
mysqlDs.setPassword(password);
mysqlDs.setUser(username);
return wrapWithPool(mysqlDs);
And it magically starts to work. This is some driver specific issue, since oracle datasource does not have the getPassword method, but works. So some very specific implementation details of those 2 libraries just make it incompatible in my use case.

How to structure application to aid testing of sql

I am writing an application - to which I think my design is not great. I use a class called ExposedFunctions which houses my #WebMethod logic for requests to my web service. These methods often need to read from an internal cache or go to the database. Therefore I have a DBFactory class which has individual methods for database operations. An example of this structure can be seen below
ExposedFunctions
#WebService
public class ExposedFunctions {
private static Logger logger = Logger.getLogger(ExposedFunctions.class);
private DBFactory factory = new DBFactory();
#WebMethod
public String register(String username, String password, String email, String firstname, String lastname) {
if(StringUtilities.stringEmptyOrNull(username) ||
StringUtilities.stringEmptyOrNull(password) ||
StringUtilities.stringEmptyOrNull(email) ||
StringUtilities.stringEmptyOrNull(firstname) ||
StringUtilities.stringEmptyOrNull(lastname)){
logger.error("String was null or empty when registering");
}
RegistrationStatus status = factory.register(username, password, email, firstname, lastname);
return status.getValue();
}
}
DBFactory
public class DBFactory {
private final BasicDataSource source = new BasicDataSource();
private final Logger logger = Logger.getLogger(DBFactory.class);
public DBFactory() {
try {
setupConnections();
} catch (DatabasePropertyException e) {
logger.info("Unable to load the properties file", e);
System.exit(1);
}
}
private void setupConnections() throws DatabasePropertyException{
Properties props = DatabaseUtilities.getDatabaseConnectionProps("/swiped.properties");
if(props != null){
source.setDriverClassName("com.mysql.jdbc.Driver");
source.setUsername(props.getProperty("username"));
source.setPassword(props.getProperty("password"));
source.setUrl(props.getProperty("url_local"));
source.setMaxActive(-1);
}else{
throw new DatabasePropertyException("Unable to load the proeprties file in order to connect to the database - exiting application");
}
}
public RegistrationStatus register(String username, String password, String email, String firstname, String lastname) {
String sql = "INSERT INTO Users (username, password, email, firstname, lastname) values (?,?,?,?,?)";
RegistrationStatus status = null;
Connection conn = null;
PreparedStatement st = null;
try {
conn = source.getConnection();
st = conn.prepareStatement(sql);
st.setString(1, username);
st.setString(2, password);
st.setString(3, email);
st.setString(4, firstname);
st.setString(5, lastname);
st.executeUpdate();
status = RegistrationStatus.SUCCESSFUL;
}
catch (MySQLIntegrityConstraintViolationException e) {
logger.warn("Unable to register user " + username + " as they are already registered");
return RegistrationStatus.USER_ALREADY_REGISTERED;
}catch(Exception e){
logger.error("Unable to insert a new user in to the database", e);
status = RegistrationStatus.FAILED;
}finally{
DatabaseUtilities.closeConnection(conn);
DatabaseUtilities.closePreparedStatement(st);
}
return status;
}
This setup makes it very difficult for me to test because 1) DBFactory is tied to a particular database connection - meaning that its very likely the data inside will change and tests will pass and fail irregularly. There is also another concern that DBFactory can reach 2000+ lines of code which isn't ideal either.
Please can someone suggest some way of improving this design so as to increase / maximise my testing efforts and also to help ensure better maintainability and extendability.
Thanks

Well, there are at least 3 levels that you need to test and there needs to be some minor modifications to your code to make it easier:
1. Unit testing your code that calls DbFactory.
It's nice that you have already abstracted the code that actually interacts with the database as this makes it easier to test. If you modify ExposedFunctions so you can pass in or inject a different DbFactory instance during tests, then your tests can use a mock DbFactory. The mock factory can make sure your code passes in the correct parameters and you can test various error conditions without actually touching a real database.
All you probably need to do is add a constructor or setter method to modify your dbFactory field:
public class ExposedFunctions {
...
private DBFactory factory;
public ExposedFunctions(){
this(new DBFactory());
}
public ExposedFunctions(DbFactory factory){
Objects.requireNonNull(factory);
this.factory = factory;
}
...
}
2. Modify your DbFactory class to mock out the Connection
If you do something similar in your DbFactory class so you can mock out the Connection then your tests against DbFactory also don't have to hit a real database. By Mocking the Connection and the Statements it returns, you can verify that the correct SQL code is getting executed for the given parameters as well as testing various SQLException conditions to simulate connection or data problems that are otherwise hard to test in the real world.
3. Test against a test (in memory? ) database
It is also a good idea to actually connect to a real database to make sure everything works as part of an integration test. Don't connect to production though! You can swap out your connection either by using a different property file or by injecting a different connection object like in #2. There are libraries such as dbUnit to help working with test databases. There are also in-memory databases like derby or hsqldb which are lightweight and let you run tests faster than connecting to a "regular" database. However, one word of caution with in-memory dbs, it's not exactly the same enviornment or SQL interpreter as your database vender of choice. There may be differences in SQL or schema design that might make you think your code works since the test passes, but then something wrong happens in production with the same code.
Hope this helps

How can I write string to container to be used after a loop?

I have an aplication which create a number of query (update or insert) and then each query is executed.
The whole code is working fine but I've saw that my server IO latency is too much during this proccess.
The code execute a loop which is taking arround 1 minute.
Then what I wanted to do is write each query in memory instead to execute it, and then, once I have the whole list of query to execute, use "LOAD DATA LOCAL INFILE" from mysql, which will take less time.
My question is: How can I write all my query (String object) in a "File" or "any other container" in java to use it after the loop?.
#user3283548 This is my example code:
Class1:
import java.util.ArrayList;
public class Class1 {
public static void main(String[] args) throws Exception {
// TODO Auto-generated method stub
ArrayList<String> Staff=new ArrayList<String>();
Staff.add("tom");
Staff.add("Laura");
Staff.add("Patricia");
for (int x = 0; x < Staff.size(); x++) {
System.out.println(Staff.get(x));
Class2 user = new Class2 (Staff.get(x));
user.checkUser();
}
}
}
Class2:
public class Class2 {
private String user;
public Class2(String user){
this.user=user;
}
public void checkUser() throws Exception{
if (user.equals("tom")){
String queryUser="update UsersT set userStatus='2' where UserName='"+user+"';";
Class3 updateUser = new Class3(queryUser);
updateUser.UpdateQuery();;
}else{
String queryUser="Insert into UsersT (UserName,userStatus)Values('"+user+"','1');";
Class3 updateUser = new Class3(queryUser);
updateUser.InsertQuery();
System.out.println(user+" is not ton doing new insert");
}
}
}
Class3:
public class Class3 {
public String Query;
public Class3(String Query){
this.Query = Query;
}
public void UpdateQuery() throws Exception{
/*// Accessing Driver From Jar File
Class.forName("com.mysql.jdbc.Driver");
//DB Connection
Connection con = DriverManager.getConnection("jdbc:mysql://localhost:3306/default","root","1234567");
String sql =Query;
PreparedStatement pst = con.prepareStatement(sql);*/
System.out.println(Query); //Just to test
//pst.execute();
}
public void InsertQuery() throws Exception{
/*// Accessing Driver From Jar File
Class.forName("com.mysql.jdbc.Driver");
//DB Connection
Connection con = DriverManager.getConnection("jdbc:mysql://localhost:3306/default","root","1234567");
String sql =Query;
PreparedStatement pst = con.prepareStatement(sql);*/
System.out.println(Query); //Just to test
//pst.execute();
}
}
Then, what I wanted to do is create an ArraList in Class1 and use it in Class3 to collect all the queries which has to be executed.
The idea is to execute the list of queries in one time, once the main process is finished, istead to do it for each element within in loop of the Class1. I wanted to do it, because I think it will be take less resource IO from the server HD

Your loop is probably too slow because you're building up Strings using String
I'd hazard a guess you're doing things like
String query = "SELECT * FROM " + variablea + " WHERE + variableb + " = " ...
If you're doing a lot of string concatenation then use StringBuilder as every time you change a string it is actually re-created which is expensive. Simply changing your code to use StringBuilder instead of string will probably cut your loop executed time to a couple of MS. Simply call .toString() method of StringBuilder obj to get the string.
Storing objects
If you want to store anything for later use you should store it in a Collection. If you want a a key-value relationship then use a Map (HashMap would suit you fine). If you just want the values use an List (ArrayList is most popular).
So for example if I wanted to store query strings for later use I would...
Construct the string using StringBuilder.
Put the string (by calling .toString() into a HashMap
Get the query string from the HashMap...
You should never store things on disk if you don't need them to be persistent over application restarts and even then I'd store them in a database not in a file.
Hope this helps.
Thanks
David
EDIT: UPDATE BASED ON YOU POSTING YOUR CODE:
OK this needs some major re-factoring!
I've kept it really simple because I don't have a lot of time to re-write comprehensively.
I've commented where I have made corrections.
Your major issue here is creating objects in loops. You should just create the object once as creating objects is expensive.
I've also corrected other coding issues and replaced the for loop as you shouldn't be writing it like that.I've also renamed the classes to something useful.
I've not tested this so you may need to do some work to get it to work. But this should be a lot faster.
OLD CLASS 1
import java.util.ArrayList;
import java.util.List;
public class StaffChecker {
public static void main(String[] args) throws Exception {
// Creating objects is expensive, you should do this as little as possible
StaffCheckBO staffCheckBO = new StaffCheckBO();
// variables should be Camel Cased and describe what they hold
// Never start with ArrayList start with List you should specific the interface on the left side.
List<String> staffList = new ArrayList<String>();
staffList.add("tom");
staffList.add("Laura");
staffList.add("Patricia");
// use a foreach loop not a (int x = 0 ... ) This is the preffered method.
for (String staffMember : staffList) {
// You now dont need to use .get() you can access the current variable using staffMember
System.out.println(staffMember);
// Do the work
staffCheckBO.checkUser(staffMember);
}
}
}
OLD CLASS 2
/**
* Probably not really any need for this class but I'll assume further business logic may follow.
*/
public class StaffCheckBO {
// Again only create our DAO once...CREATING OBJECTS IS EXPENSIVE.
private StaffDAO staffDAO = new StaffDAO();
public void checkUser(String staffMember) throws Exception{
boolean staffExists = staffDAO.checkStaffExists(staffMember);
if(staffExists) {
System.out.println(staffMember +" is not in database, doing new insert.");
staffDAO.insertStaff(staffMember);
} else {
System.out.println(staffMember +" has been found in the database, updating user.");
staffDAO.updateStaff(staffMember);
}
}
}
OLD CLASS 3
import java.sql.*;
/**
* You will need to do some work to get this class to work fully and this is obviously basic but its to give you an idea.
*/
public class StaffDAO {
public boolean checkStaffExists(String staffName) {
boolean staffExists = false;
try {
String query = "SELECT * FROM STAFF_TABLE WHERE STAFF_NAME = ?";
PreparedStatement preparedStatement = getDBConnection().prepareStatement(query);
// Load your variables into the string in order to be safe against injection attacks.
preparedStatement.setString(1, staffName);
ResultSet resultSet = preparedStatement.executeQuery();
// If a record has been found the staff member is in the database. This obviously doesn't account for multiple staff members
if(resultSet.next()) {
staffExists = true;
}
} catch (SQLException e) {
System.out.println("SQL Exception in getStaff: " + e.getMessage());
}
return staffExists;
}
// Method names should be camel cased
public void updateStaff(String staffName) throws Exception {
try {
String query = "YOUR QUERY";
PreparedStatement preparedStatement = getDBConnection().prepareStatement(query);
// Load your variables into the string in order to be safe against injection attacks.
preparedStatement.setString(1, staffName);
ResultSet resultSet = preparedStatement.executeQuery();
} catch (SQLException e) {
System.out.println("SQL Exception in getStaff: " + e.getMessage());
}
}
public void insertStaff(String staffName) throws Exception {
try {
String query = "YOUR QUERY";
PreparedStatement preparedStatement = getDBConnection().prepareStatement(query);
// Load your variables into the string in order to be safe against injection attacks.
preparedStatement.setString(1, staffName);
ResultSet resultSet = preparedStatement.executeQuery();
} catch (SQLException e) {
System.out.println("SQL Exception in getStaff: " + e.getMessage());
}
}
/**
* You need to abstract the connection logic away so you avoid code reuse.
*
* #return
*/
private Connection getDBConnection() {
Connection connection = null;
try {
Class.forName("com.mysql.jdbc.Driver");
connection = DriverManager.getConnection("jdbc:mysql://localhost:3306/default", "root", "1234567");
} catch (ClassNotFoundException e) {
System.out.println("Could not find class. DB Connection could not be created: " + e.getMessage());
} catch (SQLException e) {
System.out.println("SQL Exception. " + e.getMessage());
}
return connection;
}
}

Store database connection as separate Class - Java

Is it possible to store a database connection as a separate class, then call the database objects from a main code? ie;
public class main{
public static void main{
try{
Class.forName("com.jdbc.driver");
Database to = new Database(1,"SERVER1","DATABASE");
Database from = new Database(2,"SERVER2","DATABASE");
String QueryStr = String.format("SELECT * FROM TABLE WHERE Id = %i", to.id)
to.results = sql.executeQuery(QueryStr);
while (to.results.next()) {
String QueryStr = String.format("INSERT INTO Table (A,B) VALUES (%s,%s)",to.results.getString(1),to.results.getString(2));
from.sql.executeQuery("QueryStr");
}
to.connection.close()
from.connection.close()
} catch (Exception ex) {
ex.printStackTrace();
{ finally {
if (to.connection != null)
try {
to.connection.close();
} catch (SQLException x) {
}
if (from.connection != null)
try {
from.connection.close();
} catch (SQLException x) {
}
}
}
public static class Database {
public int id;
public String server;
public String database;
public Connection connection;
public ResultSet results;
public Statement sql;
public Database(int _id, String _server, String _database) {
id = _id;
server = _server;
database = _database;
String connectStr = String.format("jdbc:driver://SERVER=%s;port=6322;DATABASE=%s",server,database);
connection = DriverManager.getConnection(connectStr);
sql = connection.createStatement;
}
}
}
I keep getting a "Connection object is closed" error when I call to.results = sql.executeQuery("SELECT * FROM TABLE"); like the connection closes as soon as the Database is done initializing.
The reason I ask is I have multiple databases that are all about the same that I am dumping into a master database. I thought it would be nice to setup a loop to go through each from database and insert into each to database using the same class. Is this not possible? Database will also contain more methods than shown as well. I am pretty new to java, so hopefully this makes sense...
Also, my code is probably riddled with syntax errors as is, so try not to focus on that.

Connection object is closed doesn't mean that the connection is closed, but that the object relative to the connection is closed (it could be a Statement or a ResultSet).
It's difficult to see from your example, since it has been trimmed/re-arranged, but it looks like you may be trying to use a ResultSet after having re-used its corresponding Statement. See the documentation:
By default, only one ResultSet object per Statement object can be open
at the same time. Therefore, if the reading of one ResultSet object is
interleaved with the reading of another, each must have been generated
by different Statement objects. All execution methods in the Statement
interface implicitly close a statment's current ResultSet object if an
open one exists.

In your example, it may be because autoCommit is set to true by default. You can override this on the java.sql.Connection class. Better yet is to use a transaction framework if you're updating multiple tables.