I am writing an integration test between my JPA layer and the database to check the SQL I've written is correct. The real database is Oracle, unfortunately down to reasons out my control my test database has to be Derby so naturally there are some differences. For example my JPA class has the following SQL String constant
private static final String QUERY = "Select * from Users where regexp_like(user_code, '^SS(B)?N')";
Because Derby doesn't support regexp_like I am using JMockits Deencapsulation.setField to change the SQL on the fly. eg.
#Test
public void testMyDaoFind() {
new Expectations() {
{
Deencapsulation.setField(MyClass.class, "QUERY", "Select * from Users");
}
};
dao.findUsers();
}
Now ignoring the fact that this isn't a good test as it's not testing the actual query that will be running on the real database (this is purely to satisfy my curiousity as to what is going on), I am getting a SQL exception error from Eclipselink/Derby complaining about regexp_like is not recognized as a function or a procedure.
If I place a break point on the line in the DAO that attempts to get the result list, I can see from a new watch that
JMockit has substituted the query correctly
getResultList() returns the data I'm expecting to see
If however I let the test run all the way through then I get the afformentioned exception?!
Strings in Java are not handled the way you are thinking. The Java source compiler replaces reads from fields holding string literals with the fixed "address" where the string is stored (in the class' constant pool); the field is not read anymore at runtime. So, even if JMockit replaces the string reference stored in the field, it makes no difference as that reference isn't seen by the client code using the field.
(BTW, why is the test putting the call to Deencapsulatin.setField inside an expectation block? Such blocks are only meant for recording expectations...)
Bottom line, there is no way to achieve what you're trying to. Instead, either use an Oracle database for integration testing, or make all SQL code portable, avoiding RDBMS-specific functions such as regexp_like.
Abstract:
An application I work on uses top link, I'm having trouble finding out if and when top link automatically uses bind variables.
Problem Description:
Lets say I need to do something akin to validating if a vehicle full of people can travel somewhere, where each person could invalidate the trip, and provide error messages so the person could get their restrictions removed before the trip starts. A simple way to do that is to validate each member in the list, and display a list of errors. Lets say their info is stored on an oracle database and I query for each riders info using their unique ids, This query will be executed for each member in the list. A naïve implementation would cause a hard parse, a new execution path, despite only the unique id changing.
I've been reading about bind variables in sql, and how they allow for reuse of an execution path, avoiding cpu intensive hard parses.
A couple links on them are:
http://www.akadia.com/services/ora_bind_variables.html
https://oracle-base.com/articles/misc/literals-substitution-variables-and-bind-variables
An application I work on uses toplink and does something similar to the situation described above. I'm looking to make the validation faster, without changing the implementation much.
If I do something like the following:
Pseudo-code
public class userValidator{
private static DataReadQuery GET_USER_INFO;
static{
GET_USER_INFO = "select * from schema.userInfo ui where ui.id= #accountId"
GET_USER_INFO.bindAllParameters();
GET_USER_INFO.cacheStatement();
GET_USER_INFO.addArgument("accountId", String.class);
}
void validate(){
List<String> listOfUserAccountIds = getuserAccountIdList();
list args;
for(String userAccountId: listOfUserAccountIds){
args = new ArrayList(1);
args.add(userAccountId)
doSomethingWithInfo(getUnitOfWork().executequery(GET_USER_INFO, args);
}
}
}
The Question:
Will a new execution path be parsed for each execution of GET_USER_INFO?
What I have found:
If I understand the bindAllParameters function inside of the DatabaseQuery class well enough, it simple is a type validation to stop sql injection attacks.
There is also a shouldPrepare function inside the same class, however that seems to have to do more with allowing dynamic sql usage where the number of arguments is variable. A prepared DatabaseQuery has its sql written once with just the values of the variables changing based on the argument list passed in, which sounds like simple substitution and not bind variables.
So I'm at a lost.
This seems answered by the TopLink documentation
By default, TopLink enables parameterized SQL but not prepared
statement caching.
So prepared statements are used by default, just not cached. This means subsequent queries will have the added cost of re-preparing statements if not optimized by the driver. See this for more information on optimizations within TopLink
This a part of my code java constructing a function in an anonymous plsql block:
*String p ="FUNCTION get_result RETURN varchar2 AS "
+"BEGIN "
+"sql_error:='ok';"
+query1+";"
+query2+";"
+query3+";"
+"COMMIT;"
+"RETURN sql_error; "
+"EXCEPTION "
+"when others then "
+"ROLLBACK;"
+"end;";*
where queries 1,2 and 3 are sql queries; sometimes query2 is not executed and this does not generate a transaction error and queries 1 and 3 and committed. I would like to Know how is it possible and, how can I correct it.
I would like also to know which procedure is better between calling anonymous block statements and statement.addBatch
I want to know how the manage sessions that execute transaction that manipulate many sequences.
I found that using many seauences in a transaction causes errors that oracle do not send back to the user. Do they sometimes get the same value for 2 executions on sequence.nextval ?
thank you
I've seen the bug I was creating.
I fact, all the queries were not in the same transaction, due to the fact that I was using a static attribute that was collecting the queries of various transactions and sending them to the SGBD oracle through anonymous plsql.
so I changed the static variable to simple and public one, collector of the queries assessed through an instance of the class, so that all the queries of a transaction should be always sent at once and executed in the same order.
I'm trying to use setDistinct(true) as it is described in the guide: http://mybatis.github.io/generator/generatedobjects/exampleClassUsage.html
I've written in this way:
testExample ae = new testExample();
testExample.Criteria criteriatest = ae.createCriteria();
ae.setDistinct(true);
criteriatest.andIDENTLAVEqualTo(Long.parseLong(cert.getCODINDIVID()));
ae.or(criteriatest);
List<test> listtest = testMapper.selectByExample(ae);
but the setDistinct(true) doesn't affect the results.
Where should I add the setDistinct line?
It looks like the link you referenced is for an extremely old version of MyBatis. On that page, it lists the following:
Version: 1.3.3-SNAPSHOT
The latest version is:
mybatis-3.3.0-SNAPSHOT
Grepping the 3.x code for setDistinct does not return anything:
https://github.com/mybatis/mybatis-3/search?q=setDistinct
I'm surprised you don't get a compile-time error about the method not being found. Are you using version 1.3.3 (or 1.x)?
I would recommend doing the DISTINCT right in the query. Since MyBatis is generally a sort of a close-to-the-SQL-metal type of mapping framework, I think it's best to add it in the mapper file's query itself. Plus that way, you can choose specifically what to DISTINCT by. The setDistinct method does not seem to provide any way to specify the target.
For MyBatis 3, I think the analogous style of query would be this:
http://mybatis.github.io/mybatis-3/statement-builders.html
This seems to be analogous to a jOOQ-style DSL. It has a SELECT_DISTINCT method. I personally find it easier to code/read the pure SQL with some XML markup as needed for dynamic SQL in a mapper file, but this is certainly a viable option in MyBatis 3.
Edit:
So, I did some more digging, and the reason I couldn't find the code in the MyBatis3 git repo is because setDistinct is in the mybatis-generator code base.
I think part of the issue here may stem from what is part of Mybatis-Generator's description on GitHub:
MBG seeks to make a major impact on the large percentage of database
operations that are simple CRUD (Create, Retrieve, Update, Delete).
So, it provides a way to do simple DISTINCTs, but with limited control.
The code resides in the addClassElements method of the ProviderSelectByExampleWithoutBLOBsMethodGenerator class. Searching for setDistinct won't show up on a Github search since it's an automatically generated setter.
This is the relevant code snippet:
boolean distinctCheck = true;
for (IntrospectedColumn introspectedColumn : getColumns()) {
if (distinctCheck) {
method.addBodyLine("if (example != null && example.isDistinct()) {"); //$NON-NLS-1$
method.addBodyLine(String.format("%sSELECT_DISTINCT(\"%s\");", //$NON-NLS-1$
builderPrefix,
escapeStringForJava(getSelectListPhrase(introspectedColumn))));
method.addBodyLine("} else {"); //$NON-NLS-1$
method.addBodyLine(String.format("%sSELECT(\"%s\");", //$NON-NLS-1$
builderPrefix,
escapeStringForJava(getSelectListPhrase(introspectedColumn))));
method.addBodyLine("}"); //$NON-NLS-1$
} else {
method.addBodyLine(String.format("%sSELECT(\"%s\");", //$NON-NLS-1$
builderPrefix,
escapeStringForJava(getSelectListPhrase(introspectedColumn))));
}
distinctCheck = false;
}
So, essentially, this looks like it's wrapping the SELECT_DISTINCT method I mentioned originally, and it attempts to introspect the columns and apply the DISTINCT to all of the ones it gets back.
Digging a bit deeper, it ultimately calls this code to get the columns:
/**
* Returns all columns in the table (for use by the select by primary key
* and select by example with BLOBs methods)
*
* #return a List of ColumnDefinition objects for all columns in the table
*/
public List<IntrospectedColumn> getAllColumns() {
List<IntrospectedColumn> answer = new ArrayList<IntrospectedColumn>();
answer.addAll(primaryKeyColumns);
answer.addAll(baseColumns);
answer.addAll(blobColumns);
return answer;
}
So, this is definitely essentially an all-or-nothing DISTINCT (whereas Postgres itself allows DISTINCT on just certain columns).
Try moving the setDistinct to the very last line before you actually invoke the ae object. Perhaps subsequent calls are affecting the column set (although from the code, it doesn't seem like it should -- basically once the columns are set, the setDistinct should use them).
The other thing that would be interesting would be to see what SQL it is actually generating with and without setDistinct.
Check this link out for more detail on debug/logging:
http://mybatis.github.io/generator/reference/logging.html
I'd recommend perhaps trying out the XML-based mapper file definitions which interleave SQL with XML tags for dynamic-ness. IMO, it's much easier to follow than the code Mybatis Generator code snippet above. I suppose that's one of the main tradeoffs with a generator -- easier to create initially, but more difficult to read/maintain later.
For super-dynamic queries, I could see some more advantages, but then that sort of goes against their self-description of it being for simple CRUD operations.
So, I'm using jdbc to talk to a MySQL DB. For many a table and for many queries/views, I have created one class which encapsulates one row of the table or the query/table result. Accesses to the DB return one object of such a class (when I know exactly that there's only one matching row) or a Vector of such objects.
Each class features a factory method that builds an object from a row of a ResultSet. A lot of ResultSet.getXXX() methods are needed, as is finicky bookkeeping about which value is in which column, especially after changes to the table/query/view layout.
Creating and maintaining these objects is a boring, work-intensive and mind-numbing task. In other words, the sort of task that is done by a tool. It should read SQL (the MySQL variant, alas) and generate Java code. Or, at least, give me a representation (XML? DOM?) of the table/query/view, allowing me to do the java code generation myself.
Can you name this tool?
I'm a little confused about your questions. Why don't you use an Object-relational-mapping framework like Hibernate?
I used to have the same problem having to read and write a lot of SQL directly. Eventually I started writing newer projects with Hibernate and haven't looked back. The system takes care for me of building the actual tables and running the SQL in the background, and I can mostly work with the java objects.
If you are looking for a simple framework to help with the drudge work in writing sql, I would recommend ibatis sql maps. This framework basically does exactly what you want.
Hibernate is also a good option, but it seems a bit oversized for a simple problem like yours.
You might also have a look at the spring framework. This aims to create a simple environment for writing java application and has a very usable sql abstraction as well. But be careful with spring, you might start to like the framework and spend too many happy hours with it 8)
As to your concern with reflection. Java has no major problems anymore with performance overhead of reflection (at least since Version 1.4 and with O/R mapping tools).
In my experience, it is better to care about well written and easily understandable code, than caring about some performance overhead this might perhaps cost, that is only theoretical.
In most cases performance problems will not show up, where you expect them and can only be identified with measurement tools used on your code after it has been written. The most common problems with performance are I/O related or are based on some error in your own code (i.e. massively creating new instances of classes or loops with millions of runs, that are not necessary...) and not in the jdk itself.
I created a mini-framework like that years ago, but it was for prototyping and not for production.
The idea follows and it is VERY very simple to do. The tradeoff is the cost of using reflection. Although Hibernate and others ORM tools pay this cost also.
The idea is very simple.
You have a Dao class where you execute the query.
Read the ResultSet Metadata and there you can grab the table name, fields, types etc.
Find in the class path a Class that matches the table name and / or have the same number/types of fields.
Set the values using reflection.
Return this object and cast it in the other side and you're done.
It might seem absurd to find the class at runtime. And may look too risky too, because the query may change or the table structure may change. But think about it. When that happens, you have to update your mappings anyway to match the new structure. So instead you just update the matching class and live happy with that.
I'm not aware on how does ORM tools work to reduce reflection call cost ( because the mapping the only thing it does is help you to find the matching class ) In my version the lookup among about 30,000 classes ( I added jars from other places to test it ) took only .30 ms or something like that. I saved in cache that class and the second time I didn't have to make the lookup.
If you're interested ( and still reading ) I'll try to find the library in my old PC.
At the end my code was something like this:
Employee e = ( Employee ) MagicDataSource.find( "select * from employee where id = 1 ");
or
Employee[] emps = ( Employee[] ) MagicDataSource.findAll("select * from employee ");
Inside it was like:
Object[] findAll( String query ) {
ResultSet rs = getConnection().prepareStatemet( query ).executeQuery();
ResultSetMetaData md = rs.getMetadata();
String tableName = md.getTableName();
String clazz = findClass( toCamelCase( tableName ) ); // search in a list where all the class names where loaded.
Class.forName( clazz );
while( rs.next() ) {
for each attribute etc. etc.
setter...
end
result.append( object );
}
return result.toArray();
}
If anyone knows how ORM tools deal with reflection cost please let me know. The code I have read from open source projects don't event attempt to do anything about it.
At the end it let me create quick small programs for system monitoring or stuff like that. I don't do that job anymore and that lib is now in oblivion.
Apart from the ORMs...
If you're using the rs.getString and rs.getInt routines, then you can certainly ease your maintenance burden if you rely on named columns rather than numbered columns.
Specifically rs.getInt("id") rather than rs.getInt(1), for example.
It's been rare that I've had an actual column change data type, so future SQL maintenance is little more than adding the new columns that were done to the table, and those can be simply tacked on to the end of your monster bind list in each of you little DAO objects.
Next, you then take that idiom of using column names, and you extend it to a plan of using consistent names, and, at the same, time, "unique" names. The intent there is that each column in your database has a unique name associated with it. In theory it can be as simple (albeit verbose) as tablename_columnname, thus if you have a "member" table, the column name is "member_id" for the id column.
What does this buy you?
It buys you being able to use your generic DAOs on any "valid" result set.
A "valid" result set is a result set with the columns named using your unique naming spec.
So, you get "select id member_id, name member_name from member where id = 1".
Why would you want to do that? Why go to that bother?
Because then your joins become trivial.
PreparedStatement = con.prepareStatement("select m.id member_id, m.name member_name, p.id post_id, p.date post_date, p.subject post_subject from member m, post p where m.id = p.member_id and m.id = 123");
ResultSet rs = ps.executeQuery();
Member m = null;
Post p = null;
while(rs.next()) {
if (m == null) {
m = MemberDAO.createFromResultSet(rs);
}
p = PostDAO.createFromResultSet(rs);
m.addPost(p);
}
See, here the binding logic doesn't care about the result set contents, since it's only interested in columns it cares about.
In your DAOs, you make them slightly clever about the ResultSet. Turns out if you do 'rs.getInt("member_id")' and member_id doesn't happen to actually BE in the result set, you'll get a SQLException.
But with a little work, using ResultSetMetaData, you can do a quick pre-check (by fetching all of the column names up front), then rather than calling "rs.getInt" you can call "baseDAO.getInt" which handles those details for you so as not to get the exception.
The beauty here is that once you do that, you can fetch incomplete DAOs easily.
PreparedStatement = con.prepareStatement("select m.id member_id from member m where m.id = 123");
ResultSet rs = ps.executeQuery();
Member m = null;
if (rs.next()) {
m = MemberDAO.createFromResultSet(rs);
}
Finally, it's really (really) a trivial bit of scripting (using, say, AWK) that can take the properties of a bean and convert it into a proper blob of binding code for an initial DAO. A similar script can readily take a SQL table statement and convert it in to a Java Bean (at least the base members) that then your IDE converts in to a flurry of getters/setters.
By centralizing the binding code in to the DAO, maintenance is really hardly anything at all, since it's changed in one place. Using partial binding, you can abuse them mercilessly.
PreparedStatement = con.prepareStatement("select m.name member_name, max(p.date) post_date from member m, post p where post.member_id = m.id and m.id = 123");
ResultSet rs = ps.executeQuery();
Member m = null;
Post p = null;
if (rs.next()) {
m = MemberDAO.createFromResultSet(rs);
p = MemberDAO.craateFromResultSet(rs);
}
System.out.println(m.getName() + " latest post was on " + p.getDate());
Your burden moving forward is mostly writing the SQL, but even that's not horrible. There's not much difference between writing SQL and EQL. Mind, is does kind of suck having to write a select statement with a zillion columns in it, since you can't (and shouldn't anyway) use "select * from ..." (select * always (ALWAYS) leads to trouble, IME).
But those are just the reality. I have found,though, that (unless you're doing reporting), that problem simply doesn't happen a lot. It happens at least once for most every table, but it doesn't happen over and over and over. And, naturally, once you have it once, you can either "cut and paste" your way to glory, or refactor it (i.e. sql = "select " + MemberDAO.getAllColumns() + ", " + PostDAO.getAllColumns() + " from member m, post p").
Now, I like JPA and ORMs, I find them useful, but I also find them a PITA. There is a definite love/hate relationship going on there. And when things are going smooth, boy, is it smooth. But when it gets rocky -- hoo boy. Then it can get ugly. As a whole, however, I do recommend them.
But if you're looking for a "lightweight" non-framework, this technique is useful, practical, low overhead, and gives you a lot of control over your queries. There's simply no black magic or dark matter between your queries and your DB, and when things don't work, it's not some arcane misunderstanding of the framework or edge case bug condition in someone elses 100K lines of code, but rather, odds are, a bug in your SQL -- where it belongs.
Edit: Nevermind. While searching for a solution to my own problem, I forgot to check the date on this thing. Sorry. You can ignore the following.
#millermj - Are you doing that for fun, or because there's a need? Just curious, because that sounds exactly like what Java IDEs like Eclipse and NetBeans already provide (using the Java Persistence API) with New->JPA->Entity Classes from Tables functionality.
I could be missing the point, but if someone just needs classes that match their tables and are persistable, the JPA plus some IDE "magic" might be just enough.