Is there any way to test that SQL scripts contain standard SQL with java/junit tests?
Currently we have sql scripts for creating a database etc. in a Postgres db, but when using hsqldb everything fails. That's why I wonder if any java tools exist for testing if sql statements are standard sql.
Or would it just be wise to create different sets of scripts per database vendor?
If so, is there a way to test if a given script works with postgres/hsqldb?
The H2 database supports different modes, which may help you with postgres testing, I've found that our sql often contains functions which are not supported but H2, but you can create your own "stored procedures" which actually invoke a static Java method to work around this. If you want to support different database vendors you should go down the vendor specific script route, unless you are doing really basic queries.
If you have the available resources I would recommend setting up a fully fledged UAT environment which you can use to test against a live postgres database, as even seemingly minor db configuration differences can impact query plans in unexpected ways.
I've usually made a very simple java-wrapper that tests this code
by using a localhost-connection with some standard user/pass settings.
Remember to use a temporary database or a known test-database so your tests
doesn't destroy anything important.
Reason for above is that I have had the need for specific databases (non standard
features etc).
If you only want to test standard sql-stuff for junit tests (like syntax, selects etc),
I would consider using a embedded sql database in java (ususally memory only).
That way it is easy to test lots of stuff without the need to install a db
and also without the risk of destoring other installations.
It sounds like you're looking for an SQL syntax parser and validator. The only Java SQL parser with which I'm familiar is Zql, but I've never actually used it.
A similar question was asked early last year, and the best answer there turned out to be writing your own parser with ANTLR.
The best tool for checking SQL statements for conformance to Standard SQL is HSQLDB 2.0. This is especially true with data definition statements. HSQLDB has been written to the Standard, as opposed to adopting bits of the Standard over several versions.
PostgresSQL has been slowly moving towards standard SQL. But it still has some "legacy" data types. HSQLDB allows you to define types with the CREATE TYPE statement for compatibility with other dialects. It also allows you to define functions in SQL or Java for the same purpose.
The best policy is to use standard SQL as much as possible, with user-defined types and functions to support an alternative database's syntax.
ZQL works fine with Junit..
ZqlParser parser = new ZqlParser(any input stream);
try{
parser = parser.readStatement();
}catch(ParseException e){
// if sql is not valid it caught here
}
Different vendors expose different additional features in their SQL implementation.
You may decide the set of databases to test. Then use http://dbunit.sourceforge.net to simplify the testing job.
Related
I'm trying to convert some Oracle queries using PL/SQL to jOOQ. Most package queries, stored procedures, etc. are easy, using the code generator. However there's one feature used in several places for which I haven't found a jOOQ alternative:
begin
MY_SCHEMA.MY_PACKAGE.MY_QUERY(some_param => MY_SCHEMA.MY_PACKAGE.SOME_CONSTANT)
-- more code
end;
I can call the query just fine, but I'm not sure how to pass the MY_SCHEMA.MY_PACKAGE.SOME_CONSTANT value into it. The jOOQ code generator doesn't seem to generate anything for the constant (at least, I can't find anything similarly named). Do I need to enable a feature on the generator? Or do I need to query those constants? If so, how?
Enabling PL/Scope for this to work
jOOQ can generate code for your package constants if it can find them in your ALL_IDENTIFIERS dictionary view. That's only the case if you enable PLSCOPE_SETTINGS when compiling your packages, e.g. using:
ALTER SESSION SET PLSCOPE_SETTINGS='IDENTIFIERS:ALL'
With that in place, jOOQ will generate expressions for your package constants, which you can use in routine calls, or other procedural logic.
PL/Scope independence
What Simon Martinelli referred to in the comments is issue https://github.com/jOOQ/jOOQ/issues/6504, which attempts to enable this code generation support even without the above PL/Scope setting turned on, because that's quite unreliable depending on your environment.
As of jOOQ 3.15, there's no solution yet that works on any Oracle environment. But you could use testcontainers to generate your jOOQ code from a Docker image that has PL/Scope enabled.
I am trying to understand "changing database without changing code". Currently working with micro services using springboot, java, thymeleaf and cloud foundry.
I have a spring boot application and attached a database as a service using cloud foundry.
My problem is I am seeing that the purpose of micro service is allowing the ease to change services without changing code.
Here is where I got stuck
In java I have a sql script, "select * from ORDER where Status = 'ACCEPTED';"
Images source
My database would be attached as a service on cloud foundry using CUPS
"jdbc:oracle:thin:username/password//host:port/servicename"
So let say I want to change this database to CUSTOMER table(take it as a different database). This will throw an error because CUSTOMER table will not have "select * from ORDER where Status = 'ACCEPTED';"
I've changed database, but wouldn't I still have to go back to my code and change the sql script?
My Attempt to resolve this issue
So instead of hard coding my sql script in java "select * from ORDER where Status = 'ACCEPTED';"
I created a system environment variable and set it as sqlScript with value of select * from ORDER where Status = 'ACCEPTED'
Then in java I called the env variable String sqlScript= System.getenv("sqlScript");
So now instead of going back into java to change sql script, user can change it through environment variables.
this is a very dirty method to go around my issue, what would be a better alternative?
I know my logic of understanding is really wrong. Please guide me to the right path.
I think the phrase 'changing database without changing code' doesn't mean that if you add/remove fields in DB you do not have to modify your codebase - it just doesn't make any sense.
What it really means is that you should use good database abstractions, so in case you need to change your database vendor from, let's say, MYSQL to OracleDB your Java code should stay the same. The only thing that may differ is some configurations.
A good example of it is ORM like Hibernate. You write your java code once, no matter what is the SQL Database that you are using underneath. To switch databases the only thing that you need to change is a dialect configuration property (In reality it's not that easy to do, but probably easier than if we were coupled to a one specific DB).
Hibernate gives you a good abstraction over SQL databases. Nowadays we have a new trend - having the abstraction over different DB families like SQL and NoSQL. So in the ideal world, your codebase should stay unchanged even if you want to change MySQL to MongoDB or even Neo4j. Spring Data probably is the most popular framework that tries to solve this problem. Another framework that I found recently is Kundera but I haven't used it so far.
So answering your question - you do not need to keep your SQL queries as system variables. All you need to do is to use proper abstractions in your language of choice.
In my opinion, it would be better to use something like Flyway or Liquibase, which are integrated really well in Spring Boot. You can find more information here.
I prefer Liquibase, since it uses a higher level format to describe your database migrations, allowing you to switch databases quite easily. This way, you can also use different databases per environment, for example:
HSQLDB during local development
MySQL in DEV and TEST
Oracle in Production
It's also possible to export your current database schema from an existing database to have an initial version in Flyway or Liquibase, this will give you a good baseline for your scripts.
I am working on a project which uses Activiti1 Library and, for these reason, I am using MyBatis API to execute some native queries against Activiti data base.
The problem is: For some builds I am using Oracle and, for other, MySQL database then, as the dialect of this two data bases is different, it would be necessary to use the MyBatis multi vendor support, however, I didn't like it, because I have to deal with specific statments of each type of data base which makes the maintance a hard task, mainly if necessary to add another data base in the future.
So I would like to know if somehow it is possible to use HQL together with MyBatis or if there are another generic SQL engine that can be used in this case.
Or if someone know a free Java API that converts MySQL queries to Oracle queries. I tried to use Hibernate translation, however it didn't work, because Activiti classes is not using JPA, so it is not mapped :(.
Thanks in advance!
Sandro
In Activiti, we do use the multi-vendor support of MyBatis. However, there are only a few cases where we really needed. Do you have that much 'special' queries?
This question is extracted from a comment I posted here:
What's the best strategy for unit-testing database-driven applications?
So I have a huge database schema for a legacy application (with quite an old code base) that has many tables, synonyms, triggers, and dblinks. We and we have (finally) started to test some part of the application.
Our tests are already using mocks, but in order to test the queries that we are using we have decided to use an in-memory db with short-lived test dataset.
But the setup of the in-memory database requires a specific SQL script for the db schema setup. The script is not the real DDL we have in production because we can not import it directly.
To make things harder, the database contains functions and procedures that needs to be implemented in Java (we use the h2 db, and that is the way to declare procedures).
I'm afraid that our test won't break the day the real db will change and we will spot the problem only at runtime, potentially in production.
I know that our tests are quite at the border between integration and unit. However with the current architecture it is quite hard to insulate the test from the db. And we want to have proper tests for the db queries (no ORM inside).
What would be solution to have a DDL as close as possible of the real one and without the need to manually maintain it ?
If your environments are Dockerized I would highly suggest checking out Testcontainers (https://www.testcontainers.org/modules/databases/). We have used it to replace in-memory databases in our tests with database instances created from production DDL scripts.
Additionally, you can use tmpfs mounting to get performance levels similar to in-memory databases. This is nicely explained in following post from Vlad Mihalcea: https://vladmihalcea.com/how-to-run-integration-tests-at-warp-speed-with-docker-and-tmpfs/.
This combination works great for our purposes (especially when combined with Hibernate auto-ddl option) and I recommend that you check it out.
I have an data centric & data sensitive application, which is written using java, but almost all the business logic is maintained in a .sql files.
These sql files are executed 1 by 1 , temporary table is created and updated by these sql files.
Internally these sql files fire update queries on temporary table with available data values on various conditions.
finally the temporary table is dumped into a physical table.
We are planning to move this to java rule based application as sql scripts are getting huge and hard to understand as well as maintain.
Planning to have all the data in memory using Lucene & its RAMDiirectory, what would be the preferred choice for building rules (these are nothing but update queries in sql)
Was looking # scripting languages to have dynamic rules, but scripts (rhino / groovy)
have same characteristics as that of sql files (hard to write & maintain)
Please post your suggestions.
Thanks in advance!!
Our company uses Drools. Works really great for us. Drools normally has you write your rules in an XML-based format, but we just extended some of their classes so we could write our rules in Java (allows us to debug the rules at runtime).
We also use JBoss Rules / Drools.The newer version (> 4.0.0) has a nice DSL that is perfectly readable and maintainable.No more XML is needed.