Since I'm not really proficient with databases, some details may be irrlevant, but I'll include everything:
As part of a project in my University, we're creating a website that uses JSP, servlets and uses a MySQL server as backend.
I'm in charge of setting up the tables on the DB, and creating the Java classes to interact with it. However, we can only connect to the MySQL server from inside the University, while we all (7 people) work mostly at home.
I'm creating an interface QueryHandler which has a method that takes a string (representing a query) and returns ResultSet. My question is this: How do I create a class that implements this interface which will simulate a database and allow others to use different DBHandlers and not know the difference and allow me to test different queries without connecting to the actual MySQL database?
EDIT: I'm not so sure on the differences between SQL databases, but obviously all the queries I run on MySQL should run on the mock.
Why not just install your own MySQL database for testing? It runs on Windows, Mac and Linux, and it's not too resource heavy. I have it installed on my laptop for local testing.
Your API appears to be flawed. You should not be returning ResultSets to clients. By doing so, you are forever forcing your clients to rely on a relational database backend. Your data access layer needs to hide all of the details of how your data is actually structured and stored.
Instead of returning a ResultSet, consider returning a List or allowing the client to supply a Stream that your data access component can write to.
This will make unit tests trivial for the clients of the API and will allow you to swap storage mechanisms at will.
Try derby. It's a free server you can use to test against, if you don't mind having to change drivers when you go back to SqlServer. You might be limited in the kind of queries you can run though. I'm not sure if SqlServer has any special syntax outside of standard SQL.
How about using a HSQLDB for offline tests? It wont behave exactly like a MySQL DB but is a fast in memory SQL DB that should satisfy most of your needs.
The best way in my experience is multiple database instances and or schemas. Normally you'd have one for each user to do their development against/sanity checking the running application, one for an automated build for running unit tests and ideally one for each user to run their unit tests against. And of course instances/schemas for demos, integration testing. Apart from the practial side, being able to do this ensures deploying/upgrading the app/database will be pretty near faultless too.
Assuming you have a DAO layer, the only code that needs access to a real database at the unit test level is the DAO implementation, the business layer should be using a mock DAO implementation.
Related
For our applications we have created a series of healthchecks. One of these health checks checks whether or not a MySql database is up and running. In order to unit test this functionality I need to mock or simulate a database and run the health check against it.
To establish a connection to the MySql database being checked, we are using DriverManager.getConnection(). So we first tried to mock the return value of getConnection(). However, that is a static method and Mockito is unable to mock it. Then we turned to Powermock because it is able to mock static methods. Unfortunately, we found it to be poorly supported with JUnit5 and we had to clutter up our pom with a lot of dependencies. Not to mention the code was becoming overly complicated. In the end, we decide to look for other options besides mocking.
Basically, our health check just needs to make sure that the MySQL url, username, and password are correct and the database will accept a connection. Ideally, our unit tests would reflect these requirements.
So, our next idea is to use an In-Memory database, like this one from Baeldung. This seems like a great option but I'm worried it won't behave in the same way a real MySQL database will. Is there a way to create an In-memory MySQl database and establish a connection with DriverManager.getConnection()? This would effectively simulate our real life health check
One option is to use the mysql jcabi maven plugin: mysql.jcabi.com
Please note version > 0.7 doesn't work on Windows.
This will actually setup an instance of MySQL, so it allows for one to test very mysql specific stuff, though the first run is slow.
So, a couple of questions will help drive your answer: are you doing anything highly MySQL specific? Do you want your app to be portable to other DBs?
If the answer is yes to the first one and no to the second one, probably use the jcabi plugin.
If the answer is no to the first, and yes to the second, you're probably pretty safe using H2.
If the answer is yes to both, you've got bigger problems :).
I generally try to keep my code away from DB vendor specificity and use libraries like JPA to abstract this. I can then assume that JPA (and impl) has tested across the supported DBs, and I'm good enough to run unit and integration tests with an in-memory DB like H2.
I generally also have some end-to-end testing in a prod-like env with the same DB setup as prod, in order to cover any vendor-specific quirks.
I am trying to understand "changing database without changing code". Currently working with micro services using springboot, java, thymeleaf and cloud foundry.
I have a spring boot application and attached a database as a service using cloud foundry.
My problem is I am seeing that the purpose of micro service is allowing the ease to change services without changing code.
Here is where I got stuck
In java I have a sql script, "select * from ORDER where Status = 'ACCEPTED';"
Images source
My database would be attached as a service on cloud foundry using CUPS
"jdbc:oracle:thin:username/password//host:port/servicename"
So let say I want to change this database to CUSTOMER table(take it as a different database). This will throw an error because CUSTOMER table will not have "select * from ORDER where Status = 'ACCEPTED';"
I've changed database, but wouldn't I still have to go back to my code and change the sql script?
My Attempt to resolve this issue
So instead of hard coding my sql script in java "select * from ORDER where Status = 'ACCEPTED';"
I created a system environment variable and set it as sqlScript with value of select * from ORDER where Status = 'ACCEPTED'
Then in java I called the env variable String sqlScript= System.getenv("sqlScript");
So now instead of going back into java to change sql script, user can change it through environment variables.
this is a very dirty method to go around my issue, what would be a better alternative?
I know my logic of understanding is really wrong. Please guide me to the right path.
I think the phrase 'changing database without changing code' doesn't mean that if you add/remove fields in DB you do not have to modify your codebase - it just doesn't make any sense.
What it really means is that you should use good database abstractions, so in case you need to change your database vendor from, let's say, MYSQL to OracleDB your Java code should stay the same. The only thing that may differ is some configurations.
A good example of it is ORM like Hibernate. You write your java code once, no matter what is the SQL Database that you are using underneath. To switch databases the only thing that you need to change is a dialect configuration property (In reality it's not that easy to do, but probably easier than if we were coupled to a one specific DB).
Hibernate gives you a good abstraction over SQL databases. Nowadays we have a new trend - having the abstraction over different DB families like SQL and NoSQL. So in the ideal world, your codebase should stay unchanged even if you want to change MySQL to MongoDB or even Neo4j. Spring Data probably is the most popular framework that tries to solve this problem. Another framework that I found recently is Kundera but I haven't used it so far.
So answering your question - you do not need to keep your SQL queries as system variables. All you need to do is to use proper abstractions in your language of choice.
In my opinion, it would be better to use something like Flyway or Liquibase, which are integrated really well in Spring Boot. You can find more information here.
I prefer Liquibase, since it uses a higher level format to describe your database migrations, allowing you to switch databases quite easily. This way, you can also use different databases per environment, for example:
HSQLDB during local development
MySQL in DEV and TEST
Oracle in Production
It's also possible to export your current database schema from an existing database to have an initial version in Flyway or Liquibase, this will give you a good baseline for your scripts.
This question is extracted from a comment I posted here:
What's the best strategy for unit-testing database-driven applications?
So I have a huge database schema for a legacy application (with quite an old code base) that has many tables, synonyms, triggers, and dblinks. We and we have (finally) started to test some part of the application.
Our tests are already using mocks, but in order to test the queries that we are using we have decided to use an in-memory db with short-lived test dataset.
But the setup of the in-memory database requires a specific SQL script for the db schema setup. The script is not the real DDL we have in production because we can not import it directly.
To make things harder, the database contains functions and procedures that needs to be implemented in Java (we use the h2 db, and that is the way to declare procedures).
I'm afraid that our test won't break the day the real db will change and we will spot the problem only at runtime, potentially in production.
I know that our tests are quite at the border between integration and unit. However with the current architecture it is quite hard to insulate the test from the db. And we want to have proper tests for the db queries (no ORM inside).
What would be solution to have a DDL as close as possible of the real one and without the need to manually maintain it ?
If your environments are Dockerized I would highly suggest checking out Testcontainers (https://www.testcontainers.org/modules/databases/). We have used it to replace in-memory databases in our tests with database instances created from production DDL scripts.
Additionally, you can use tmpfs mounting to get performance levels similar to in-memory databases. This is nicely explained in following post from Vlad Mihalcea: https://vladmihalcea.com/how-to-run-integration-tests-at-warp-speed-with-docker-and-tmpfs/.
This combination works great for our purposes (especially when combined with Hibernate auto-ddl option) and I recommend that you check it out.
I am developping an application that tests different WebServices, and I want it to be as generic as possible. I need to populate database to do JUnit tests, but I don't want these changes to be commited.
I know that some in-memory databases like HSQL DB allow testing on a sort of a virtual (or mock) database, but unfortunately I use oracle and I cannot change it now because of my complex data tables structure.
What is the best practice you suggest?
Thanks.
First of all, HSQL and Hibernate aren't related in any way. The question is whether you can find an embedded database which supports the same SQL as your production database (or rather the subset of SQL which your application uses).
A good candidate for this is H2 database since it emulates a lot of different SQL flavours.
On top of that: Don't test the database. Assume that the database is tested thoroughly by your vendor and just works.
In my code, I aim for:
Save and load each entity.
Generate the SQL for all the queries that I use and compare them against String literals in tests (i.e. I don't run the queries against the database all the time).
Some tests look for a System property. If it's set, then they will run the queries against the database. This happens during the night on my CI server.
The rationale for this: As long as the DB schema doesn't change, there is no point to actually run the queries. That means running them during the day while I sit in front of the computer is a huge waste of time.
To make sure that "low impact" changes don't slip through the gaps, I let a computer run them when I don't care.
Along the same lines, I have mocks for many DAOs which return various predefined results, so I don't have to query the database. The rationale here is that I want to test the processing of results from the database, not the JDBC API, the DB driver, the OS's TCP/IP stack, the network hardware (and software), or any other of the 1000 things between my code and the database records on a harddisk somewhere.
More details in my blog: http://blog.pdark.de/2008/07/26/testing-with-databases/
Our development databases (Oracle 9i) use a remote database link to a remote shared database.
This decision was made years ago when it wasn't practical to put some of the database schemas on a development machine - they were too big.
We have certain schemas on the development machines and we make the remote schemas look local by using Oracle's database links, together with some synonyms on the development machines.
The problem I have is that I would like to test a piece of SQL which joins tables in schemas on either side of the database link.
e.g. (a simplified case):
select a.col, b.col
from a, b
where a.b_id = b.id
a is on the local database
b is on the remove database
I have a synonymn on the locale DB so that 'b' actually points at b#remotedb.
Running the query takes ages in the development environment because of the link. The queries run fine in production (I don't think the Oracle cost based optimiser can cope very well with database links).
We have not been very good at writing unit tests for these types of queries in the past - probably due to the due to the poor performance - so I'd like to start creating some tests for them.
Does anyone have any strategies for writing a unit test for such a query, so as to avoid the performance problems of using the database link?
I'd normally be looking at ways of trying to mock out remote service, but since all this is in a SQL query, I can't see anyway of easily mocking out the remove database.
You should create exact copies of all the schema you need from production on development but without all the data. You should populate the schema with enough data so you can do a proper test. You can also manipulate the optimizer to behave on the test system to be like production by exporting the statistics from the production server and importing them to the development database for the schemas you are duplicating. That way the query will run with the data set you've made but the query will optimize with plans that is similar to that of production. Then you can estimate theoretically how it will scale on production.
Copy the relevant data into your development database and create the tables locally.
Ideally, just build a test case which tells you:
The SQL is correct (it parses)
It operates correctly with a few rows of test data
Don't fall for the "let's copy everything" because that means you'll have no idea what you're testing anymore (and what you're missing).
If in doubt, create a table b with just a single record. If you get an error in this area, add more rows as you learn where it can fail.
If you want to take this to the edge, create the test table (with all data) in a unit test. This way, you can document the test data you're using.
[EDIT] What you need is a test database. Don't run tests against a database which can change. Ideally, the tests should tear down the whole database and recreate it from scratch (tables, indexes, data, everything) as the first step.
In this test database, only keep well defined test data that only changes by defining new tests (and not by someone "just doing something"). If you can, try to run your tests against an in-memory database.
I would suggest materialized views. These are views that store remote data locally.
In theory to do the unit-testing you can work with any set of controlled data created and designed based on your test-cases. It doesn't have to be your live or development system. That's assuming your unit is portable enough. You would test it with your current databases/application when you come to integration testing, which might as well be on the live system anyway (so no db links will be required - I understand your live databases are in one place).
What I'm trying to say, is that you can/should test your unit (i.e. your component, query or whatever you define as a unit) on a controlled set of data that would simulate different 'use cases' and once you complete your testing to satisfactory results, then you can proceed to integration + running integration tests.
Integration tests - you could run this in the live environment, but only after you've proved by unit-testing that your component is 'bullet-proof' (if that's OK with your company's approach/philosophy :) - sys admin's reaction:"Are you flippin creazy?!")
If you are trying to go back in time and test already implemented units, then why bother? If they've been in a production use for some time without any incidents then I would argue that they're OK. However, there's always a chance that your unit/query might have some 'slowly ticking time bomb' effect on the side (cumulative effect over time). Well, analyse the impact is the answer.