Develop Reference

MySql table listener in Java

Here is my requirement:
My java service is continuous running process. When the java service start, it loads the configuration from one of the table in MySql database.
When there are any changes (can be insert/update/delete, outside the java service also possible), java service should reloaded the configuration immediately.
I saw a post in stackoverflow which is for Oracle database, but I am looking for Mysql database.
I can able to achieve this using separate thread, which polling the database table with some interval. But, polling for change is a not a good way to do. Instead, I am looking for any Watcher/Listener which will notify when there is any change in the MySql table.
Can anybody help me to achieve this?
Note: If this question is already answered somewhere, please post the link in the comment, I will remove this question.

You want to do a Change Data Capture system. Perhaps you could use one based on Kafka.
You can use kafka-connect with Debezium on it. This connector first snapshot your table, then read the mysql binlog to be able to have a consistent kafka topic with insert/modify/delete on the table.
If you don't want use it, perhaps you can fork the code to use the same thing as they do, just modifying the output to be your application listener.

Microservice - What does changing service without changing code really mean?

I am trying to understand "changing database without changing code". Currently working with micro services using springboot, java, thymeleaf and cloud foundry.
I have a spring boot application and attached a database as a service using cloud foundry.
My problem is I am seeing that the purpose of micro service is allowing the ease to change services without changing code.
Here is where I got stuck
In java I have a sql script, "select * from ORDER where Status = 'ACCEPTED';"
Images source
My database would be attached as a service on cloud foundry using CUPS
"jdbc:oracle:thin:username/password//host:port/servicename"
So let say I want to change this database to CUSTOMER table(take it as a different database). This will throw an error because CUSTOMER table will not have "select * from ORDER where Status = 'ACCEPTED';"
I've changed database, but wouldn't I still have to go back to my code and change the sql script?
My Attempt to resolve this issue
So instead of hard coding my sql script in java "select * from ORDER where Status = 'ACCEPTED';"
I created a system environment variable and set it as sqlScript with value of select * from ORDER where Status = 'ACCEPTED'
Then in java I called the env variable String sqlScript= System.getenv("sqlScript");
So now instead of going back into java to change sql script, user can change it through environment variables.
this is a very dirty method to go around my issue, what would be a better alternative?
I know my logic of understanding is really wrong. Please guide me to the right path.

I think the phrase 'changing database without changing code' doesn't mean that if you add/remove fields in DB you do not have to modify your codebase - it just doesn't make any sense.
What it really means is that you should use good database abstractions, so in case you need to change your database vendor from, let's say, MYSQL to OracleDB your Java code should stay the same. The only thing that may differ is some configurations.
A good example of it is ORM like Hibernate. You write your java code once, no matter what is the SQL Database that you are using underneath. To switch databases the only thing that you need to change is a dialect configuration property (In reality it's not that easy to do, but probably easier than if we were coupled to a one specific DB).
Hibernate gives you a good abstraction over SQL databases. Nowadays we have a new trend - having the abstraction over different DB families like SQL and NoSQL. So in the ideal world, your codebase should stay unchanged even if you want to change MySQL to MongoDB or even Neo4j. Spring Data probably is the most popular framework that tries to solve this problem. Another framework that I found recently is Kundera but I haven't used it so far.
So answering your question - you do not need to keep your SQL queries as system variables. All you need to do is to use proper abstractions in your language of choice.

In my opinion, it would be better to use something like Flyway or Liquibase, which are integrated really well in Spring Boot. You can find more information here.
I prefer Liquibase, since it uses a higher level format to describe your database migrations, allowing you to switch databases quite easily. This way, you can also use different databases per environment, for example:
HSQLDB during local development
MySQL in DEV and TEST
Oracle in Production
It's also possible to export your current database schema from an existing database to have an initial version in Flyway or Liquibase, this will give you a good baseline for your scripts.

Migrating from Apache Derby to MySQL

I have a JDBC application that uses Apache Derby. How can I migrate my entire database system to use MySQL?
I have 3 Java programs that access the database
I have 3 tables and 2 views
I am using Netbeans. I have never used MySQL before and do not know where to begin. Is there nice integration with Java and MySQL in Netbeans? How can I get nice integration with NetBeans and MySQL?
All help is greatly appreciated!

Looks like this plugin would probably help you:
http://netbeans.org/kb/docs/ide/mysql.html

I found this tutorial on the Spring site, but I think it is only a partial solution.
Tutorial
In it they are relying on hibernate to drop and create the tables, and I really don't like that. You have to go through special coding to add static data. For example, if your app is tracking devices, you probably want a table of device_types. At least some o those device types will be in the db, as well as devices, users, etc.
What I intend to do, is to use Derby until I am somewhat stable. From it, I will get the database schema and create it in mysql. It seems that the DB look utility can be used for that. DB Look
As added security I intend to run my web app with a db user that does not have the ability to add or drop tables. Also it is possible to remove the permission to delete rows if you use the concept of making rows "inactive" So instead of deleting a no longer used device type, you set the "active" flag to F. So your device type query would look like:
select * from device_type where active = 'T'

The simplest Java SQL provider?

I need SQL functionalities for a Java JSE application, but dont need a whole SQL server, with things like listen on a port, connection string or even a standalone process to be runned or configured.
I also would prefer to work with files as storages, so that file path identifies DB data.
So given to the DB API the name of a file, I would need perform SQL with the file as DB storage, supporting tables, search, joins and inserts, without thinking to things like ports, external processes, server installation, ecc..
Without any other configurration action, since any other configurable feature is not needed.
Is there some library, preferrably installable as single .jar, that provides this functionality?
If there is not this library, which file-based DB is the simplest to configure and use within JSE, and which configuration steps are needed to perform a query in the provided DB and deploy it with (working) the java .jar application?

I suggest Derby db.apache.org/derby/

I like H2 Database very much. It compares very well with other database engines.

Sure, Hypersonic SQL or Derby, the database that comes bundled with Java 7, will both fill the bill. SQLite would be a third alternative.

you may want to check this out:
http://www.sqlite.org/

I have used HSQLDB in the past and liked it. Depending on what your needs are, you might also be interested in JoSQL which allows you to do SQL type of queries on java collections.

Getting events from a database

I am not very familiar with databases and what they offer outside of the CRUD operations.
My research has led me to triggers. Basically it looks like triggers offer this type of functionality:
(from Wikipedia)
There are typically three triggering events that cause triggers to "fire":
INSERT event (as a new record is being inserted into the database).
UPDATE event (as a record is being changed).
DELETE event (as a record is being deleted).
My question is: is there some way I can be notified in Java (preferably including the data that changed) by the database when a record is Updated/Deleted/Inserted using some sort of trigger semantics?
What might be some alternate solutions to this problem? How can I listen to database events?
The main reason I want to do this is a scenario like this:
I have 5 client applications all in different processes/existing across different PCs. They all share a common database (Postgres in this case).
Lets say one client changes a record in the DB that all 5 of the clients are "interested" in. I am trying to think of ways for the clients to be "notified" of the change (preferably with the affected data attached) instead of them querying for the data at some interval.

Using Oracle you can setup a Trigger on a table and then have the trigger send a JMS message. Oracle has two different JMS implementations. You can then have a process that will 'listen' for the message using the JDBC Driver. I have used this method to push changes out to my application vs. polling.
If you are using a Java database (H2) you have additional options. In my current application (SIEM) I have triggers in H2 that publish change events using JMX.

Don't mix up the database (which contains the data), and events on that data.
Triggers are one way, but normally you will have a persistence layer in your application. This layer can choose to fire off events when certain things happen - say to a JMS topic.
Triggers are a last ditch thing, as you're operating on relational items then, rather than "events" on the data. (For example, an "update", could in reality map to a "company changed legal name" event) If you rely on the db, you'll have to map the inserts & updates back to real life events.... which you already knew about!
You can then layer other stuff on top of these notifications - like event stream processing - to find events that others are interested in.
James

Hmm. So you're using PostgreSQL and you want to "listen" for events and be "notified" when they occur?
http://www.postgresql.org/docs/8.3/static/sql-listen.html
http://www.postgresql.org/docs/8.3/static/sql-notify.html
Hope this helps!

Calling external processes from the database is very vendor specific.
Just off the top of my head:
SQLServer can call CLR programs from
triggers,
postgresql can call arbitrary C
functions loaded dynamically,
MySQL can call arbitrary C functions,
but they must be compiled in,
Sybase can make system calls if set
up to do so.

The simplest thing to do is to have the insert/update/delete triggers make an entry in some log table, and have your java program monitor that table. Good columns to have in your log table would be things like EVENT_CODE, LOG_DATETIME, and LOG_MSG.
Unless you require very high performance or need to handle 100Ks of records, that is probably sufficient.

I think you're confusing two things. They are both highly db vendor specific.
The first I shall call "triggers". I am sure there is at least one DB vendor who thinks triggers are different than this, but bear with me. A trigger is a server-side piece of code that can be attached to table. For instance, you could run a PSQL stored procedure on every update in table X. Some databases allow you to write these in real programming languages, others only in their variant of SQL. Triggers are typically reasonably fast and scalable.
The other I shall call "events". These are triggers that fire in the database that allow you to define an event handler in your client program. IE, any time there are updates to the clients database, fire updateClientsList in your program. For instance, using python and firebird see http://www.firebirdsql.org/devel/python/docs/3.3.0/beyond-python-db-api.html#database-event-notification
I believe the previous suggestion to use a monitor is an equivalent way to implement this using some other database. Maybe oracle? MSSQL Notification services, mentioned in another answer is another implementation of this as well.
I would go so far as to say you'd better REALLY know why you want the database to notify your client program, otherwise you should stick with server side triggers.

What you're asking completely depends on both the database you're using and the framework you're using to communicate with your database.
If you're using something like Hibernate as your persistence layer, it has a set of listeners and interceptors that you can use to monitor records going in and out of the database.

There are a few different techniques here depending on the database you're using. One idea is to poll the database (which I'm sure you're trying to avoid). Basically you could check for changes every so often.
Another solution (if you're using SQL Server 2005) is to use Notification Services, although this techonology is supposedly being replaced in SQL 2008 (we haven't seen a pure replacement yet, but Microsoft has talked about it publicly).

This is usually what the standard client/server application is for. If all inserts/updates/deletes go through the server application, which then modifies the database, then client applications can find out much easier what changes were made.

If you are using postgresql it has capability to listen notifications from JDBC client.

I would suggest using a timestamp column, last updated, together with possibly the user updating the record, and then let the clients check their local record timestamp against that of the persisted record.
The added complexity of adding a callback/trigger functionality is just not worth it in my opinion, unless supported by the database backend and the client library used, like for instance the notification services offered for SQL Server 2005 used together with ADO.NET.