Develop Reference

Does deleting from a table of a h2 database handled by Hibernate corrupts the table?

Here a quick description of the system:
A java 7 REST client receives jsons and write their parsed content into an h2 database via Hibernate.
Some Pentaho Kettle Spoon 4 ETLs directly connect to the same database to read and delete a lot of entries at once.
This solution worked fine in our test environment, but in production (where the traffic is really higher because of course it is) the ETLs are often failing with the following error
Error inserting/updating row
General error: "java.lang.ArrayIndexOutOfBoundsException: -1"; SQL statement:
DELETE FROM TABLE_A
WHERE COLUMN_A < ? [50000-131]
and if I navigate the database I can indeed see that that table is not readable (apparently because it thinks its lenght is -1?). The error code 50000 is for "Generic" so is no use.
Apart from the trivial "maybe h2 is not good for an Event Handler", I've been thinking that the corruption could possible be caused by a confict between Kettle and Hibernate, or in other words that no one should delete from an Hibernate handled database without him knowing.
My questions to those more experienced then me with Hibernate are:
Is my sopposition correct?
Should I re-design my solution to also use the same restful Hibernate to perform deletes?
Should I resign using h2 for such a system?
Thanks for the help!
EDIT:
The database is created by a simple sh script that runs the following command that basically uses the provided Shell tool to connect to a non existing db which by defalts creates it.
$JAVA_HOME/bin/java -cp *thisIsAPath*/h2database/h2/main/h2-1.3.168-redhat-2.jar org.h2.tools.Shell -user $DB_USER -password $DB_PASSWORD -url jdbc:h2:$DB_FOLDER/Temp_SD_DS_EventAgent<<END
So all its parameters are set to version 1.3.168's defaults. Unfortunately while I can find the current URL setting I can't find where to look for that version's defauts and experimentals.
I also found the followings:
According to the tutorial When using Hibernate, try to use the H2Dialect if possible. which I didn't.
The tutorial also says Please note MVCC is enabled in version 1.4.x by default, when using the MVStore. Does that mean cuncurrency is disabled/unsupported by default in this older case and this is the problem?
The database is created with h2 version 1.3.168 but the consumer uses 1.4.197. Is this a big deal?

I cannot comment on the credibility of h2 db.
But from application perspective, I think you should use locking mechanism - Optimistic or Pessimistic lock. This will avoid the conflict situations. Hope this answer helps to point in correct direction
Article on Optimistic and Pessimistic locking

What's the recommended way to migrate from H2 1.3.175 to 1.4.195

Now that H2 1.4 is out of beta, I'd like to migrate my old 1.3.175 database to 1.4.195.
Background info:
In the docs, database upgrade does not mention 1.4 yet.
The roadmap still lists "Automatic migration from 1.3 databases to 1.4." as "planned change".
The current state of MVStore is still labeled as "experimental".
So, what's the recommended way to migrate?
Additional aspects/bonus questions:
Should I enable MVStore or stick with PageStore (pros/cons)? Which one delivers better performance (multithreading is not important for me), which one better stability, especially resilience against OutOfMemoryErrors?

A database created with 1.3.175 can be read and opened with 1.4.195 without any additional work. H2 will automatically detect that it is using the Page Store and treat it as such. There will be no problems with doing this.
The advantage to doing this is that while the MVStore was being developed, the Page Store continued to receive performance improvements and bug fixes. Consequently H2 with the Page Store has become an extremely stable database store.
There is as yet no automatic upgrade procedure for converting a database from using the Page Store to using the MVStore. If you do want to do this, you'll need to do it manually. With the latest H2 Jar, use H2's SCRIPT command to export SQL from your 1.3 database, then use RUNSCRIPT into a freshly created db with 1.4.195.
If your H2 JDBC URL doesn't explicitly specify ;mv_store=false, note that H2 will first look to see if a page store database already exists. If it doesn't then it will create an MVStore database. This will appear seamless to you, your app, and your users. The only superficial difference you'll notice is that the database file on disk has a different file extension.
Finally, a suggestion. If your customer databases are large, consider only using the page store. I'm a heavy user of H2. (My commercial product built on H2 has thousands of users who typically have databases many gigabytes in size.) I still use the page store for all my customers, even though I use the latest H2 Jar. There are still some performance issues with the MVStore that start to appear as databases get large. With time, I expect the cause of the problems to be identified and fixed.

#Steve McLeod's answer is on point. For completeness, here are the exact commands:
//Do a backup of current .h2.db file
//Connect to current DB with same URL as always
SCRIPT TO 'fileName'
//Rename the .h2.db to something else, possibly another backup
//Connect to database with same URL as before. The new MVStore engine will be chosen by default, and the .mv.db file will be created
RUNSCRIPT FROM 'fileName'
Documentation, H2 Grammar
Moreover, if you prefer using the H2 jar for this, refer to Thomas's answers (1 and 2). Concretely, the corresponding classes are org.h2.tools.Script and org.h2.tools.RunScript

To highlight another alternative for similar requests I would like to mention a tool which allows an automatized migration of an old H2 database into a new H2 database:
https://github.com/manticore-projects/H2MigrationTool

Java Embedded Derby Database Plugin?

I am developing a very large scale J2EE application and we chose to use Derby as an embedded database for junit testing since hitting the actual prod database will slow down our tests. When I bootstrap my application, the Derby DB will create all the tables so I can run JDBC queries against it. It works fine but the drawback is I cannot actually query any of the tables except through JDBC calls at runtime, so if I need to make changes to my queries, I need to stop the app, modiify my query statements, then restart the application and run in debug. This process makes it very difficult when it comes to analyzing complex queries. Does anyone know of some kind of Derby plugin that can help me to query the DB without doing it through my java code?

If you are using Maven for your build, you can use the derby-maven-plugin, which I wrote and is available on GitHub and via Maven Central. It will take care of starting and stopping the database for you before your tests. You will need to populate this database, yourself of course. You will also have the database in your target/derby folder after the tests execute, so you can always query the data yourself afterwards. This will help you work in a separate development environment which doesn't affect the production database.
You can check here for my answer to a similar question.

Most MySQL-like hibernate-compatible in-memory database?

For a project I am working on(Spring/struts 2/hibernate), we decided to use h2 for unit testing with MySQL for the production store and manage the scheme in liquibase, pretty standard fare, but the issue we keep on running into is that h2 and MySQL differ in a lot of ways, for example how they handle timestamps and triggers. It's getting to the point that I am starting to regret using h2 as the extra headaches the mis-matches are causing are starting to outweigh its benefits. My question is this, is there any other in-memory/local file database that behaves more like MySQL? Obviously for integration testing we will still use MySQL, but being able to do unit testing without either making the liquibase files into a giant hack or having to ensure the local MySQL db is running would be nice.

I don't think there is another in-memory Java database that is more compatible to MySQL than H2. If you have a lot of code that only works with MySQL, then you should probably also use MySQL for testing.
Just be aware that it will be difficult in the future to switch to another database. Relying too much on features of one product will result in a "vendor lock in". In case of MySQL at least you have the option to switch to MariaDB, so it's not all that bad.

You may use a ram drive, copy your testing tables and datas into that drive, and start your mysql configured to load from that drive, all that in a script at boot time.
Then your unit tests will run insanely faster. We used it for developpers workstations and the level of frustation went three steps down.

I think that as of right now the correct approach is to use MySQL as a Docker image.
Once you create the image you can easily spin it up from your tests, and it's going to take seconds. Your hibernate will dynamically initialize DB schema and there you go!
The only issue is that CI servers need to have Docker installed.

Large ResultSet on postgresql query

I'm running a query against a table in a postgresql database. The database is on a remote machine. The table has around 30 sub-tables using postgresql partitioning capability.
The query will return a large result set, something around 1.8 million rows.
In my code I use spring jdbc support, method JdbcTemplate.query, but my RowCallbackHandler is not being called.
My best guess is that the postgresql jdbc driver (I use version 8.3-603.jdbc4) is accumulating the result in memory before calling my code. I thought the fetchSize configuration could control this, but I tried it and nothing changes. I did this as postgresql manual recomended.
This query worked fine when I used Oracle XE. But I'm trying to migrate to postgresql because of the partitioning feature, which is not available in Oracle XE.
My environment:
Postgresql 8.3
Windows Server 2008 Enterprise 64-bit
JRE 1.6 64-bit
Spring 2.5.6
Postgresql JDBC Driver 8.3-603

In order to use a cursor to retrieve data you have to set the ResultSet type of ResultSet.TYPE_FORWARD_ONLY (the default) and autocommit to false in addition to setting a fetch size. That is referenced in the doc you linked to but you didn't explicitly mention that you did those steps.
Be careful with PostgreSQL's partitioning scheme. It really does very horrible things with the optimizer and can cause massive performance issues where there should not be (depending on specifics of your data). In any case, is your row only 1.8M rows? There is no reason that it would need to be partitioned based on size alone given that it is appropriately indexed.

I'm betting that there's not a single client of your app that needs 1.8M rows all at the same time. You should think of a sensible way to chunk the results into smaller pieces and give users the chance to iterate through them.
That's what Google does. When you do a search there might be millions of hits, but they return 25 pages at a time with the idea that you'll find what you want in the first page.
If it's not a client, and the results are being massaged in some way, I'd recommend letting the database crunch all those rows and simply return the result. It makes no sense to return 1.8M rows just to do a calculation on the middle tier.
If neither of those apply, you've got a real problem. Time to rethink it.
After reading the later responses it sounds to me like this is more of a reporting solution that ought to be crunched in batch or calculated in real time and stored in tables that are not part of your transactional system. There's no way that bringing 1.8M rows to the middle tier for calculating moving averages can scale.
I'd recommend reorienting yourself - start thinking about it as a reporting solution.

The fetchSize property worked as described at postgres manual.
My mistake was that I was setting auto commit = false to a connection from a connection pool that was not the connection being used by the prepared statement.
Thanks for all the feedback.

I did everything above, but I needed one last piece: be sure the call is wrapped in a transaction and set the transaction to read only, so that no rollback state is required.
I added this: #Transactional(readOnly = true)
Cheers.