Develop Reference

Tools to do data processing from Java

I've got a legacy system that uses SAS to ingest raw data from the database, cleanse and consolidate it, and then score the outputted documents.
I'm wanting to move to a Java or similar object oriented solution, so I can implement unit testing, and otherwise general better code control. (I'm not talking about overhauling the whole system, but injecting java where I can).
In terms of data size, we're talking about around 1 TB of data being both ingested and created. In terms of scaling, this might increase by a factor of around 10, but isn't likely to increase on massive scale like a worldwide web project might.
The question is - what tools would be most appropriate for this kind of project?
Where would I find this information - what search terms should be used?
Is doing processing on an SQL database (creating and dropping tables, adding columns, as needed) an appropriate, or awful, solution?
I've had a quick look at Hadoop - but due to the small scale of this project, would Hadoop be an unnecessary complication?
Are there any Java packages that do similar functionality as SAS or SQL in terms of merging, joining, sorting, grouping datasets, as well as modifying data?

It's hard for me to prescribe exactly what you need given your problem statement.
It sounds like a good database API (i.e. native JDBC might be all you need with a good open source database backend)
However, I think you should take some time to check out Lucene. It's a fantastic tool and may meet your scoring needs very well. Taking a search engine indexing approach to your problem may be fruitful.

I think the question you need to ask yourself is
what's the nature of your data set, how often it will be updated.
what's the workload you will have on this 1TB or more data in the future. Will there be mainly offline read and analysis operations? Or there will also have a lot random write operations?
Here is an article talking about if to choose using Hadoop or not which I think is worth reading.
Hadoop is a better choice if you only have daily or weekly update of your data set. And the major operations on the data is read-only operations, along with further data analysis. For the merging, joining, sorting, grouping datasets operation you mentioned, Cascading is a Java library running on top of Hadoop which supports this operation well.

Need for Hibernate in the legacy world [closed]

Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 9 years ago.
Improve this question
I have a several questions about hibernate.
In many questions here in stackoverflow, several people are saying that hibernate is not a good choise for very complex databases. If we have very complex database, hibernate is not the right choice. It better suits for green field projects, but it is not so good for complex legacy database.
Is this true?
Also hibernate generates queries.
Every project manager will like to have optimized queries (hibernate cannot generate more optimized queries than sql specialist!). So for big project it is not a problem to hire sql specialist. The sql specialist will optimize the queries (use explain sql, use joins ...)
My question is how come a huge and expensive project does not care about sql optimization?
(you will say that you can write HQL but as I have seen in a lot of posts that explains that HQL is not so powerful than sql and a lot of programmers get headache and several hours of tuning) (you like all your organs in your body to work ideally don't you?)
Also the second level cache helps hibernate a lot because hibernate knows to generate a lot of queries instead of complex join.
My question is: Is really a complex db only modified by one system (example the web site)? If we are talking about the enterprise system the db can be accessed via several processes, sharing different programming languages and platforms.
So in this case the second level cache does not help very much.
For what kind of projects hibernate is suitable for?
Is it for back office projects where nobody cares about the sql ?
What happens when your administrator says: please use memcached for caching and please use this optimized queries instead of yours?
If you are using oracle database, orache has the most advanced sql syntax. They have spend a lot of time and money on the syntax that is very powerful. What for is this syntax if it is not used.
The software is written only once (and then maintained) and used for a long time.
If I am a company that orders software I will say: I will use the software for a couple of years and I like to be fast, and if you spend 1 month for writing software with hibernate I will pay one more month for software that uses example IBATIS knowing that it will work better for years
(when you are buying a car you are interested in the car economy 1kg-oil/km, not how short and easy the manufacturer produced the car!). So as a software consumer I do not interested in your productivity, just how fast the software is. Of course also the price is relevant but if we are speaking about the price there are more complex mathematics.
Can we call something engineering when we really cannot predict some part of the system?
(can electrical engineer be really a engineer if he cannot predict the current)
Please share your opinion.
Regards

1) (...) Is this true?
No it isn't, Hibernate can deal with pretty complex databases, including existing ones. However, it might not deal very well with an heavily denormalized database or an exotic schema. This is different.
2) (...) My question is how come a huge and expensive project does not care about sql optimization?
This is non-sense, using Hibernate doesn't mean you don't care about optimization. I have worked on a huge and complex STP system (several hundreds millions € budget) and performance was definitely an important concern and we actually introduced Hibernate to benefit from things like lazy loading, second level cache (and speed up development).
Here is the deal when using an ORM like Hibernate (when suitable):
You'll be done faster with ORM than without ORM (or there wouldn't be any point at using them).
The vast majority of the generated queries will behave correctly (and the fact is that Hibernate generates better SQL than the average developer).
You can (and have to) tune queries and Hibernate to a certain degree.
Even if you spend some time on performance optimization (including falling back to native SQL for really problematic queries), you'll still be done faster.
3) (...) So in this case the second level cache does not help very much.
Well, you are right about the fact that using the second level cache ideally means using Hibernate APIs (although you can still evict the cache "manually" and although I tend to prefer using it for "mostly read" entities). But, more important, to my experience sharing data between many applications through the database just leads to unmaintainable applications (changing a single bit becomes impossible as it may impact several applications) and and should be avoided. Use an EAI/ESB and expose services of the main system through it. This way, you can reuse the business logic, the 2nd level cache, etc.
4) (...) For what kind of projects hibernate is suitable for? Is it for back office projects where nobody cares about the sql ?
Hibernate is indeed very nice for CRUD applications, but not only (see above) and your question shows some ignorance as I already said. However, it isn't suitable for any project:
I would probably not use it for a data warehouse or a big reporting application.
I might not use it with a heavily denormalized or exotic legacy database (a data mapper like mybatis might be a better choice in this case).
I might not use it with an existing system using stored procedure for everything.
I would not use it with a non RDBMS datastore :)
5) (...) What happens when your administrator says: please use memcached for caching and please use this optimized queries instead of yours?
I tell him that memcached is maybe not the best solution in our context (no, I don't want to always send my data over the wire and I don't care that Facebook/LiveJournal/Twitter/whatever are using it, our app might have different needs), there are other better cache implementations when working with Hibernate, I ask him to discuss problems with me and we discuss the various solutions, etc. We work as a team, not against each other.
To sum up, ORM solutions are not always suitable but I think that you currently have a biased opinion and my experience is different from the opinions (misbeliefs?) expressed in your question.
See also
When NOT to use O/R mapping in Java

It's good for green field projects, but it's also good for legacy projects. You may need to do some mapping tricks, but it offers reasonably flexible mapping.
Since you can use native queries, and since you can integrate it with your favorite caching solution, you don't need to suffer any performance problems just because you're using Hibernate. When your db administrator says that you should use memcached, you can use this memcached/Hibernate integration. You can write a caching implementation using your favorite cache and plug in into Hibernate. When she says you should use this optimized query, you say "great! Hibernate has a native SQL facility that will let me use that query". You can use native Oracle syntax, you can use the native syntax of whatever RDBMS you've chosen.
A multiple-application environment poses the same challenges to Hibernate as it does to any solution. If you want your application to perform well, you will use what amounts to a second-level cache. Hibernate happens to offer an ORM that is integrated with the cache. It doesn't solve the problem of coordinating a cache across multiple applications, but you'll have to solve that problem even if you don't use Hibernate.

Your question is probably too broad. I can tell you about my experience.
I worked on a project that adopted the .NET version (NHibernate). A naive implementation of loading a single row from a single table was almost two orders of magnitude slower than a raw ADO query. After much optimization I believe they got it down to merely one order of magnitude slower.
In java where the start up time is probably less of a factor. The web server loads java and hibernate at server start instead of while a user waits for a desktop app to start.
Personally I really dislike it. It hides implementation details that are necessary to efficiently manage your data. I've found no real world application that could perform acceptably with a vanilla implementation of a data layer that hides database details.
But that may be sour grapes on my part since I was forced to use it and blamed for
not being able to put enough lipstick on the pig.

No matter how complex database is. The most important question is how complex domain model of application is.
Is query select * from anytable where anycol = #anyvalue optimized? I have no idea. Nobody has. Because there is only one true criteria of optimization - this is performance of such queries. You can save a lot of time with hibernate or other ORM, then use this time to find actually slow queries. As far as I know Hibernate has some ways to use optimized query.
Third your question is good. But also there is no one answer to the question 'Is dirty data good every time everywhere?'. Strictly saying, until locked, any data read from database are dirty, no matter how its were read and where its were stored. Data blocking is not good thing for performance, so usually you should find compromisse between actual data and performance.
There is no silver bullet. ORM has a lot of advantages, but there is only one serious case when it is not suitable: it is dynamic resultsets depends of parameters (when different parameters returns data with different column sets). Because object structure are static at compile time (in static typed languages) ORM can't help in this case.
Every other case can be solved. Entity sevices (changes tracking etc.) can be off, second-level cache can be disabled, and optimized query can be used instead of generated. I have no idea how to do all that things in Hibernate, but I'm sure it is possible.
ORM has a great advantage it concentrate all data access logic in manageable form, and put it in specific place. Also it supports few things are not so easy and direct to implement in your own data access library, like transaction management (including nested transactions, etc), identity mapping (one row - one object), complex hierarchy persisting (if you use objects and object hierarchies), optimistic locking etc, and ORM can greatly helps you with it.

Recommendations for an in memory database vs thread safe data structures

TLDR: What are the pros/cons of using an in-memory database vs locks and concurrent data structures?
I am currently working on an application that has many (possibly remote) displays that collect live data from multiple data sources and renders them on screen in real time. One of the other developers have suggested the use of an in memory database instead of doing it the standard way our other systems behaves, which is to use concurrent hashmaps, queues, arrays, and other objects to store the graphical objects and handling them safely with locks if necessary. His argument is that the DB will lessen the need to worry about concurrency since it will handle read/write locks automatically, and also the DB will offer an easier way to structure the data into as many tables as we need instead of having create hashmaps of hashmaps of lists, etc and keeping track of it all.
I do not have much DB experience myself so I am asking fellow SO users what experiences they have had and what are the pros & cons of inserting the DB into the system?

Well a major con would be the mismatch between Java and a DB. That's a big headache if you don't need it. It would also be a lot slower for really simple access. On the other hand, the benefits would be transactions and persistence to the file system in case of a crash. Also, depending on your needs, it allows for querying in a way that might be difficult to do with a regular Java data structure.
For something in between, I would take a look at Neo4j. It is a pure Java graph database. This means that it is easily embeddable, handles concurrency and transactions, scales well, and does not have all of the mismatch problems that relational DBs have.
Updated If your data structure is simple enough - a map of lists, map of maps, something like that, you can probably get away with either the concurrent collections in the JDK or Google Collections, but much beyond that, and you will likely find yourself recreating an in memory database. And if your query constraints are even remotely difficult, you're going to have to implement all of those facilities yourself. And then you'll have to make sure that they work concurrently etc. If this requires any serious complexity or scale(large datasets), I would definitely not roll your own unless you really want to commit to it.
If you do decided to go with an embedded DB there are quite a few choices. You might want to start by considering whether or not you want to go the SQL or the NoSQL route. Unless you see real benefits to go SQL, I think it would also greatly add to the complexity of your app. Hibernate is probably your easiest route with the least actual SQL, but its still kind of a headache. I've done it with Derby without serious issues, but it's still not straightforward. You could try db4o which is an object database that can be embedded and doesn't require mapping. This is a good overview. Like I had said before, if it were me if I would likely try Neo4j, but that could just be me wanting to play with new and shiny things ;) I just see it as being a very transparent library that makes sense. Hibernate/SQL and db4o just seems like too much hand waving to feel lightweight.

You could use something like Space4J and get the benefits of both a collections like interface and an in memory database. In practical use something as basic as a Collection is an in memory database with no index. A List is an in memory database with a single int index. A Map is an in memory database with a single index type T based index and no concurrency unless synchronized or a java.util.concurrency.* implementation.

I was once working for a project which has been using Oracle TimesTen. This was back in early 2006 when Java 5 was just released and java.util.concurrent classes were barely known. The system we have developed had reasonably big scalability and throughput requirements (it was one of the core telco boxes for SMS/MMS messaging).
Briefly speaking, reasoning for TimesTen was fair: "let's outsource our concurrency/scalability problems to somebody else and focus on our business domain" and made perfect sense then. But this was back in 2006. I don't think such a decision would be made today.
Concurrency is hard, but so is handling of in-memory databases. Freeing yourself of concurrency problems you'd have to become an expert of in-memory database world. Fine tuning TimesTen for replication is hard (we had to hire a professional consultant from Oracle to do this). License(s) don't come for free. You also need to worry about additional layer which is not open source and/or might be written in a different language than the one you understand.
But it is really hard to make any judgement without knowing your experience, budget, time requirements, etc. Do a shopping around, spend some time for looking into decent concurrency frameworks (such as http://akkasource.org/) ...and let us know what you have decided ;)

Below are few questions which could facilitate a decision.
Queries - do you need to query/reproject/aggregate your data in different forms?
Transactions - do you ever need to rollback added data?
Persistence - do you only need to present the gathered data or do you also need to store it in some way?
Scalability - will your data always fit in the memory?
Performance - how fast should it be?

It is unclear to me why you feel that an in memory database cannot be thread safe.
Why don't you look at JDO and DataNucleus? They have a lot of different datastores where you get to plug in what your back end persistence provider is at run time as a configuration step. Your application code is dependent on an ORM but that ORM might be plugged into an RDBMS, DB40, NeoDatis, LDAP, etc. If one backend doesn't work for you, then switch to another.

Should we drop stored procedures and run database calls from java programs [closed]

Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 3 years ago.
Improve this question
I am fighting to keep the use of stored procedures in our company. There are a few people who say they are bad and we should not use them. We are using DB2 on the i-series.
Please help in my argument to keep stored procedures alive in my company.

You're not going to like this, and I'm probably going to get downvoted into oblivion, but I'm with the rest of your comapny.
Stored Procedures used to offer many benefits (security, performance, etc.) but with parameterized queries and better query optimization, stored procedures really just add another layer of overhead to your application and give you another place you need to update/modify code.
I prefer to keep everything in a single spot so that when I need to edit code, I can go to one place and make my changes there.
If you want more details about the arguments for moving away from Stored Prcoedures, check out this CodingHorror article:
Coding Horror: Who Needs Stored Procedures Anyway?
...and I just noticed that the article is from 2004. I have to believe that databases have gotten better since then which means this would ring even more true today than it did then.

Doing everything over JDBC essentially means that you are inserting a network layer between you and the database. All in all it means that data is more "remote" and come to you slower. Stored procedures can work directly on the data inside the database, and the resulting difference in speed may astonish you.
Please note that you can write stored procedures in any IBM i language including Java, in case it is a matter of programmings skills. Also, you have access to the FULL machine, not just some database internals. Here the AS/400 is so vastly different from any other database product, that experiences from other databases simply - in my opinion - does not apply.
I would recommend the Midrange mailing lists as they have the largest concentration of AS/400 programming skills I know of.

This is one of those Marmite issues. if you are primarily a database programmer you will think that stored procedures should be used extensively. If you are an application programmer - say a Java or a .Net coder - the chances are you will say that they should be avoided completely.
Not that this meets application programmers want to write their own SQL statements. No, these days they tend to want to abstract everything behind convoluted ORM services. These are not easier to understand than stored procedures but are available within the same IDE, so they require less context switching.
There are two big things in favour of stored procedures. The first is that people who know PL/SQL are likely to be familiar with Oracle databases (T-SQL & SQL Server, etc), and so will tend to write better programs for that database (defined as programs which take advantage of the platform's features and are fitted to its functionality) than people who don't.
The second thing is that data persists. Application developers are fond of talking about "database independence" but what really matters is application independence. Front-ends come and go but the data model endures forever. In the last ten years Java applications have been written as Applets, Servlets, JSPs, Tiles and Faces, with add-ons in JavaScript, Groovy, AJAX and JSON, connecting to the database through hand-rolled JDBC, EJB (v1,2,3), TopLink, Hibernate and IBatis... forgive me if I've missed a few. Applications whose UI is a skin over a layer of stored procedures are easier to upgrade to the latest and greatest than applications where the business logic has to be re-written every time. And they will perform better too.
However, in the long run applications which interact directly with the database are probably going to die away. Everything is going to talk to the service bus, and that will decide from where to get the data. Of course, shops where the database is exposed through a well-designed API of stored procedures may find it easier to move to this brave new world than those places which are going to have to extract everything out of their ORM logic.

OK I'll come out in favor of stored procs.
First if you use them exclusively, they make refactoring the database much simpler as you can use the dependencies stored in the database to find out what would be affected by a change (well in SQL Server anyway, can't speak for other datbases).
Second, if you need to change just the query, they are far simpler to deploy.
They are also easier to performance tune as they can easily be called without firing up the application.
If you have complex logic then you save some performance by not having to send all that over the network to the database server. May not seem like a big gain, but if the complex query is run thousands of times a day, it can add up.
Security is also extremely important. If you do not use store procedures, you must set rights at the table or view level. This opens up the database to internal fraud. Yes, parameterized queries reduce the risk of sql injection, but that is not the only threat you need to guard against. If you have personal or financial data and you do not use stored procs (and ones with NO dynamic SQl) and limit your users to only being able to do things through the procs, then your system is in extreme danger from internal employees who can steal data or bypass internal controls to steal money. Read about internal controls in the accounting standards to see why this is a problem.
ORMs also tend to write just downright bad SQL code especially if the query is complex. Further as people start to use them instead of stored procs, I have found that the people who have never used stored procs have a poorer understanding of how to get data out of the database and frequently get the wrong data. Using an ORM is fine if you already understand SQL and can determine when to rewrite the autogenerated code into something that works better. But too many users don't have the skill to write complex code because they never learned the basics.
Finally since you already have stored procs for your application, getting rid of them altogether is a way to introduce new bugs becasue you had to generate new code.

They're useful when you have a layered set of apps. For example, a single core DB with web services offering the atomic operations (which happen to be stored procedures) and a ESB or a set of applications consuming those WSs.
In a single-app/single-db case, the idea is to keep the code in one place as others suggested.
But well, that's just me.

I am a long-time Java developer who has recently come across several projects that made heavy use of stored procedures that have put the use of stored procedures in a really bad light for me.
Having said that, I am reluctant to make a blanket statement that stored procedures are bad as a system design option, because really it depends on the project in question and what the particular stored procedures are trying to accomplish.
My preference is to avoid any kind of stored procedure for simple CRUD operations (it may sound laughable to some to have stored procedures handle these operations, but I've encountered several systems that were doing this) -- This ends up resulting in a lot of code having to be written (and tested and maintained) on the Java side to manage these procedure calls from what I've observed. It's better to just use Hibernate (or some other ORM library) to handle these kinds of operations...if for no other reason than it tends to reduce the amount of code needing to be maintained. It also can cause problems when trying to refactor or make any significant changes to the system, as you're not just having to concern yourself with class/table changes, but stored procedures that handle CRUD ops as well. And this can be exacerbated further if you're in a situation where developers cannot make changes to the database themselves, or there is some formal process in place to coordinate changes between the two parts of the system.
On the other hand, having stored procedures that require limited interaction with the Java code (basically, you just fire off a call to one with a few arguments), and run in a semi-autonomous fashion is not a terrible thing either. I've encountered a few situations (particularly where we were migrating or importing data into a system) where using a stored procedure was a much better route than writing a bunch of Java code to handle the functionality.
I guess the real answer here would be that you should be examining what each store procedure in the system is doing currently and evaluate them on a case-by-case basis to determine if perhaps it's easier to handle the operation in Java or the database. Some may very well work better in Java (either by ORM library, or actual hand-written code), some may not. In either case, the goal should always be to make sure the system is understandable and easy to maintain for everyone, not just whether stored procedures are good or bad in and of themselves.

How to Call Java Code from MySQL?

I found an article from 2008 discussing how to call Java code from MySQL. There were a lot of caveats and disclaimers because the process involved working with an experimental branch of MySQL.
For a project I have in mind, it would be very useful to be be able to access Java libraries within MySQL, analogous to Oracle's Java Stored Procedures. Does this capability now exist as a standard feature of MySQL? If not, what open source RDBMSs support something similar to Oracle's Java Stored Procedures?

PostgreSQL supports pluggable procedure languages, and a project exists to extend PostgreSQL with PL/Java as the language.
I don't recommend putting too much code in the RDBMS. Tools to develop, test, and debug code in the application layer are better than tools for code in the RDBMS.
Also many developers don't understand that code inside the RDBMS should obey transaction isolation. They try to send emails from triggers and so forth. I think code with side effects should be in the application layer, so you don't create phantom effects (e.g. an email may notify of a database change, even though the change was rolled back).

If you can use HSQLDB then you can call java methods directly from SQL: http://hsqldb.org/doc/2.0/guide/sqlroutines-chapt.html#N1240C

I fully agree with Bill, but I can imagine business rules being stored (not processed) in the database. I'm thinking of drools here. The engine would be in the application, but the rules could be in the database with a management front-end.
Such a beast would be interesting for scenarios where not only the parameters change, but also the formulas can change.

It is difficult to give good advice based on the limited information that you have provided so far. However:
... the example involves a graph-based data type (chemical structures) that can't be matched to a query using built-in MySQL functions. The Java library would convert the query and contents of a text field into an in-memory object that can by matched. Keeping this logic in the DB layer would, for example, keep joins within the database, which seems like where they belong. That's the idea, at least.
I don't think I would use database-side Java in MySQL for this. Instead, I think I would consider the following options:
Use an object-relational mapping such as JDO or JPA (for example using Hibernate) to deal with the mapping between your graph-based data model and what the database provides. You don't necessarily have to use an RDBMS as the backend, but that is probably the best place to start ... unless you've already found that this is a performance issue.
Take another look at your data model and data access patterns. See if you can figure out some transformation that allows your application's main queries to be implemented as (efficient) table joins without resorting to server-side application logic.
If you do need to use server-side application logic (for performance reasons!) stick with the mechanisms supported by your RDBMS. For example, in Oracle you'd use PL/SQL and PostgreSQL you have a number of options. Be prepared to switch to a different RDBMS that better suits your application requirements.
I (personally) would avoid depending on an experimental branch of some database:
Consider what happens if the experimental branch is not merged back into the main branch. You would be stuck with your code base depending on a branch that is not supported, and is likely to stop being maintained and fizzle out.
Using a (currently) unsupported RDBMS branch will be an impediment to other folks who might want to use your software.
Now obviously, if the long term viability of your software is not a primary concern, you could choose to ignore this advice. But it probably matters to someone; e.g. your research supervisor.

I realise that this is quite an old article, but it bears updating. The ability to call java from a database trigger is is part of the "SQL Routines and Types for the Java Programming Language" (SQL/JRT) standard.
Read more about this on Wikipedia at https://en.wikipedia.org/wiki/SQL/JRT.
Amongst the compliant database engines are..
HyperSQL: http://hsqldb.org/
Oracle: https://www.oracle.com/database/