Develop Reference

Trying to understand what gradle and maven actually do

What exactly do Gradle and Maven "do" that eclipse or sts doesn't? From what I've read it builds, runs, etc which can all be done in eclipse easily.
If I have an existing project I've created, built, and currently runs in eclipse via tomcat, what would I use gradle for?

There's not a lot of benefit to using Maven or Gradle on a small project that you never share with anyone; an IDE can do the build just as well. But as the number of developers increases and the complexity of the build increases, it becomes very useful to separate the build instructions from the IDE. Let's drill into these a little bit.
With the increase in developers, you don't want everyone to have to come by and use your IDE to get a build done. That would be really annoying! So they're on their own machines, but then they tend to have different setups (how dare they have different user account names!) and probably have their IDE installations set up a little differently too. So we need some kind of build instructions that people can use to get things going, and it helps if everyone can use the same build instructions repeatably so that you don't get too many instances of “but it works on my machine!”. It's also very helpful if those instructions are simple enough to use that a new programmer to the team can get up to speed rapidly.
But the other thing that often happens as projects grow in scope is that their builds become more complicated. They very often gain additional dependencies (they didn't start out needing a high-performance date parser and MIME-type identifier, buit they've become required since and you don't want to have to write all those from scratch) and that means you've got to make sure that when the build is done, the right version of those dependencies is used. But that's not the only way that complexity increases. It's also very often the case that you find you're using more automatically-generated code. You might find yourself working with XML schemas or WSDL a lot, or maybe your using Hibernate, or Spring, or … well, there's lots of ways in which things can get complicated, OK? Getting the various steps to do all the build right, reliably, in these sorts of scenarios can be a bit tricky, but encoding them as instructions to something like Maven makes life a lot easier once you've taken the jump in the first place. (It gets even more important when you start trying to deal with projects which need many different sub-programs that work in concert; some of those are plain hard to build even with Maven or Gradle or any other tool.)
And then there's the possibility of offloading work to a build server, running tests automatically, managing dependencies cleanly, etc. IDEs don't handle these all that well by themselves; where they do a reasonable job of it, it's usually because they're using a tool like Maven under the covers to do the heavy lifting.
tl;dr
You don't have to make your code work with a build system, but it helps if you do and in many ways.

Maven and Gradle can do many things that Eclipse doesn't. However, the most important thing they do, is to decouple the bulding and testing processes from the IDE you choose (i.e. Eclipse). When you work on a large environment, with many programmers, usually you can not control the IDE they use. So, it's better to use a tool like Maven and Gradle to standardize these tasks. The same happens with the code examples of a book: instead of the authors having to provide the instructions for configuring any IDE to execute them, they provide the Maven or Gradle files, so the reader can build and test them on any IDE he's using.
Another very important feature that Maven and Gradle give you, is the fact that dependencies are managed without the need of having the executable code under source version control. Instead of having the executable code you depend on as part of the project, you declare the dependencies on a text file (which is under source version control), and then get them from a repository.
However, you may only see the real advantages of using tools like Maven or Gradle (and even Jenkins or Hudson), when you think in large scale projects, developed along many months by teams composed of many developers).

Gradle and Maven are build tools. Maven was first and is a bit older, Gradle is newer and has redefined a way of how projects are built and maintained. In my opinion it's also much easier to use, more readable and easier to maintain. I prefer Gradle ;)
You use eclipse or STS (any other IDE) for development. And while You finish this process You need to provide a configured artifact (war, ear, whatever...) to production and deploy it there. These artifacts have well defined format and the application won't be run from eclipse or STS at the production environment. It's tiresome and error prone to prepare such artifacts by hand.
Gradle or Maven can take responsbility of building and preparing these artifacts (in fact such tools can do much more) off Your shoulders, they make this process automated.

What scripting frameworks exist for database-based reporting?

I have to run a lot of relational-database-based reports for my job. In order to make my life easier, I have written a handful of scripts and programs to run a report (simple SQL query, complicated super-specific cross-cutting query, and full JasperReports anything-goes-style reports), compress and/or encrypt it, and deliver it via SMTP to one or more recipients.
Some of this has been written in Java, some in PHP, and some as bash shell scripts. I'd like to unify a lot of this, plus add a single report-description format that can include configuration information like which report (or reports: sometimes I have to run several reports at once and combine them into a single "package" for the client) to run, what any parameters for the report should be set to (customer id, for instance), who should receive the report via email/SFTP/etc.
Note that tools like JasperReports are only part of the solution: the actual report-generation part. I'm more interested in something that can be used to script the whole process and be run from a cron job.
Do any products (OSS would be great) exist to help me with this kind of thing? Or, is this kind of thing so unique to a particular environment as to require the kind of customized tools that I have already built?

Use BIRT that integrates with your Java/J2EE application to produce compelling reports. And the second option is DataVision that is similar to popular Crystal Reports.

It might be a little off what you were thinking about, but have you considered (or heard of) R ? R is an open source statistical programming language that has many features, but specific to your problems is;
Is open source
Has packages that enable interacting with SQL databases (generic ODBC, or specific Oracle, MySQL, Postgres etc)
Can do many kinds of data processing steps with the data
Can produce
tables and graphs
Can produce documents in several ways (LaTex,
Markdown, ODF)
Can be used from the command line, (for instance I
produce documents with make)
Can be integrated into a web server
(RApache)
Can run Java code (RJava)
the only problem is this would be roll your own, there's no specific implementation that offers the features you want.

some frameworks that might help you with configuring report delivery workflow are.
Apache Service Mix http://servicemix.apache.org/ see achitecture diagram froma a previous version http://servicemix.apache.org/home.data/ServiceMix3.png
Spring integration http://static.springsource.org/spring-integration/reference/htmlsingle/
Both should give you a framework for building a decoupled architecture so that the part does report compression and the part that does report encryption and report generation don't know about each other but can be configured via the framework to work in a particular workflow.

Maybe you can check Gradle http://www.gradle.org/. Since it is based on Groovy you can send Mails, package archives and call JasperReports easily.
From Gradle web site:
Gradle is build automation evolved. Gradle can automate the building,
testing, publishing, deployment and more of software packages or other
types of projects such as generated static websites, generated
documentation or indeed anything else.
Gradle combines the power and flexibility of Ant with the dependency
management and conventions of Maven into a more effective way to
build. Powered by a Groovy DSL and packed with innovation, Gradle
provides a declarative way to describe all kinds of builds through
sensible defaults. Gradle is quickly becoming the build system of
choice for many open source projects, leading edge enterprises and
legacy automation challenges.

You might want to take a look at Pentaho: http://www.pentaho.com/explore/pentaho-business-analytics/

TFS for Java - bad idea?

We're considering TFS for our .NET based projects and as a task management platform.
Some teams develop exclusively in Java and they're quite happy with SVN (Subclipse).
Our managers came up with the following questions:
Should we migrate the Java teams to TFS as well?
Does TFS (source control only) handles well Java projects?
Is it a pain to migrate our Java code base and history from Subclipse to TFS?
Currently we are looking to use TFS as a sole source control platform for maintainability reasons. We would like to avoid having our IT guys supporting multiple systems.
Thanks

Full disclosure, I work on the team that write the Java tooling for TFS so take this answer as appropriately biased :-)
As far as TFS is concerned - all code is created equal. It's just bytes in files that it checks in to version control. Like all SCM systems it doesn't care what language the files are written in.
Microsoft provide a full, rich TFS Plug-in for Eclipse (called Team Explorer Everywhere). This provides full source control, work item tracking, build, sharepoint, reports access etc into TFS from Eclipse based IDE's. It's written in 100% Java and talks directly to the web services exposed by TFS.
In addition we also provide a cross-platform command line client for TFS so that you can talk to TFS from the command line on your operating system of choice (Mac, Linux, Solaris, HP-UX, Aix etc all fully supported).
Finally, if you have tools written in Java that want to talk to TFS then they can make use of the TFS SDK for Java which is the full API that we used to create the Eclipse integration and cross-platform command line client but packaged up with samples and snippets and ready for you to redistribute with your applications.
When it comes to build you have a couple of choices. If you want to stick with your current build server then it is likely that this already supports talking to TFS (all the popular open source build servers do). In addition to that, Microsoft provide the TFS Build Extensions which allow you to run Ant or Maven based builds on the Team Foundation Build server. The build results (along with any warnings or errors) are published back into TFS along with any JUnit test data if you execute JUnit tests as part of your build. Also you get to create and manage the build definitions in the Eclipse IDE and have one place to manage access to them etc.
So - the level of support for Java is very high and Microsoft has shown consistent investment in this area. We recently shipped some TFS 2010 Power Tools for Eclipse and we've also been shipping preview releases of Team Explorer Everywhere 11 alongside Team Foundation Server 11 (we're the same team inside the company).
To import history from SVN, that's the same as importing history from any SCM tool into TFS (or TFS into any SCM tool). You have a couple of options. You can take a snapshot and cut over at a particular point (such as a release) or you can migrate history. To Migrate history from SVN there are some partner solutions available including one from Timely Migration that I've seen a lot of customers have success with.
Hope that helps.

After a year of working on a Java/JVM project using TFS, I would like to dissuade anyone from doing this. While TFS may be considered top-of-the-line for .NET developers, you won't find any Java Developers with any experience with it. There is the plug-in for Eclipse and a port to IntelliJ, but I've had terrible luck with both, though I'm guessing it's mainly because TFS does not work like any other VCS I've used.
On our team, we've estimated a 10-15% overhead due to TFS and complications caused by it. Days of work lost because TFS decided to overwrite files, days of troubleshooting issues caused by incomplete TFS Updates. We have done a branch in 6 months because the entire team lost two days the last time we did. It's common to hear the phrase "I just updated with your latest changes, can you come check to make sure nothing disappeared in the merge?". Instead of using Jira, we're stuck using the terrible issue-tracking in TFS, causing more yet more issues.
Several of the developers on the team have taken to either using git, either standalone or the git-tfs bridge. Others just copy the source tree prior to any 'risky' activities, like updating or checking in.
Either way, I wouldn't recommend it for a team that does not have experience with it...

I like the answer of #Martin_Woodward a lot, but it is too much biased in my opinion, so I add my 2 cents here. We in our company are in a similar situation, and the decision (in my opinion) depends on the context. I can see 3 different situations, and the decisions may be different in each one:
You are mostly developing .NET solutions, and the Java parts are integrated in the .NET solutions.
Your .NET solutions are independent developed from the Java solutions, and they are half .NET, half Java.
Most of your solutions are developed with Java, and only a small percentage is developed with .NET
I would agree with Martin only in the first case. You will gain profit from the common development environment, source code control, build process ... Your Java guys will learn the differences to TFS Source Control (does it have a name??). And your future will look bright ;-)
If your .NET solutions and Java solutions are independent from each other, the only argument to use TFS for developing Java solutions is cost in operation. And you should carefully look at it, if the savings for operating the development environment only TFS will out weight the additional cost of switching your Subversion projects to TFS.
In the last case, it would be an awful decision to switch with a lot of people just to have a common environment to develop. You may integrate Subversion into VisualStudio (using e.g. VisualSVN or other plugins), and you have nearly no invest at all.
The migration of source code including history is normally a pain, and it depends on the source and target if that works well. We have good experiences with CSV and SVN, but no (good) experience with others. But that is normally not a problem, you may use your old SVN repositories (read-only) and just migrate the last milestone. After some time, SVN repos may be let alone ...

After 1 year working with TFS/Java I completely agree with Dusty J (Yes, TFS/Java is bad) and completely disagree with Martin Woodward about great Microsoft support. Although for my duty as a developer the Eclipse TFS is OK, the problems are for my build/release duties.
First, this Eclipse plugin does not allow creating a branch for several projects at once as in CVS/SVN. One needs to create a branch separately for every project. Then we cannot keep the same project names in the branch – one needs changing a project name and after checking out from the branch to rename to the original name. See also my post How to associate an Eclipse Workspace with TFS workspace?, there is no way to associate an Eclipse workspace with TFS workspace. Thus, the mapping for a local folder cannot be saved; it needs to be done again after opening another Eclipse workspace for branch building. And since the local mapping is the same there is a possibility of erasing a local folder with unsaved work as Dusty J wrote.
This removing local files without warning is a terrible feature of TFS (see the post Why command get from a command line in TFS removes parallel projects?). What Microsoft thinks about the possibility of erasing local files just under regular option "Remove Local Mapping" in Eclipse?
So, despite my effort to learn TFS I still spend 10 times more time for various builds as compared to CVS I used before.

(Another biased MS employee)
TFS formed a team about 18 months ago to focus solely on making the Java experience great in TFS/Team Services and across all platforms. I am on that team and I think we have made a ton of great progress. I won't disagree that the end to end story was pretty bad when this question was asked, but I think the answer has changed quite a bit in the last year.
My team provides build and deployment tasks for TFS as well the plugins for Eclipse and IntelliJ to make the end to end experience as complete as possible. We are also working hard to make sure we document how to get the best out of TFS if you are a Java developer.
If you want more details, checkout http://java.visualstudio.com.
Thanks,
Jason Prickett

Why not use SVN for the .NET projects? Is there any reason for that? There are multiple plugins for SVN in Visual Studio as well as a windows shell extension.

Java EE: Need better deployment system

We are currently using JDeveloper to build our production EARs. One problem with this is that if the developer doesn't add new files to a VCS, then that developer is the only one capable of making EARS, and therefore he can use unversioned files as well.
What would be a good system that seperates this, so that EAR files can be correctly produced without depending on the local developers workspace (this would also ensure that they add their files to a VCS before allowing to make a deployment/check-in).

One problem with this is that if the developer doesn't add new files to a VCS, then that developer is the only one capable of making EARS,
If the developer doesn't use the VCS, this is not your only problem:
You cannot reproduce things in another environment, you're tied to the developer machine (but you're aware of that). What if he is sick?
Not versioning files means you don't have any history of modifications and that you don't know what you put into production ("Hmm, what is in this version? Wait, let's open the EAR to check that.").
And last but not least, in case of hardware failure (e.g. a hard drive crash), you can say good bye to everything that is not in the VCS.
So, the first thing to fix is to ALWAYS version files, even if there is only one developer as working alone doesn't save you from the mentioned problems. These points should be reminded (the developer needs to be aware of them to understand their importance).
To make sure this happens, you should actually not rely on the developer machine to build the EAR but rather rely on an "external" process that would be the reference. You want to avoid this syndrome:
alt text http://img716.imageshack.us/img716/9901/worksonmymachinestarbur.png
To put such a process in place, you need to automate the build (i.e. your whole build can be run in one command) and to break the dependency with your IDE. In other words, do not use the IDE to build your EAR but rather use a tool like Maven or Ant (which are IDE agnostic). That would be the second thing to fix.
Once you'll have automated your build process, you could go one step further and run it continuously: this is called Continuous Integration (CI) and allows to get frequent, ideally immediate, feedback about changes (to avoid big bang integration problems). That would be the third thing to fix.
Given your actual toolset (which is far from ideal, there is not much community support for the tools you are using), my recommendation would be to use Ant (or Maven if you have some knowledge of it) for the build and Hudson for the continuous integration (because it's extremely easy to install and to use, and it has a Dimensions plugin).

Here's my recommendation:
Get a better IDE - IntelliJ, Eclipse, or NetBeans. Nobody uses JDeveloper
Have developers check into a central version control system like Subversion or Git.
Set up a continuous integration facility using Ant and either Cruise Control or Hudson to automate your builds.

What we do is use cruisecontrol. It does two things, it lets us do continuous integration builds, so that we have nightly builds as well as lightweight builds that get built every time a change is checked it.
We also use it to more specifically address your issue. When we want to ship, we use cruisecontrol to kick off a build, that is tagged with the proper production build version. It will grab the code from our version control system (we use SVN) and will build on that, so it is not dependent on developers local environments.
One thing you might also want to consider is creating a production branch to build out of. So, production ears for a particular release are always built from that branch. This way you have even have a bit more control over what goes into the build.

Instead of doing builds from developer workspaces, setup Maven, and have something like Hudson run your Maven build. The artificats of this build (your ear) gets deployed.

Is there good deployment and administration for Java environments?

We already have a good build server in Hudson but we want something that would let us startup and shutdown servers, push out new deployments of software (which is much more involved than just a single WAR or EAR going somewhere, there is copying, extracting, renaming, etc.), and various other tasks we would like to automate.
I've looked at SmartFrog (seems largely academic rather than commonly used), ControlTier (a dose of heavy complexity to go with your complexity), Capistrano (Ruby and Rails focused), and Func (no docs at all, their MediaWiki has been wiped). Is there nothing that is a good 80/20 solution for this kind of thing. Surely you could have a tool that would offer a lot of what ControlTier offers without all the overhead.
The alternative is Ant scripts to do everything (possibly available through our existing Hudson server) or even a Hudson plugin of some type but that feels kludgy to me and I'm just not liking it as a solution (plus we might be rebuilding something when we don't need to).
Please, tell me about a good alternative.

We are using Ubuntu only and use Debian packages to distribute builds across our servers and control them with init scripts. It's also great to setup a new server. Just add dependencies to Java, MySql and whatever you need and you'll be up and running a new deployment within seconds.

You might want to try Cargo. You can use ant and maven for the deployment definition you want and then set Hudson to point at your ant's build.xml or maven's pom.xml.

Chef was mentioned here but I have no experience with it.

There is also Puppet that you need to check out. You might need to read this great whitepaper on achieving fully automated provisioning
-Ken

LiveRebel might be a suitable tool for this task. It provides CLI API and also a Hudson/Jenkins plugin for automation