My company is trying to determine the best strategy for implementing batch Java programs. We have a few hundred (and growing) separate Java programs. Most of them are individual Jasper Reports but some are bigger batch Java jobs. Currently, each Java Project is packaged an independent JAR file using Eclipse's export option. Those JARs are then deployed to our Linux server manually where they are tested. If they pass testing, they are then migrated up through QA and onto Production through a home grown source code control system.
Is this the best strategy for doing batch Java? Ongoing maintenance can be a hassle since searching Jar files is not easy and different developers are creating new Java Projects (new reports) every week.
Importing existing projects from the Jar files into Eclipse is a tricky process as well. We would like these things to be easier. We have thought about packaging all the code into 1 big project and writing an interface to be able to execute the desired "package" (aka program) maybe using a Web Server.
What are other people/companies doing out there with their batch Java programs? Are there any best practices out there on this stuff? Any help/ideas/working models would be appreciated.
I would say that you should be able to create one web based app for access Jasper reports, rather than a bunch of batch processes. Then, when you need to deploy a new report, just deploy a minor update that accesses a new compiled Jasper report file.
That said, you should be checking your code, not your binaries, into a Subversion or Git repository. Dump the "home grown" source control repository. Life is too short to try to home grow stuff like that. Just use Git or Subversion, they're proven, simple, and functional. When you import a new project, just pull it down from Subversion, don't try to import the JAR file from your Eclipse IDE.
Put your JAR files into a Maven repository such as Nexus, and deploy to QA and Production from there. Create automated builds for every project (be that with Maven or something else). Don't depend upon an IDE to export your JAR files. IDE's change and exporting from an IDE introduces more opportunity for human error. Also, different developers will prefer different IDE's. By standardizing on something like Maven, you're a bit more IDE agnostic.
Mhy company has standardized Java Batch execution using IBM Websphere Extended Deployment.
Here http://www.ibm.com/developerworks/websphere/techjournal/0801_vignola/0801_vignola.html is an article introducing techniques for programming and deploying java batch.
Introduction to batch programming using WebSphere Extended Deployment Compute Grid
Christopher Vignola, WebSphere
Architect, IBM
Commonly thought of as a
legacy "mainframe" technology, batch
processing is showing itself to be a
venerable workload style with growing
demand in Java™ and distributed
environments. This article introduces
an exciting new capability for Java
batch processing from IBM®, the leader
in batch processing systems for the
last 40 years. This content is part of
the IBM WebSphere Developer Technical
Journal.
WebSphere Extended Deployment Compute
rid provides a simple abstraction of
a batch job step and its inputs and
outputs. The programming model is
concise and straightforward to use.
The built-in checkpoint/rollback
mechanism makes it easy to build
robust, restartable Java batch
applications.
The Batch Simulator utility provided
with this article offers an
alternative test environment that runs
inside your Eclipse (or Rational
Application Developer) development
environment. Its xJCL generator can
help jump start you to the next phase
of testing in the Compute Grid unit
test server.
But even if you are not interested in the product, the article is a must read anyway.
Related
I have a Java application (a quite large one with many external .jar dependencies as well as dependencies on images) and I need to package it up so that someone can double click to run, for example. Or something easy like that.
It uses Java Persistence, so it requires a sql connection which is specified in the Persistence.xml file in the Java Project.
How can I package this up? I was thinking:
the installation process should validate that the user has MySQL installed and if not, direct them to install it
the installation process could ask the user to enter credentials for any database and then I could update the Persistence.xml at run time
These were two ideas I had...but I wasn't sure if there was a known solution to this problem. Any help would be much appreciated!
I think you should take a look at embedded database solutions, like H2. Also, you can package your application using maven's shadowing or jar plugin, having the jar-with-dependencies profile activated.
This will nicely rid you of checking for database servers running on the client machine, and also will give you the proper means of bundling the application in one nice JAR, albeit a little large.
Maven is a build ecosystem and toolset especially designed for building Java applications and executing the code -- and generally doing whatever else you can imagine that's possible to do with and to your code.
It has a rich API for developing plugins and many developers have exploited this feature. There are numerous plugins for building -- and launching -- and packaging your application as well as helping you manage your applications dependencies.
Maven's shadowing comes in the form of maven-shade-plugin, available here. What it does is that it helps you create a single JAR file from all your dependencies. Also, there is the maven-jar-plugin which offers a profile jar-with-dependencies. It is also accessible from here.
H2, on the other hand is a full-fledged RDBMS. This is the website: http://www.h2database.com/html/main.html, and here is a tutorial.
You can find information on embedding the database here:
How to embed H2 database into jar file delivered to the client?
Embedding the Java h2 database programmatically
h2 (embedded mode ) database files problem
I would also suggest you use a combination of H2/Hibernate/Spring which is a very easy setup and provides you with really rich features and an easy-to-use API.
I hope this helps you :)
Building a sophisticated installer that checks lots of dependencies, and runs on lots of different platforms (which I assume you want) is complicated.
I suggest that you look at an installer generator; see What is the best installation tool for java?
Another alternative that I've seen in a few products is to write a (non-GUI) installer or configurer in a scripting language like Perl.
I wrote an installer using ANT, but has no GUI. Also, I used Iz Pack (good option), so I think that depends on how smart do you want it to be, if you are supposed to use it, or a non-technical person, etc.
I have to run a lot of relational-database-based reports for my job. In order to make my life easier, I have written a handful of scripts and programs to run a report (simple SQL query, complicated super-specific cross-cutting query, and full JasperReports anything-goes-style reports), compress and/or encrypt it, and deliver it via SMTP to one or more recipients.
Some of this has been written in Java, some in PHP, and some as bash shell scripts. I'd like to unify a lot of this, plus add a single report-description format that can include configuration information like which report (or reports: sometimes I have to run several reports at once and combine them into a single "package" for the client) to run, what any parameters for the report should be set to (customer id, for instance), who should receive the report via email/SFTP/etc.
Note that tools like JasperReports are only part of the solution: the actual report-generation part. I'm more interested in something that can be used to script the whole process and be run from a cron job.
Do any products (OSS would be great) exist to help me with this kind of thing? Or, is this kind of thing so unique to a particular environment as to require the kind of customized tools that I have already built?
Use BIRT that integrates with your Java/J2EE application to produce compelling reports. And the second option is DataVision that is similar to popular Crystal Reports.
It might be a little off what you were thinking about, but have you considered (or heard of) R ? R is an open source statistical programming language that has many features, but specific to your problems is;
Is open source
Has packages that enable interacting with SQL databases (generic ODBC, or specific Oracle, MySQL, Postgres etc)
Can do many kinds of data processing steps with the data
Can produce
tables and graphs
Can produce documents in several ways (LaTex,
Markdown, ODF)
Can be used from the command line, (for instance I
produce documents with make)
Can be integrated into a web server
(RApache)
Can run Java code (RJava)
the only problem is this would be roll your own, there's no specific implementation that offers the features you want.
some frameworks that might help you with configuring report delivery workflow are.
Apache Service Mix http://servicemix.apache.org/ see achitecture diagram froma a previous version http://servicemix.apache.org/home.data/ServiceMix3.png
Spring integration http://static.springsource.org/spring-integration/reference/htmlsingle/
Both should give you a framework for building a decoupled architecture so that the part does report compression and the part that does report encryption and report generation don't know about each other but can be configured via the framework to work in a particular workflow.
Maybe you can check Gradle http://www.gradle.org/. Since it is based on Groovy you can send Mails, package archives and call JasperReports easily.
From Gradle web site:
Gradle is build automation evolved. Gradle can automate the building,
testing, publishing, deployment and more of software packages or other
types of projects such as generated static websites, generated
documentation or indeed anything else.
Gradle combines the power and flexibility of Ant with the dependency
management and conventions of Maven into a more effective way to
build. Powered by a Groovy DSL and packed with innovation, Gradle
provides a declarative way to describe all kinds of builds through
sensible defaults. Gradle is quickly becoming the build system of
choice for many open source projects, leading edge enterprises and
legacy automation challenges.
You might want to take a look at Pentaho: http://www.pentaho.com/explore/pentaho-business-analytics/
Currentyl i am working on a Java Web Application something simple JSF + Spring + Hibernate textbook application architecture, but i find my self losting time on uploading my WAR into production every time i have to do it and also in updating MySQL scripts, so
is there any good practice to deployed Java Web Application quickly?
Edit
This is the current Process:
I work on localhost where i quickly
deploy with eclipse
If everything work fines i export a WAR with eclipse
Upload this WAR through FTP into (hosted) production server
Move this into tomcat webapps
Thanks.
Tip. I'm Using eclipse as my IDE and ain't using Maven nor Ant
There are a lot of tools you can use to automate the build and deployment process:
Compile application => deploy to local server => execute unit- and integration tests => upload WAR file to web server if all local tests pass -- this is exactly the kind of functionality that Maven and Ant were invented for!
If you want to get rid of the tedious manual work involved in these tasks, perhaps checking them out once more is not such a bad idea.
Also, there are more advanced Continuous Integration solutions, like Hudson, which help you to better integrate source code management and deployment processes.
But none of those will speed up data delivery.
It seems like your most pressing concern is to find a provider to make your internet connection faster: However you manage your build process, uploading the content to the server will quite probably be the most time-consuming part.
The single most important thing is being able to script the stuff, so you can let the computer automate it.
You may be able to get Netbeans to create build scripts for your eclipse projects so you can just run ant to create your war. If not, keep doing it manually.
Then you need to script the ftp-session. That is easy - many scriptable programs exist (or you can just use a modern Norton Commander clone supporting ftp), so script it.
Then you need to script the MySQL-script changes. This is not easy to suggest, as you did not mention what needs to be done, but you can probably get very far with a Perl script or perhaps a small stand-alone PHP snippet expanding a macro.
THen you need to collect all the steps in a single script you can then easily invoke...
(or you could just figure out how to talk to the Tomcat administrative console which allows you to update WAR files remotely - but that may not be feasible or I would have expected you to do so already)
Currently we have a Java Restlet API with dependencies controlled via Maven. When we update the API we run maven assembly:assembly which does the unit tests etc and produces a single jar file. We then upload this to the production server and run it using nohup.
Is there a better or more automated way of doing this? Is this where something like Hudson would come in?
Thanks
My experience goes with webapp-deployment. But same should hold true here. Use Maven, Cargo, Nexus (or Artifactory), Hudson and probably, Jira in conjunction of product release.
Automated release process are more reliable because there is no human factor involved that may forget a step.
We also use Liquibase for database versioning. And, if you are dealing with database changes in your application deployment. You'll realize Liquibase boosts so much confidence while running alter scripts.
I would suggest to go through the following resources
Automated Deployment with Maven - going the whole nine yards If you can, literally follow this pattern.
Maven 2 Effective Implementation -- this book really helped us a lot.
There are several Maven plugins to help deployment. The most general of them is Cargo, but there are also app server specific plugins for some concrete servers like JBoss.
Most companies I have worked for (actually, all) have had some sort of custom in-house built deployment system; even if build was done using a standard framework (like Maven in use at my current company).
Part of this is because there are many aspects that tie closely to company-specific infrastructure, capacity management and monitoring systems; and so even though there are open-source systems, there is usually something that needs to be tweaked.
It sounds like you are running your app on its own--it isn't part of any application server. If you aren't using an application server, there are probably some ways to get cargo and maven to deploy it for you, but you may be better off just using some shell scripts to deploy and run the application.
However, as your application grows, you may find a need for an application server like Jetty, JBoss, Glassfish, Tomcat, etc. When this happens, take a look at the cargo plugin for Maven because it will allow you to do something like:
mvn cargo:redeploy
That will package up your application, send it to the server and restart the app. If you want Hudson to do this for you automatically you can add it as a target to build.
Cargo can save you a lot of time when you have to frequently update an application server.
Is it possible to set up continuous build of projects written in .NET and Java on single build server with single set of applications?
I've seen CruiseControl has support for both world but as far as I know these are two separate applications. Should I go with a separate machine to build Java projects (a machine for .NET projects is already set up)?
Hudson has support for both Ant (out of the box IIRC) and NAnt (through plugin) so that's an option for you.
CruiseControl supports several different build options include Ant, Maven, NAnt, Phing (php), Rake, XCode, and then the generic "exec" which you can use to invoke any command-line script/tool you want. Not a problem mixing Java and .NET on the same server.
Of course this is not unique to CruiseControl. There are lots of CI tools that support multiple build technologies as you can see on this matrix of features.
Disclaimer: I work on CruiseControl. OTOH since I don't make money on it I don't much care which tool people choose. I care more about advancing the state of CI practices which is why I organize the Continuous Integration and Testing Conference (CITCON).
Ant and NAnt can reasonably easily execute arbitrary processes, including each other, so the actual build part shouldn't be too hard.
I'd expect the tricky bit to be getting the reports (including unit test results) into an appropriate format. How's your XSLT? ;)
EDIT: Now that I think about it, my first agile project had a continuous build server (just CruiseControl, I believe) which must have been doing some of this... I suspect at the time we directly invoked Visual Studio to build the code and NUnit to test it. If I were at the same company I'd check, but that was two jobs ago :(
You could checkout Atlassian Bamboo. Unfortunately, its not free, unless you are applying for an opensource/community license for use with opensource software.
You can use two different products on the same machine. Or you can run a single builds system across multiple machines. It is really up to the load you place on your CI.