I am doing some experiments with my thesis involving the cold start problem that occurs with containers. My test application is a spring boot application that is build on the openjdk image. The first thing I want to try to resolve the problem of cold start, is the following:
Have a container ready, in the container is the openjdk and the libraries that the springboot app uses. I start my other container, using the ipc and networknamespace of the already existing container, and then be able to use the openjdk and the libraries of this container to run the jar file.
I am not exactly sure on how to achieve this? Can i achieve this by using volumes or should i be looking for a completely different approach?
On a another note, if i want x containers to run, i will make sure there are x pre-existing containers running. This is to make sure that every container has its own specific librarycontainer to work with. Would this be okay ?
In short, any way that I can speed up the spring boot application by using a second containers that is connected through ipc/net; would be helpfull to my problem.
Spring boot is a purely "runtime" framework.
If I've got your question right, you describe the following situation:
So, say you have a container A with JDK and some jars. This alone doesn't mean that you have a running process though. So its more like a volume with files ready to be reused (or maybe a layer in terms of docker images).
In addition you have another container B with a spring boot application that should be started somehow (probably with the open jdk from container A or its dedicated JDK).
Now what exactly would like like to "speed up"? The size of image (smaller image means faster deployment in CI/CD pipeline for example)? The spring boot application startup time (the time interval between the point of spawning the JVM till the Spring boot application is up and running)? Or maybe you're trying to load less classes in runtime?
The techniques that solve the arised issues are different. But all in all I think you might need to check out the Graal VM integration that among other things might create native images and speed up the startup time. This stuff is fairly new, I by myself haven't tried this yet. I believe its work-in-progress and spring will put an effort on pushing this forward (this is only my speculation so take it with a grain of salt).
Anyway, you may be interested in reading this article
However, I doubt it has something to do with your research as you've described it.
Update 1
Based on your comments - let me give some additional information that might help. This update contains information from the "real-life" working experience and I post it because it might help to find directions in your thesis.
So, we have a spring boot application on the first place.
By default its a JAR and its Pivotal's recommendation there is also an option of WARs(as Josh Long, their developer advocate says: "Make JAR not WAR")
This spring boot application usually includes some web server - Tomcat for traditional Spring Web MVC applications by default, but you can switch it to Jetty, or undertow. If you're running a "reactive applcation" (Spring WebFlux Supported since spring boot 2) your default choice is Netty.
One side note that not all the spring boot driven applications have to include some kind of embedded web server, but I'll put aside this subtle point since you seem to target the case with web servers (you mention tomcat, a quicker ability to serve requests etc, hence my assumption).
Ok, now lets try to analyze what happens when you start a spring boot application JAR.
First of all the JVM itself starts - the process is started, the heap is allocated, the internal classes are loaded and so on and so forth. This can take some time (around a second or even slightly more depending on server, parameters, the speed of your disk etc).
This thread address the question whether the JVM is really slow to start I probably won't be able to add more to that.
Ok, So now, its time to load the tomcat internal classes. This is again can take a couple of seconds on modern servers. Netty seems to be faster, but you can try to download a stanalone distribution of tomcat and start it up on your machine, or create a sample application wihout spring boot but with Embedded Tomcat to see what I'm talking about.
So far so good, not comes our application. As I said in the beginning, spring boot is purely runtime framework. So The classes of spring/spring boot itself must be loaded, and then the classes of the application itself. If The application uses some libraries - they'll be also loaded and sometimes even custom code will be executed during the application startup: Hibernate may check schema and/or scan db schema definitions and even update the underlying schema, Flyway/Liquidbase can execute schema migrations and what not, Swagger might scan controllers and generate documentation and what not.
Now this process in "real life" can even take a minute and even more, but its not because of the spring boot itself, but rather from the beans created in the application that have some code in "constructor"/"post-construct" - something that happens during the spring boot application context initialization. Another side note, I won't really dive into the internals of spring boot application startup process, spring boot is an extremely power framework that has a lot of things happening under the hood, I assume you've worked with spring boot in one way or another - if not, feel free to ask concrete questions about it - I/my colleagues will try to address.
If you go to start.spring.io can create a sample demo application - it will load pretty fast. So it all depends on your application beans.
In this light, what exactly should be optimized?
You've mentioned in comments that there might be a tomcat running with some JARs so that they won't be loaded upon the spring boot application starts.
Well, like our colleagues mentioned, this indeed more resembles a "traditional" web servlet container/application server model that we, people in the industry, "used for ages" (for around 20 years more or less).
This kind of deployment indeed has an "always up-and-running" a JVM process that is "always" ready to accept WAR files - a package archive of your application.
Once it detects the WAR thrown into some folder - it will "deploy" the application by means of creating the Hierarchical class-loader and loading up the application JARs/classes. Whats interesting in your context is that it was possible to "share" the libraries between multiple wars so that were loaded only once. For example, if your tomcat hosts, say, 3 applications (read 3 WARs) and all are using, oracle database driver, you can put this driver's jar to some shared libs folder and it will be loaded only once by the class loader which is a "parent" for the class loaders created per "WAR". This class loader hierarchy thing is crucial, but I believe its outside the scope of the question.
I used to work with both models (spring boot driven with embedded server, an application without spring boot with embedded Jetty server and "old-school" tomcat/jboss deployments ).
From my experience, and, as time proves, many of our colleagues agree on this point, spring boot applications are much more convenient operational wise for many reasons (again, these reasons are out of scope for the question IMO, let me know if you need to know more on this), that's why its a current "trend" and "traditional" deployments are still in the industry because or many non pure technical reasons (historical, the system is "defined to be" in the maintenance mode, you already have a deployment infrastructure, a team of "sysadmins" that "know" how to deploy, you name it, but bottom line nothing purely technical).
Now with all this information you probably understand better why did I suggest to take a look at Graal VM that will allow a faster application startup by means of native images.
One more point that might be relevant. If you're choosing the technology that will allow a fast startup, probably you're into Amazon Lambda or the alternative offered by other cloud providers these days.
This model allows virtually infinite scalability of the "computational" power (CPU) and under the hood they "start" containers and "kill" them immediately once they detect that the container does actually nothing. For this kind of application spring boot simple is not a good fit, but so is basically Java, again, because the JVM process is relatively slow-to-start, so once they start the container like this it will take too long till the time it becomes operational.
You can read Here about what spring ecosystem has to offer at this field, but its not really relevant to your question (I'm trying to provide directions).
Spring boot shines when you need an application that might take some time to start, but once it starts it can do its job pretty fast. And yes, its possible to stop the application (we use the term scale out/scale in) if its not "occupied" by doing an actual work, this approach is also kind of new (~3-4 years) and works best in "managed" deployment environments like kubernetes, amazon ECS, etc.
So if speed-up application start is your goal i think you would need a different approach here a summary of why i think so:
docker: a container is a running instance of an image, you can see an image as a filesystem (actually is more than that but we are talking about libraries). In a container you have jdk (and i guess your image is based on tomcat). Docker engine has a very well designed cache system so containers starts very quickly, if no changes are made on a container docker only need to retrieve some info from a cache. These containers are isolated and for good reasons (security, modularity and talking about libraries isolation let you have more version of a library in different containers). Volumes do not what you think, they are not designed to share libraries, they let you break isolation to make some things for example you can create a volume for your codebase so you have not to rebuild an image for each change during the programming phase, but usually you won't see them in a production environment (maybe for some config files).
java/spring: spring is a framework based on java, java is based on a jdk and java code runs on a vm. So to run a java program you have to start that vm(no other way to do that) and of course you cannot reduce this startup time. Java environment is very powerfull but this is why a lot of people prefer nodejs expecially for little services, java code is slow in startup (minutes versus seconds). Spring as said before is based on java, servelets and context. Spring application lives in that context, so to run a spring application you have to initialize that context.
You are running a container, on top of that you are running a vm, then you are initializing a spring context and finally you are initializing beans of your application. These steps are sequential for dependencies reasons. You cannot initialize docker,vm and a spring context and run somewhere else your application, for example if you in a spring application add a chainfilter you would need to restart application because you would need to add a servlet to your system. If you want to speed-up the process of startup you would need to change the java vm or make some changes in the spring initialization. In summary you are tryng to deal with this problem at a high level instead of low level.
To answer your first question:
I am not exactly sure on how to achieve this? Can I achieve this by using volumes or should I be looking for a completely different approach?
This has to be balanced with the actually capabilities of your infrastructure.
One could have a full optical fibre cable network, a Docker repository in this network and so a bandwidth than can allow for "big" images
One could have a poor connection to its Docker repository, and so need extra small images but a quick connection with the libraries repositories
One could have a blue/green deployment technique and care about neither the size of the images and layer nor the boot time
One thing is, if you care about image and layer size this is good and this is definitely a good practice advised by Docker, but all depends of your needs. The recommendation about keeping images and layers small if for images that you'll distribute. If this is your own image for your own application, then, you should act upon your needs.
Here is for a little bit of my own experience: in a company I was working on, we need the database to be synchronised back from the production to user acceptance test and developer environment.
Because of the size of the production environment, importing the data from an SQL file in the entrypoint of the container took around twenty minutes. This might have been alright for the UAT environment, but, was not for the developer's one.
So after trying all sort of minor improvement in the SQL file (like disabling foreign keys checks and the like), I came up with a totally new approach: I created a big fatty image, in a nightly build that would contain the database already. This is, indeed, against all good practice of Docker, but the bandwidth at the office allowed the container to start in a matter of 5 minutes at worse, compared to the twenty that was before.
So I indeed ended up with a build time of my Docker SQL image being humongous, but a download time acceptable, considering the bandwidth available and a run time reduced to the maximum.
This is taking the advantage of the fact that the build of an image only happens for once, while the start time will happens for all the containers derivating from this image.
To answer your second question:
On a another note, if I want x containers to run, I will make sure there are x pre-existing containers running. This is to make sure that every container has its own specific librarycontainer to work with. Would this be okay?
I would say the answer is: no.
Even in a micro services architecture, each service should be able to do something. As I understand it, you actual not-library-container are unable to do anything because they are tightly coupled to the pre-existence of another container.
This said there are two things that might be of interest to you:
First: remember that you can always build from another pre-existing image, even your own.
Given this would be your library-container Dockerfile
FROM: openjdk:8-jdk-alpine
ARG JAR_FILE=target/*.jar
COPY ${JAR_FILE} app.jar
ENTRYPOINT ["java","-jar","/app.jar"]
Credits: https://spring.io/guides/topicals/spring-boot-docker/#_a_basic_dockerfile
And that you build it via
docker build -t my/spring-boot .
Then you can have another container build on top of that image:
FROM: my/spring-boot
COPY some-specific-lib lib.jar
Secondly: there is a nice technique in Docker to deal with libraries that is called multi-stage builds and that can be used exactly for your case.
FROM openjdk:8-jdk-alpine as build
WORKDIR /workspace/app
COPY mvnw .
COPY .mvn .mvn
COPY pom.xml .
COPY src src
RUN ./mvnw install -DskipTests
RUN mkdir -p target/dependency && (cd target/dependency; jar -xf ../*.jar)
FROM openjdk:8-jdk-alpine
VOLUME /tmp
ARG DEPENDENCY=/workspace/app/target/dependency
COPY --from=build ${DEPENDENCY}/BOOT-INF/lib /app/lib
COPY --from=build ${DEPENDENCY}/META-INF /app/META-INF
COPY --from=build ${DEPENDENCY}/BOOT-INF/classes /app
ENTRYPOINT ["java","-cp","app:app/lib/*","hello.Application"]
Credits: https://spring.io/guides/topicals/spring-boot-docker/#_multi_stage_build
And as you can see in the credits of this multi-stage build, there is even a reference to this technique in the guide of the Spring website.
The manner you are attempting to reach your goal, defies the entire point of containerisation.
We may cycle back to firmly focus on the goal -- you are aiming to "resolve the problem of cold start" and to "speed up the spring boot application".
Have you considered actually compiling your Java application to a native binary?
The essence of the JVM is to support Java's feature of interoperability in a respective host environment. Since containers by their nature inherently resolves interoperability, another layer of resolution (by the JVM) is absolutely irrelevant.
Native compilation of your application will factor out the JVM from your application runtime, therefore ultimately resolving the cold start issue. GraalVM is a tool you could use to do native compilation of a Java application. There are GraalVM Container Images to support your development of your application container.
Below is a sample Dockerfile that demonstrates building a Docker image for a native compiled Java application.
# Dockerfile
FROM oracle/graalvm-ce AS builder
LABEL maintainer="Igwe Kalu <igwe.kalu#live.com>"
COPY HelloWorld.java /app/HelloWorld.java
RUN \
set -euxo pipefail \
&& gu install native-image \
&& cd /app \
&& javac HelloWorld.java \
&& native-image HelloWorld
FROM debian:10.4-slim
COPY --from=builder /app/helloworld /app/helloworld
CMD [ "/app/helloworld" ]
# .dockerignore
**/*
!HelloWorld.java
// HelloWorld.java
public class HelloWorld {
public static void main(String[] args) {
System.out.println("Hello, Native Java World!");
}
}
Build the image and run the container:
# Building...
docker build -t graalvm-demo-debian-v0 .
# Running...
docker run graalvm-demo-debian-v0:latest
## Prints
## Hello, Native Java World!
Spring Tips: The GraalVM Native Image Builder Feature is an article that demos building a Spring Boot application with GraalVM.
Related
It may sound no-brainer. But I wanted to package a Spring Boot Web application and a Java Program (aka Batch job) together and deploy it to PCF.
The Batch will just be sitting there doing nothing. But I will be running it from the Command Line. (pcf run-task).
How do we package this?
I think there are multiple ways that you could do this, but from a high level what you need is to have the application packaged in a way that is consistent with the way you are trying to then execute the applications.
For your Web Application, you are going to want that to be the primary command. When the Java buildpack runs, it'll detect a Spring Boot application and create a start command that is capable of running your application. The simplest path is to not change your packaging in a way that would impact or break this. Just let the buildpack pick the start command and use it.
If you can java -jar your application locally and it launches your web application, then you should be all set. That should be enough for the buildpack.
Now, for your Batch application. If you have a joint code base, where the web and batch code is together in the same project & gets packaged up into the same JAR file, then it is just a matter of taking the command that is generated by the Java buildpack for your web application and modifying it to start a different class (and possibly also memory constraints). The rest of the command should be OK.
If you have separate codebases, then you need to somehow join the two. When you run cf push -p path/to/jar, the cf CLI is going to extract and upload all of the files in your JAR. This is why you need to join the two codebases because you can only upload one JAR file for your application.
I think a simple way to do this would be to just add all of the compiled code for your batch application under a subdirectory in the JAR like batch/... (Do be careful about the subdirectory you pick, you want to ensure that you're not exposing your batch application via download through your web application). It should upload just fine that way and it shouldn't impact the Java buildpacks ability to detect & run your web application.
You can then take the start command generated by the Java buildpack, adjust the classpath to point to the subdirectory, adjust the start class, optionally adjust memory settings and use that to start your batch application.
Again, I'm sure there are probably other ways you could compose things, but I hope this covers the constraining factors of Cloud Foundry & the Java buildpack so that you can mix things together in a way that is suitable for your project.
I'm using the Liferay platform to develop a company portal (version 6.1.1). This portal already have a considerable implementation and database size (174 tables).
As expected, from the beginning the build services and deploys were getting slower as the project were growing.
The problem is that with the current implementation it takes like 20 minutes to perform the 'build services' and about 3/4 minutes to perform a deploy which happens even if i change a simple string in the code. And for every 3 deploys it´s necessary to restart the server because it seems to froze.
My machine specs are:
Intel core i5-3210M
8GB RAM
64bits
And this are the memory args of my liferay server:
-Xms1024m -Xmx1024m -XX:PermSize=1024m -XX:MaxPermSize=1024m
As you know this waiting times have a huge drop of performance in the implementation.
My questions are: is this normal? If yes, what kind of alternatives do i have in a future portal implementation?
Thank you.
174 tables are quite a lot - more than Liferay itself brings. I'd recommend to spread out your application into separately deployable plugins - they don't (technically) need to be in the same plugin, service builder allows you to use the services across different plugins.
Proper dependency management should help you to isolate the functionality that you'll extract into separate applications. Declare which application needs which other application deployed before, and you can access the services cross-context.
To answer your comment-question, sampling with only two projects: Create them, both with service-builder. Let's call them common-portlet and app1-portlet. Obviously, app1-portlet uses components (and services) from common-portlet.
In app1-portlet, edit docroot/WEB-INF/liferay-plugin-package.properties and add the line
required-deployment-contexts=common-portlet
This will make sure that app1-portlet is only deployed when common-portlet is available. Also, common-service.jar, the API of common-portlet, generated with service-biulder, will automatically be put on the classpath of app1-portlet, in other words, you can call the services that you implemented in common-portlet.
Assuming your more abstract portlets have a more stable interface (typically this indicates a proper architecture), changes to app1-portlet (or app2-portlet etc.) will only affect the portlet where you make a change in. Even if you have a change in common-portlet, service-builder will be relatively quick, however, on interface changes you still need to recompile everything, but that's the nature of dependencies. If you don't change your interfaces, you'll only need a redeploy.
We have a web application made in Java, which uses struts2, spring and JasperReport. This application runs on glassfish 4.0.
The libraries of the application are in the WEB-INF/lib folder, and also in glassfish are installed 4 more than uses the same libraries.
Glassfish is configured to use 1024mb for heapspace and 512m for permgen, and the most of the memory consumption when i use libraries per application is in the struts actions and spring aop classes (using netbeans profiler).
The problem we are having is the amount of memory consumed by having libraries in the classloader per application because is to high and generates PermGen errors and we have also noticed that the application run slower with more users.
because of that we try to use shared-libraries, put it in domain1/lib folder and found that with a single deployed application the load time and memory consumption is much lower, and the application works faster in general. But when we deploy the rest of the applications on the server only the first application loaded works well and the rest has errors when we calls struts2 actions.
We believe that is because each application has slightly different settings on struts2 and log4j.
We have also tried to put only certain libraries on glassfish and leaving only struts2 in the application but it shows InvocationTargetException errors because all libraries depend the lib from apache-common and it dont matter if we put those lib on one place or another. Also if we put it in both places the application don’t start.
there any special settings or best practices for using shared-libraries?
Is there a way to use shared-libraries but load settings per application? or we have to change the settings to make them all the same?
Is there any special settings or best practices for using shared-libraries? Is there a way to use shared-libraries but load settings per application? or we have to change the settings to make them all the same?
These are actually interesting questions... I don't use GlassFish but, according to the documentation :
Application-Specific Class Loading
[...]
You can specify module- or application-specific library classes [...] Use the asadmin deploy command with the --libraries option and specify comma-separated paths
[...]
Circumventing Class Loader Isolation
Since each application or individually deployed module class loader universe is isolated, an application or module cannot load classes from another application or module. This prevents two similarly named classes in different applications or modules from interfering with each other.
To circumvent this limitation for libraries, utility classes, or individually deployed modules accessed by more than one application, you can include the relevant path to the required classes in one of these ways:
Using the Common Class Loader
Sharing Libraries Across a Cluster
Packaging the Client JAR for One Application in Another Application
Using the Common Class Loader
To use the Common class loader, copy the JAR files into the domain-dir/lib or as-install/lib directory or copy the .class files (and other needed files, such as .properties files) into the domain-dir/lib/classes directory, then restart the server.
Using the Common class loader makes an application or module accessible to all applications or modules deployed on servers that share the same configuration. However, this accessibility does not extend to application clients. For more information, see Using Libraries with Application Clients. [...]
Then I would try:
Solution 1
put all the libraries except Struts2 jars under domain1/lib ,
put only Struts2 jars under domain1/lib/applibs,
then run
$ asadmin deploy --libraries struts2-core-2.3.15.2.jar FooApp1.war
$ asadmin deploy --libraries struts2-core-2.3.15.2.jar FooApp2.war
To isolate Struts2 libraries classloading while keeping the rest under Common Classloader's control.
Solution 2
put all the libraries except Struts2 jars under domain1/lib ,
put only Struts2 jars under domain1/lib/applibs, in different copies with different names, eg appending the _appname at the jar names
then run
$ asadmin deploy --libraries struts2-core-2.3.15.2_FooApp1.jar FooApp1.war
$ asadmin deploy --libraries struts2-core-2.3.15.2_FooApp2.jar FooApp2.war
To prevent sharing of the libraries by istantiating (mock) different versions of them.
Hope that helps, let me know if some of the above works.
You can try to create what is known as a skinny WAR. Pack all your WARs inside an EAR and move all the common JARs from WEB-INF/lib to the lib/ folder in the EAR (don't forget to set <library-directory> in the application.xml).
I'd bet that placing the libs under lib/ or lib/ext won't resolve your performance issues. You did not write anything about the applications or server settings, like size of application, available Heap and PermGen space, but nonetheless I would recommend to stay with separate libs per app.
If you place the libs in server dirs, they will be shared among all apps. You will loose the option to upgrade only one of your applications to a new framework or to get rid away of any of them. Your deployment will be bound to a specific server architecture.
And you wrote it did not solve your problems, it even may raise new ones.
I would recommend to invest some hours into tuning the server. If it runs with defaults, allocate more PermGen and HeapSpace.
If this does not help, you should analyze in deep what's going wrong. Shared libs might be a solution, but you don't know the problem, yet. IBM offer some cool and free tools to analyze heap dumps, this could be a good starting point.
I came here in search of guidance about installing libraries that are shared among multiple applications or projects. I am deeply disappointed to read that the accepted practice favors installing a copy of every shared library into each project. So, if you have ten Web application, all of which use, e. g., httpcomponents-client, mysql-connector-java, etc., then your installation contains ten copies of each.
This behavior reminds me, painfully, of the way of thinking that motivated me to abandon the mainframe in favor of the PC; the thinking seemed to be "I don't care how many resources my application consumes. In fact, I'd like to be able to brag about what a resource hog it is." Excuse me, please, while I hurl.
The interface exposed by a library is an immutable contract that is not subject to change at the developer's whim.
There is this concept called backwards compatibility. If you break it, you create a new interface.
I know of at least two types of interfaces that adhere to the letter and spirit of these rules.
By far the oldest is the IBM System/370 system libraries. You might have Foo and Foo2, where the latter extends and/or breaks the contract made by the Foo interface in some way that made it incompatible.
From its beginnings in the Bell Labs Unix project, the standard C runtime library has adhered to the above rules.
Though it is much newer, the Microsoft COM interface specification enforces the same rule.
To their credit, Microsoft generally adheres to those rules in the Win32 API, too, although there are a handful of exceptions in that API. To a degree, they went backwards with the .NET Framework, which seems slavishly to follow in the footsteps of the Java environment that it so eagerly seeks to replace.
I've been using libraries since 1978, and my understanding was and is that the goal of putting code into a library was to make it reusable. While maintaining copies of the library code in each application eliminates the need to implement it again for each new project, it severely complicates upgrading, since you now have ten (or more) copies of the library, each of which must be updated.
If libraries adhere to the rule that an interface is an immutable contract, why shouldn't they live in a shared library directory, as do the Unix system libraries that live in its /lib directory, from which everything that runs on the host shares a single copy of the standard C runtime library, Zlib, and so forth.
Color me seriously disappointed.
Our build/deploy process is very tedious, sufficiently manual and error-prone. Could you give proposals for improvement?
So let me describe our deployment strategy and build process.
We are developing system called Application Server (AS for short). It is essentially servlet-based web application hosted on JBoss Web server. AS can be installed in two "environments". Each environment is a directory with webapp's code. This directory is placed on network storage. Storage is mounted to several production servers where JBoss instances are installed. Directory is linked to JBoss' webapps directory. Thus all JBoss instances use the same code for environment. Configuration of JBoss is separate from environment and updated on per instance basis.
So we have two types of patches: webapp patches (for different environments) and configuration patches (for per instance configuration)
Patch is an executable file. In fact it is bash script with embedded binary rpm package. Installation is pretty straight-forward: you just execute file and optionally answer some questions. Important point is that the patch is not a system as a whole - it contains only some classes with fixes and/or scripts that modify configuration files. Classes are copied into WEB-INF/classes (AS is deployed as exploded directory).
The way we build those pathes is:
We take some previous patch files and copy them.
We modify content of patch. The most important part of it is RPM spec. There we change name of patch, change its prerequisite rpm packages and write down actual bash commands for backing up, copying and modifying files. This is one of the most annoying parts because we not always can get actual change-set. That is especially true for new complex features which are spanned among multiple change requests and commits. Also, writing those commands for change-set is tedious and error-prone.
For webapp patches we also modify spec for other environment. Usually they are identical excepting rpm package name.
We put all rpm related files to VCS
We modify build.xml by adding a couple of targets for building new patch. Modification is done by copypasting and editing.
We modify CruiseControl's config by copypasting project and changing ant targets in it
At last, we build a system
Also, I'm interested in any references on patch preparation and deployment practices, preferably for Java applications. I haven't succeed googling that.
The place I work had similar problems, but perhaps not as complex.
We responded by eliminating the concept of patch altogether. We stopped patching, and started simply installing the whole app (even if we do a just a small change).
We now have Cruise Control build complete install kits that happen to contain the build timestamp in the install-kit name. This is a Cruise Control build artifact.
Cruise Control autoinstalls them on a test server, and runs some automated smoke tests. We then run manual tests on the test server. Then we install the artifact on a staging, then production server.
Getting rid of patching caused some people to splutter, "isn't that wasteful if you're just changing a couple of things?" and "why would you overwrite all the software just to patch something?"
But the truth is that good source control, automated install-kit building, and one-step installation has saved us tons of time. It does take a few seconds longer to install, but we can do it far more repeatedly and with less developer labor.
Our current app runs in a single JVM.
We are now splitting up the app into separate logical services where each service runs in its own JVM.
The split is being done to allow a single service to be modified and deployed without impacting the entire system. This reduces the need to QA the entire system - just need to QA the interaction with the service being changed.
For inter service communication we use a combination of REST, an MQ system bus, and database views.
What I don't like about this:
REST means we have to marshal data to/from XML
DB views couple the systems together which defeats the whole concept of separate services
MQ / system bus is added complexity
There is inevitably some code duplication between services
You have set up n JBoss server configurations, we have to do n number of deployments, n number of set up scripts, etc, etc.
Is there a better way to structure an internal application to allow modular development and deployment while allowing the app to run in a single JVM (and achieving the associated benefits)?
I'm a little confused as to what you're really asking here. If you split your application up into different services running across the network, then data marshalling has to occur somewhere.
Having said that, have you investigated OSGi ? You can deploy different bundles (basically, jar files with additional metadata defining the interfaces) into the same OSGi server, and the server will facilitate communication between these bundles transparently, since everything is running within the same JVM - i.e. you call methods on objects in different bundles as you would normally.
An OSGi server will permit unloading and upgrades of bundles at runtime and applications should run normally (if in a degraded fashion) provided the OSGi bundle lifecycle states are respected.
It sounds like your team has a manual QA process and the real issue is automating regression tests so that you can deploy new releases quickly and with confidence. Breaking up the code into separate servers is a workaround for that.
If you're willing to restart the server then one approach might be to compile the code into separate jar files, and deploy a module by dropping in a new jar and restarting. This is largely a matter of structuring your code base so that bad dependencies don't creep in and the calls between jars are made via interfaces that don't change. (Or alternately, use abstract classes so you can add a new method with a default implementation.) Your build system could help by making sure that separately deployed modules can only depend on common interfaces and anything else is a compile error. But note that your compiler isn't going to help you detect incompatibilities when you're swapping in jars that you didn't compile against, so I'm not sure this really avoids having a good QA process.
If you want to deploy new code without restarting the JVM then OSGI is the standard way to do that. (But one that I know little about.)