Spring boot 2 Embedded tomcat 8 - slow request handling [duplicate] - java

I have built a web application that uses
SpringBoot v1.3.6.RELEASE
Tomcat 8.0.36
Java 1.8u101
on CentOS 7.2
The web application is also a SOAP client that calls out to another web application.(JAX-WS RI 2.2.9) If the applications remains idle for 15 seconds the first webservice call stalls for nearly 2 seconds. It appears that the stall happens in o.a.c.loader.WebappClassLoaderBase.
After idle 15 seconds
16:02:36.165 : Delegating to parent classloader org.springframework.boot.loader.LaunchedURLClassLoader#45283ce2
16:02:36.170 : Searching local repositories
16:02:36.170 : findResource(META-INF/services/javax.xml.soap.MetaFactory)
16:02:38.533 : --> Resource not found, returning null
16:02:38.533 : --> Resource not found, returning null
Next request no idle time
16:07:09.981 : Delegating to parent classloader org.springframework.boot.loader.LaunchedURLClassLoader#45283ce2
16:07:09.984 : Searching local repositories
16:07:09.985 : findResource(META-INF/services/javax.xml.soap.MetaFactory)
16:07:09.986 : --> Resource not found, returning null
16:07:09.986 : --> Resource not found, returning null
16:07:09.988 : findResources(META-INF/services
All above messages produced by o.a.c.loader.WebappClassLoaderBase and they are apparently being caused by ClientSOAPHandlerTube.processRequest which is from JAX-WS RI.
You'll notice the first call takes over 2 seconds but subsequent calls take only milliseconds.
I'm wondering if anyone has experienced this behavior?
Possible solutions:
Is it possible to change out the classloader used by tomcat in springboot to use ParallelWebappClassLoader
Or maybe this is a product of the reloadable flag on the classloader but I don't see how to change that flag in springboot.
When run using Jetty as the container this does not occur.
Final Solution: (thanks to Gergely Bacso)
#Bean
public EmbeddedServletContainerCustomizer servletContainerCustomizer() {
return new EmbeddedServletContainerCustomizer() {
#Override
public void customize(ConfigurableEmbeddedServletContainer container) {
if (container instanceof TomcatEmbeddedServletContainerFactory) {
customizeTomcat((TomcatEmbeddedServletContainerFactory) container);
}
}
private void customizeTomcat(TomcatEmbeddedServletContainerFactory tomcatEmbeddedServletContainerFactory) {
tomcatEmbeddedServletContainerFactory.addContextCustomizers(new TomcatContextCustomizer() {
#Override
public void customize(Context cntxt) {
cntxt.setReloadable(false);
}
});
}
};
}

Actually your findings are quite good and you have 90% answered your question already. These two facts:
"it appears that the stall happens in o.a.c.loader.WebappClassLoaderBase"
"when run using Jetty as the container this does not occur."
show that it is going be a Tomcat-related problem because:
o.a.c. stands for org.apache.catalina
Your code works well on another container. (Jetty)
You also observed, that the issue is happening after 15 seconds of idle time. This perfectly corresponds to Tomcat's default checkInterval setting, which is:
The number of seconds between checks for modified classes and
resources, if reloadable has been set to true. The default is 15
seconds.
So in short: currently your reloadable flag is ON, and Tomcat tries to reload your classes which is handy during development, but unacceptable in any other case. The way to switch it off is not via Spring-boot though.
SOLUTION:
You need to locate your context.xml / server.xml where you will find your Context defined like this:
<Context ... reloadable="true">
Remove the reloadable flag, and you have solved the problem. The file itself can be either in $CATALINA_BASE/conf of $CATALINE_HOME/conf, but in reality these locations can be a bit tricky to find if you are using some IDE to manage Tomcat for you.
In case of embedded Tomcat with Spring-boot:
The class you can use to manipulate Tomcat settings is: EmbeddedServletContainerCustomizer.
Through this you can add a TomcatContextCustomizer (addContextCustomizers) so that you can call setReloadable on the context itself.
I do not see any reason for Spring-boot needing this flag on true.

Related

Spring Event for all Contexts being initialized

A similar question has been answered...
Here: Listener for server starup and all spring bean loaded completely
Here: How to add a hook to the application context initialization event?
and several other places.
In my primary application, when the Root Spring context initializes it will trigger three child contexts to initialize. When listening for an event to fire (based on the ContextRefreshedEvent), this results in four total events. This does not give the consumer of these events accurate information regarding the overall state of the application's Spring context.
Unfortunately I don't have the ability to change the primary application. The code that I'm wrestling with now is packaged as a jar and loaded into the primary application using a plugin architecture. I am able to hook into the application's Spring context without issue, and I can successfully receive ContextRefreshEvent triggers.
What I'm looking for is a way to understand if all Spring application contexts in the primary application have completed. One thing I tried was to manually keep track of the starting and completing of the application context (using a map) so that I could tell when all known application contexts were finished initializing. This didn't work for me, as I found that the ContextStartedEvent didn't trigger during Spring's refresh operation.
Additional things that could work for me would be knowing how many application contexts exist or how many beans there are that need to be loaded. That way I could keep track of them as they finished and ultimately know that all application contexts are complete.
Any ideas would be appreciated.
EDIT
In an attempt to ask as pointed a question as possible, I made this question too prescriptive in terms of the implementation. Here is the actual problem that I was trying to solve (in the form of a question):
How might a plugin, packaged as a jar inside of a webapp, tell when the context of the webapp has been deployed successfully in Tomcat?
Through testing, I thought it was fairly safe to assert that the initialization of the Spring context was synonymous with the deploy phase of the application through Tomcat. This may be false, but it lined up well enough.
Broadening the scope of the fix from exclusively using Spring, I was able to come up with a solution to the problem. I will post that as a separate answer.
As of Spring 2.5.X < 4.X
One thing you could do is listen for ContextStartedEvent-s, add them to a map and then once ContextRefreshedEvent-s are triggered, wait while all contexts are refreshed.
This still might be not safe if the context is refreshed multiple times though...but at least a place to start.
As of Spring 4+
Reading ContextRefreshedEvent javadocs:
Event raised when an {#code ApplicationContext} gets initialized or refreshed.
What this means is that you can have your own logic to track whether that context was initialized or refreshed.
#EventListener( { ContextRefreshedEvent.class } )
void contextRefreshedEvent( ContextRefreshedEvent e ) {
ApplicationContext context = ( ApplicationContext ) e.getSource();
System.out.println( "a context refreshed event happened for context: " + context.getDisplayName() );
}
Meaning - the first time you'll get an event with that context - you will know that it was initialized. The following times - you will know that it was refreshed.
Being brutally simple ( no optimizations ) you can do the following:
private final Map<String, String> contextStatuses = new HashMap<>();
#EventListener( { ContextRefreshedEvent.class } )
public void contextRefreshedEvent( ContextRefreshedEvent e ) {
ApplicationContext context = ( ApplicationContext ) e.getSource();
if ( !contextStatuses.containsKey( context.getDisplayName() ) ) { // initialized
contextStatuses.put( context.getDisplayName(), "initialized" );
} else { // refreshed
contextStatuses.put( context.getDisplayName(), "refreshed" );
}
checkAllContextsRefreshed();
}
private void checkAllContextsRefreshed() {
for ( String status : contextStatuses.values() ) {
if ( !"refreshed".equals( status ) ) {
return;
}
}
doWhenAllContextsRefreshed();
}
Per the edit on my question, here is the solution that I went with:
In the plugin that is packaged inside the primary application, I added a check using JMX against the stateName attribute of the WebModule type under Catalina. This isn't the exact code I used, but for simplicity the logic looks like this:
ObjectName name = new ObjectName("Catalina:j2eeType=WebModule,name=//localhost/" + contextPath + ",J2EEApplication=none,J2EEServer=none");
MBeanServer mx = ManagementFactory.getPlatformMBeanServer();
String state = String.valueOf(mx.getAttribute(name, "stateName"));
if (state.equals("STARTED")) {
LOGGER.warn("SUCCESS!");
}
As I mentioned in the edit, equating Tomcat deploy to Spring context initialization was close, but ultimately it wasn't a fair comparison. Using this JMX endpoint gives me a very accurate representation of the state of the Tomcat deploy.

Spring Boot 1.4.0 : REST Controllers don't set reason to HTTP response

I have 2 projects created with Spring Tool Suite "Spring Starter Project":
First project : Spring Boot 1.3.7 (Release) + Web
Second project : Spring Boot 1.4.0 (Release) + Web
I replace the #SpringBootApplication class of each project with this code:
#SpringBootApplication
#RestController
public class TestWebApplication {
public static void main(String[] args) {
SpringApplication.run(TestWebApplication.class, args);
}
#RequestMapping("/hello")
public String sayHello() {
return "hello!";
}
}
When I call GET /hello on each server I get the following responses:
First project (1.3.7) : Status 200 (no reason)
Second project (1.4.0) : Status 200 OK
Is it a bug or am I missing something?
Besides, I can't set a reason to the 1.4.0 project.
The change in behaviour is due to Spring Boot 1.4 using Tomcat 8.5 and Spring Boot 1.3 using Tomcat 8.0. Tomcat 8.5 doesn't send a reason phrase.
Strictly speaking, the reason phrase isn't required. In RFC 7230 the reason phrase is defined as:
reason-phrase = *( HTAB / SP / VCHAR / ons-text )
Where * means zero or more. In other words it's legal for the reason phrase to be empty.
Tomcat 8.5 takes advantage of this and saves some bandwidth by dropping the reason phrase. If this is causing a problem for your client, then it arguably isn't spec compliant as the section of the RFC linked to above has this to say:
The reason-phrase element exists for the sole purpose of providing a textual description associated with the numeric status code, mostly out of deference to earlier Internet application protocols that were
more frequently used with interactive text clients. A client SHOULD
ignore the reason-phrase content.
You haven't said if the absence of the reason phrase is causing a problem. If it's not, then your best course of action is not to worry about it and to move on. If you really want to keep the reason phrase for some reason then switching to Undertow or Jetty is your best bet.
I had raised a bug with feign and it was fixed almost immediately..
Please check https://github.com/OpenFeign/feign/issues/382
Thanks,
KK
Initially the reason phrase was removed by Tomcat-8.5.
However, later a patch was provided to configure it with an attribute in the Http Connector.
All you have to do is set sendReasonPhrase="true" attribute and it will work as before.
Use any of the below method to configure :
Method-1 : Update the .../apache-tomcat-8.5.x/conf/server.xml to include the attribute as below :
<Connector port="8081" protocol="HTTP/1.1"
connectionTimeout="20000"
redirectPort="8443" sendReasonPhrase="true" />
Method-2 : Configure through bean. (add in the config file)
#Bean
public EmbeddedServletContainerFactory servletContainerFactory() {
TomcatEmbeddedServletContainerFactory factory = new TomcatEmbeddedServletContainerFactory();
factory.addConnectorCustomizers(connector ->
((AbstractProtocol) connector.getProtocolHandler()).setSendReasonPhrase(true));
return factory;
}
NOTE : This config will be dropped in Tomcat-9.0.x (ie : response message will not be sent)
Refs :
Bug Report & Discussion
Tomcat documentation for configuring the attribute

WildFly EJB UnknownSessionID

I just started learning ejb and now have problems.
My ejb code is
#Stateful
#StatefulTimeout(value=80, unit=TimeUnit.SECONDS)
public class HelloWorldBean
{
int i = 0;
public int SayHello()
{
return i++;
}
}
It's only for testing. Ok, so, after 80 sec I get problem from WildFly
javax.ejb.NoSuchEJBException: WFLYEJB0168: Could not find EJB with id UnknownSessionID [5156495653657051576570495270526865695251507057526654654868486852]
1) I thought that after 80 sec ejb should be deleted and after refresh page I'll get new instance. Or after 80 sec instance going to passivation (saving on hard drive) ?
2) What is this problem with UnknownSessionID? Why WildFly doesn't want assign ID to session ?
3) With this code example - Why if I use two different browsers at the same time I have the same instance? I thought stateful bean works as one_bean-to-one_user ? So in Google Chrome and Firefox, for example, I should start from i=0 and shouldn't have any similar data between browser sessions
I'll be appreciated for your help!
1) The specification says exactly that removal will happen after a stateful timeout. Wildfly follows the EJB specification.
2) An Unknown session ID is a way of saying that the bean does not exist.
3) You must have been using the same servlet to access the bean code above. This means that you must have been using just a single client to access the bean.

Can a deployment stop itself?

I'm currently searching for a way to stop a deployment on wildfly programmatically.
Background:
The application does some health checks in its startup phase using an MBean.
If the app determines that the environment is not valid because some resources are missing, it needs to stop its own deployment.
The way it was:
The application was formerly running on JBoss 4 and simply stopped the whole app server calling the shutdown command using JMX.
In case this command failed, it simply terminated the whole JVM using System.exit(1).
Current problems:
Calling shutdown() via JMX does not work on wildfly since the whole server hangs when trying to stop it from within a deployed application.
System.exit() will also not work since wildly must be catching the command in any way.
So does anyone know how to stop the server from within the deployment or stop the deployment process or undeploy the app?
Thanks a lot!
I assume the core question is stopping the deployment process if some health checks fail. Throwing a run-time exception during app startup is enough to do the job.
#Startup
#Singleton
public class StartupBean {
#PostConstruct
public void start() {
//your checks
boolean check = doHealthCheck();
if(!check){
throw new RuntimeException("Your error message");
}
}
}
or
#Startup
#Singleton
public class StartupBean {
#PostConstruct
public void start() {
//your checks
boolean check = doHealthCheck();
if(!check){
throw new Error("Your error message");
}
}
}
I suggest you to try WildFly CLI:
Running the CLI
or use Marker Files.
But in any case, I'm not sure how the server will behave. For example what will happen when You add file myWarName.dodeploy when there is myWarName.isdeploying. So let us know when You will earn some experience in this topic (it is quite interesting).
Ok, I did not yet manage to undeploy the app but I've been able to shutdown the server in case of an error. This is not perfect but matches the behavior of the app on the older version of JBoss, so I think it's not too bad at all.
I'm now calling the CLI interface like so
try {
String jbossBinDir = System.getProperty("jboss.server.base.dir").replace("standalone", "bin");
Runtime.getRuntime().exec("sh " + jbossBinDir + "/jboss-cli.sh -c command=:shutdown");
} catch(IOException e) {
...
}
This works reliable for us.
In my comment above I stated that the execution returns with an error code but this was probably the case because I must have had a typo in the command call.
We're using a CDI Extension to abort the deployment if our DB schema doesn't match the application's expectation:
class MyValidatingExtension implements javax.enterprise.inject.spi.Extension {
void deploymentValidationFinished(#Observes AfterDeploymentValidation afterDeploymentValidation) {
if (!stateExpected) {
afterDeploymentValidation.addDeploymentProblem(new IDontLikeThisException());
}
}
}
The deployment of the WAR will fail with the stacktrace of the exception listed as DeploymentProblem, leaving your WAR in an undeployed state. This solution is independent of your container implementation, it uses a CDI standard mechanism only. Note that this will not stop/shutdown the server!

Abort Tomcat start-up

Specifically, I want Tomcat to abort start-up when my web app can't initialize (due to a configuration error).
I'm tried to use the Tomcat shutdown port. That's not ready when my web app is starting up.
-Dorg.apache.catalina.startup.EXIT_ON_INIT_FAILURE=true, doesn't work. Tomcat starts up just fine.
Throw an uncaught Exception. I've tried throwing an exception. Tomcat still initializes correctly but my web app is in a non-functional state.
Use the Tomcat JMX API. None of the shutdown methods will cause Tomcat to shut down.
So far, the only other option I've got is System.exit(-1). That does work, but I'm concerned that doing that to Tomcat will cause problems.
Tomcat is designed to be resilient against badly written applications that don't start up.
org.apache.catalina.startup.EXIT_ON_INIT_FAILURE is for stopping Tomcat if one of Tomcat's components, such as one of the connectors, doesn't start (e.g. because the port is in use).
System.exit() is ugly but it will work and it is safe as far as Tomcat is concerned (as for that matter is kill -9). It is applications that may not like this rather brutal approach to shutdown.
There is a Tomcat specific way to do this more cleanly. Add a custom LifecycleListener to the Server component and have it respond to AFTER_START_EVENT. Have the listener navigate the Container hierarchy from the Server to the Contexts [Server->Service(s)->Engine-Host(s)->Context(s)] and check that each context is in the state STARTED. If not, call stop() and then destroy() on the Server.
Here's an implementation of the listener which aborts the Tomcat startup, as mentioned by Mark Thomas. StrictStateCheckListener is an after-start server listener. It checks the state of all Container, including engine, host, and context. The server start-up will be aborted if there's any component failed to start (state different from STARTED).
package com.example.lifecycle;
import java.util.logging.*;
import org.apache.catalina.*;
public class StrictStateCheckListener implements LifecycleListener {
private static final Logger LOGGER = Logger.getLogger(StrictStateCheckListener.class.getName());
private boolean hasFailures;
private String containerState;
#Override
public void lifecycleEvent(LifecycleEvent event) {
String type = event.getType();
Lifecycle lifecycle = event.getLifecycle();
if (lifecycle instanceof Server && type.equals(Lifecycle.AFTER_START_EVENT)) {
Server server = (Server) lifecycle;
hasFailures = false;
// Check status of each container
StringBuilder sb = new StringBuilder("Status:").append(System.lineSeparator());
for (Service service : server.findServices()) {
checkState(service.getContainer(), sb, "");
}
if (LOGGER.isLoggable(Level.INFO)) {
LOGGER.info(sb.toString());
}
if (hasFailures) {
/*
* The server will not be stopped by this listener. Indeed,
* an exception is raised to trigger shutdown hook. See:
* `org.apache.catalina.startup.Catalina`.
*/
throw new IllegalStateException(containerState);
}
}
}
private void checkState(Container container, StringBuilder report, String indent) {
if (!hasFailures && container.getState() != LifecycleState.STARTED) {
hasFailures = true;
containerState = stateOf(container);
}
report.append(indent).append(stateOf(container));
for (Container child : container.findChildren()) {
checkState(child, report, indent + " ");
}
}
private String stateOf(Container container) {
String className = container.getClass().getSimpleName();
String stateName = container.getStateName();
return String.format("%s[%s]: %s%n", className, container.getName(), stateName);
}
}
Register this listener to your server component via server.xml:
<Server>
...
<Listener className="com.example.lifecycle.StrictStateCheckListener" />
...
</Server>
Your point No 1
Do you mean when uninitialized state exception is thrown from application you are calling tomcat shutdown.bat file.Have you tried to call shutdown.bat file from your program.
I know this question is a bit old, but I'm going a bit further here thinking out of the box to solution you are seeking. It looks like your requirement is to stop tomcat when your webapp fails to initialize. This is a bit different than "abort Tomcat start-up". Tomcat itself is already started and will start to deploy your webapp right after it started up. Based on your requirement, I assume that you don't have any other webapp. If that's the case, why don't you just use Spring Boot? Spring boot provides a way to deploy self contained web application. In this case, if initialization fails, application will not start.

Categories

Resources