On our production (tomcat, java8) servers, from time to time, we're seeing the following stacktrace in the logfiles:
java.lang.Exception
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:142)
at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:102)
at org.springframework.web.method.annotation.ModelAttributeMethodProcessor.createAttribute(ModelAttributeMethodProcessor.java:139)
at org.springframework.web.servlet.mvc.method.annotation.ServletModelAttributeMethodProcessor.createAttribute(ServletModelAttributeMethodProcessor.java:82)
at org.springframework.web.method.annotation.ModelAttributeMethodProcessor.resolveArgument(ModelAttributeMethodProcessor.java:106)
at org.springframework.web.method.support.HandlerMethodArgumentResolverComposite.resolveArgument(HandlerMethodArgumentResolverComposite.java:121)
at org.springframework.web.method.support.InvocableHandlerMethod.getMethodArgumentValues(InvocableHandlerMethod.java:158)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:128)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:635)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:728)
at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:591)
at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:527)
at org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.java:380)
at org.apache.catalina.core.StandardHostValve.throwable(StandardHostValve.java:323)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:166)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:660)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:798)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:806)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
What could be the cause of this exception and how can I trace the origin?
Thanks
Based on e.g.: Spring Boot custom error controller is showing the incorrect exception this
might be an Exception thrown from your own code, so maybe search your own codebase for places where Exception is thrown.
To complicate matters a bit this is all implemented in native code, see: class newInstance propagates checked and unchecked exceptions - is it true?
But maybe it is further obfuscated because the original exception on your production system has already been generated many times (since startup) and has been converted to extremely optimized native code.
In that case you can:
see this in the logging when the application just is started; or
if you start Java with this extra parameter: -XX:-OmitStackTraceInFastThrow (before your applications jar/Main class name and application arguments)
I hope that helps, I'm curious if this is the case, so please let me (and all other readers that might come here later know).
I don’t have spring mvc code in front of me right now, but it looks like there is a faulty end point.
ModelAttributeMethodProcessor tries to create a model attribute (that should be basically something in your code, and this class should have constructor) and fails to instantiate it.
It’s possible that calls like this will respond with 500 http status (or at least not 200). Now, probably you have an access log (log with all the requests / response statuses) somewhere - it can be a gateway, an applicative load balancer, ELB maybe, it really depends where you’re running.
In this case you might be able to examine these logs and try to find the endpoints that sometimes return 5XX codes. Maybe there will be the same endpoint that sometimes doesn’t work. Now as you say, it shouldn’t really cause an issue to your server, but the client who expects to get the result by calling this endpoint obviously doesn’t get the desired response.
Related
We had an outage recently, and in the log of our application, we can see this exception.
Looks like the data in a table is being modified while primefaces is sorting ¿How is this possible?
This is something that doesn't happen frequently, it just happened once in a production enviroment.
Also this same stack trace repeats like ten or more times in seconds.
05-Jul-2022 19:13:53.583 SEVERE [http-nio-8080-exec-746] com.sun.faces.application.view.FaceletViewHandlingStrategy.handleRenderException Error Rendering View[/pages/example.xhtml]
java.util.ConcurrentModificationException
at java.util.ArrayList.sort(ArrayList.java:1466)
at java.util.Collections.sort(Collections.java:177)
at org.primefaces.component.datatable.feature.SortFeature.singleSort(SortFeature.java:167)
at org.primefaces.component.datatable.DataTableRenderer.preRender(DataTableRenderer.java:141)
at org.primefaces.component.datatable.DataTableRenderer.encodeEnd(DataTableRenderer.java:93)
at javax.faces.component.UIComponentBase.encodeEnd(UIComponentBase.java:949)
at javax.faces.component.UIComponent.encodeAll(UIComponent.java:1912)
at javax.faces.component.UIComponent.encodeAll(UIComponent.java:1908)
at javax.faces.render.Renderer.encodeChildren(Renderer.java:176)
at javax.faces.component.UIComponentBase.encodeChildren(UIComponentBase.java:918)
at javax.faces.component.UIComponent.encodeAll(UIComponent.java:1905)
at com.sun.faces.context.PartialViewContextImpl$PhaseAwareVisitCallback.visit(PartialViewContextImpl.java:638)
at com.sun.faces.component.visit.PartialVisitContext.invokeVisitCallback(PartialVisitContext.java:183)
at javax.faces.component.UIForm.visitTree(UIForm.java:405)
at javax.faces.component.UIComponent.visitTree(UIComponent.java:1747)
at javax.faces.component.UIComponent.visitTree(UIComponent.java:1747)
at com.sun.faces.context.PartialViewContextImpl.processComponents(PartialViewContextImpl.java:423)
at com.sun.faces.context.PartialViewContextImpl.processPartial(PartialViewContextImpl.java:342)
at org.primefaces.context.PrimePartialViewContext.processPartial(PrimePartialViewContext.java:65)
at javax.faces.context.PartialViewContextWrapper.processPartial(PartialViewContextWrapper.java:252)
at org.omnifaces.context.OmniPartialViewContext.processPartial(OmniPartialViewContext.java:122)
at javax.faces.component.UIViewRoot.encodeChildren(UIViewRoot.java:1124)
at javax.faces.component.UIComponent.encodeAll(UIComponent.java:1905)
at com.sun.faces.application.view.FaceletViewHandlingStrategy.renderView(FaceletViewHandlingStrategy.java:465)
at com.sun.faces.application.view.MultiViewHandler.renderView(MultiViewHandler.java:194)
at javax.faces.application.ViewHandlerWrapper.renderView(ViewHandlerWrapper.java:151)
at javax.faces.application.ViewHandlerWrapper.renderView(ViewHandlerWrapper.java:151)
at org.omnifaces.viewhandler.OmniViewHandler.renderView(OmniViewHandler.java:155)
at com.sun.faces.lifecycle.RenderResponsePhase.execute(RenderResponsePhase.java:126)
at com.sun.faces.lifecycle.Phase.doPhase(Phase.java:100)
at com.sun.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:223)
at javax.faces.webapp.FacesServlet.service(FacesServlet.java:671)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at ar.com.dux.jsf.filters.CacheFilter.doFilter(CacheFilter.java:27)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at ar.com.dux.jsf.filters.SecurityFilter.doFilter(SecurityFilter.java:84)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.primefaces.webapp.filter.FileUploadFilter.doFilter(FileUploadFilter.java:89)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.omnifaces.filter.GzipResponseFilter.doFilter(GzipResponseFilter.java:183)
at org.omnifaces.filter.HttpFilter.doFilter(HttpFilter.java:108)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:196)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:698)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:364)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:624)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:831)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1650)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
We are using JSF 2.3.9, Primefaces 8.0, Omnifaces 3.11, Tomcat 8 and the bean is ViewScoped (javax.faces.view.ViewScoped not omnifaces).
The page is very complex, but his basic function is to generate an invoice so we have a section for the client data, another section for products that are added to the invoice and another section for payment info (using facelets include).These last two sections have datatables with sorting, and are updated from all the other sections.
There are two possible causes for that issue:
Multiple threads working on the same list
A single thread that modifies the list while iterating over it (e.g. counted for or foreach)
The second option should not be the cause here, as the stacktrace contains just a call to ArrayList.sort. I don't think, that the JDK is faulty at that place.
The first option is more likely:
Simultaneous AJAX requests
com.sun.faces.context.PartialViewContextImpl.processPartial should be an indication, that this happens during AJAX
Shared list due to wrong bean scope - e.g. session or even application
Correct bean scope, but the list comes from a shared source
You can try 1 of 2 solutions...
Wrap your backing bean list Collections.synchronizedList.... List<Object> objList = Collections.synchronizedList(new ArrayList<Object>());
Try java.util.concurrent.CopyOnWriteArrayList https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/concurrent/CopyOnWriteArrayList.html
it's expensive to modify, but wait-free to read.
I'm working on a Java8 Tomcat8 Jersey2 web app and it's randomly logging a bizarre exception after successfully responding to requests. I have a filter in place for all requests to do some logging and add some values to the MDC (e.g. requestId) and the exception is coming from here. Here's the stack:
- HTTP 404 Not Found
javax.ws.rs.NotFoundException: HTTP 404 Not Found
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:323)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154)
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.xyz.gatorws.webapp.LoggingFilter.doFilter(LoggingFilter.java:34)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:543)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:678)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:609)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:810)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1623)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
The odd thing is though that the request was successfully processed and a success response (200 sent back to the calling client before this stack is showing up. I'm at a loss. Also, the line in the stack coming from my code (com.xyz.gatorws.webapp.LoggingFilter.doFilter(LoggingFilter.java:34)) is the following:
chain.doFilter(httpRequest, response);
This is bizarre given that the filter runs before the request is processed. Argh! I must be missing something. I clear the logging context in the LoggingFilter.destroy() method so I can't see that another request is involved, but I'm really at a loss.
Any thoughts would be much appreciated.
In case this can help the asker or others
My configuration was Java9 Tomcat9 Jersey2. I got a problem like this one cause the matched method wasn't annotated with any HTTP method annotation. In my case the #GET was missing. The method was returning a Response object and after the last statement before the return Response ... it was throwing NotFoundException. I think Jersey thought this method was a subresource locator that was found due to the #Path annotation.
So i changed from this:
#Path("unique/page")
#Consumes("*/*")
#RolesAllowed("owner")
public Response getUnique(#PathParam("userUUID") String userUUID) throws RuntimeException {
LogNdc.log("userUUID: " + userUUID);
SearchCompanieService scs = new SearchCompanieService();
CompanyPagePojo cpp = scs.searchByFieldNameAutoCommit("userPojoOwnerId", userUUID);
System.out.println(cpp);
return Response.status(Status.OK).entity(cpp).build();
}
To this:
#Path("unique/page")
#GET
#Consumes("*/*")
#RolesAllowed("owner")
public Response getUnique(#PathParam("userUUID") String userUUID) throws RuntimeException {
LogNdc.log("userUUID: " + userUUID);
SearchCompanieService scs = new SearchCompanieService();
CompanyPagePojo cpp = scs.searchByFieldNameAutoCommit("userPojoOwnerId", userUUID);
System.out.println(cpp);
return Response.status(Status.OK).entity(cpp).build();
}
But in my case the request was not generating any 200 response code due to the exception, the client was getting 500 because I had a RuntimeExceptionMapper.
I'm facing some problems with my project (probably because windows, antivirus or both).
What's happening is the following error:
SEVERE: Exception Processing ErrorPage[errorCode=404, location=/404.jsp]
org.apache.jasper.JasperException: org.apache.jasper.JasperException: Unable to compile class for JSP
at org.apache.jasper.servlet.JspServletWrapper.handleJspException(JspServletWrapper.java:565)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:389)
at org.apache.jasper.servlet.JspServlet._serviceJspFile(JspServlet.java:386)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:40001)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:330)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:728)
at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:472)
at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:395)
at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:316)
at org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.java:395)
at org.apache.catalina.core.StandardHostValve.status(StandardHostValve.java:254)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:177)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:650)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:800)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:800)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1471)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.jasper.JasperException: Unable to compile class for JSP
at org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:610)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:368)
... 26 more
Caused by: java.io.IOException: classFile.delete() failed
at org.apache.jasper.compiler.SmapUtil$SDEInstaller.install(SmapUtil.java:204)
at org.apache.jasper.compiler.SmapUtil.installSmap(SmapUtil.java:166)
at org.apache.jasper.compiler.JDTCompiler.generateClass(JDTCompiler.java:510)
at org.apache.jasper.compiler.Compiler.__compile(Compiler.java:379)
at org.apache.jasper.compiler.Compiler.compile(Compiler.java:41002)
at org.apache.jasper.compiler.Compiler.compile(Compiler.java:351)
at org.apache.jasper.compiler.Compiler.compile(Compiler.java:335)
at org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:595)
... 27 more
Looks like the file that classFile.delete() is trying to delete is in use.
We cannot add a exception in the antivirus....
I found a post from 2006 that says "My workaround was to modify the SmapUtil class. If the rename fails on the first try, it waits 50ms then retries (loops as needed). So far it always works on the second try after the 50ms delay."
My question is, where that SmapUtil class is located and how to edit that delete (and rename) methods?
I can't find this info anywhere
Thanks
Ensure you have Tomcat and Jasper configured for running in production.
To solve your specific issue you could set the Jasper option suppressSmap to true or precompile your JSP files, or both.
SMAP provides JSR-045 support for Jasper; It creates mappings from the generated bytecode back to the source JSP files to aid in debugging.
See this page for information on Jasper configuration.
I was able to "fix" it by changing that Tomcat method (delete and rename)
Looks like that whichever file it was trying to delete/rename, was being user by another process (likely antivirus)
So I cloned the source-code from Tomcat's Github, changed the class and voilá, it's working now.
That method was something like:
if ( !classPath.delete(..) )
throw Exception...
then I changed for something like:
while ( !classPath.delete(..) && count < 10){
Thread.wait(50);
count++
}
if (count == 10)
Throw Exception
We use tomcat 7 with Spring 4.2.5 and Spring-MVC. We have several REST interfaces that some of our customers can also call. Recently we included jackson XML (from fasterxml) so XML interfaces could be handled with jackson instead of JAXB. Since then after a tomcat restart the CPU load grows more and more as time goes on, resulting in 100% CPU load after a few hours.
We were able to pinpoint the cause to one particular REST XML POST interface that one of our customers calls frequently. We also were able to observe that such requests fail with a HTTP 499 ClientClosedRequest error. The number of those 499 coincides with the number of locked threads that we retrieved via kill -3.
The stacktrace of those locked threads looks like this:
"catalina-exec-1" #50 daemon prio=5 os_prio=0 tid=0x00007f11c0001800 nid=0x1d89 runnable [0x00007f11f46e6000]
java.lang.Thread.State: RUNNABLE
at com.sun.xml.stream.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:353)
at com.sun.xml.stream.XMLReaderImpl.next(XMLReaderImpl.java:565)
at com.fasterxml.jackson.dataformat.xml.XmlFactory._initializeXmlReader(XmlFactory.java:653)
at com.fasterxml.jackson.dataformat.xml.XmlFactory._createParser(XmlFactory.java:536)
at com.fasterxml.jackson.dataformat.xml.XmlFactory._createParser(XmlFactory.java:29)
at com.fasterxml.jackson.core.JsonFactory.createParser(JsonFactory.java:820)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3084)
at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.readJavaType(AbstractJackson2HttpMessageConverter.java:222)
at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.read(AbstractJackson2HttpMessageConverter.java:209)
at org.springframework.web.servlet.mvc.method.annotation.AbstractMessageConverterMethodArgumentResolver.readWithMessageConverters(AbstractMessageConverterMethodArgumentResolver.java:197)
at org.springframework.web.servlet.mvc.method.annotation.RequestResponseBodyMethodProcessor.readWithMessageConverters(RequestResponseBodyMethodProcessor.java:147)
at org.springframework.web.servlet.mvc.method.annotation.RequestResponseBodyMethodProcessor.resolveArgument(RequestResponseBodyMethodProcessor.java:125)
at org.springframework.web.method.support.HandlerMethodArgumentResolverComposite.resolveArgument(HandlerMethodArgumentResolverComposite.java:99)
at org.springframework.web.method.support.InvocableHandlerMethod.getMethodArgumentValues(InvocableHandlerMethod.java:161)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:128)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:110)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:817)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:731)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:959)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:893)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:968)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:870)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:646)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:844)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:316)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:126)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:90)
We were unable to reproduce this with our own requests. We even tried to provoke a 499 error by aborting our requests prematurely, but to no avail.
Even if we were to reproduce this - how could we actually fix this, since Spring and Jackson are doing all the work.
It is unfortunately impossible to go back to JAXB since some of our new features rely on jackson.
My application get multiple request per second we have bot's crawling our site. I use Lucene for Indexing and searching. For the 1st request when the site is restart application opens the Lucene indexed file and store it. So from second request it will look into the stored object.
But the issue is till the file is completely open and store there are multiple request which will try to open the file again.
This causes the site to go out of memory after 5-10 minutes.
This are the following errors.
java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.TreeMap.put(Unknown Source)
at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:61)
at org.apache.lucene.codecs.lucene42.Lucene42FieldInfosReader.read(Lucene42FieldInfosReader.java:96)
at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:121)
at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:56)
at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:62)
at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:783)
at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:52)
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:66)
at com.webjaguar.web.frontend.LuceneCategery.getLuceneProduct(LuceneCategery.java:166)
at com.webjaguar.web.frontend.CategoryController.handleRequest(CategoryController.java:1034)
at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:923)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:852)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:778)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:624)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:312)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:116)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:324)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:324)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:324)
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
SECOND ERROR
Exception in thread "Lucene Merge Thread #9" org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: Java heap space
at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:541)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:514)
Caused by: java.lang.OutOfMemoryError: Java heap space
java.lang.IllegalStateException: this writer hit an OutOfMemoryError; cannot commit
at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2661)
at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2827)
at org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:981)
at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:883)
at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:845)
at com.webjaguar.thirdparty.lucene.LuceneProductIndexer.reIndex(LuceneProductIndexer.java:750)
at com.webjaguar.web.quartz.LuceneProductJob.autoIndex(LuceneProductJob.java:90)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:273)
at org.springframework.scheduling.quartz.MethodInvokingJobDetailFactoryBean$MethodInvokingJob.executeInternal(MethodInvokingJobDetailFactoryBean.java:311)
at org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:113)
at org.quartz.core.JobRunShell.run(JobRunShell.java:223)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:549)
THIS LINE IS THE ERROR LINE
reader = DirectoryReader.open(NIOFSDirectory.open(indexFile));
Is there a way to lock the file till it is store. Any solution to improve the way it should be implemented
You should have a look at the LockFactory of the NIOFSDirectory (inherited from parent Directory).
See LockFactory Javadoc for little more informations
Additional to this, your requirements look like a NRT (near-real-time) use case for me. If you like to index and search within a short time period and indexing will be done continuous a NRT implementation would make sense. I'm not sure if this is already a feature of lucene v4.2.
See Simple NRT tutorial for additional information.