We use tomcat 7 with Spring 4.2.5 and Spring-MVC. We have several REST interfaces that some of our customers can also call. Recently we included jackson XML (from fasterxml) so XML interfaces could be handled with jackson instead of JAXB. Since then after a tomcat restart the CPU load grows more and more as time goes on, resulting in 100% CPU load after a few hours.
We were able to pinpoint the cause to one particular REST XML POST interface that one of our customers calls frequently. We also were able to observe that such requests fail with a HTTP 499 ClientClosedRequest error. The number of those 499 coincides with the number of locked threads that we retrieved via kill -3.
The stacktrace of those locked threads looks like this:
"catalina-exec-1" #50 daemon prio=5 os_prio=0 tid=0x00007f11c0001800 nid=0x1d89 runnable [0x00007f11f46e6000]
java.lang.Thread.State: RUNNABLE
at com.sun.xml.stream.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:353)
at com.sun.xml.stream.XMLReaderImpl.next(XMLReaderImpl.java:565)
at com.fasterxml.jackson.dataformat.xml.XmlFactory._initializeXmlReader(XmlFactory.java:653)
at com.fasterxml.jackson.dataformat.xml.XmlFactory._createParser(XmlFactory.java:536)
at com.fasterxml.jackson.dataformat.xml.XmlFactory._createParser(XmlFactory.java:29)
at com.fasterxml.jackson.core.JsonFactory.createParser(JsonFactory.java:820)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3084)
at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.readJavaType(AbstractJackson2HttpMessageConverter.java:222)
at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.read(AbstractJackson2HttpMessageConverter.java:209)
at org.springframework.web.servlet.mvc.method.annotation.AbstractMessageConverterMethodArgumentResolver.readWithMessageConverters(AbstractMessageConverterMethodArgumentResolver.java:197)
at org.springframework.web.servlet.mvc.method.annotation.RequestResponseBodyMethodProcessor.readWithMessageConverters(RequestResponseBodyMethodProcessor.java:147)
at org.springframework.web.servlet.mvc.method.annotation.RequestResponseBodyMethodProcessor.resolveArgument(RequestResponseBodyMethodProcessor.java:125)
at org.springframework.web.method.support.HandlerMethodArgumentResolverComposite.resolveArgument(HandlerMethodArgumentResolverComposite.java:99)
at org.springframework.web.method.support.InvocableHandlerMethod.getMethodArgumentValues(InvocableHandlerMethod.java:161)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:128)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:110)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:817)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:731)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:959)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:893)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:968)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:870)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:646)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:844)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:316)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:126)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:90)
We were unable to reproduce this with our own requests. We even tried to provoke a 499 error by aborting our requests prematurely, but to no avail.
Even if we were to reproduce this - how could we actually fix this, since Spring and Jackson are doing all the work.
It is unfortunately impossible to go back to JAXB since some of our new features rely on jackson.
Related
On our production (tomcat, java8) servers, from time to time, we're seeing the following stacktrace in the logfiles:
java.lang.Exception
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:142)
at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:102)
at org.springframework.web.method.annotation.ModelAttributeMethodProcessor.createAttribute(ModelAttributeMethodProcessor.java:139)
at org.springframework.web.servlet.mvc.method.annotation.ServletModelAttributeMethodProcessor.createAttribute(ServletModelAttributeMethodProcessor.java:82)
at org.springframework.web.method.annotation.ModelAttributeMethodProcessor.resolveArgument(ModelAttributeMethodProcessor.java:106)
at org.springframework.web.method.support.HandlerMethodArgumentResolverComposite.resolveArgument(HandlerMethodArgumentResolverComposite.java:121)
at org.springframework.web.method.support.InvocableHandlerMethod.getMethodArgumentValues(InvocableHandlerMethod.java:158)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:128)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:635)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:728)
at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:591)
at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:527)
at org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.java:380)
at org.apache.catalina.core.StandardHostValve.throwable(StandardHostValve.java:323)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:166)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:660)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:798)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:806)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
What could be the cause of this exception and how can I trace the origin?
Thanks
Based on e.g.: Spring Boot custom error controller is showing the incorrect exception this
might be an Exception thrown from your own code, so maybe search your own codebase for places where Exception is thrown.
To complicate matters a bit this is all implemented in native code, see: class newInstance propagates checked and unchecked exceptions - is it true?
But maybe it is further obfuscated because the original exception on your production system has already been generated many times (since startup) and has been converted to extremely optimized native code.
In that case you can:
see this in the logging when the application just is started; or
if you start Java with this extra parameter: -XX:-OmitStackTraceInFastThrow (before your applications jar/Main class name and application arguments)
I hope that helps, I'm curious if this is the case, so please let me (and all other readers that might come here later know).
I don’t have spring mvc code in front of me right now, but it looks like there is a faulty end point.
ModelAttributeMethodProcessor tries to create a model attribute (that should be basically something in your code, and this class should have constructor) and fails to instantiate it.
It’s possible that calls like this will respond with 500 http status (or at least not 200). Now, probably you have an access log (log with all the requests / response statuses) somewhere - it can be a gateway, an applicative load balancer, ELB maybe, it really depends where you’re running.
In this case you might be able to examine these logs and try to find the endpoints that sometimes return 5XX codes. Maybe there will be the same endpoint that sometimes doesn’t work. Now as you say, it shouldn’t really cause an issue to your server, but the client who expects to get the result by calling this endpoint obviously doesn’t get the desired response.
My application get multiple request per second we have bot's crawling our site. I use Lucene for Indexing and searching. For the 1st request when the site is restart application opens the Lucene indexed file and store it. So from second request it will look into the stored object.
But the issue is till the file is completely open and store there are multiple request which will try to open the file again.
This causes the site to go out of memory after 5-10 minutes.
This are the following errors.
java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.TreeMap.put(Unknown Source)
at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:61)
at org.apache.lucene.codecs.lucene42.Lucene42FieldInfosReader.read(Lucene42FieldInfosReader.java:96)
at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:121)
at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:56)
at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:62)
at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:783)
at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:52)
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:66)
at com.webjaguar.web.frontend.LuceneCategery.getLuceneProduct(LuceneCategery.java:166)
at com.webjaguar.web.frontend.CategoryController.handleRequest(CategoryController.java:1034)
at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:923)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:852)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:778)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:624)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:312)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:116)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:324)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:324)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:324)
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
SECOND ERROR
Exception in thread "Lucene Merge Thread #9" org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: Java heap space
at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:541)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:514)
Caused by: java.lang.OutOfMemoryError: Java heap space
java.lang.IllegalStateException: this writer hit an OutOfMemoryError; cannot commit
at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2661)
at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2827)
at org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:981)
at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:883)
at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:845)
at com.webjaguar.thirdparty.lucene.LuceneProductIndexer.reIndex(LuceneProductIndexer.java:750)
at com.webjaguar.web.quartz.LuceneProductJob.autoIndex(LuceneProductJob.java:90)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:273)
at org.springframework.scheduling.quartz.MethodInvokingJobDetailFactoryBean$MethodInvokingJob.executeInternal(MethodInvokingJobDetailFactoryBean.java:311)
at org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:113)
at org.quartz.core.JobRunShell.run(JobRunShell.java:223)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:549)
THIS LINE IS THE ERROR LINE
reader = DirectoryReader.open(NIOFSDirectory.open(indexFile));
Is there a way to lock the file till it is store. Any solution to improve the way it should be implemented
You should have a look at the LockFactory of the NIOFSDirectory (inherited from parent Directory).
See LockFactory Javadoc for little more informations
Additional to this, your requirements look like a NRT (near-real-time) use case for me. If you like to index and search within a short time period and indexing will be done continuous a NRT implementation would make sense. I'm not sure if this is already a feature of lucene v4.2.
See Simple NRT tutorial for additional information.
I am new to Jersey & tomcat.
I have a webservice hosted on tomcat7 which returns a zip file after some processing.I am getting below exception
org.glassfish.jersey.server.internal.process.MappableException: org.apache.catalina.connector.ClientAbortException: java.net.SocketException: Connection timed out (Write failed)
at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:92)
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162)
at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1130)
at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:711)
at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:444)
at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:434)
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:329)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154)
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:221)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:956)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:436)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1078)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.catalina.connector.ClientAbortException: java.net.SocketException: Connection timed out (Write failed)
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:407)
at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:371)
at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:432)
at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:420)
at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:91)
at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:84)
at org.glassfish.jersey.servlet.internal.ResponseWriter$NonCloseableOutputStreamWrapper.write(ResponseWriter.java:320)
at org.glassfish.jersey.message.internal.CommittingOutputStream.write(CommittingOutputStream.java:218)
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$UnCloseableOutputStream.write(WriterInterceptorExecutor.java:294)
at org.glassfish.jersey.message.internal.ByteArrayProvider.writeTo(ByteArrayProvider.java:96)
at org.glassfish.jersey.message.internal.ByteArrayProvider.writeTo(ByteArrayProvider.java:60)
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:265)
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:250)
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162)
at org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:106)
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162)
at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:86)
... 39 more
Caused by: java.net.SocketException: Connection timed out (Write failed)
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
at org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:215)
at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:480)
at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:366)
at org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:240)
at org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:119)
at org.apache.coyote.http11.AbstractOutputBuffer.doWrite(AbstractOutputBuffer.java:192)
at org.apache.coyote.Response.doWrite(Response.java:499)
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:402)
... 55 more
This code works fine with smaller file created in shorter time but gives exception in case file having size 700-800kb, after some processing of approximately 6-8minutes.
PS : The request comes to webservice hosted on tomcat & then goes for RMI call to different VM via load balancer.
According to the exception information & your description, based on my understanding, I think the issue was caused by the synchronous processing mode of Jersey which require as below.
Request processing on the server works by default in a synchronous processing mode, which means that a client connection of a request is processed in a single I/O container thread. Once the thread processing the request returns to the I/O container, the container can safely assume that the request processing is finished and that the client connection can be safely released including all the resources associated with the connection. This model is typically sufficient for processing of requests for which the processing resource method execution takes a relatively short time. However, in cases where a resource method execution is known to take a long time to compute the result, server-side asynchronous processing model should be used. In this model, the association between a request processing thread and client connection is broken. I/O container that handles incoming request may no longer assume that a client connection can be safely closed when a request processing thread returns. Instead a facility for explicitly suspending, resuming and closing client connections needs to be exposed. Note that the use of server-side asynchronous processing model will not improve the request processing time perceived by the client. It will however increase the throughput of the server, by releasing the initial request processing thread back to the I/O container while the request may still be waiting in a queue for processing or the processing may still be running on another dedicated thread. The released I/O container thread can be used to accept and process new incoming request connections.
So in the case without your codes, my suggestion is that please refer to the jersey offical document 11.1. Asynchronous Server API to rewrite the code for processing zip files. Hope it helps.
I am trying to load some Freemarker templating from the DB using stringLoader and executing them through template.process, but here is the problem, freemarker stuff like <#if... is executing but when I try to execute assign JspTaglibs it breaks.
Here is a simple example:
From the DB I get <#assign bean=JspTaglibs["/WEB-INF/struts-bean.tld]>, which i put in a in StringTemplateLoader. I create configuration, template.. so far everything is working. When I do template.process(data, out), the page reports:
Starting output:FreeMarker template error: The following has evaluated to null or missing: ==> JspTaglibs [in template "temp" at line 1, column 15] Tip: If the failing expression is known to be legally null/missing, either specify a default value with myOptionalVar!myDefault, or use <#if myOptionalVar??>when-present<#else>when-missing. (These only cover the last step of the expression; to cover the whole expression, use parenthessis: (myOptionVar.foo)!myDefault, (myOptionVar.foo)?? The failing instruction (FTL stack trace): ---------- ==> #assign bean = JspTaglibs["/WEB-INF/s... [in template "temp" at line 1, column 1] ---------- Java stack trace (for programmers): ---------- freemarker.core.InvalidReferenceException: [... Exception message was already printed; see it above ...] at freemarker.core.InvalidReferenceException.getInstance(InvalidReferenceException.java:98) at freemarker.core.DynamicKeyName._eval(DynamicKeyName.java:87) at freemarker.core.Expression.eval(Expression.java:111) at freemarker.core.Assignment.accept(Assignment.java:106) at freemarker.core.Environment.visit(Environment.java:265) at freemarker.core.Environment.process(Environment.java:243) at freemarker.template.Template.process(Template.java:277) at org.apache.jsp.test5_jsp._jspService(test5_jsp.java:109) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70) at javax.servlet.http.HttpServlet.service(HttpServlet.java:727) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:432) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:390) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:334) at javax.servlet.http.HttpServlet.service(HttpServlet.java:727) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1070) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611) at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.doRun(AprEndpoint.java:2440) at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(AprEndpoint.java:2429) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Unknown Source)
Scenario that I have is that I have some code stored in the database and I would like to execute it by freemarker during template.process().
Is that even possible, and if it is, what am I doing wrong?
The problem is that JspTaglibs is not a core FreeMarker feature, but something that the FreemarkerServlet puts into the data-model, and something that requires Servlet environment and a HTTP request (as JSP requires that). Out of the box, there's no way to separate this feature from the FreemarkerServlet. However, maybe you can extend it so that you can still use it for your purpose. Most of the magic happens inside createModel, where among others, it adds JspTaglibs to the data-model.
It might also useful to know that FreeMarker (and thus FreemarkerServlet) can load templates directly from the database, if you give it a such TemplateLoader implementation through the FreeMarker Configuration. If all you want is serving templates from the database instead of where Struts traditionally stores them, you just need to provide the proper TemplateLoader.
I have implement a certain spring-ws service and when I sent a soap message I receive in spring webapplication 1 (running on tomcat 6.0.32):
Request:
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:gen="http://www.foo.com/bar/webservice/generated">
<soapenv:Header/>
<soapenv:Body>
<gen:GetAllPanelWSRequest/>
</soapenv:Body>
</soapenv:Envelope>
Response
<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/">
<SOAP-ENV:Header/>
<SOAP-ENV:Body>
<ns2:GetAllPanelWSResponse xmlns:ns2="http://www.foo.com/bar/webservice/generated">
<ns2:PanelWS ns2:Id="5">
<ns2:name>all</ns2:name>
<ns2:code>all</ns2:code>
</ns2:PanelWS>
<ns2:PanelWS ns2:Id="9">
<ns2:name>test</ns2:name>
<ns2:code>test1</ns2:code>
</ns2:PanelWS>
</ns2:GetAllPanelWSResponse>
</SOAP-ENV:Body>
</SOAP-ENV:Envelope>
So far no problem. From a webappliction 2 (not using spring MVC etc.) I made a function and when I run this function it works:
Main.java:
...
public class Main {
public static void main(String[] args) {
List<PanelWS> panelWSs = CallServices.getPanelWSs();
for (PanelWS p : panelWSs) {
System.out.println("\t" + makeString(p));
}
}
...
CallServices.java
public static List<PanelWS> getPanelWSs() {
ApplicationContext ctx = new ClassPathXmlApplicationContext("applicationContext.xml");
PanelClient client = (PanelClient) ctx.getBean("client");
PanelWS panelWS = null;
// Get all panelWS
System.out.println("Get all panels...");
List<PanelWS> panelWSs = client.getAllPanelWS();
return panelWSs;
}
Output:
Get all panels...
[id=5, name=all, code=all]
[id=9, name=test, code=test1]
So this works great and as expected. But when I want to call the function
<%
List<PanelWS> panelWSs = CallServices.getPanelWSs();
%>
within a JSP (runnning on the same tomcat server) or a simple class I get the following error:
Server:
13-mei-2011 17:57:37 com.sun.xml.internal.messaging.saaj.soap.MessageImpl saveChanges
SEVERE: SAAJ0539: Unable to get header stream in saveChanges
13-mei-2011 17:57:37 com.sun.xml.internal.messaging.saaj.soap.MessageImpl saveChanges
SEVERE: SAAJ0540: Error during saving a multipart message
Client:
org.springframework.ws.soap.saaj.SaajSoapMessageException: Could not write message to OutputStream: Error during saving a multipart message; nested exception is com.sun.xml.internal.messaging.saaj.SOAPExceptionImpl: Error during saving a multipart message
I'm using the following Java version pointing to myeclipse and tomcat (runs on):
*JDK 1.6.0_13*
Jars:
xercesImpl-2.9.1.jar
xalan-2.7.1.jar
serializer-2.7.1.jar
spring-ws-1.5.9-all.jar
spring.jar(2.5.6)
xml-apis-1.3.04.jar
xml-apis-ext-1.3.04.jar
Full stacktrace of client
Message: org.springframework.ws.soap.saaj.SaajSoapMessageException: Could not write message to OutputStream: Error during saving a multipart message; nested exception is com.sun.xml.internal.messaging.saaj.SOAPExceptionImpl: Error during saving a multipart message
Error ID: #9yb3mzb7
Stack trace: org.springframework.ws.soap.saaj.SaajSoapMessage.writeTo(SaajSoapMessage.java:169)
org.springframework.ws.client.core.WebServiceTemplate.sendRequest(WebServiceTemplate.java:580)
org.springframework.ws.client.core.WebServiceTemplate.doSendAndReceive(WebServiceTemplate.java:549)
org.springframework.ws.client.core.WebServiceTemplate.sendAndReceive(WebServiceTemplate.java:502)
org.springframework.ws.client.core.WebServiceTemplate.marshalSendAndReceive(WebServiceTemplate.java:351)
org.springframework.ws.client.core.WebServiceTemplate.marshalSendAndReceive(WebServiceTemplate.java:345)
org.springframework.ws.client.core.WebServiceTemplate.marshalSendAndReceive(WebServiceTemplate.java:337)
com.foo.bar.webservice.client.PanelMGMClientImpl.getAllPanelWS(PanelMGMClientImpl.java:22)
com.foo.bar.webservice.client.CallServices.getPanelWSs(CallServices.java:22)
org.apache.jsp.workspace.aclusers_jsp._jspService(aclusers_jsp.java:786)
org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:386)
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313)
org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260)
javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
Fullstacktrace of server
**
**SEVERE: Servlet.service() for servlet spring-ws threw exception
java.io.IOException: org.apache.xml.serializer.ToXMLSAXHandler cannot be cast to org.apache.xml.serializer.SerializationHandler
at com.sun.xml.internal.messaging.saaj.soap.impl.EnvelopeImpl.output(EnvelopeImpl.java:298)
at com.sun.xml.internal.messaging.saaj.soap.impl.EnvelopeImpl.output(EnvelopeImpl.java:309)
at com.sun.xml.internal.messaging.saaj.soap.SOAPPartImpl.getContentAsStream(SOAPPartImpl.java:305)
at com.sun.xml.internal.messaging.saaj.soap.MessageImpl.getHeaderBytes(MessageImpl.java:947)
at com.sun.xml.internal.messaging.saaj.soap.MessageImpl.saveChanges(MessageImpl.java:1098)
at org.springframework.ws.soap.saaj.Saaj13Implementation.writeTo(Saaj13Implementation.java:268)
at org.springframework.ws.soap.saaj.SaajSoapMessage.writeTo(SaajSoapMessage.java:165)
at org.springframework.ws.transport.AbstractWebServiceConnection.send(AbstractWebServiceConnection.java:45)
at org.springframework.ws.transport.support.WebServiceMessageReceiverObjectSupport.handleConnection(WebServiceMessageReceiverObjectSupport.java:97)
at org.springframework.ws.transport.http.WebServiceMessageReceiverHandlerAdapter.handle(WebServiceMessageReceiverHandlerAdapter.java:57)
at org.springframework.ws.transport.http.MessageDispatcherServlet.doService(MessageDispatcherServlet.java:230)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:571)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:511)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.foo.bar.support.gzip.CompressionFilter.doFilter(CompressionFilter.java:30)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:96)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.springframework.security.util.FilterChainProxy.doFilter(FilterChainProxy.java:168)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:236)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.springframework.orm.hibernate3.support.OpenSessionInViewFilter.doFilterInternal(OpenSessionInViewFilter.java:198)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
at org.apache.coyote.http11.Http11AprProcessor.process(Http11AprProcessor.java:864)
at org.apache.coyote.http11.Http11AprProtocol$Http11ConnectionHandler.process(Http11AprProtocol.java:579)
at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1665)
at java.lang.Thread.run(Thread.java:619)
16-mei-2011 14:20:12 com.sun.xml.internal.messaging.saaj.soap.EnvelopeFactory createEnvelope
SEVERE: SAAJ0511: Unable to create envelope from given source**
**
hopefully this is enough information to solve the problem. I found some messages and bugs the pointing to the fact that I need to delete some Jars (xalan,xercesImpl and serializer) from the projects and add these jars to the jdk\lib\endorsed\ folder and add to tomcat the argument: -Djava.endorsed.dirs=. This works but I can't do this because the applications is running on several servers already.
Thx in advance
Alternative solution
Solved It for now while using the Apache CXF on the client. And spring-ws 1.5.9 on the server... apparently spring-ws sucks with jdk > 1.5
Maybe you should try considering updating to xerces-2.10.x, the problem can be that in jdk6 (rt.jar) there is a com.sun.org.apache.xerces package which seem to contain a jaxp 1.4 implementation, so does xerces 2.10+:
"It also contains an implementation of
the parser related portions of JAXP
1.4"
quote from here.
Are you building the binary containing the jsp as a separate process or you are building everything together? If it's separated, you can try to remove xerces and xalan from the buildpath of that package only? - If you cannot, try upgrading maybe it'll help.
Still think it would be better to have a service calling your ws and you'd invoke that service from the jsp, that way you could test it separately, but perhaps you can't do that for some reason.
If both of these fail, can you post a) full stacktrace b) some of the errors which appear if you remove xerces and xalan alltogether?
Solved It for now using the Apache CXF 2.4.0 on the client. And spring-ws 1.5.9 on the server... apparently spring-ws sucks with jdk > 1.5
This is a known issue with Spring-WS and Java 6, see ticket SWS-175. According to this ticket the solution is to remove Xalan and Xerces from your webapp classpath.