It seems the JMSProducer is not getting garbage collected and keeps alive after delivering messages to queue, I'm using Spring 3.2.2 and CachingConnectionFactory with Keep-alive setting for sending message.
Producers count keeps increasing every time I send message.
Is it related to spring version I am using?
or am I doing something wrong in my configuration?
You need to call close() method on your MessageProducer. As per the Java docs:-
void close()
throws JMSException
Closes the message producer.
Since a provider may allocate some resources on behalf of a MessageProducer outside the Java virtual machine, clients should close them when they are not needed. Relying on garbage collection to eventually reclaim these resources may not be timely enough.
As per the spring CachingConnectionFactory docs :-
NOTE: This ConnectionFactory requires explicit closing of all Sessions
obtained from its shared Connection. This is the usual recommendation
for native JMS access code anyway. However, with this
ConnectionFactory, its use is mandatory in order to actually allow for
Session reuse.
So you need to call getCachedSessionProxy instead of getSession and once done with sending message call the close() (in finally block) . As per the source code, the close call to this Session proxy is handled such that the session and messageproducer is reused. Gary's comments states the same.
Related
I am trying to identify where a suspected memory / resource leak is occurring with regards to a JMS Queue I have built. I am new to JMS queues, so I have used many of the standard JMS class objects to ensure stability. But somewhere in my code or configuration I am doing something wrong, and my queue is filling up or resources are slowing down, perhaps inherent to unknown deficiencies within the architecture I am attempting to implement.
When load testing my API (using Gatling), I can run 20 messages a second through (which is a tiny load) for most of a ten minute duration. But after that, the messages seem to back up, and the ability to process them slows to a crawl. Generally time-out errors begin to occur once the overall requests exceed 60 seconds to complete. There is more business logic that processes data and persists it to a relational database, but none of that appears to be an issue.
Interestingly, subsequent test runs continue with the poor performance, indicating that whatever resource is leaking is transcending the tests. A restart of the application clears out whatever has become bloated leaking. Then the tests run fast again, for the first seven or eight minutes... upon which the cycle repeats itself. Only a restart of the App clears the issue. Since the issue doesn't self-correct itself, even after waiting for a period of time, something has filled up resources.
When pulling the JMS calls from the logic, I am able to process hundreds of messages a second. And I can run back-to-back tests runs without leaking or filling up the queue.
Although this is a Spring project, I am not using Spring's JMS Template, so I wrote my own Connection object, which I injected as a Spring Bean and implemented as a single connection to avoid creating a new connection for every JMS message I sent through.
Likewise, I configured my JMS Session to also be an injected Bean, in which I use the Connection Bean. That way I can persist my Connection and Session objects for sending all of my JMS messages through, which are sent one at a time. A Qpid Server I am calling receives these messages. While it is possible I am exceeding it's capacity to consume the messages I am producing, I expect that the resource leak is associated with my code, and not the JMS Server.
Here are some code snippets to give you an idea of my approach. Any feedback is appreciated.
JmsConfiguration (key methods)
#Bean
public ConnectionFactory jmsConnectionFactory() {
return new JmsConnectionFactory(user, pass, host);
}
#Bean(name="jmsSession")
public Session jmsConnection() throws JMSException {
Connection conn = jmsConnectionFactory().createConnection();
Session session = conn.createSession(false, Session.AUTO_ACKNOWLEDGE);
return session; //Injected as Singleton
}
#Bean(name="jmsQueue")
public Queue jmsQueue() throws JMSException {
return jmsConnection().createQueue(queue);
}
//Jackson's objectMapper is heavy enough to warrant injecting and re-using it.
#Bean
public ObjectMapper objectMapper() {
return new ObjectMapper();
}
JmsMessageEnqueuer
#Component
public class MessageJmsEnqueuer extends CommonThreadScope {
#Autowired
#Qualifier("Session")
private Session jmsSession;
#Autowired
#Qualifier("jmsQueue")
private Queue jmsQueue;
#Value("${acme.jms.queue}")
private String jmsQueueName;
#Autowired
#Qualifier("jmsObjectMapper")
private ObjectMapper jmsObjectMapper;
public void enqueue(String message, String dataType) {
try {
String messageAsJson = objectMapper.writeValueAsString(message);
MessageProducer jmsMessageProducer = jmsSession.createProducer(jmsQueue);
TextMessage message = jmsSession.createTextMessage(message);
message.setStringProperty("dataType", dataType.name());
jmsMessageProducer.send(message);
logger.log(Level.INFO, "Message successfully sent. Queue=" + jmsQueueName + ", Message -> " + message);
} catch (JMSRuntimeException | JsonProcessingException jmsre) {
String msg = "JMS Message Processing encountered an error...";
logService.severe(logger, messagesBuilder() ... msg)
}
//Skip the close() method to persist connection...
//Reconnect logic exists to reset an expired connection from server.
}
}
I was able to solve my resource leak / deadlock issue simply by rewriting my code to use the simplified API provided with the release of JMS 2.0. Although I was never able to determine which of the Connection / Session / Queue objects was giving my code grief, using the Context object to build my connection and session was the golden ticket in this case.
Upon switching to the simplified API (since I was already pulling in the JMS 2.0 dependency), the resource leak immediately vanished! This leads me to believe that the simplified API does more than just make life easier by providing an easier API for the developer to code against. While that is already an advantage to begin with (even without the few features that the simplified API doesn't support), it is now clear to me that the underlying connection and session objects are being managed by the API, and thus resolved whatever was filling up or deadlocking.
Furthermore, because the resource build-up was no longer occurring, I was able to triple the number of messages I passed through, allowing me to process 60 users a second, instead of 20. That is a significant increase, and I have fixed the compatibility issues that prevented me from using the simplified JMS API to begin with.
While I would have liked to identify precisely what was fouling up the code, this works as a solution. Plus, the fact that version 2.0 of JMS was released in April of 2013 would indicate that the simplified API is definitely the preferred solution.
Just a guess, but a MessageProducer extends AutoClosable, suggesting it to be closed after it is no longer of use. Since you're not using a try-with-resources or explicitly close it afterwards, the jmsSession may contain more and more producers over time. Although I am not sure whether you should close per method call, or re-use the created producer.
Have you tried using a profiler such as VisualVM to visualize the heap and metaspace? If so, did you find any significant changes over time?
I have a spring integration JMS outbound gateway that I'm using to push messages to multiple queues in my queue manager.
#Bean
public IntegrationFlow sendTo101flow() {
return IntegrationFlows.from("sendTo101Channel")
.handle(Jms.outboundAdapter(context.getBean("connection101", ConnectionFactory.class))
.destinationExpression("headers." + HeaderKeys.DESTINATION_NAME)
.configureJmsTemplate(jmsOutboundTemplateSpec())
.get(), jmsOutboundEndpointSpec())
.get();
}
I'm facing problems when we get concurrent requests with huge payloads which need to be inserted into the same queue. On inspection it looks like even though the threads trying to insert the message are separate, they're only allowed to do the insertion sequentially.
I have checked the mq documentation and it looks like actual parallel insertion will only work if a new connection is opened for each message.
Is there a way to make a JMS outbound gateway open a new connection per message? Or set the number of concurrent connections opened through it (like on the inbound side)?
That is the default behavior, as long as you don't use a CachingConnectionFactory (or its parent SingleConnectionFactory) which shares a single connection across all operations.
Connection (and session, producer) caching is generally recommended, to avoid the overhead of creating connection, session, and producer for each send. But there may be cases, like yours, where this is unavoidable.
I am using hystrix to handle falback scenarios when my messages are not delivered to RabbitMQ server. My fallback is getting invoked when RabbitMQ server is down(as AMQPException is thrown).
If broker is unable to accept/route the messages then returnCallback/returnConfirm(with nack) is invoked.
What I understand is that RabbitTemplate returnCallbacks/returnConfirms will be executed in different thread than the Hystrix thread.
Is it possible to throw Exception in these scenarios so that Hystrix fallback gets executed?
I have referred these q's : Spring AMQP return callback vs retry callback
Spring RabbitTemplate- How to get hold of the published message for NACKs in Publisher confirm mode
Any pointer to handle this scenario is much appreciated.
No; returns are completely asynchronous; even if you enable transactions - from the rabbit mq documentation...
AMQP does not specify when errors (e.g. lack of permissions, references to unknown exchanges) in transactional basic.publish and basic.ack commands should be detected. RabbitMQ performs the necessary checks immediately (rather than, say, at the time of commit), but note that both basic.publish and basic.ack are asynchronous commands so any errors will be reported back to the client asynchronously.
If you publish to a non-existent exchange (and setChannelTransacted(true)*) you will get an exception on the commit, but publishing to an exchange with no routable queue will never get an exception (only an async return callback).
enabling transactions is quite expensive for all operations so consider it carefully if you want to catch this particular scenario
I struggeling with JTA, two-phase-commit, JMS- and JDBC-transactions. The idea is (in short) to
receive a message on a queue
perform some database operations
acknowledge the message, when the db operations have been successful
So I got the XAQueueConnectionFactory, create the XAQueueSession, create a receiver from the session and set a message listener.
Inside the listener, in the onMessage method, I begin my user transaction, do the jdbc stuff and commit the transaction or do a rollback, if something went wrong. Now I expected (aka "hoped") that the message would be acknowledged, when the user transaction commits.
But that doesn't happen, the messages are still on the queue and get redelivered again and again.
What am I missing? I double-checked the session and the acknowledge mode really is "SESSION_TRANSACTED" and getTransacted returns true.
I don't have a Java EE container, no spring, no message driven beans. I use the standalone JTA bitronix.
You don't really need XA for this. Just following your algorithm: receive the message, perform the DB operations, then acknowledge the message... Literally, that's the solution. (And instead a transacted session, you probably would just choose explicit CLIENT_ACKNOWLEDGE.) If your application should fail while performing the DB operations, don't ack the JMS msg and it will be redelivered. If your app fails after the DB txn and before the ack, then the message will be redelivered -- but you can detected this (redelivered flag will be set to true on the message), and you can decide to reprocess the message or not, based on the state of the database.
When you say that inside the listener, you begin your user transaction, this seems to hint that you are using Bean Managed Transaction (BMT). Is there a good reason for doing so?
If you used Container Managed Transaction (CMT), what you want would come for free.
As far as I remember, it is not possible with BMT, since the UserTransaction will not participate and will not be able to participate in the transaction created for the message. But you might want to double check with the Java EE spec.
Edit:
Sorry, I realized too late that you are not using a Java EE container.
Are you sure that the user transaction that you start inside the listener is part of the transaction started for the message? It seems that you start an independent transaction for the db work.
If you use no container, who provides the JMS implementation, i.e. XAQueueConnectionFactory etc?
I think with XA you shoulndn't use transacted session.
We currently have a Stateful bean that is injected into a Servlet. The problem is that sometimes we get a Caused by: javax.ejb.ConcurrentAccessException: SessionBean is executing another request. [session-key: 7d90c02200a81f-752fe1cd-1] when executing a method on the stateful bean.
public class NewServlet extends HttpServlet {
#EJB
private ReportLocal reportBean;
protected void processRequest(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
response.setContentType("text/html;charset=UTF-8");
PrintWriter out = response.getWriter();
try {
String[] parameters = fetchParameters(request);
out.write(reportBean.constructReport(parameters));
} finally {
out.close();
}
}
}
In the above code, constructReport will check if it needs to open a new connection to the database specified in the Report after which a Report in HTML is constructed from a query which is built from the parameters specified.
The reason why we chose to use a stateful bean over a stateless bean was because we need to open a database connection to an unknown database and perform queries on it. With a stateless bean it seems terribly inefficient to repeatedly open and close database connections with each injected instance of the bean.
A few more details regarding the ConcurrentAccessException: as per the EJB spec, access to SLSB is synchronized by the app. server. However, this is not the case with SFSB. The burden of making sure that the SFSB is not accessed concurrently is on the application developer's shoulders.
Why? Well, synchronization of SLSB is only necessary at the instance-level. That is, each particular instance of the SLSB is synchronized, but you may have multiple instances in a pool or on different node in a cluster, and concurrent requests on different instances is not a problem. This is unfortunately not so easy with SFSB because of passivation/activation of instances and replication across the cluster. This is why the spec doesn't enforce this. Have a look at this dicussion if you are interested in the topic.
This means that using SFSB from servlet is complicated. A user with multiple windows from the same session, or reloading page before the rendering finished can lead to concurrent access. Each access to the EJB that is done in a servlet needs theoretically to be synchronized on the bean itself. What I did was to to create an InvocationHandler to synchronize all invocations on the particular EJB instance:
public class SynchronizationHandler implements InvocationHandler {
private Object target; // the EJB
public SynchronizationHandler( Object bean )
{
target = bean;
}
public Object invoke(Object proxy, Method method, Object[] args) throws Throwable
{
synchronized( target )
{
// invoke method to the target EJB
}
}
}
Then, right after you obtain the remote reference to the EJB, you wrap it with the SynchronizationHandler. This way you are sure that this particular instance will not be accessed concurrently from your app (as long as it runs in only one JVM). You can also write a regular wrapper class which synchronizes all the methods of the bean.
My conclusion is nevertheless: use SLSB whenever possible.
EDIT:
This answer reflects the EJB 3.0 specs (section 4.3.13):
Clients are not allowed to make concurrent calls to a stateful session
object. If a client-invoked business method is in progress on an
instance when another client-invoked call, from the same or different
client, arrives at the same instance of a stateful session bean class,
if the second client is a client of the bean’s business interface, the
concurrent invocation may result in the second client receiving the
javax.ejb.ConcurrentAccessException
Such restrictions have been removed in EJB 3.1 (section 4.3.13):
By default, clients are allowed to make concurrent calls to a stateful
session object and the container is required to serialize such
concurrent requests.
[...]
The Bean Developer may optionally specify that concurrent client
requests to a stateful session bean are prohibited. This is done using
the #AccessTimeout annotation or access-timeout deployment descriptor
element with a value of 0. In this case, if a client-invoked business
method is in progress on an instance when another client-invoked call,
from the same or different client, arrives at the same instance of a
stateful session bean, if the second client is a client of the bean’s
business interface or no-interface view, the concurrent invocation
must result in the second client receiving a
javax.ejb.ConcurrentAccessException
This is not what stateful session beans (SFSB) are intended to be used for. They are designed to hold conversation state, and are to be bound to the user's http session to hold that state, like a heavyweight alternative to storing state in the session directly.
If you want to hold things like database connections, then there are better ways to go about it.
Best option is to use a connection pool. You should always use a connection pool, and if you're running inside an application server (which, if you're using EJBs, then you are), then you can easily use your appserver's datasource configuration to create a connection pool, and use that inside your stateless session bean (SLSB).
Until you provide some code and the stacktrace, I'd suggest that you consider using a connection pool.
If by "unknown database" you mean a database whose parameters are supplied by the end user, and hence no preconfigured connection pool is possible, you can still use the connection pool concept, rather than opening a new connection each time.
Also, theck this thread.
Session beans cannot be used concurrently, like skaffman said they were meant to handle state corresponding to the client session and are typically stored in the session object per client.
Refactoring to use a database pool to handle concurrent requests to your resources is the way to go.
In the meantime, if all you need is this trivial use, you could synchronise the call to constructReport as in:
synchronised (reportBean) {
out.write(reportBean.constructReport(parameters));
}
Note that this is no solution if constructReport takes a significant amount of time relative to your number of clients.
you should never syncronize servlet or ejb access since this cause requests queue and if you have N concurrently users the last one will wait for a long time and often get a timeout response!!! Syncronize method is not intended for this reason!!!