We recently had a very unexpected issue in PROD which we manage to reproduce but we still don't have a clear explanation of what is causing it and more importantly how to fix it.
Our system is a vendor workflow framework that offers us hooks to add our custom behavior. It is running on WebSphere Application Server and it is a mix of Camel and Spring and EJB. Initially it was pure EJB but a few years ago the vendor added Camel with a view of getting rid of application server.
Now the problem:
The start of processing is a Camel route which is transacted with PROPAGATION_REQUIRED attribute. Part of the processing there is a local call to an EJB method. There is not explicit setting for any of EJB methods exposed apart from being Container managed. This means the EJB call should get the implicit value of REQUIRED. Part of the EJB call there is data change happening. The reference to the EJB is obtained using some code like below:
String repositoryName = MBLLookupUtil.getInstance().getRepositoryName();
RepositoryInstanceFacadeHome facadeHome = MBLLookupUtil.getInstance().getRepositoryInstanceFacadeHome(
repositoryName);
RepositoryInstanceFacade facade = facadeHome.create();
// Here is when the data change happens:
facade.doSomeWork();
The application logs show clearly that the EJB processing is happening in the same thread as the client. After making the call there is more work in the calling client when a Timeout exception happens.
We expect that both data change on client side and that changed as part of the EJB call will be roll backed, however the data changed by EJB call stays committed while the client side (Camel) gets roll backed.
In the spring application context we use WebSphereUowTransactionManager which is the IBM recommended way of configuring transaction manager:
<bean id="txManager" class="org.springframework.transaction.jta.WebSphereUowTransactionManager"/>
Any hints about what could cause this or how we could further investigate would be great appreciated. Thank you in advance.
UPDATE:
As advised in tomgeraghty3's comment I made a code change to log the value of TransactionSynchronizationManager.isActualTransactionActive() and the result was true
UPDATE 2:
We found out where the issue was by spending hours of analyzing decompiled vendor code. The transaction propagation was configured in an application-context.xml file but it was also overwritten straight in the POJO code with an #Transactional(propagation=REQUIRED_NEW) annotation. We proved that was the problem by removing the POJO class from the vendor jar and adding one with no annotation with the same package in one of ours. Very dirty just to prove our suspicion.
Now our bigger challenge would be to persuade the vendor address the issue. Hope it is just a bug and not an impacting change.
Related
I have this Exception, please I need your support.
there are one wildfly20 instance to services and other wildfly20 instance to tier web.
Web tier consume jdni as service in remote enviroment. It's testing and work well with some methods.
Now, right here come the case. Some methods from jndi tier work without problem and web call it, and get resources well, But, some others when Web tier call jndi call receive this exception (javax.ejb.EJBException: Failed to read response).
Please, Let's me know witch could be the cause for this exception triggered?.
Any hints that help with that.
every one.
I'm answer my own quetion for anyone with same issue, because I dont find information that I could use to resolve it.
Like I said I have two instance of wildfly.
1 web tier project.
2 JNDI tier with services.
In the web tier just add in the jboss-deployment-struture.xml hibernate dependency module. It's default wildfly in modoles set org.hibernate.
That's it.
We have a Spring-Boot application exposing some REST endpoints. We allow for this application to be operated standalone (as executable jar) or as a war to be deployed in a wildfly-11 application-server.
The class defining the REST-endpoints is marked #RestController #Transactional(REQUIRES_NEW) (both on class level, obviously). When running standalone, everything works as expected but when deployed in wildfly, the rollback on exceptions does not work. We established this by sending the exact same REST-message while operating on the exact same database.
We have confirmed via debugging that the final frames of the stacktrace is identical in both cases and especially in both cases we see a transactional-proxy around our REST-controller bean.
One difference would be, that within wildfly the application will use a jndi-datasource, prepared by wildfly while standalone the spring-boot will manage the database-connections.
Any idea what is wrong here?
Edit
I just tried explicitly invoking setRollbackOnly on the JtaTransactionmanager from within my code. The transaction will still commit. This sort of looks like a bug in Spring Boot to me.
Edit 2
Debugging further reveals that the transaction seems to be set to autocommit - every statement is immediately written to the database. This seems to be in violation to the annotation #Transactional and also to the fact that Spring creates a transactional proxy around my bean.
It's not a full answer - just a reasoning. JNDI is usally used at the app server layer whereas JDBC - at the application layer. At the App server layer are used global transaction settins that are overriding app settings. Follow the spring doc to get more
For reasons beyond my understanding the default transactional behaviour when deploying a spring-boot webapp to an application-server is auto-commit.
The solution to this problem is to enrich your application-configuration with the property spring.datasource.tomcat.default-auto-commit=false
In our project, we have several Spring-based modules which are deployed on WAS as web applications. We need to skip deployment, or stop a module if its Spring context initialization fails (i.e. ContextLoaderListener#contextInitialized or DispatcherServlet#init throws an exception). Now, if such happens, app is got deployed and starts, but returns HTTP 500 for any request.
Websphere 8.5.5
Related question: https://stackoverflow.com/a/272747/3459206
This APAR seems to be relevant:
https://www-01.ibm.com/support/docview.wss?uid=swg1PI58875
From the APAR text:
Listener exceptions typically should not stop the application
from starting up for service. However, some applications depend
on their listeners to do the necessary setup before the
application is started for service. Such applications prefer to
stop the application from starting up when there is any
exception in their listeners.
Problem conclusion
The WebContainer Container code was modified to provide an
option to stop the application when there is any listener
exception during the application starting up process.
A new WebContainer custom property needs to be set to enable the
behavior provided by this APAR:
For Full Profiles
com.ibm.ws.webcontainer.stopappstartuponlistenerexception = true
(default is false)
For Liberty Profile
stopappstartuponlistenerexception=true
The fix for this APAR is currently targeted for inclusion in
WebSphere Application Server fix packs 8.5.5.11 and 9.0.0.2,
and Liberty 16.0.0.3
See the APAR link for additional information.
You can use jenkins + maven.
Add the part you need to check under your test like junit.
Then if this module do not pass test, jenkins would not deploy it.
But I prefer fix bugs before deployment
Had a very similar issue.
The thing is - webfear - sorry could not resist ;-) does not initialize everything on startup.
To trigger a controlled request, I added a ScheduledEJB to the startup of the application. This bean itself triggered a http-request to a defined URL, which itself triggered:
any filters to get initialized in the chain
any contexts which are needed are initialized
And this itself ensured that my application (EAR or WAR) got very quickly tested after deployment. This works well with some small amout of requests per minute
If you work with high load, means tons of requests per second, you need to choose a different approach.
In this case I added a polling mechanism into the #Startup of the application, which polled every second or 250ms (depends on the load of the application).
This firing to the server ensured, that my #Startup bean was the very first which triggered the possible init issues in the application. If this happened I initialized a filter which always reported a 500 (or better fitting error) to the requestor.
Of course stop your firing bean, as soon as you get the 500, else your admins may like to kill you. (happend to me, since I produced tons or monitoring issues ;-) )
And of course on the regular operation, after your application started properly, you should also disable the polling
Look for a try-catch in the top level of your application code that is catching the Spring exception and allowing the application to continue running.
If the Spring exceptions being thrown are permitted to propagate to the top of the stack, the JVM will stop and there's no way it can keep running, far as I know.
I'm fairly new to Java--though, coming from PHP + JS. I aim to make a CRUD (+MVC) with Java Servlets.
I am using Java EE 7. I based some of my patterns here with this and that (both have the same final files).
So, I am implementing MVC, and I can't seem to make my code to work. JPA seems to be the main issue; I can't fetch anything from the database.
Here's the repository (really small project -- 7~8 classes). Models; DAO.
I had setup controllers.Test(url: /test) to test if I could actually communicate with the database via JPA. However, when I go to /test(controllers.Test), an exception is thrown. The same goes for controllers.NewsEdit(url: /edit).
java.lang.NullPointerException
controllers.Test.doGet(Test.java:36)
javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
When I visit home(url: /; controller.NewsIndex), I receive no exceptions -- I get the homepage, but no data at all. Is there anything I'm doing wrong?
My SQL files are located in the /sql folder. I got persistent.xml on META-INF/.
Any help or reference would be appreciated. Really need to learn it..
Thanks!
Ok,
A lil explanation about NullPointerException.
When the variable (in this case private NewsDAO newsDAO;) hasn't been initialized, that "pointer" is pointing to nothing in memory. Then when you try to call a method, there is nothing there and the NPE happens.
The problem here is that private NewsDAO newsDAO; is not being injected by the container and therefore when you call newsDAO.all(); newDAO is null and throws the exception.
CDI Injection only happens for components managed by the containers. I'm a spring guy so I'm not sure where to go here. Try to figure out if your NewsDAO is being loaded by the container.
Tomcat is a web container... You need a full Application server with an EJB container to run your example. Try to download and run on glassfish, JBoss or any other full app server.
more info here.
http://en.wikipedia.org/wiki/Application_server#Java_application_servers
and
http://www.oracle.com/technetwork/java/javaee/overview/compatibility-jsp-136984.html
I am working on a project that has Spring based Web Services and Spring Jdbc based persistence.
I have configured a DataSourceTransactionManager to manage the transactions and applied this to the service layer using a Pointcut.
Transaction propagation level is set REQUIRED.
The queries are issued through Spring provided JdbcTemplate.
The problem is that in case of multiple concurrent request to a service i get a MySQLTransactionRollbackException("Deadlock found when trying to get lock; try restarting transaction").
Apparently one of the transactions has obtained a lock which makes the second one fail.
My question is - How should configure Spring to delay the execution of the service until a lock is obtainable, instead of just giving up and throwing an exception?
I can't even catch the exception and try to re-execute the query because I've applied transaction on service layer as an advice keeping my DAOs clean.
I'm hoping to get a declarative solution only (since I am an AOP fanatic & crusader against boilerplate code :-) ). But even programmatic solutions are welcome.
Thanks for your suggestions.
Update -
#ninjalj Yes it was actually a real deadlock. Turns out I was writing my test case incorrectly. Silly me :-(
If I'm not mistaken, that indicates a real deadlock, so you should really retry the transaction. IIRC, a timeout waiting for a lock has "timeout" in the exception message.