Develop Reference

How to avoid "failed to send operations" errors after switching to the "exactly-once" delivery strategy?

I recently tried to switch my subscriptions in GCP Pub/Sub to the "exactly-once" delivery strategy. However, I started seeing the following warnings ~10 times every 30 minutes in my application logs:
com.google.api.gax.rpc.InvalidArgumentException: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: Some acknowledgement ids in the request were invalid. This could be because the acknowledgement ids have expired or the acknowledgement ids were malformed.
at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:92)
at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:98)
at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:66)
at com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:97)
at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:67)
at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1041)
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1215)
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983)
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:771)
at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:574)
at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:544)
at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
at com.google.api.gax.grpc.ChannelPool$ReleasingClientCall$1.onClose(ChannelPool.java:535)
at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:563)
at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:744)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:723)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: Some acknowledgement ids in the request were invalid. This could be because the acknowledgement ids have expired or the acknowledgement ids were malformed.
at io.grpc.Status.asRuntimeException(Status.java:535)
... 17 more
They're immediately followed by the following INFO log messages in the same thread:
Permanent error invalid ack id message, will not resend
I didn't see any problems caused by these warnings, but it's a bit hard to tell because my application is processing a decent number of messages (~1000/hour). I initially thought that these warnings were just short-term "aftershocks" from switching to the "exactly-once" strategy. However, I waited for about 2 hours and they kept occurring with the same frequency, showing no sign of disappearing. I then disabled the "exactly-once" strategy and they went away immediately after. Can anyone tell me whether these warnings are dangerous, what side effects I can expect, and most importantly how I can get rid of them?
I'm using version 3.4.0 of spring-cloud-gcp-dependencies and spring-cloud-gcp-starter-pubsub. I'm also using Spring Cloud Stream to process the incoming messages and I rely on it to automatically acknowledge the messages.
I have the following configuration set in my application.yaml file:
spring:
cloud:
gcp:
pubsub:
subscriber:
executor-threads: 15
max-ack-extension-period: 23400 # 6 hours and 30 minutes
acknowledgement-deadline: 600 # Maximum value
For context: The messages in my application represent jobs for execution, and they could take quite a while to finish - hence the 6h30m maximum acknowledgement extension period.
I also saw the following StackOverflow question:
How to handle errors during message acknowledgement using google pubsub java library?
From what I understand, the consequence of these warnings is that the messages will be redelivered to my application, but this is exactly what I want to avoid.

Thanks for the question, Alexander.
The errors you are seeing happen when the modifyAckDeadline or Acknowledgement request to the service fail because the acknowledgement id is already expired. In such cases, the service considers the expired acknowledgement id as invalid, since a newer delivery might already be in-flight. This is as-per the guarantees for exactly once delivery. There could be a few reasons for it:
The request was delayed due to network delays and by the time it arrived at the server, the acknowledgement id lease is already expired.
The tasks issuing the modifyAckDeadline or Acknowledgement request is overwhelmed (high CPU/memory/network usage), leading to delay in issuing those requests.
I suggest setting min-duration-per-ack-extension to a high number to reduce issues mentioned above. This will help mitigate the cases where the acknowledgement id lease expired. The highest value you can set for this field is 600 seconds.
Additionally, as mentioned in the other stack overflow question, you should consider checking the response of your acknowledgement operations. This can be used to guide your application if it can expect a redelivery. Sample.

Login fails after new user registration | Weblogic security

My application is deployed on Weblogic and I'm extending Unified User Profile(UUP) from Weblogic API for new user registration into the portal but I'm facing an issue where after completion of user registration, user gets redirected to login page instead of landing page.
Hereby I'm attaching my implementation details and exception logs:
I'm using ServletAuthentication.login(); each time a user logs in
ProfileWrapper profile = null;
try {
profile = ProfileFactory.getProfile((String) userAttributes.get(USERNAME), null);
}catch(RemoteException rex){
logger.error("RemoteException while setting profile: " + (String)userAttributes.get(USERNAME), rex);
createErrorResponse(request, response,
REQUEST_FAILED, rex.getMessage(), rex
.getClass().getCanonicalName(),HttpServletResponse.SC_INTERNAL_SERVER_ERROR);
}catch(ProfileNotFoundException pex){
logger.error("ProfileNotFoundException while setting profile: " + (String)userAttributes.get(USERNAME),pex);
createErrorResponse(request, response,
REQUEST_FAILED, pex.getMessage(), pex
.getClass().getCanonicalName(),HttpServletResponse.SC_INTERNAL_SERVER_ERROR);
}
Error logs:
2017-06-22 21:06:49,747 [[ACTIVE] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'] ERROR UserLoginServlet - ProfileNotFoundException while setting profile: ************#********.com
com.bea.p13n.usermgmt.profile.ProfileNotFoundException: ************#********.com
at com.bea.p13n.usermgmt.profile.internal.PhantomProfileWrapperImpl.validateProfile(PhantomProfileWrapperImpl.java:406)
at com.bea.p13n.usermgmt.profile.ProfileFactory.getProfile(ProfileFactory.java:205)
at com.bea.p13n.usermgmt.profile.ProfileFactory.getProfile(ProfileFactory.java:122)
at com.bea.p13n.usermgmt.profile.ProfileFactory.getProfile(ProfileFactory.java:99)
at nz.co.vodafone.portal.utility.security.UserLoginServlet.authenticateNFetchProfile(Use
JDK Version: 6
Weblogic Version: WebLogic Server 9.2(deprecated)
Need help from Weblogic(9.2) experts. Help would be highly appreciated.
PS: I'm aware that this Oracle is no more supporting Weblogic 9.2 but I cant change version..
Googled out and found that this is product issue. Need to confirm.

Random disconnects from master node NoNodeAvailableException using Elastic Cloud/Found

I'm using elastic cloud (former found) with shield and the transport java client. The app communicating with ES runs on heroku. I'm running a stress test on a staging environment with one node
{
"cluster_name": ...,
"status": "yellow",
"timed_out": false,
"number_of_nodes": 1,
"number_of_data_nodes": 1,
"active_primary_shards": 19,
"active_shards": 19,
"relocating_shards": 0,
"initializing_shards": 0,
"unassigned_shards": 7,
"delayed_unassigned_shards": 0,
"number_of_pending_tasks": 0,
"number_of_in_flight_fetch": 0
}
A the beginning everything works perfectly. But after some time (3-4 minutes) I begin to get some errors. I've set the log level to trace and these are the errors I've been getting (I've replaced with ... everything that is irrelevant.
org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes were available: [[...][...][...][inet[...]]{logical_availability_zone=..., availability_zone=..., max_local_storage_nodes=1, region=..., master=true}]
at org.elasticsearch.client.transport.TransportClientNodesService$RetryListener.onFailure(TransportClientNodesService.java:242)
at org.elasticsearch.action.TransportActionNodeProxy$1.handleException(TransportActionNodeProxy.java:78)
at org.elasticsearch.transport.TransportService$3.run(TransportService.java:290)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.transport.SendRequestTransportException: [...][inet[...]][indices:data/read/search]
at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:286)
at org.elasticsearch.shield.transport.ShieldClientTransportService.sendRequest(ShieldClientTransportService.java:41)
at org.elasticsearch.action.TransportActionNodeProxy.execute(TransportActionNodeProxy.java:57)
at org.elasticsearch.client.transport.support.InternalTransportClient$1.doWithNode(InternalTransportClient.java:109)
at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:205)
at org.elasticsearch.client.transport.support.InternalTransportClient.execute(InternalTransportClient.java:106)
at org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:334)
at org.elasticsearch.client.transport.TransportClient.search(TransportClient.java:416)
at org.elasticsearch.action.search.SearchRequestBuilder.doExecute(SearchRequestBuilder.java:1122)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:91)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:65)
...
Caused by: org.elasticsearch.transport.NodeNotConnectedException: [...][inet[...]] Node not connected
at org.elasticsearch.transport.netty.NettyTransport.nodeChannel(NettyTransport.java:936)
at org.elasticsearch.transport.netty.NettyTransport.sendRequest(NettyTransport.java:629)
at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:276)
...
These are my properties
settings = ImmutableSettings.settingsBuilder()
.put("client.transport.nodes_sampler_interval", "5s") //Tried it with 30s, same outcome
.put("client.transport.ping_timeout", "30s")
.put("cluster.name", clusterName)
.put("action.bulk.compress", false)
.put("shield.transport.ssl", true)
.put("request.headers.X-Found-Cluster", clusterName)
.put("shield.user", user + ":" + password)
.put("transport.ping_schedule", "1s") //Tried with 5s, same outcome
.build();
I've also set for every query I make:
max_query_response_size=100000
timeout_seconds=30
I'm using ElasticSearch 1.7.2 and Shield 1.3.2 with corresponding (same version) clients, Java 1.8.0_65 on my machine - Java 1.8.0_40 on the node.
I was getting the same errors without a stress test, but the errors happened very randomly so I wanted to reproduce. That's why I'm running this in a single node.
I spotted another error in my logs
2016-03-07 23:35:52,177 DEBUG [elasticsearch[Vermin][transport_client_worker][T#7]{New I/O worker #16}] ssl.SslHandler (NettyInternalESLogger.java:debug(63)) - Swallowing an exception raised while writing non-app data
java.nio.channels.ClosedChannelException
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:433)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:373)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Hot threads
0.0% (111.6micros out of 500ms) cpu usage by thread 'elasticsearch[...][transport_client_timer][T#1]{Hashed wheel timer #1}'
10/10 snapshots sharing following 5 elements
java.lang.Thread.sleep(Native Method)
org.elasticsearch.common.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:445)
org.elasticsearch.common.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:364)
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
java.lang.Thread.run(Thread.java:745)
After reading this http://blog.trifork.com/2015/04/08/dealing-with-nodenotavailableexceptions-in-elasticsearch/ I came to understand a little better how the whole communication works. I haven't tested this yet, but I believe that the problem lies there. The thing is though, even if I confirm that the problem is closed query connections, how do I handle this? Keep the config as is and just reconnect? Do I disable keepAlive? If yes, should I be worrying over something else?

Citing this link:
https://discuss.elastic.co/t/nonodeavailableexception-with-java-transport-client/37702 by Konrad Beiske
your application could be resolving the ip address at boot time. The
ELB can change ip's at any point in time. For the best reliability
your application should add all ip's of the ELB to the client and
periodically check the DNS service for changes.
The connection timeout of our ELB's are 5 minutes.
Following should help you fix it:
Creating a new TransportClient for every request is not ideal as it
will imply a new connection handshake for every request and this will
hurt your response time. You could have a pool of TransportClients if
you prefer, but it will most likely be an unnecessary overhead as the
client is thread safe.
My suggestion is that you create a small singleton service that
periodically checks for changes to the DNS service and adds any new
ip's to your existing transport client. In theory it could be as naive
as just adding all ip's discovered every time it checks as the
transport client will discard duplicate addresses and also purges old
addresses no longer reachable.

Exceptions in the weblogic server - Illegal attempt to call EJBContext.setRollbackOnly()

I get the following exception in the Weblogic server,from the logs, it is definitely something internal to the weblogic, but, not sure which part of the application the weblogic is trying to archive, do you know why it is happening?
<Nov 4, 2015 10:32:07 AM CST> <Info> <EJB> <BEA-010213> <Message-Driven EJB: WLIArchiverSchedulerMDB's transaction was rolledback. The transaction details are: Xid=BEA1-0482AE5EBACD7EBCD75C(5835637),Status=Rolled back. [Reason=weblogic.transaction.internal.AppSetRollbackOnlyException],numRepliesOwedMe=0,numRepliesOwedOthers=0,seconds since begin=0,seconds left=60,XAServerResourceInfo[JMS_cgJMSStore]=(ServerResourceInfo[JMS_cgJMSStore]=(state=rolledback,assigned=msolvint101-prd01-z),xar=JMS_cgJMSStore,re-Registered = false),XAServerResourceInfo[cgPool]=(ServerResourceInfo[cgPool]=(state=rolledback,assigned=msolvint101-prd01-z),xar=cgPool,re-Registered = false),SCInfo[m6intdomain+msolvint101-prd01-z]=(state=rolledback),properties=({ISOLATION LEVEL=2}),local properties=({weblogic.jdbc.jta.cgPool=weblogic.jdbc.wrapper.TxInfo#1ff65e5}),OwnerTransactionManager=ServerTM[ServerCoordinatorDescriptor=(CoordinatorURL=msolvint101-prd01-z+msolvint101-prd01-z.tds.local:7001+m6intdomain+t3+, XAResources={JMS_FileStore, JMS_pluginStore, cgPool, NetExpert Pool, MSLVPool, JMS_cgJMSStore, MSLVwliPool, bpmArchPool},NonXAResources={})],CoordinatorURL=msolvint101-prd01-z+msolvint101-prd01-z.tds.local:7001+m6intdomain+t3+).>
<Nov 4, 2015 10:32:07 AM CST> <Warning> <EJB> <BEA-010065> <MessageDrivenBean threw an Exception in onMessage(). The exception was:
java.lang.IllegalStateException: [EJB:010158]Illegal attempt to call EJBContext.setRollbackOnly() from an EJB that was not participating in a transaction..
java.lang.IllegalStateException: [EJB:010158]Illegal attempt to call EJBContext.setRollbackOnly() from an EJB that was not participating in a transaction.
at weblogic.ejb20.internal.BaseEJBContext.setRollbackOnly(BaseEJBContext.java:348)
at weblogic.ejb20.internal.MessageDrivenEJBContextImpl.setRollbackOnly(MessageDrivenEJBContextImpl.java:56)
at com.bea.wli.management.archiving.WLIArchiverSchedulerMDB.onMessage(WLIArchiverSchedulerMDB.java:164)
at com.bea.wli.management.archiving.WLIArchiverSchedulerMDB.onMessage(WLIArchiverSchedulerMDB.java:75)
at weblogic.ejb20.internal.MDListener.execute(MDListener.java:400)
at weblogic.ejb20.internal.MDListener.transactionalOnMessage(MDListener.java:333)
at weblogic.ejb20.internal.MDListener.onMessage(MDListener.java:298)
at weblogic.jms.client.JMSSession.onMessage(JMSSession.java:2686)
at weblogic.jms.client.JMSSession.execute(JMSSession.java:2598)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:224)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:183)

Make sure your MDB transaction attribute is set to Required and not to NotSupported

Blank Page in JSF

If my code throws an exception, sometimes - not everytime - the jsf presents a blank page. I´m using facelets for layout.
A similar error were reported at this Sun forumn´s post, but without answers.
Anyone else with the same problem, or have a solution?
;)
Due to some requests. Here follow more datails:
web.xml
<error-page>
<exception-type>com.company.ApplicationResourceException</exception-type>
<location>/error.faces</location>
</error-page>
And the stack related to jsf is printed after the real exception:
####<Sep 23, 2008 5:42:55 PM GMT-03:00> <Error> <HTTP> <comp141> <AdminServer> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1222202575662> <BEA-101107> <[weblogic.servlet.internal.WebAppServletContext#6d46b9 - appName: 'ControlPanelEAR', name: 'ControlPanelWeb', context-path: '/Web'] Problem occurred while serving the error page.
javax.servlet.ServletException: viewId:/error.xhtml - View /error.xhtml could not be restored.
at javax.faces.webapp.FacesServlet.service(FacesServlet.java:249)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:226)
at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:124)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:283)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:175)
at weblogic.servlet.internal.RequestDispatcherImpl.invokeServlet(RequestDispatcherImpl.java:525)
at weblogic.servlet.internal.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:261)
at weblogic.servlet.internal.ForwardAction.run(ForwardAction.java:22)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic.security.service.SecurityManager.runAs(Unknown Source)
at weblogic.servlet.internal.ErrorManager.handleException(ErrorManager.java:144)
at weblogic.servlet.internal.WebAppServletContext.handleThrowableFromInvocation(WebAppServletContext.java:2201)
at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2053)
at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1366)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:200)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:172)
javax.faces.application.ViewExpiredException: viewId:/error.xhtml - View /error.xhtml could not be restored.
at com.sun.faces.lifecycle.RestoreViewPhase.execute(RestoreViewPhase.java:180)
at com.sun.faces.lifecycle.LifecycleImpl.phase(LifecycleImpl.java:248)
at com.sun.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:117)
at javax.faces.webapp.FacesServlet.service(FacesServlet.java:244)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:226)
at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:124)
I´m using the jsf version Mojarra 1.2_09, richfaces 3.2.1.GA and facelets 1.1.13.
Hope some help :(

I think this largely depends on your JSF implementation. I've heard that some will render blank screens.
The one we were using would throw error 500's with a stack trace. Other times out buttons wouldn't work without any error for the user. This was all during our development phase.
But the best advice I can give you is to catch the exceptions and log them in an error log so you have the stack trace for debugging later. For messages that we couldn't do anything about like a backend failing we would just add a fatal message to the FacesContext that gets displayed on the screen and log the stack trace.

I fixed a similar problem in my error.jsp page today. This won't be exactly the same as yours, but it might point someone in the right direction if they're having a similar problem. My problem seemed to be coming from two different sources.
First, the message exception property wasn't being set in some of the servlets that were throwing exceptions caught by the error page. The servlets were catching and rethrowing exceptions using the ServletException(Throwable rootCause) constructor.
Second, in the error page itself, the original author had used scriptlet code to parse the message using String.split(message, ";"); Since the message was null this failed. I was getting a NullPointerException in my error log, along with the message "Problem occurred while serving the error page."
These two things combined to give me a blank page at the URL of the servlet that was throwing the original exception. I fixed my problem by providing my own error message when I rethrow exceptions in my servlets using the ServletException(String message, Throwable rootCause) constructor, so the error message will no longer be null. I also rewrote the error.jsp page using EL instead of scriptlet code, but that wasn't strictly necessary.

For a blank page on JSF 2, place a breakpoint in ExceptionHandlerWrapper.handle or a class overriding this method. In my case it was due to custom code which was a too restrictive and the error was not logged.