If I throw enough traffic at a performance environment, I can cause out of memory errors. If I do a thread dump on this instance, it will report a deadlock:
Found one Java-level deadlock:
=============================
"http-nio-8080-exec-9":
waiting to lock monitor 0x00007fb528003098 (object 0x00000000826d6018, a com.mysql.jdbc.LoadBalancedConnectionProxy),
which is held by "Finalizer"
"Finalizer":
waiting to lock monitor 0x00007fb528002fe8 (object 0x00000000826d5f98, a com.mysql.jdbc.ReplicationConnectionProxy),
which is held by "http-nio-8080-exec-9"
I think the OOM is happening because the Finalizer is deadlocked and can't garbage collect unused instances. I'm not sure how to troubleshoot the deadlock though. The http-nio-8080-exec-9 thread is doing this:
"http-nio-8080-exec-9":
at com.mysql.jdbc.MultiHostConnectionProxy$JdbcInterfaceProxy.invoke(MultiHostConnectionProxy.java:104)
- waiting to lock <0x00000000826d6018> (a com.mysql.jdbc.LoadBalancedConnectionProxy)
at com.sun.proxy.$Proxy60.setPingTarget(Unknown Source)
at com.mysql.jdbc.ReplicationConnectionProxy.invokeMore(ReplicationConnectionProxy.java:311)
at com.mysql.jdbc.MultiHostConnectionProxy.invoke(MultiHostConnectionProxy.java:457)
- locked <0x00000000826d5f98> (a com.mysql.jdbc.ReplicationConnectionProxy)
at com.sun.proxy.$Proxy57.prepareStatement(Unknown Source)
at org.apache.commons.dbcp2.DelegatingConnection.prepareStatement(DelegatingConnection.java:291)
at org.apache.commons.dbcp2.DelegatingConnection.prepareStatement(DelegatingConnection.java:291)
my code here
On this line of my code, my code is doing this:
preparedStatement = connect.prepareStatement(sqlQuery);
The finalizer thread is doing this:
"Finalizer":
at com.mysql.jdbc.ResultSetImpl.realClose(ResultSetImpl.java:6597)
- waiting to lock <0x00000000826d5f98> (a com.mysql.jdbc.ReplicationConnectionProxy)
at com.mysql.jdbc.ResultSetImpl.close(ResultSetImpl.java:851)
at sun.reflect.GeneratedMethodAccessor112.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.mysql.jdbc.MultiHostConnectionProxy$JdbcInterfaceProxy.invoke(MultiHostConnectionProxy.java:108)
- locked <0x00000000826d6018> (a com.mysql.jdbc.LoadBalancedConnectionProxy)
at com.sun.proxy.$Proxy61.close(Unknown Source)
at sun.reflect.GeneratedMethodAccessor112.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.mysql.jdbc.MultiHostConnectionProxy$JdbcInterfaceProxy.invoke(MultiHostConnectionProxy.java:108)
- locked <0x00000000826d6018> (a com.mysql.jdbc.LoadBalancedConnectionProxy)
at com.sun.proxy.$Proxy62.close(Unknown Source)
at org.apache.commons.dbcp2.DelegatingResultSet.close(DelegatingResultSet.java:169)
at org.apache.commons.dbcp2.DelegatingStatement.close(DelegatingStatement.java:149)
at org.apache.commons.dbcp2.DelegatingStatement.finalize(DelegatingStatement.java:549)
at java.lang.System$2.invokeFinalize(System.java:1270)
at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:98)
at java.lang.ref.Finalizer.access$100(Finalizer.java:34)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:210)
The ResultSetImpl.realClose is doing this when it deadlocks:
synchronized (locallyScopedConn.getConnectionMutex()) {
Both seem related to mysql jdbc drivers. We are using org.apache.commons.dbcp2.BasicDataSource for our connection pooling. Here's the code where we set up our connection:
private static final BasicDataSource dataSource = new BasicDataSource();
private void setUpConnectionPool()
{
final String JDBC_CONNECTION_STRING = System.getProperty("JDBC_CONNECTION_STRING");
final String DB_USER_STRING = System.getProperty("DB_USER_STRING");
final String DB_PASSWORD_STRING = System.getProperty("DB_PASSWORD_STRING");
final int MAX_CONNECTIONS = System.getProperty("MAX_CONNECTIONS") == null ? 100 : Integer.valueOf(System.getProperty("MAX_CONNECTIONS"));
try {
ReplicationDriver driver = new ReplicationDriver();
dataSource.setUrl(JDBC_CONNECTION_STRING);
dataSource.setDriver(driver);
dataSource.setUsername(DB_USER_STRING);
dataSource.setPassword(DB_PASSWORD_STRING);
dataSource.setMaxTotal(MAX_CONNECTIONS);
dataSource.setConnectionProperties("autoReconnect=true;roundRobinLoadBalance=true;");
} catch (Exception e) {
e.printStackTrace();
}
}
The above method is called when the context is initialized. Then, all code gets its connection by calling this method:
public static Connection getConnection() throws SQLException {
return dataSource.getConnection();
}
I'm hoping there's something obviously wrong with this code, but I don't see an obvious reason why it would cause a deadlock.
Related
I have some Java app using Spring Batch. I've got a table used as a queue which contains information on jobs that were requested by clients (as a client requests for a task to be executed, a row is added to this queue).
In one of my classes a while loop is run until someone deactivates some flag :
protected void runJobLaunchingLoop() {
while (!isTerminated()) {
try {
if (isActivated()) {
QueueEntryDTO queueEntry = dequeueJobEntry();
launchJob(queueEntry);
}
}
catch (EmptyQueueException ignored) {}
catch (Exception exception) {
logger.error("There was a problem while de-queuing a job ('" + exception.getMessage() + "').");
}
finally {
pauseProcessor();
}
}
}
The pauseProcessor() method calls Thread.sleep(). When I run this app in a Docker container it looks like the number of threads run by the application keep on increasing. The threads have the name "Timer-X" with X some integer that auto-increments.
I looked at the stack trace of one of these :
"Timer-14" - Thread t#128
java.lang.Thread.State: WAITING
at java.base#11.0.6/java.lang.Object.wait(Native Method)
- waiting on <25e60c31> (a java.util.TaskQueue)
at java.base#11.0.6/java.lang.Object.wait(Unknown Source)
at java.base#11.0.6/java.util.TimerThread.mainLoop(Unknown Source)
- locked <25e60c31> (a java.util.TaskQueue)
at java.base#11.0.6/java.util.TimerThread.run(Unknown Source)
Locked ownable synchronizers:
- None
Any idea what could be the cause of this? I'm not sure but if I don't run the app in a container but locally from IntelliJ, it seems like the problem does not occur. I'm not sure because sometimes it takes a while before thread count starts increasing.
EDIT : Some relevant code ...
protected QueueEntryDTO dequeueJobEntry() {
Collection<QueueEntryDTO> collection = getQueueService().dequeueEntry();
if (collection.isEmpty())
throw new EmptyQueueException();
return collection.iterator().next();
}
#Transactional
public Collection<QueueEntryDTO> dequeueEntry() {
Optional<QueueEntry> optionalEntry = this.queueEntryDAO.findTopByStatusCode(QueueStatusEnum.WAITING.getStatusCode());
if (optionalEntry.isPresent()) {
QueueEntry entry = (QueueEntry)optionalEntry.get();
QueueEntry updatedEntry = this.saveEntryStatus(entry, QueueStatusEnum.PROCESSING, (String)null);
return Collections.singleton(this.queueEntryDTOMapper.toDTO(updatedEntry));
} else {
return new ArrayList();
}
}
private void pauseProcessor() {
try {
Long sleepDuration = generalProperties.getQueueProcessingSleepDuration();
sleepDuration = Objects.requireNonNullElseGet(
sleepDuration,
() -> Double.valueOf(Math.pow(2.0, getRetries()) * 1000.0).longValue());
Thread.sleep(sleepDuration);
if (getRetries() < 4)
setRetries(getRetries() + 1);
}
catch (Exception ignored) {
logger.warn("Failed to pause job queue processor.");
}
}
It seems like this was caused by a bug that was resolved in a more recent version of DB2 than I was using.
Applications are getting large number of timer threads when API
timerLevelforQueryTimeout value is not set explicitly in an
application using JCC driver version 11.5 GA (JCC 4.26.14) or
later.
This issue is fixed in 11.5 M4 FP0(JCC 4.27.25).
I updated the version to a newer one (11.5.6) in my POM file, but this didn't fix the issue. Turns out my K8s pod was still using 11.5.0 and Maven acted weird. I then applied this technique (using dependencyManagement in the POM file) and the newer version was loaded.
I have a Grails app where controller calls transactional service. The service has the following body
def rep = Rep.findById(2708280)
def r = Rep.findById(2708280)
r.accountNumber = "123423565476"
r.save(failOnError: true)
def list = Rep.findAllByRtnAndAccountNumber(
rep.rtn, rep.accountNumber)
When the last line in the method is called, I get the following hibernate exception:
ERROR an assertion failure occured (this may indicate a bug in Hibernate, but is more likely due to unsafe use of the session) [AssertionFailure]
org.hibernate.AssertionFailure: collection [com.mydomain.InnerObject.assignedTests] was not processed by flush()
at org.hibernate.engine.CollectionEntry.postFlush(CollectionEntry.java:228)
at org.hibernate.event.def.AbstractFlushingEventListener.postFlush(AbstractFlushingEventListener.java:352)
at org.hibernate.event.def.DefaultAutoFlushEventListener.onAutoFlush(DefaultAutoFlushEventListener.java:65)
at org.hibernate.impl.SessionImpl.autoFlushIfRequired(SessionImpl.java:1185)
at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1709)
at org.hibernate.impl.CriteriaImpl.list(CriteriaImpl.java:347)
at org.codehaus.groovy.grails.orm.hibernate.metaclass.FindAllByPersistentMethod$1.doInHibernate(FindAllByPersistentMethod.java:113)
at org.springframework.orm.hibernate3.HibernateTemplate.doExecute(HibernateTemplate.java:411)
at org.springframework.orm.hibernate3.HibernateTemplate.executeFind(HibernateTemplate.java:348)
at org.codehaus.groovy.grails.orm.hibernate.metaclass.FindAllByPersistentMethod.doInvokeInternalWithExpressions(FindAllByPersistentMethod.java:73)
at org.codehaus.groovy.grails.orm.hibernate.metaclass.AbstractClausedStaticPersistentMethod.doInvokeInternal(AbstractClausedStaticPersistentMethod.java:543)
at org.codehaus.groovy.grails.orm.hibernate.metaclass.AbstractClausedStaticPersistentMethod.doInvokeInternal(AbstractClausedStaticPersistentMethod.java:417)
at org.codehaus.groovy.grails.orm.hibernate.metaclass.AbstractStaticPersistentMethod.invoke(AbstractStaticPersistentMethod.java:79)
at org.codehaus.groovy.grails.orm.hibernate.metaclass.AbstractStaticPersistentMethod.invoke(AbstractStaticPersistentMethod.java:72)
at sun.reflect.GeneratedMethodAccessor9819.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoCachedMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:233)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:58)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:141)
at org.grails.datastore.gorm.GormStaticApi$_methodMissing_closure2.doCall(GormStaticApi.groovy:102)
at org.grails.datastore.gorm.GormStaticApi$_methodMissing_closure2.call(GormStaticApi.groovy)
at org.codehaus.groovy.runtime.metaclass.ClosureStaticMetaMethod.invoke(ClosureStaticMetaMethod.java:62)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at org.codehaus.groovy.runtime.callsite.StaticMetaMethodSite.invoke(StaticMetaMethodSite.java:46)
at org.codehaus.groovy.runtime.callsite.StaticMetaMethodSite.call(StaticMetaMethodSite.java:91)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:133)
at com.mydomaint.MyService.$tt__process(MyService.groovy:118)
where 'com.mydomain.InnerObject' is an inner object of my domain object above.
static belongsTo = [owner: InnerObject]
InnerObject owner
and assignedTests is a property of InnerObject domain:
static hasMany = [assignedVouchers: AssignedTests]
Is there anything I am missing here? What does this exception mean?
That happens probably because you have two objects of the same instance in the session with different values.
rep is unmodified and r is modified (and persisted)
When you call list, Hibernate tries to flush the session and it gets confused
at org.hibernate.engine.CollectionEntry.postFlush(CollectionEntry.java:228)
at org.hibernate.event.def.AbstractFlushingEventListener.postFlush(AbstractFlushingEventListener.java:352)
at org.hibernate.event.def.DefaultAutoFlushEventListener.onAutoFlush(DefaultAutoFlushEventListener.java:65)
at org.hibernate.impl.SessionImpl.autoFlushIfRequired(SessionImpl.java:1185)
at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1709)
Instead you could try to hold the parameters you pass to the list in separate variables. Try that:
def r = Rep.findById(2708280)
def p1 = r.rtn
def p2 = r.accountNumber
r.accountNumber = "123423565476"
r.save(failOnError: true)
def list = Rep.findAllByRtnAndAccountNumber(p1, p2)
$ java -version
java version "1.7.0_65"
Java(TM) SE Runtime Environment (build 1.7.0_65-b17)
Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)
Neo4j version = 2.2.6.
Two threads tried to access the same Node within the same milli, and iterate approx 500+ Relationships. The result is that both threads hang forever.
What might be the fix, or correct approach, please? (Apart from, "Don't do it twice.")
Update:
Using JStack to get a thread dump, it appears that it has nothing to with neo4j after all, but something to do with HashSet / HashMap. See update below ....
Log output - note the sequence of threads 1 & 2:
17:12:12.798 (neo4j-ro-pool-2) DEBUG [db.neo4j.Neo4jDao] - Open neo4j tx
17:12:12.798 (neo4j-ro-pool-2) DEBUG [db.neo4j.Neo4jDao] - Getting Node for [XYZ]
17:12:12.798 (neo4j-ro-pool-1) DEBUG [db.neo4j.Neo4jDao] - Open neo4j tx
17:12:12.798 (neo4j-ro-pool-1) DEBUG [db.neo4j.Neo4jDao] - Getting Node for [XYZ]
17:12:12.800 (neo4j-ro-pool-2) DEBUG [db.neo4j.Neo4jDao] - Got Node[284172852]
17:12:12.800 (neo4j-ro-pool-2) DEBUG [db.neo4j.Neo4jDao] - Getting Relationships for Node[284172852]
17:12:12.800 (neo4j-ro-pool-1) DEBUG [db.neo4j.Neo4jDao] - Got Node[284172852]
17:12:12.800 (neo4j-ro-pool-1) DEBUG [db.neo4j.Neo4jDao] - Getting Relationships for Node[284172852]
Code inside Neo4jDao:
public Data addNamesToData(Data data) {
LOG.debug("Open neo4j tx");
try (Transaction tx = graphdb.beginTx()) {
LOG.debug("Getting Node for [{}]", data);
Node node = getNode(data);
LOG.debug("Got {}", node);
if (node == null) {
LOG.debug("Existing Node for [{}] NOT FOUND", data);
tx.success();
return data;
}
addNames(node, data);
tx.success();
return data;
} finally {
LOG.debug("Close neo4j tx");
}
}
private Data addNames(Node node, Data data) {
LOG.debug("Getting Relationships for {}", node);
Iterable<Relationship> rels = node.getRelationships(RelType.HAS_NAME_DATA, Direction.OUTGOING);
int count = 0;
for (Relationship rel : rels) { // Approx 500+ Relationships
String name = NodeUtils.getNameFromNode(rel.getEndNode());
LOG.trace("Adding Name [{}] to Data [{}]", name, data );
data.addName(name);
count++;
}
LOG.debug("Got {} Relationships for [{}]", count, node);
return data;
}
Data.addName():
private final Set<String> nameSet = new HashSet<>();
public void addName(String name) {
if(name != null) {
this.nameSet.add(name);
}
}
Update:
Using JStack to get a thread dump, it appears that it has nothing to with neo4j after all, but something to do with HashSet / HashMap.
Note: The instance of Data is different on each thread ....
"neo4j-ro-pool-2" prio=10 tid=0x00007f572c002000 nid=0x6ebf runnable [0x00007f4eb40d8000]
java.lang.Thread.State: RUNNABLE
at java.util.HashMap.put(HashMap.java:494)
at java.util.HashSet.add(HashSet.java:217)
at data.Data.addName(Data.java:162)
at db.neo4j.Neo4jDao.addNames(Neo4jDao.java:231)
at db.neo4j.Neo4jDao.addNamesToData(Neo4jDao.java:216)
at bus.subscriber.NamesSubscriber.handle(NamesSubscriber.java:39)
at bus.subscriber.NamesSubscriber.handle(NamesSubscriber.java:18)
at bus.adaptor.SubscriberAdaptor$Task.run(SubscriberAdaptor.java:102)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x00007f5845c439d8> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"neo4j-ro-pool-1" prio=10 tid=0x00007f572c001000 nid=0x6ebe runnable [0x00007f4eb3cd4000]
java.lang.Thread.State: RUNNABLE
at java.util.HashMap.put(HashMap.java:494)
at java.util.HashSet.add(HashSet.java:217)
at data.Data.addName(Data.java:162)
at db.neo4j.Neo4jDao.addNames(Neo4jDao.java:231)
at db.neo4j.Neo4jDao.addNamesToData(Neo4jDao.java:216)
at bus.subscriber.NamesSubscriber.handle(NamesSubscriber.java:39)
at bus.subscriber.NamesSubscriber.handle(NamesSubscriber.java:18)
at bus.adaptor.SubscriberAdaptor$Task.run(SubscriberAdaptor.java:102)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x00007f58494161d8> (a java.util.concurrent.ThreadPoolExecutor$Worker)
Using a tip from this blog I've identified which thread is chewing CPU.
$ top -n1 -H | grep -m1 java
28350 ubuntu 20 0 95.143g 0.013t 2.383g R 99.9 45.7 1154:16 java
$ printf "%x\n" 28350
6ebe
From the thread dump we see that nid=0x6ebe is thread "neo4j-ro-pool-1". Not sure how this helps.
I tried to create a lotus notes client using Java. Now, I have a problem to list all the user defined databases. What I tried to do is
// testing purpose
private void printAllDb() throws NotesException
{
DbDirectory dir = session.getDbDirectory(host);
String server = dir.getName();
if(server.equals(""))
{
server = "Local";
}
System.out.println("database direcory list on server (" + server + ")");
Database db = dir.getFirstDatabase(DbDirectory.DATABASE);
do
{
System.out.println("file name: " + db.getFileName().toUpperCase() + " - " + db.getTitle());
} while((db = dir.getNextDatabase()) != null);
}
However, the program will throw the exception:
Exception in thread "main" NotesException: Server access denied
at lotus.domino.NotesExceptionHelper.read(Unknown Source)
at lotus.domino.NotesExceptionHolder._read(Unknown Source)
at lotus.priv.CORBA.iiop.RepImpl.invoke(Unknown Source)
at lotus.priv.CORBA.portable.ObjectImpl._invoke(Unknown Source)
at lotus.domino.corba._IDbDirectoryStub.getFirstDatabase(Unknown Source)
at lotus.domino.cso.DbDirectory.getFirstDatabase(Unknown Source)
at nz.co.sylresearch.sylsearch.agents.lotusnotes.LotusNotesAPIHandler.printAllDb(LotusNotesAPIHandler.java:58)
at nz.co.sylresearch.sylsearch.agents.lotusnotes.LotusNotesAPIHandler.main(LotusNotesAPIHandler.java:44)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
The issue is pretty clear. You are getting an access denied error to the server. The tricky part now is figuring out why.
You should start with making sure the username and password used to create the session object is correct. Then make sure that user has access to the server and has access to run Java code on the server. You'll have to check the server document in the Directory for that.
I am using Spring Hibernate with app engine and cloudSQL for my project but i am getting one error frequently. This occurs when application becomes ideal for sometimes.
For every query (fetching or save/update to database), i open session and close session after its use. like this --
My code for database fetching is
try{
Session session = getSessionFactory().openSession();
if(session != null)
{
List<Account> accounts = session.createQuery("from " + this.clazz.getName() + " where subDomainName = '"+subDomain+"'").list();
session.close();
if(accounts != null)
{
if(accounts.size()>0){
return accounts.get(0);
}
else{
return null;
}
}
else
{
return null;
}
}
else {
return null;
}
}
catch(Exception e){
log.info("Error in retriving subdomain details :: in Account Dao");
return null;
}
I am using Autowired structuring. but when application becomes ideal for sometime and after some time when i refresh page its display the error as Stream Closed of CloudSQL on app engine, error is shown below...
Error is :
6 Jan, 2014 6:21:04 AM com.google.appengine.repackaged.org.apache.http.impl.client.DefaultRequestDirector handleResponse
WARNING: Authentication error: Unable to respond to any of these challenges: {bearer=WWW-Authenticate: Bearer realm="https://www.google.com/accounts/AuthSubRequest", error=invalid_token}
6 Jan, 2014 6:21:04 AM com.google.appengine.api.rdbms.dev.LocalRdbmsServiceRemoteDriver openConnection
WARNING: openConnection
java.sql.SQLException: Stream closed
at com.google.cloud.sql.jdbc.internal.googleapi.RpcGoogleApi.newOpenConnectionIOException(RpcGoogleApi.java:187)
at com.google.cloud.sql.jdbc.internal.googleapi.RpcGoogleApi.openConnection(RpcGoogleApi.java:105)
at com.google.appengine.api.rdbms.dev.LocalRdbmsServiceRemoteDriver.openConnection(LocalRdbmsServiceRemoteDriver.java:206)
at com.google.appengine.api.rdbms.dev.LocalRdbmsService.openConnection(LocalRdbmsService.java:119)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.google.appengine.tools.development.ApiProxyLocalImpl$AsyncApiCall.callInternal(ApiProxyLocalImpl.java:498)
at com.google.appengine.tools.development.ApiProxyLocalImpl$AsyncApiCall.call(ApiProxyLocalImpl.java:452)
at com.google.appengine.tools.development.ApiProxyLocalImpl$AsyncApiCall.call(ApiProxyLocalImpl.java:430)
at java.util.concurrent.Executors$PrivilegedCallable$1.run(Executors.java:461)
at java.security.AccessController.doPrivileged(Native Method)
at java.util.concurrent.Executors$PrivilegedCallable.call(Executors.java:458)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Stream closed
at java.util.zip.GZIPInputStream.ensureOpen(GZIPInputStream.java:42)
at java.util.zip.GZIPInputStream.read(GZIPInputStream.java:85)
For this to engage app engine and cloudSQL i have wriiten cron job of certains request but this is not feasible solution, i can not gets why cloudSQL closes its stream for app engine.