I received an OOM exception at one point in Cassandra. Mine is a single instance running on a modestly powered server, and i was doing some load testing, so no surprise there.
But, i have subsequently been unable to use the instance. When i list the keyspaces, only "system" is shown. But when i try to recreate the keyspace i was testing with, Hector responds with the dreaded "All host pools marked down. Retry burden pushed out to client." message, and the Cassandra log has the following stack trace:
ERROR [MigrationStage:1] 2012-04-27 20:47:00,863 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[MigrationStage:1,5,main]
java.lang.AssertionError
at org.apache.cassandra.db.DefsTable.updateKeyspace(DefsTable.java:441)
at org.apache.cassandra.db.DefsTable.mergeKeyspaces(DefsTable.java:339)
at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:269)
at org.apache.cassandra.service.MigrationManager$1.call(MigrationManager.java:214)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
ERROR [Thrift:9] 2012-04-27 20:47:00,864 CustomTThreadPoolServer.java (line 204) Error occurred during processing of message.
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError
at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:372)
at org.apache.cassandra.service.MigrationManager.announce(MigrationManager.java:191)
at org.apache.cassandra.service.MigrationManager.announceNewKeyspace(MigrationManager.java:129)
at org.apache.cassandra.thrift.CassandraServer.system_add_keyspace(CassandraServer.java:987)
at org.apache.cassandra.thrift.Cassandra$Processor$system_add_keyspace.getResult(Cassandra.java:3370)
at org.apache.cassandra.thrift.Cassandra$Processor$system_add_keyspace.getResult(Cassandra.java:3358)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:186)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:368)
... 11 more
Caused by: java.lang.AssertionError
at org.apache.cassandra.db.DefsTable.updateKeyspace(DefsTable.java:441)
at org.apache.cassandra.db.DefsTable.mergeKeyspaces(DefsTable.java:339)
at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:269)
at org.apache.cassandra.service.MigrationManager$1.call(MigrationManager.java:214)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
... 3 more
The old keyspace was still in the data dir, so i moved it, but that didn't help. It seems that the system data still has an invalid reference somewhere. Does anyone know how to fix this?
Edit: from the CLI, a "describe cluster;" only describes the "system" keyspace. But when i "use system;" and then "list schema_keyspaces;" the following is displayed:
Using default limit of 100
-------------------
RowKey: mango
=> (column=durable_writes, value=true, timestamp=29127788177516974)
=> (column=name, value=mango, timestamp=29127788177516974)
=> (column=strategy_class, value=org.apache.cassandra.locator.SimpleStrategy, timestamp=29127788177516974)
=> (column=strategy_options, value={"replication_factor":"1"}, timestamp=29127788177516974)
1 Row Returned.
Elapsed time: 1107 msec(s).
"mango" is the keyspace that i can no longer access, but it is still in there to some degree. Is there any way to fix it?
The problem almost certainly is that the recreated keyspace is inconsistent with the commit log or data stored with the original definition. Shut down the Cassandra server and clear out the commitlog, saved_caches, and data directory corresponding to the keyspace. The locations of these directories are in cassandra.yaml - look for data_file_directories, saved_caches_directory, and commitlog_directory.
This problem is due to inconsistency and you can go for following steps.
1) In your case this is OK to clear "data" , "saved_caches" and "commitlog" directories as you dont have any critical data and other Keyspaces.
2) In the scenarios where you have some critical data and you can not delete above mentioned directories do the following.
Use nodetool drain to Empty the commitlog on all the nodes of the cluster.
Then delete all the "LocationInfo*" files from "/data/system" directories and restart the cluster.
Related
How do deal with h2 database inability to deal with interrupts, I was occasionally seeing that my embedded h2 database appeared to get corrupted, in particular I had amended an ExecutorService so that if a task took too long it would cancel the task. The task would be cancelled okay but then subsequent database access failed with exceptions such as
23/07/2019 14.23.31:BST:DeleteDuplicatesController:start:SEVERE: commit failed
org.hibernate.TransactionException: commit failed
at org.hibernate.engine.transaction.spi.AbstractTransactionImpl.commit(AbstractTransactionImpl.java:187)
at com.jthink.songkong.db.ReportCache.save(ReportCache.java:46)
at com.jthink.songkong.reports.AbstractReport.setReportDatabaseObject(AbstractReport.java:365)
at com.jthink.songkong.reports.DeleteDuplicatesReport.setReportDatabaseObject(DeleteDuplicatesReport.java:333)
at com.jthink.songkong.reports.DeleteDuplicatesReport.closeReport(DeleteDuplicatesReport.java:377)
at com.jthink.songkong.analyse.toplevelanalyzer.DeleteDuplicatesController.deleteAnyDups(DeleteDuplicatesController.java:606)
at com.jthink.songkong.analyse.toplevelanalyzer.DeleteDuplicatesController.start(DeleteDuplicatesController.java:665)
at com.jthink.songkong.ui.swingworker.DeleteDuplicates.doInBackground(DeleteDuplicates.java:43)
at com.jthink.songkong.ui.swingworker.DeleteDuplicates.doInBackground(DeleteDuplicates.java:20)
at javax.swing.SwingWorker$1.call(SwingWorker.java:295)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at javax.swing.SwingWorker.run(SwingWorker.java:334)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.hibernate.TransactionException: unable to commit against JDBC connection
at org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction.doCommit(JdbcTransaction.java:116)
at org.hibernate.engine.transaction.spi.AbstractTransactionImpl.commit(AbstractTransactionImpl.java:180)
... 14 more
Caused by: org.h2.jdbc.JdbcSQLNonTransientException: General error: "java.lang.IllegalStateException: Reading from nio:C:/Users/Paul/AppData/Roaming/SongKong/Database/Database.mv.db failed; file length -1 read length 4096 at 1541494 [1.4.199/1]"; SQL statement:
COMMIT [50000-199]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:502)
at org.h2.message.DbException.getJdbcSQLException(DbException.java:427)
at org.h2.message.DbException.get(DbException.java:194)
at org.h2.message.DbException.convert(DbException.java:347)
at org.h2.command.Command.executeUpdate(Command.java:280)
at org.h2.jdbc.JdbcConnection.commit(JdbcConnection.java:542)
at com.mchange.v2.c3p0.impl.NewProxyConnection.commit(NewProxyConnection.java:1284)
at org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction.doCommit(JdbcTransaction.java:112)
... 15 more
Caused by: java.lang.IllegalStateException: Reading from nio:C:/Users/Paul/AppData/Roaming/SongKong/Database/Database.mv.db failed; file length -1 read length 4096 at 1541494 [1.4.199/1]
at org.h2.mvstore.DataUtils.newIllegalStateException(DataUtils.java:883)
at org.h2.mvstore.DataUtils.readFully(DataUtils.java:420)
at org.h2.mvstore.FileStore.readFully(FileStore.java:98)
at org.h2.mvstore.MVStore.readBufferForPage(MVStore.java:1048)
at org.h2.mvstore.MVStore.readPage(MVStore.java:2186)
at org.h2.mvstore.MVMap.readPage(MVMap.java:554)
at org.h2.mvstore.Page$NonLeaf.getChildPage(Page.java:1086)
at org.h2.mvstore.Page.get(Page.java:221)
at org.h2.mvstore.MVMap.get(MVMap.java:402)
at org.h2.mvstore.MVMap.get(MVMap.java:389)
at org.h2.mvstore.MVStore.getMapName(MVStore.java:2737)
at org.h2.mvstore.MVStore.renameMap(MVStore.java:2650)
at org.h2.mvstore.tx.TransactionStore.commit(TransactionStore.java:453)
at org.h2.mvstore.tx.Transaction.commit(Transaction.java:389)
at org.h2.engine.Session.commit(Session.java:691)
at org.h2.command.dml.TransactionCommand.update(TransactionCommand.java:46)
at org.h2.command.CommandContainer.update(CommandContainer.java:133)
at org.h2.command.Command.executeUpdate(Command.java:267)
... 18 more
Caused by: java.nio.channels.ClosedChannelException
at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:110)
at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:721)
at org.h2.store.fs.FileNio.read(FilePathNio.java:74)
at org.h2.mvstore.DataUtils.readFully(DataUtils.java:406)
... 34 more
23/07/2019 14.23.31:BST:Errors:addError:SEVERE: Adding Error:commit failed
I have since found this issue
Basically if using H2 in embedded mode, and it receives an interrupt then all subsequent access fails until the thread pool is close and reopened. In the example I give of a process having to be cancelled because it appears to be stuck there is no solution except for interrupting
I also have another case whereby usually the controller thread that doesn't directly do a database work itself so I was struggling to see why when an interrupt occurred why this would cause database errors since this is handled by controller thread. I have now worked out the issue is that Im using an ExecutorService with a fixed size BlockingQueue (so that we dont have a big queue build up in memory), but if the queue gets full then new task is actually executed by the controller thread (because of CallerRunsPolicy), so the controller thread can be making calls to database after all.
Im using H2 with hibernate and in both cases calling the following immediately after the interrupt
HibernateUtil.closeFactory();
seems to solve the issue, however I guess this means that any other threads with hibernate sessions will be broken, but at least newly opened sessins will be okay. So im not particularly happy with this workaround, any other ideas ?
Using H2 as a server is not a solution since the whole point of H2 was an embedded db self contained within application.
Although not properly documented using the async protocol allows a connection to be interrupted without breaking all other connections.
I've got a spring boot rest service with embedded tomcat deployed on an Azure App Service and I am experiencing intermittent outages, every few weeks.
Every time there is an outage the logs contain the following entries:
Message: java.util.zip.ZipException: ZIP_Read: error reading zip file ZIP_Read: error reading zip file
Exception type: java.util.zip.ZipException
Failed method: java.util.zip.ZipFile.access$1400
With the following call stack.
java.lang.IllegalStateException:
at org.apache.catalina.webresources.JarWarResourceSet.getArchiveEntries (JarWarResourceSet.java133)
at org.apache.catalina.webresources.AbstractArchiveResourceSet.getResource (AbstractArchiveResourceSet.java256)
at org.apache.catalina.webresources.StandardRoot.getResourceInternal (StandardRoot.java281)
at org.apache.catalina.webresources.CachedResource.validateResource (CachedResource.java97)
at org.apache.catalina.webresources.Cache.getResource (Cache.java69)
at org.apache.catalina.webresources.StandardRoot.getResource (StandardRoot.java216)
at org.apache.catalina.webresources.StandardRoot.getResource (StandardRoot.java206)
at org.apache.catalina.mapper.Mapper.internalMapWrapper (Mapper.java1027)
at org.apache.catalina.mapper.Mapper.internalMap (Mapper.java842)
at org.apache.catalina.mapper.Mapper.map (Mapper.java698)
at org.apache.catalina.connector.CoyoteAdapter.postParseRequest (CoyoteAdapter.java679)
at org.apache.catalina.connector.CoyoteAdapter.service (CoyoteAdapter.java336)
at org.apache.coyote.http11.Http11Processor.service (Http11Processor.java803)
at org.apache.coyote.AbstractProcessorLight.process (AbstractProcessorLight.java66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process (AbstractProtocol.java868)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun (NioEndpoint.java1459)
at org.apache.tomcat.util.net.SocketProcessorBase.run (SocketProcessorBase.java49)
at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run (TaskThread.java61)
at java.lang.Thread.run (Thread.java748)
Inner exception java.util.zip.ZipException handled at org.apache.catalina.webresources.JarWarResourceSet.getArchiveEntries:
at java.util.zip.ZipFile.access$1400 (ZipFile.java60)
at java.util.zip.ZipFile$ZipFileInputStream.read (ZipFile.java734)
at java.io.FilterInputStream.read (FilterInputStream.java133)
at java.io.PushbackInputStream.read (PushbackInputStream.java186)
at java.util.zip.ZipInputStream.readFully (ZipInputStream.java403)
at java.util.zip.ZipInputStream.readLOC (ZipInputStream.java278)
at java.util.zip.ZipInputStream.getNextEntry (ZipInputStream.java122)
at java.util.jar.JarInputStream.<init> (JarInputStream.java83)
at java.util.jar.JarInputStream.<init> (JarInputStream.java62)
at org.apache.catalina.webresources.TomcatJarInputStream.<init> (TomcatJarInputStream.java37)
at org.apache.catalina.webresources.JarWarResourceSet.getArchiveEntries (JarWarResourceSet.java108)
at org.apache.catalina.webresources.AbstractArchiveResourceSet.getResource (AbstractArchiveResourceSet.java256)
at org.apache.catalina.webresources.StandardRoot.getResourceInternal (StandardRoot.java281)
at org.apache.catalina.webresources.CachedResource.validateResource (CachedResource.java97)
at org.apache.catalina.webresources.Cache.getResource (Cache.java69)
at org.apache.catalina.webresources.StandardRoot.getResource (StandardRoot.java216)
at org.apache.catalina.webresources.StandardRoot.getResource (StandardRoot.java206)
at org.apache.catalina.mapper.Mapper.internalMapWrapper (Mapper.java1027)
at org.apache.catalina.mapper.Mapper.internalMap (Mapper.java842)
at org.apache.catalina.mapper.Mapper.map (Mapper.java698)
at org.apache.catalina.connector.CoyoteAdapter.postParseRequest (CoyoteAdapter.java679)
at org.apache.catalina.connector.CoyoteAdapter.service (CoyoteAdapter.java336)
at org.apache.coyote.http11.Http11Processor.service (Http11Processor.java803)
at org.apache.coyote.AbstractProcessorLight.process (AbstractProcessorLight.java66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process (AbstractProtocol.java868)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun (NioEndpoint.java1459)
at org.apache.tomcat.util.net.SocketProcessorBase.run (SocketProcessorBase.java49)
at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run (TaskThread.java61)
at java.lang.Thread.run (Thread.java748)
As soon as this occurs, all calls to the rest service returns a 500 and the log gets another entry like above. This continues until I manually reboot the app service.
I'm struggling to figure out what is the issue. Googling the exception only returns results for issues with opening zip files. The app does not do any zipping/unzipping in itself and the call stack seems to indicate that it's tomcat things(that could be caused by something I've done on Azure, in SpringBoot, the JVM or something else entirely).
CPU or memory usage appear to be fine preceding the outages so that doesn't seem to be a factor.
This issue is not preceded by any deployments or platform changes.
I'm stumped regarding what to do next, if anyone can point me in the right direction to investigate, it'll be much appreciated.
I run the following code as a java program. The query runs infinite times even though it is called only once which I am not able to figure out why. It drops the keyspace during the first query and when it runs again (which it should not), it keeps giving exception numerous times that the keyspace does not exist.
What should I do to make the query run only once?
Session session=cassandraSessionFactory.getSession();
String query = "drop keyspace "+keyspace_name";
session.execute(query);
session.close();
Here is the exception stack trace which keeps on getting printed numerous times and stops only when manually halted.
23:24:55,654 INFO BackendOperation:75 - Temporary exception during backend operation [messageReading#0:0]. Attempting backoff retry.
com.thinkaurelius.titan.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at com.thinkaurelius.titan.diskstorage.cassandra.astyanax.AstyanaxKeyColumnValueStore.getNamesSlice(AstyanaxKeyColumnValueStore.java:114)
at com.thinkaurelius.titan.diskstorage.cassandra.astyanax.AstyanaxKeyColumnValueStore.getNamesSlice(AstyanaxKeyColumnValueStore.java:78)
at com.thinkaurelius.titan.diskstorage.cassandra.astyanax.AstyanaxKeyColumnValueStore.getSlice(AstyanaxKeyColumnValueStore.java:67)
at com.thinkaurelius.titan.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:769)
at com.thinkaurelius.titan.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:766)
at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:133)
at com.thinkaurelius.titan.diskstorage.util.BackendOperation$1.call(BackendOperation.java:147)
at com.thinkaurelius.titan.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:56)
at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:42)
at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:144)
at com.thinkaurelius.titan.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:703)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.netflix.astyanax.connectionpool.exceptions.BadRequestException: BadRequestException: [host=127.0.0.1(127.0.0.1):9160, latency=0(0), attempts=1]InvalidRequestException(why:Keyspace keyspace_name does not exist)
at com.netflix.astyanax.thrift.ThriftConverter.ToConnectionPoolException(ThriftConverter.java:159)
at com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:65)
at com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:28)
at com.netflix.astyanax.thrift.ThriftSyncConnectionFactoryImpl$ThriftConnection.execute(ThriftSyncConnectionFactoryImpl.java:153)
at com.netflix.astyanax.connectionpool.impl.AbstractExecuteWithFailoverImpl.tryOperation(AbstractExecuteWithFailoverImpl.java:119)
at com.netflix.astyanax.connectionpool.impl.AbstractHostPartitionConnectionPool.executeWithFailover(AbstractHostPartitionConnectionPool.java:352)
at com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$4.execute(ThriftColumnFamilyQueryImpl.java:538)
at com.thinkaurelius.titan.diskstorage.cassandra.astyanax.AstyanaxKeyColumnValueStore.getNamesSlice(AstyanaxKeyColumnValueStore.java:112)
... 17 more
Caused by: InvalidRequestException(why:Keyspace keyspace_name does not exist)
at org.apache.cassandra.thrift.Cassandra$multiget_slice_result$multiget_slice_resultStandardScheme.read(Cassandra.java:14678)
at org.apache.cassandra.thrift.Cassandra$multiget_slice_result$multiget_slice_resultStandardScheme.read(Cassandra.java:14633)
at org.apache.cassandra.thrift.Cassandra$multiget_slice_result.read(Cassandra.java:14559)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
at org.apache.cassandra.thrift.Cassandra$Client.recv_multiget_slice(Cassandra.java:741)
at org.apache.cassandra.thrift.Cassandra$Client.multiget_slice(Cassandra.java:725)
at com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$4$1.internalExecute(ThriftColumnFamilyQueryImpl.java:544)
at com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$4$1.internalExecute(ThriftColumnFamilyQueryImpl.java:541)
at com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:60)
... 23 more
We are trying a bunch of operations like SELECT -> store row key into a collection ->then split the collection into each worker thread -> Each thread again created connection using phoenix jdbc -> perform SELECT then depending on the result UPSERT into a different phoenix table.
I am using ExecutorService with a fixed thread pool of 4 I am seeing exceptions as below.
org.apache.phoenix.exception.PhoenixIOException: org.apache.phoenix.exception.PhoenixIOException: The system cannot find the path specified
at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:108)
at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:538)
at org.apache.phoenix.iterate.ConcatResultIterator.getIterators(ConcatResultIterator.java:50)
at org.apache.phoenix.iterate.ConcatResultIterator.currentIterator(ConcatResultIterator.java:97)
at org.apache.phoenix.iterate.ConcatResultIterator.next(ConcatResultIterator.java:117)
at org.apache.phoenix.jdbc.PhoenixResultSet.next(PhoenixResultSet.java:764)
at com.vonage.test.PopulateStagingGWCDRWorker.run(MyCode.java:74)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException: org.apache.phoenix.exception.PhoenixIOException: The system cannot find the path specified
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:206)
at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:534)
... 8 more
Caused by: org.apache.phoenix.exception.PhoenixIOException: The system cannot find the path specified
at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:108)
at org.apache.phoenix.iterate.SpoolingResultIterator.<init>(SpoolingResultIterator.java:122)
at org.apache.phoenix.iterate.SpoolingResultIterator.<init>(SpoolingResultIterator.java:73)
at org.apache.phoenix.iterate.SpoolingResultIterator$SpoolingResultIteratorFactory.newIterator(SpoolingResultIterator.java:67)
at org.apache.phoenix.iterate.ChunkedResultIterator.<init>(ChunkedResultIterator.java:92)
at org.apache.phoenix.iterate.ChunkedResultIterator$ChunkedResultIteratorFactory.newIterator(ChunkedResultIterator.java:72)
at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:92)
at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:83)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Caused by: java.io.IOException: The system cannot find the path specified
at java.io.WinNTFileSystem.createFileExclusively(Native Method)
at java.io.File.createTempFile(File.java:2024)
at org.apache.commons.io.output.DeferredFileOutputStream.thresholdReached(DeferredFileOutputStream.java:176)
at org.apache.phoenix.iterate.SpoolingResultIterator$1.thresholdReached(SpoolingResultIterator.java:98)
at org.apache.commons.io.output.ThresholdingOutputStream.checkThreshold(ThresholdingOutputStream.java:224)
at org.apache.commons.io.output.ThresholdingOutputStream.write(ThresholdingOutputStream.java:92)
at java.io.DataOutputStream.writeByte(DataOutputStream.java:153)
at org.apache.hadoop.io.WritableUtils.writeVLong(WritableUtils.java:273)
at org.apache.hadoop.io.WritableUtils.writeVInt(WritableUtils.java:253)
at org.apache.phoenix.util.TupleUtil.write(TupleUtil.java:146)
at org.apache.phoenix.iterate.SpoolingResultIterator.<init>(SpoolingResultIterator.java:107)
... 10 more
enter code here
But If I am using a pool size of 2 or less it works fine. I was wondering if there is a property at the client side that can be changed ?
I my case I solved this by using below dependency in pom.xml
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-protocol</artifactId>
<version>1.1.11</version>
</dependency>
Just to update you i am having Hbase version : 1.1 and phoenix is at 4.7
phoenix.spool.directory in the hbase-site.xml fixes this. Thanks
Following is our cypher for a find by ID kind of service:
START n=node:PATIENTS('MEMBER_PLAN_ID:(1 2)') return n
Where 1 2 are the ids passed. When we pass around 2000 ids following error occurs:
java.lang.RuntimeException: org.apache.lucene.queryParser.ParseException: Cannot parse 'MEMBER_PLAN_ID:(1 2)': too many boolean clauses
at org.neo4j.index.impl.lucene.IndexType.query(IndexType.java:304)
at org.neo4j.index.impl.lucene.LuceneIndex.query(LuceneIndex.java:227)
at org.neo4j.index.impl.lucene.LuceneIndex.query(LuceneIndex.java:238)
at org.neo4j.cypher.internal.spi.gdsimpl.GDSBackedQueryContext$$anon$1.indexQuery(GDSBackedQueryContext.scala:87)
at org.neo4j.cypher.internal.executionplan.builders.IndexQueryBuilder$$anonfun$getNodeGetter$2.apply(IndexQueryBuilder.scala:83)
at org.neo4j.cypher.internal.executionplan.builders.IndexQueryBuilder$$anonfun$getNodeGetter$2.apply(IndexQueryBuilder.scala:81)
at org.neo4j.cypher.internal.pipes.StartPipe$$anonfun$internalCreateResults$1.apply(StartPipe.scala:36)
at org.neo4j.cypher.internal.pipes.StartPipe$$anonfun$internalCreateResults$1.apply(StartPipe.scala:35)
at scala.collection.Iterator$$anon$13.__AW_hasNext(Iterator.scala:371)
at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala)
at org.neo4j.cypher.internal.ClosingIterator$$anonfun$hasNext$1.apply$mcZ$sp(ClosingIterator.scala:36)
at org.neo4j.cypher.internal.ClosingIterator$$anonfun$hasNext$1.apply(ClosingIterator.scala:35)
at org.neo4j.cypher.internal.ClosingIterator$$anonfun$hasNext$1.apply(ClosingIterator.scala:35)
at org.neo4j.cypher.internal.ClosingIterator.failIfThrows(ClosingIterator.scala:86)
at org.neo4j.cypher.internal.ClosingIterator.hasNext(ClosingIterator.scala:35)
at org.neo4j.cypher.PipeExecutionResult.hasNext(PipeExecutionResult.scala:157)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at scala.collection.convert.Wrappers$IteratorWrapper.hasNext(Wrappers.scala:29)
at org.neo4j.cypher.PipeExecutionResult$$anon$1.hasNext(PipeExecutionResult.scala:73)
at net.ahm.graph.dao.PatientDAO.__AW_findPatients(PatientDAO.java:376)
Caused by: org.apache.lucene.queryParser.ParseException: Cannot parse 'MEMBER_PLAN_ID:(1 2)': too many boolean clauses
at org.apache.lucene.queryParser.QueryParser.parse(QueryParser.java:221)
at org.neo4j.index.impl.lucene.IndexType.query(IndexType.java:300)
... 38 more
Caused by: org.apache.lucene.search.BooleanQuery$TooManyClauses: maxClauseCount is set to 1024
at org.apache.lucene.search.BooleanQuery.add(BooleanQuery.java:136)
at org.apache.lucene.queryParser.QueryParser.getBooleanQuery(QueryParser.java:958)
at org.apache.lucene.queryParser.QueryParser.getBooleanQuery(QueryParser.java:933)
at org.apache.lucene.queryParser.QueryParser.Query(QueryParser.java:1281)
at org.apache.lucene.queryParser.QueryParser.Clause(QueryParser.java:1323)
at org.apache.lucene.queryParser.QueryParser.Query(QueryParser.java:1245)
at org.apache.lucene.queryParser.QueryParser.TopLevelQuery(QueryParser.java:1234)
at org.apache.lucene.queryParser.QueryParser.parse(QueryParser.java:206)
... 39 more
We see this in the stack trace: maxClauseCount is set to 1024
Is there a way to configure this limit from Neo4j while using Cypher?
However, if i add this line to our service, things work fine:
BooleanQuery.setMaxClauseCount(20000);
We did some load tests where 1000 concurrent users hit the service with 10,000 ids each. Not seeing service crash / any unexpected performance problems afterwards.
Try this (untested):
System.setProperty("org.apache.lucene.maxClauseCount", "3000");