Develop Reference

Apache Calcite is not working in Web Application

I'm using Apache Calcite in My Project to do CSV,Excel and Other Database management.
Its working when i execute through main method but its giving an error while executing through web service
Caused by: java.lang.RuntimeException: org.codehaus.commons.compiler.CompileException: Line 1, Column 0: package org.apache.calcite.rel.metadata does not exist (compiler.err.doesnt.exist)
at com.google.common.base.Throwables.propagate(Throwables.java:160)
at org.apache.calcite.rel.metadata.JaninoRelMetadataProvider.load3(JaninoRelMetadataProvider.java:361)
at org.apache.calcite.rel.metadata.JaninoRelMetadataProvider.access$000(JaninoRelMetadataProvider.java:94)
at org.apache.calcite.rel.metadata.JaninoRelMetadataProvider$1.load(JaninoRelMetadataProvider.java:113)
at org.apache.calcite.rel.metadata.JaninoRelMetadataProvider$1.load(JaninoRelMetadataProvider.java:110)
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
at org.apache.calcite.rel.metadata.JaninoRelMetadataProvider.create(JaninoRelMetadataProvider.java:448)
at org.apache.calcite.rel.metadata.JaninoRelMetadataProvider.revise(JaninoRelMetadataProvider.java:460)
at org.apache.calcite.rel.metadata.RelMetadataQuery.revise(RelMetadataQuery.java:186)
at org.apache.calcite.rel.metadata.RelMetadataQuery.collations(RelMetadataQuery.java:484)
at org.apache.calcite.rel.metadata.RelMdCollation.project(RelMdCollation.java:207)
at org.apache.calcite.rel.logical.LogicalProject$1.get(LogicalProject.java:117)
at org.apache.calcite.rel.logical.LogicalProject$1.get(LogicalProject.java:115)
at org.apache.calcite.plan.RelTraitSet.replaceIfs(RelTraitSet.java:238)
at org.apache.calcite.rel.logical.LogicalProject.create(LogicalProject.java:113)
at org.apache.calcite.rel.logical.LogicalProject.create(LogicalProject.java:103)
at org.apache.calcite.rel.core.RelFactories$ProjectFactoryImpl.createProject(RelFactories.java:120)
at org.apache.calcite.tools.RelBuilder.project(RelBuilder.java:853)
at org.apache.calcite.plan.RelOptUtil.createProject(RelOptUtil.java:2881)
at org.apache.calcite.plan.RelOptUtil.createProject(RelOptUtil.java:2839)
at org.apache.calcite.plan.RelOptUtil.createProject(RelOptUtil.java:2783)
at org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectList(SqlToRelConverter.java:3495)
at org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectImpl(SqlToRelConverter.java:665)
at org.apache.calcite.sql2rel.SqlToRelConverter.convertSelect(SqlToRelConverter.java:622)
at org.apache.calcite.sql2rel.SqlToRelConverter.convertQueryRecursive(SqlToRelConverter.java:2852)
at org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:556)
at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:229)
at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:193)
at org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:733)
at org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:597)
at org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:567)
at org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:215)
at org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:594)
at org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:613)
at org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:139)
... 44 more

I think you're hitting CALCITE-1461, which will be fixed in Calcite 1.11. For now, disable shading.

Apache phoenix concurrent queries failing with exception

We are trying a bunch of operations like SELECT -> store row key into a collection ->then split the collection into each worker thread -> Each thread again created connection using phoenix jdbc -> perform SELECT then depending on the result UPSERT into a different phoenix table.
I am using ExecutorService with a fixed thread pool of 4 I am seeing exceptions as below.
org.apache.phoenix.exception.PhoenixIOException: org.apache.phoenix.exception.PhoenixIOException: The system cannot find the path specified
at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:108)
at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:538)
at org.apache.phoenix.iterate.ConcatResultIterator.getIterators(ConcatResultIterator.java:50)
at org.apache.phoenix.iterate.ConcatResultIterator.currentIterator(ConcatResultIterator.java:97)
at org.apache.phoenix.iterate.ConcatResultIterator.next(ConcatResultIterator.java:117)
at org.apache.phoenix.jdbc.PhoenixResultSet.next(PhoenixResultSet.java:764)
at com.vonage.test.PopulateStagingGWCDRWorker.run(MyCode.java:74)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException: org.apache.phoenix.exception.PhoenixIOException: The system cannot find the path specified
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:206)
at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:534)
... 8 more
Caused by: org.apache.phoenix.exception.PhoenixIOException: The system cannot find the path specified
at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:108)
at org.apache.phoenix.iterate.SpoolingResultIterator.<init>(SpoolingResultIterator.java:122)
at org.apache.phoenix.iterate.SpoolingResultIterator.<init>(SpoolingResultIterator.java:73)
at org.apache.phoenix.iterate.SpoolingResultIterator$SpoolingResultIteratorFactory.newIterator(SpoolingResultIterator.java:67)
at org.apache.phoenix.iterate.ChunkedResultIterator.<init>(ChunkedResultIterator.java:92)
at org.apache.phoenix.iterate.ChunkedResultIterator$ChunkedResultIteratorFactory.newIterator(ChunkedResultIterator.java:72)
at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:92)
at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:83)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Caused by: java.io.IOException: The system cannot find the path specified
at java.io.WinNTFileSystem.createFileExclusively(Native Method)
at java.io.File.createTempFile(File.java:2024)
at org.apache.commons.io.output.DeferredFileOutputStream.thresholdReached(DeferredFileOutputStream.java:176)
at org.apache.phoenix.iterate.SpoolingResultIterator$1.thresholdReached(SpoolingResultIterator.java:98)
at org.apache.commons.io.output.ThresholdingOutputStream.checkThreshold(ThresholdingOutputStream.java:224)
at org.apache.commons.io.output.ThresholdingOutputStream.write(ThresholdingOutputStream.java:92)
at java.io.DataOutputStream.writeByte(DataOutputStream.java:153)
at org.apache.hadoop.io.WritableUtils.writeVLong(WritableUtils.java:273)
at org.apache.hadoop.io.WritableUtils.writeVInt(WritableUtils.java:253)
at org.apache.phoenix.util.TupleUtil.write(TupleUtil.java:146)
at org.apache.phoenix.iterate.SpoolingResultIterator.<init>(SpoolingResultIterator.java:107)
... 10 more
enter code here
But If I am using a pool size of 2 or less it works fine. I was wondering if there is a property at the client side that can be changed ?

I my case I solved this by using below dependency in pom.xml
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-protocol</artifactId>
<version>1.1.11</version>
</dependency>
Just to update you i am having Hbase version : 1.1 and phoenix is at 4.7

phoenix.spool.directory in the hbase-site.xml fixes this. Thanks

org.h2.jdbc.JdbcSQLException: General error: "java.lang.NullPointerException" [50000-178]

I have a small 30 MB h2 database file. Driver version is 1.4.178.
Everything worked fine but recently the DB stop to work with
exception:
org.h2.jdbc.JdbcSQLException: General error: "java.lang.NullPointerException" [50000-178]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:344)
at org.h2.message.DbException.get(DbException.java:167)
at org.h2.message.DbException.convert(DbException.java:294)
at org.h2.engine.Database.openDatabase(Database.java:293)
at org.h2.engine.Database.<init>(Database.java:256)
at org.h2.engine.Engine.openSession(Engine.java:57)
at org.h2.engine.Engine.openSession(Engine.java:164)
at org.h2.engine.Engine.createSessionAndValidate(Engine.java:142)
at org.h2.engine.Engine.createSession(Engine.java:125)
at org.h2.server.TcpServerThread.run(TcpServerThread.java:150)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.NullPointerException
at org.h2.mvstore.DataUtils.parseMap(DataUtils.java:630)
at org.h2.mvstore.MVStore.openMap(MVStore.java:411)
at org.h2.mvstore.db.TransactionStore.<init>(TransactionStore.java:96)
at org.h2.mvstore.db.MVTableEngine$Store.<init>(MVTableEngine.java:161)
at org.h2.mvstore.db.MVTableEngine.init(MVTableEngine.java:94)
at org.h2.engine.Database.getPageStore(Database.java:2355)
at org.h2.engine.Database.open(Database.java:659)
at org.h2.engine.Database.openDatabase(Database.java:262)
... 7 more
at org.h2.engine.SessionRemote.done(SessionRemote.java:610)
at org.h2.engine.SessionRemote.initTransfer(SessionRemote.java:129)
at org.h2.engine.SessionRemote.connectServer(SessionRemote.java:434)
at org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:315)
at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:107)
at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:91)
at org.h2.Driver.connect(Driver.java:74)
at org.h2.server.web.WebServer.getConnection(WebServer.java:684)
at org.h2.server.web.WebApp.test(WebApp.java:896)
at org.h2.server.web.WebApp.process(WebApp.java:222)
at org.h2.server.web.WebApp.processRequest(WebApp.java:171)
at org.h2.server.web.WebThread.process(WebThread.java:138)
at org.h2.server.web.WebThread.run(WebThread.java:94)
at java.lang.Thread.run(Thread.java:744)
The problem occurs in my application and using H2 web frontend.

It's unclear if you're trying to fix the cause of this NPE or if you're trying to retrieve the content of the database.
I suggest the following:
retrieve your data using procedure described at http://h2database.com/html/tutorial.html#upgrade_backup_restore
upgrade your stack to move to the last version of 1.3 (176) or the latest 1.4 (188).
restore the data with this new stack
run and test your server
If the NPE problem still occurs, you'll need to provide more details in this forum.

Apache James 2.3.2 won't write to MySQL

I had a working Apache James mail server up and running, but just converted it to use a MySQL database instead of the filesystem.
It created (some) the tables, but it won't write to them. This makes it not deliver any mail.
Since it will interact with the database and insert users, I think I have my database data-source and user configured correctly...
<data-source name="maildb" class="org.apache.james.util.dbcp.JdbcDataSource">
<driver>com.mysql.jdbc.Driver</driver>
<dburl>jdbc:mysql://127.0.0.1/james</dburl>
<user>xxxx</user>
<password>yyyy</password>
<max>20</max>
</data-source>
But when I try to send an email, I get no error from the client, but I get nothing in the database in any table.
All I get is this error on James console...
exception! javax.mail.MessagingException: Exception caught while storing mail Container: ;
nested exception is:
com.mysql.jdbc.MysqlDataTruncation: Data truncation: Incorrect datetime value: '' for column 'last_updated' at row 1
Here's the full stack trace from mailet log...
10/12/14 16:23:18 INFO James.Mailet: Error while storing mail.
javax.mail.MessagingException: Exception caught while storing mail Container: ;
nested exception is:
com.mysql.jdbc.MysqlDataTruncation: Data truncation: Incorrect datetime value: '' for column 'last_updated' at row 1
at org.apache.james.mailrepository.JDBCMailRepository.store(JDBCMailRepository.java:764)
at org.apache.james.transport.mailets.ToMultiRepository.storeMail(ToMultiRepository.java:230)
at org.apache.james.transport.mailets.ToMultiRepository.service(ToMultiRepository.java:154)
at org.apache.james.transport.mailets.LocalDelivery.service(LocalDelivery.java:66)
at org.apache.james.transport.LinearProcessor.service(LinearProcessor.java:424)
at org.apache.james.transport.JamesSpoolManager.process(JamesSpoolManager.java:405)
at org.apache.james.transport.JamesSpoolManager.run(JamesSpoolManager.java:309)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.mysql.jdbc.MysqlDataTruncation: Data truncation: Incorrect datetime value: '' for column 'last_updated' at row 1
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2973)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1600)
at com.mysql.jdbc.ServerPreparedStatement.serverExecute(ServerPreparedStatement.java:1129)
at com.mysql.jdbc.ServerPreparedStatement.executeInternal(ServerPreparedStatement.java:681)
at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:803)
at org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:168)
at org.apache.james.mailrepository.JDBCMailRepository.store(JDBCMailRepository.java:752)
10/12/14 16:23:18 INFO James.Mailet: ToRepository: Storing mail Mail1418257398530-1 in dbfile://maildb/deadletter/error
Here is how I specified the repository...
<mailet match="All" class="ToRepository">
<repositoryPath> db://maildb/deadletter/error </repositoryPath>
</mailet>
Since it seems to be complaining about an empty last_updated datetime, I made that column null allowed, but that didn't help.
I'm not changing the java in any way, so there shouldn't be anything I messed up. I don't even know where it gets the datetime.
Can anyone see something I've missed?

Solved: I found an obscure post in the Apache bug archives that mentions adding this parm...
<dburl>jdbc:mysql://127.0.0.1/james?useServerPrepStmts=false</dburl>
...to the data-source area. Actually, the post says to add 2 parms here, but I found that it will only accept one parm.
I don't know how I was supposed to know about those parms or what they do, but this made everything magically work. Sometimes using open source software teaches you more about researching than programming.

How to tweak lucene maxClauseCount from Neo4j?

Following is our cypher for a find by ID kind of service:
START n=node:PATIENTS('MEMBER_PLAN_ID:(1 2)') return n
Where 1 2 are the ids passed. When we pass around 2000 ids following error occurs:
java.lang.RuntimeException: org.apache.lucene.queryParser.ParseException: Cannot parse 'MEMBER_PLAN_ID:(1 2)': too many boolean clauses
at org.neo4j.index.impl.lucene.IndexType.query(IndexType.java:304)
at org.neo4j.index.impl.lucene.LuceneIndex.query(LuceneIndex.java:227)
at org.neo4j.index.impl.lucene.LuceneIndex.query(LuceneIndex.java:238)
at org.neo4j.cypher.internal.spi.gdsimpl.GDSBackedQueryContext$$anon$1.indexQuery(GDSBackedQueryContext.scala:87)
at org.neo4j.cypher.internal.executionplan.builders.IndexQueryBuilder$$anonfun$getNodeGetter$2.apply(IndexQueryBuilder.scala:83)
at org.neo4j.cypher.internal.executionplan.builders.IndexQueryBuilder$$anonfun$getNodeGetter$2.apply(IndexQueryBuilder.scala:81)
at org.neo4j.cypher.internal.pipes.StartPipe$$anonfun$internalCreateResults$1.apply(StartPipe.scala:36)
at org.neo4j.cypher.internal.pipes.StartPipe$$anonfun$internalCreateResults$1.apply(StartPipe.scala:35)
at scala.collection.Iterator$$anon$13.__AW_hasNext(Iterator.scala:371)
at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala)
at org.neo4j.cypher.internal.ClosingIterator$$anonfun$hasNext$1.apply$mcZ$sp(ClosingIterator.scala:36)
at org.neo4j.cypher.internal.ClosingIterator$$anonfun$hasNext$1.apply(ClosingIterator.scala:35)
at org.neo4j.cypher.internal.ClosingIterator$$anonfun$hasNext$1.apply(ClosingIterator.scala:35)
at org.neo4j.cypher.internal.ClosingIterator.failIfThrows(ClosingIterator.scala:86)
at org.neo4j.cypher.internal.ClosingIterator.hasNext(ClosingIterator.scala:35)
at org.neo4j.cypher.PipeExecutionResult.hasNext(PipeExecutionResult.scala:157)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at scala.collection.convert.Wrappers$IteratorWrapper.hasNext(Wrappers.scala:29)
at org.neo4j.cypher.PipeExecutionResult$$anon$1.hasNext(PipeExecutionResult.scala:73)
at net.ahm.graph.dao.PatientDAO.__AW_findPatients(PatientDAO.java:376)
Caused by: org.apache.lucene.queryParser.ParseException: Cannot parse 'MEMBER_PLAN_ID:(1 2)': too many boolean clauses
at org.apache.lucene.queryParser.QueryParser.parse(QueryParser.java:221)
at org.neo4j.index.impl.lucene.IndexType.query(IndexType.java:300)
... 38 more
Caused by: org.apache.lucene.search.BooleanQuery$TooManyClauses: maxClauseCount is set to 1024
at org.apache.lucene.search.BooleanQuery.add(BooleanQuery.java:136)
at org.apache.lucene.queryParser.QueryParser.getBooleanQuery(QueryParser.java:958)
at org.apache.lucene.queryParser.QueryParser.getBooleanQuery(QueryParser.java:933)
at org.apache.lucene.queryParser.QueryParser.Query(QueryParser.java:1281)
at org.apache.lucene.queryParser.QueryParser.Clause(QueryParser.java:1323)
at org.apache.lucene.queryParser.QueryParser.Query(QueryParser.java:1245)
at org.apache.lucene.queryParser.QueryParser.TopLevelQuery(QueryParser.java:1234)
at org.apache.lucene.queryParser.QueryParser.parse(QueryParser.java:206)
... 39 more
We see this in the stack trace: maxClauseCount is set to 1024
Is there a way to configure this limit from Neo4j while using Cypher?

However, if i add this line to our service, things work fine:
BooleanQuery.setMaxClauseCount(20000);
We did some load tests where 1000 concurrent users hit the service with 10,000 ids each. Not seeing service crash / any unexpected performance problems afterwards.

Try this (untested):
System.setProperty("org.apache.lucene.maxClauseCount", "3000");