Develop Reference

Tomcat webapp hangs at startup due to db-connection

We have deployed a war file to tomcat container (java/hibernate web application). When starting tomcat, the deployment process freezes at the point where the Database Connection is being initialized. And then, tomcat server never starts. It's the company's test database which fails on startup, when changing to the development server, it works fine. However, I can connect to the test database using the db-visualiser tool.
Here is what we can see in the catalina log, when launching tomcat.
15:16:59.481 [Thread-1] INFO o.h.c.DriverManagerConnectionProvider.configure 64 - Using Hibernate built-in connection pool (not for production use!)
15:16:59.481 [Thread-1] INFO o.h.c.DriverManagerConnectionProvider.configure 65 - Hibernate connection pool size: 20
15:16:59.482 [Thread-1] INFO o.h.c.DriverManagerConnectionProvider.configure 68 - autocommit mode: false
15:16:59.562 [Thread-1] INFO o.h.c.DriverManagerConnectionProvider.configure 103 - using driver: com.sybase.jdbc3.jdbc.SybDriver at URL: jdbc:sybase:Tds:servername:5000/dbname
15:16:59.563 [Thread-1] INFO o.h.c.DriverManagerConnectionProvider.configure 109 - connection properties: {user=username, password=password}
The environments are as follows: Java6, hibernate, tomcat6, sybase

Turned out that the log file on the server was takling too much space. It was deleted and the server became responsive again

Unable to connect to remote cassandra from titan

I am using cassandra 2.0.7 sitting on a remote server listening on non-default port
<code>
---cassandra.yaml
rpc_address: 0.0.0.0
rpc_port: 6543
</code>
I am trying to connect to the server using titan-0.4.4 (java API, also tried with rexster) using the following config:
<code>
storage.hostname=172.182.183.215
storage.backend=cassandra
storage.port=6543
storage.keyspace=abccorp
</code>
It does not connect and I see the the following exceptions below. However, if I use cqlsh on the same host from where I am trying to execute my code/rexster, I am able to connect without any issues. Anybody seen this?
<code>
0 [main] INFO com.netflix.astyanax.connectionpool.impl.ConnectionPoolMBeanManager - Registering mbean: com.netflix.MonitoredResources:type=ASTYANAX,name=ClusterTitanConnectionPool,ServiceType=connectionpool
49 [main] INFO com.netflix.astyanax.connectionpool.impl.CountingConnectionPoolMonitor - AddHost: 172.182.183.215
554 [main] INFO com.netflix.astyanax.connectionpool.impl.ConnectionPoolMBeanManager - Registering mbean: com.netflix.MonitoredResources:type=ASTYANAX,name=KeyspaceTitanConnectionPool,ServiceType=connectionpool
555 [main] INFO com.netflix.astyanax.connectionpool.impl.CountingConnectionPoolMonitor - AddHost: 172.182.183.215
999 [main] INFO com.netflix.astyanax.connectionpool.impl.CountingConnectionPoolMonitor - AddHost: 127.0.0.1
1000 [main] INFO com.netflix.astyanax.connectionpool.impl.CountingConnectionPoolMonitor - RemoveHost: 172.182.183.215
2366 [main] INFO com.thinkaurelius.titan.diskstorage.Backend - Initiated backend operations thread pool of size 16
41523 [RingDescribeAutoDiscovery] WARN com.netflix.astyanax.impl.RingDescribeHostSupplier - Failed to get hosts from abccorp via ring describe. Will use previously known ring instead
61522 [RingDescribeAutoDiscovery] WARN com.netflix.astyanax.impl.RingDescribeHostSupplier - Failed to get hosts from abccorp via ring describe. Will use previously known ring instead
63080 [main] INFO com.thinkaurelius.titan.diskstorage.util.BackendOperation - Temporary storage exception during backend operation. Attempting backoff retry
com.thinkaurelius.titan.diskstorage.TemporaryStorageException: Temporary failure in storage backend
at com.thinkaurelius.titan.diskstorage.cassandra.astyanax.AstyanaxOrderedKeyColumnValueStore.getNamesSlice(AstyanaxOrderedKeyColumnValueStore.java:138)
at com.thinkaurelius.titan.diskstorage.cassandra.astyanax.AstyanaxOrderedKeyColumnValueStore.getSlice(AstyanaxOrderedKeyColumnValueStore.java:88)
at com.thinkaurelius.titan.graphdb.configuration.KCVSConfiguration$1.call(KCVSConfiguration.java:70)
at com.thinkaurelius.titan.graphdb.configuration.KCVSConfiguration$1.call(KCVSConfiguration.java:64)
at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:30)
at com.thinkaurelius.titan.graphdb.configuration.KCVSConfiguration.getConfigurationProperty(KCVSConfiguration.java:64)
at com.thinkaurelius.titan.diskstorage.Backend.initialize(Backend.java:277)
at com.thinkaurelius.titan.graphdb.configuration.GraphDatabaseConfiguration.getBackend(GraphDatabaseConfiguration.java:1174)
at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.<init>(StandardTitanGraph.java:75)
at com.thinkaurelius.titan.core.TitanFactory.open(TitanFactory.java:40)
at com.thinkaurelius.titan.core.TitanFactory.open(TitanFactory.java:29)
at com.abccorp.grp.graphorm.GraphORM.<init>(GraphORM.java:23)
at com.abccorp.grp.graphorm.GraphORM.getInstance(GraphORM.java:47)
at com.abccorp.grp.utils.dataloader.MainLoader.main(MainLoader.java:150)
Caused by: com.netflix.astyanax.connectionpool.exceptions.NoAvailableHostsException: NoAvailableHostsException: [host=None(0.0.0.0):0, latency=0(0), attempts=0]No hosts to borrow from
at com.netflix.astyanax.connectionpool.impl.RoundRobinExecuteWithFailover.<init>(RoundRobinExecuteWithFailover.java:30)
at com.netflix.astyanax.connectionpool.impl.TokenAwareConnectionPoolImpl.newExecuteWithFailover(TokenAwareConnectionPoolImpl.java:83)
at com.netflix.astyanax.connectionpool.impl.AbstractHostPartitionConnectionPool.executeWithFailover(AbstractHostPartitionConnectionPool.java:256)
at com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$4.execute(ThriftColumnFamilyQueryImpl.java:519)
at com.thinkaurelius.titan.diskstorage.cassandra.astyanax.AstyanaxOrderedKeyColumnValueStore.getNamesSlice(AstyanaxOrderedKeyColumnValueStore.java:136)
... 13 more
91522 [RingDescribeAutoDiscovery] WARN com.netflix.astyanax.impl.RingDescribeHostSupplier - Failed to get hosts from abccorp via ring describe. Will use previously known ring instead
121522 [RingDescribeAutoDiscovery] WARN com.netflix.astyanax.impl.RingDescribeHostSupplier - Failed to get hosts from abccorp via ring describe. Will use previously known ring instead
</code>
Any help greatly appreciated. I am evaluating titan on cassandra and am a bit stuck on this as previously I was using cassandra (same version) on localhost and everything was fine.
thanks

Changing the listen_address to 172.182.183.215 in the configuration had done the trick. Initially it was not clear if just setting the rpc_address was enough.

Thrift and the drivers that support Thrift are deprecated as of C* 1.2. You should switch to the DataStax Java Driver (currently at 2.0.2).
Alternately, ensure this is set properly in cassandra.yaml
start_rpc: true

Getting error in apche Pig when running over yarn "org.apache.hadoop.ipc.Client - Retrying connect to server: tasktracker3/10.201.2.169:50000"

I am running Apache Pig 0.11.2 with Hadoop 2.2.0.
Most simple jobs that I run in Pig work perfectly fine.
However, whenever I try to use GROUP BY on a large dataset, or the LIMIT operator, I get these connection errors:
2013-12-18 11:21:28,400 [main] INFO org.apache.hadoop.ipc.Client -
Retrying connect to server: tasktracker2/10.201.2.145:54957. Already
tried 0 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=1 SECONDS)
2013-12-18 11:21:29,402 [main] INFO org.apache.hadoop.ipc.Client -
Retrying connect to server: tasktracker2/10.201.2.145:54957. Already
tried 1 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=1 SECONDS)
2013-12-18 11:21:30,403 [main] INFO org.apache.hadoop.ipc.Client -
Retrying connect to server: tasktracker2/10.201.2.145:54957. Already
tried 2 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=1 SECONDS)
2013-12-18 11:21:30,507 [main] INFO
org.apache.hadoop.mapred.ClientServiceDelegate - Application state is
completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job
history server 2013-12-18 11:21:31,703 [main] INFO
org.apache.hadoop.ipc.Client - Retrying connect to server:
tasktracker1/10.201.2.20:49528. Already tried 0 time(s); retry policy
is RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=1
SECONDS) 2013-12-18 11:21:32,704 [main] INFO
org.apache.hadoop.ipc.Client - Retrying connect to server:
tasktracker1/10.201.2.20:49528. Already tried 1 time(s); retry policy
is RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=1
SECONDS) 2013-12-18 11:21:33,705 [main] INFO
org.apache.hadoop.ipc.Client - Retrying connect to server:
tasktracker1/10.201.2.20:49528. Already tried 2 time(s); retry policy
is RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=1
SECONDS) 2013-12-18 11:21:33,809 [main] INFO
org.apache.hadoop.mapred.ClientServiceDelegate - Application state is
completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job
history server 2013-12-18 11:21:34,890 [main] INFO
org.apache.hadoop.ipc.Client - Retrying connect to server:
tasktracker3/10.201.2.169:50000. Already tried 0 time(s); retry policy
is RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=1
SECONDS) 2013-12-18 11:21:35,891 [main] INFO
org.apache.hadoop.ipc.Client - Retrying connect to server:
tasktracker3/10.201.2.169:50000. Already tried 1 time(s); retry policy
is RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=1
SECONDS) 2013-12-18 11:21:36,893 [main] INFO
org.apache.hadoop.ipc.Client - Retrying connect to server:
tasktracker3/10.201.2.169:50000. Already tried 2 time(s); retry policy
is RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=1
SECONDS) 2013-12-18 11:21:36,996 [main] INFO
org.apache.hadoop.mapred.ClientServiceDelegate - Application state is
completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job
history server 2013-12-18 11:21:37,152 [main] INFO
org.apache.hadoop.mapred.ClientServiceDelegate - Application state is
completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job
history server
The strange thing is that after these errors keeping appearing for about minutes, they'll stop, and the correct output shows up at the bottom.
So Hadoop is running fine and computing the proper output. The problem is just these connection errors that keep popping up. and that causing increase in execution time of the script.
One thing that I have noticed is that whenever this error appears, the job had created and ran multiple JAR files during the job. However, after a few minutes of these message popping up, the correct output finally appears.
I have 5 nodes cluster 1 namenode and 4 datanode. All the daemons are running fine.
Any suggestions on how to get rid of these messages?

Looks like your job history server is not running.
Turn on log aggregation (you may have already done that and you're only missing the server) - put this to your yarn-site.xml:
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
Run the job history server:
$HADOOP_INSTALL/sbin/mr-jobhistory-daemon.sh start historyserver
Try running the Pig script again

Cassandra-Cli refusing connection

I am trying to connect to cassandra. I installed the latest stable version that is apache-cassandra-1.2.4 and extracted it on my desktop. As I run cassandra it sets up nicely listening for thrift client and displaying following :
sudo cassandra -f
log :
INFO 15:30:34,646 Cassandra version: 1.0.12
INFO 15:30:34,646 Thrift API version: 19.20.0
INFO 15:30:34,646 Loading persisted ring state
INFO 15:30:34,650 Starting up server gossip
INFO 15:30:34,661 Enqueuing flush of Memtable-LocationInfo#1117603949(29/36 serialized/live bytes, 1 ops)
INFO 15:30:34,661 Writing Memtable-LocationInfo#1117603949(29/36 serialized/live bytes, 1 ops)
INFO 15:30:34,877 Completed flushing /var/lib/cassandra/data/system/LocationInfo-hd-54-Data.db (80 bytes)
INFO 15:30:34,892 Starting Messaging Service on port 7000
INFO 15:30:34,901 Using saved token 143186062733850112297005303551620336860
INFO 15:30:34,903 Enqueuing flush of Memtable-LocationInfo#1282534304(53/66 serialized/live bytes, 2 ops)
INFO 15:30:34,904 Writing Memtable-LocationInfo#1282534304(53/66 serialized/live bytes, 2 ops)
INFO 15:30:35,102 Completed flushing /var/lib/cassandra/data/system/LocationInfo-hd-55-Data.db (163 bytes)
INFO 15:30:35,106 Node localhost/127.0.0.1 state jump to normal
INFO 15:30:35,107 Bootstrap/Replace/Move completed! Now serving reads.
INFO 15:30:35,108 Will not load MX4J, mx4j-tools.jar is not in the classpath
INFO 15:30:35,150 Binding thrift service to localhost/127.0.0.1:9160
INFO 15:30:35,155 Using TFastFramedTransport with a max frame size of 15728640 bytes.
INFO 15:30:35,160 Using synchronous/threadpool thrift server on localhost/127.0.0.1 : 9160
INFO 15:30:35,168 Listening for thrift clients...
Now as I run : cassandra-cli -h localhost -p 9160, it throws up the error. I have checked for the port to be free and cassandra is listening at the port. :
**
org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused
at org.apache.thrift.transport.TSocket.open(TSocket.java:183)
at org.apache.thrift.transport.TFramedTransport.open(TFramedTransport.java:81)
at org.apache.cassandra.cli.CliMain.connect(CliMain.java:80)
at org.apache.cassandra.cli.CliMain.main(CliMain.java:256)
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
at java.net.Socket.connect(Socket.java:579)
at org.apache.thrift.transport.TSocket.open(TSocket.java:178)
... 3 more
Exception connecting to localhost/9160. Reason: Connection refused.
**

I had the same error.Now, it is OK.
The main problem is that the configuration is wrong.
My configuration is as following:
My visual machine ip is 192.168.11.11.My cassandra was installed into the machine.So, I configurate thar
listen_address: 192.168.11.11
rpc_address: 0.0.0.0
broadcast_rpc_address: 192.168.11.11
That is OK。

The documentation of cassandra-stress seems to be sketchy. Maybe in due course that would be corrected. As of now, this command worked for me
./cassandra-stress write -node <IP_OF_NODE1>
Once this works, we could try putting in the other optional parameters to tweak our command.

Option 1:
Run jps command under root user and kill CassandraDaemon if you will see it. After this you will start Cassandra again.
Option2:
Try to connect Cassandra with CQL
./cqlsh 10.234.31.232 9042
Final Check:
An intermediate firewall is blocking the JVM from making the connection.
An operating system firewall, or antivirus that is causing the problems as well.
I think you installed in windows and looks like firewall is blocking your connection.

how to enable NIO / atmosphere on tomcat 6

I'm trying out atmosphere on tomcat 6 but only long polling works not websockets. The message i receive in the console output is
Tomcat failed to detect this is a Comet application because context.xml is missing or the Http11NioProtocol Connector is not enabled.
Any info on this would be appreciated, thanks
Update: After modifying web.xml and server.xml the error message goes but the chat application always fallsback to long polling - I've tried several different .war deployments of websocket demos and they all behave the same.
I tried Tomcat 7 also, but it still uses long polling.
Some log info for startup
10:42:30.423 [main] INFO o.atmosphere.cpr.AtmosphereFramework - Installed Default AtmosphereInterceptor [Android Interceptor Support, SSE Interceptor Support, JSONP Interceptor Support]. Set org.atmosphere.cpr.AtmosphereInterceptor.disableDefaults in your xml to disable them.
10:42:30.423 [main] WARN o.atmosphere.cpr.AtmosphereFramework - No BroadcasterCache configured. Broadcasted message between client reconnection will be LOST. It is recommended to configure the HeaderBroadcasterCache.
10:42:30.423 [main] INFO o.atmosphere.cpr.AtmosphereFramework - HttpSession supported: false
10:42:30.423 [main] INFO o.atmosphere.cpr.AtmosphereFramework - Using BroadcasterFactory: org.atmosphere.cpr.DefaultBroadcasterFactory
10:42:30.423 [main] INFO o.atmosphere.cpr.AtmosphereFramework - Using WebSocketProcessor: org.atmosphere.websocket.DefaultWebSocketProcessor
10:42:30.423 [main] INFO o.atmosphere.cpr.AtmosphereFramework - Using Broadcaster: org.atmosphere.jersey.JerseyBroadcaster
10:42:30.428 [main] INFO o.atmosphere.cpr.AtmosphereFramework - Atmosphere Framework 1.0.0.beta5 started.
27/08/2012 10:42:30 AM org.apache.coyote.http11.Http11NioProtocol start
INFO: Starting Coyote HTTP/1.1 on http-8080
27/08/2012 10:42:30 AM org.apache.catalina.startup.Catalina start
INFO: Server startup in 11568 ms
When the chat client connects
27/08/2012 10:47:57 AM org.apache.tomcat.util.net.NioSelectorPool getSharedSelector
INFO: Using a shared selector for servlet write/read
When a message is sent
10:48:09.256 [http-8080-exec-2] DEBUG o.a.cpr.AsynchronousProcessor - Cancelling the connection for request AtmosphereRequest{ contextPath=/atmosphere-rest-chat servletPath=/chat pathInfo=null requestURI=/atmosphere-rest-chat/chat requestURL=http://mogwai:8080/atmosphere-rest-chat/chat destroyable=true}

In context.xml:
<?xml version="1.0" encoding="UTF-8"?>
<Context>
<Loader delegate="true"/>
</Context>
In server.xml:
<Connector connectionTimeout="20000" port="8080"
protocol="org.apache.coyote.http11.Http11NioProtocol"
redirectPort="8443"/>
Make sure you only have one implementation of CometProcessor defined in your classpath. You may want to remove catalina.jar from your webpass if it's bundled in. See this thread for more info:
http://mail-archives.apache.org/mod_mbox/tomcat-users/200707.mbox/%3C11785700.post#talk.nabble.com%3E