I'm trying to write a remote HBase client using Java. Here is the code for reference :
package ttumdt.app.connector;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.MasterNotRunningException;
import org.apache.hadoop.hbase.ZooKeeperConnectionException;
import org.apache.hadoop.hbase.client.HBaseAdmin;
import org.apache.hadoop.hbase.client.HTable;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.client.ResultScanner;
import org.apache.hadoop.hbase.client.Scan;
import org.apache.hadoop.hbase.filter.BinaryComparator;
import org.apache.hadoop.hbase.filter.CompareFilter;
import org.apache.hadoop.hbase.filter.Filter;
import org.apache.hadoop.hbase.filter.FilterList;
import org.apache.hadoop.hbase.filter.RowFilter;
import org.apache.hadoop.hbase.filter.SingleColumnValueFilter;
import org.apache.hadoop.hbase.util.Bytes;
import java.io.IOException;
import java.util.Arrays;
import java.util.HashSet;
import java.util.Set;
public class HBaseClusterConnector {
private final String MASTER_IP = "10.138.168.185";
private final String ZOOKEEPER_PORT = "2181";
final String TRAFFIC_INFO_TABLE_NAME = "TrafficLog";
final String TRAFFIC_INFO_COLUMN_FAMILY = "TimeStampIMSI";
final String KEY_TRAFFIC_INFO_TABLE_BTS_ID = "BTS_ID";
final String KEY_TRAFFIC_INFO_TABLE_DATE = "DATE";
final String COLUMN_IMSI = "IMSI";
final String COLUMN_TIMESTAMP = "TIME_STAMP";
private final byte[] columnFamily = Bytes.toBytes(TRAFFIC_INFO_COLUMN_FAMILY);
private final byte[] qualifier= Bytes.toBytes(COLUMN_IMSI);
private Configuration conf = null;
public HBaseClusterConnector () throws MasterNotRunningException, ZooKeeperConnectionException {
conf = HBaseConfiguration.create();
conf.set("hbase.zookeeper.quorum",MASTER_IP);
conf.set("hbase.zookeeper.property.clientPort",ZOOKEEPER_PORT);
HBaseAdmin.checkHBaseAvailable(conf);
}
/**
* This filter will return list of IMSIs for a given btsId and ime interval
* #param btsId : btsId for which the query has to run
* #param startTime : start time for which the query has to run
* #param endTime : end time for which the query has to run
* #return returns IMSIs as set of Strings
* #throws IOException
*/
public Set<String> getInfoPerBTSID(String btsId, String date,
String startTime, String endTime)
throws IOException {
Set<String> imsis = new HashSet<String>();
//ToDo : better exception handling
HTable table = new HTable(conf, TRAFFIC_INFO_TABLE_NAME);
Scan scan = new Scan();
scan.addColumn(columnFamily,qualifier);
scan.setFilter(prepFilter(btsId, date, startTime, endTime));
// filter to build where timestamp
Result result = null;
ResultScanner resultScanner = table.getScanner(scan);
while ((result = resultScanner.next())!= null) {
byte[] obtainedColumn = result.getValue(columnFamily,qualifier);
imsis.add(Bytes.toString(obtainedColumn));
}
resultScanner.close();
return imsis;
}
//ToDo : Figure out how valid is this filter code?? How comparison happens
// with eqaul or grater than equal etc
private Filter prepFilter (String btsId, String date,
String startTime, String endTime)
{
byte[] tableKey = Bytes.toBytes(KEY_TRAFFIC_INFO_TABLE_BTS_ID);
byte[] timeStamp = Bytes.toBytes(COLUMN_TIMESTAMP);
// filter to build -> where BTS_ID = <<btsId>> and Date = <<date>>
RowFilter keyFilter = new RowFilter(CompareFilter.CompareOp.EQUAL,
new BinaryComparator(Bytes.toBytes(btsId+date)));
// filter to build -> where timeStamp >= startTime
SingleColumnValueFilter singleColumnValueFilterStartTime =
new SingleColumnValueFilter(columnFamily, timeStamp,
CompareFilter.CompareOp.GREATER_OR_EQUAL,Bytes.toBytes(startTime));
// filter to build -> where timeStamp <= endTime
SingleColumnValueFilter singleColumnValueFilterEndTime =
new SingleColumnValueFilter(columnFamily, timeStamp,
CompareFilter.CompareOp.LESS_OR_EQUAL,Bytes.toBytes(endTime));
FilterList filterList = new FilterList(FilterList.Operator.MUST_PASS_ALL, Arrays
.asList((Filter) keyFilter,
singleColumnValueFilterStartTime, singleColumnValueFilterEndTime));
return filterList;
}
public static void main(String[] args) throws IOException {
HBaseClusterConnector flt = new HBaseClusterConnector();
Set<String> imsis= flt.getInfoPerBTSID("AMCD000784", "26082013","104092","104095");
System.out.println(imsis.toString());
}
}
I'm currently using Cloudera quick start VM to test this.
The problem is; if i run this very code on VM it works absolutely fine. But it fails with below error if it is run from outside. And I'm suspecting it has something to do with the VM setting rather than anything else. Please note that I've already checked if I can connect to the node manager / job tracker of the VM from host machine and it works absolutely fine. When I run the code from my host OS instead of running it on VM; I get the below error :
2013-10-15 18:16:04.185 java[652:1903] Unable to load realm info from SCDynamicStore
Exception in thread "main" org.apache.hadoop.hbase.MasterNotRunningException: Retried 1 times
at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:138)
at org.apache.hadoop.hbase.client.HBaseAdmin.checkHBaseAvailable(HBaseAdmin.java:1774)
at ttumdt.app.connector.HBaseClusterConnector.<init>(HBaseClusterConnector.java:47)
at ttumdt.app.connector.HBaseClusterConnector.main(HBaseClusterConnector.java:117)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
Process finished with exit code 1
Please note that; the master node is actually running. The zookeper log shows that it has established connection with the host OS :
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6:16:03.274 PM INFO org.apache.zookeeper.server.ZooKeeperServer
Client attempting to establish new session at /10.138.169.81:50567
6:16:03.314 PM INFO org.apache.zookeeper.server.ZooKeeperServer
Established session 0x141bc2487440004 with negotiated timeout 60000 for client /10.138.169.81:50567
6:16:03.964 PM INFO org.apache.zookeeper.server.PrepRequestProcessor
Processed session termination for sessionid: 0x141bc2487440004
6:16:03.996 PM INFO org.apache.zookeeper.server.NIOServerCnxn
Closed socket connection for client /10.138.169.81:50567 which had sessionid 0x141bc2487440004
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
But I see no trace of any activity in Master or RegionServer log.
Please note that my host OS is Mac OSX 10.7.5
As per the resource available; this should work fine; though some suggest the simple HBase java client never works. I'm confused; and eagerly waiting for pointers !!! Please reply
start your hiveserver2 on different port and then try connecting
command to connect hiveserver2 on different port (make sure hive is in path ) :
hive --service hiveserver2 --hiveconf hive.server2.thrift.port=13000
The HBase Java client certainly does work!
The most likely explanation is that your client can't see the machine that the master is running on for some reason.
One possible explanation is that, although you are connecting to Zookeeper using an IP address, the HBase client is attempting to connect to the master using its hostname.
So, if you ensure that you have entries in your hosts file (on the client) that match the hostname of the machine running the master, this may resolve the problem.
Check that you can access the master Web UI at <hostname>:60010 from your client machine.
Related
I am trying to retrive a userCertificate associated with a domain name from Windows Active Directory but having difficulties by using Java API
for example when I use 'ldapsearch' command tool, I am able to retrieve the certificate as you can see below
ldapsearch -h 192.xx.2.xx -D "CN=Administrator,CN=Users,DC=mmo,DC=co,DC=ca" -w Password -b "CN=rsa0,CN=Users,DC=mmo,DC=co,DC=ca" "userCertificate"
# extended LDIF
#
# LDAPv3
# base <CN=rsa0,CN=Users,DC=mmo,DC=co,DC=ca> with scope subtree
# filter: (objectclass=*)
# requesting: userCertificate
#
# rsa0, Users, mmo.co.ca
dn: CN=rsa0,CN=Users,DC=mmo,DC=co,DC=ca
userCertificate:: MIIDbTCCAlWgAwIBAgIEFbvHazANBgkqhkiG9w0BAQsFADBnMQswCQYDVQQG
EwJ1azEQMA4GA1UECBMHVW5rbm93bjEWMBQGA1UEBxMNcmlja21hbnN3b3J0aDERMA8GA1UEChMId
m9jYWxpbmsxDDAKBgNVBAsTA2lwczENMAsGA1UEAxMEcnNhMDAeFw0xOTExMjExNDUwNDNaFw0yOT
ExMTgxNDUwNDNaMGcxCzAJBgNVBAYTAnVrMRAwDgYDVQQIEwdVbmtub3duMRYwFAYDVQQHEw1yaWN
rbWFuc3dvcnRoMREwDwYDVQQKEwh2b2NhbGluazEMMAoGA1UECxMDaXBzMQ0wCwYDVQQDEwRyc2Ew
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA0R0yCr0uU80oFG3Zg0vTbR4NSR2St+w4f
DOmoHQ27z1Q2JwhiNh1XkwC8MtVeGzRJw0pe+jXc2fMVbIqONHImOZuX6p1UMWof7fxMAIEfWq98u
OqVbvbXVLeCE9+BJGsOaiJ70Q76e8tDNTH3vg1orXAvb0O7R0Vz9I0iXjJKzUtmFEBju/m3eoa+WI
6OaBr64hJw7oz1CzPIKj0OcapFypFjr4+QKpRsHA4Nn21XrYSsT00Dk9SVK3NTjHm661crvTR6jSx
j1GrCpVdQGCQ25a2RrHIi0cmclNJmy81PngW0cpdO3p9ZsZ2vPUy5/CNbVwqPEPSlIjJtVa0Xf9O1
QIDAQABoyEwHzAdBgNVHQ4EFgQU1U7VOM/vAHL0pqZgi6TS1f0SAt8wDQYJKoZIhvcNAQELBQADgg
EBAC7fK81BHDbF8PSQO2YznZtnzCMs46TwCezyqIFzQljwYA5wxKIytV7GtV4aEUrfIFIeQIMW812
pMol9xIotULSl1I/2WI18QTIJfRAnkCZZPJIa9MU6nEGCouF1LwW9bzQzHOeI07NgCIyBryojXaxc
L/epJtVxYialdI9mBWB8KDytINrylOcP9sXYaUtkOOiU7h0sBF9XBfzXgtTkF8pB7ObX9YJnyvzTn
y2zVfeZD8Q7BtDL7AvIDcUjoHtYx5B0oD86aCNTSShmtB/ZEyqt8Kynqf+QUYQIWA3wVFjgZjCCwc
NxiXuf6H8KGW8hP+ETKnc7u9XP9GCHINf9K0I=
# search result
search: 2
result: 0 Success
# numResponses: 2
# numEntries: 1
however when I try to use the Java program, I am unable to retrive it, below is the sample java program
package CertStore;
import javax.naming.AuthenticationException;
import javax.naming.AuthenticationNotSupportedException;
import javax.naming.Context;
import javax.naming.NamingException;
import javax.naming.directory.DirContext;
import javax.naming.directory.InitialDirContext;
import javax.security.auth.x500.X500Principal;
import java.security.cert.*;
import java.util.*;
import java.io.*;
class CertStoreTest {
CertStoreTest() {
try {
LDAPCertStoreParameters lcsp =
new LDAPCertStoreParameters("192.xx.2.xx", 389);
String referenceID = "CN=rsa0,CN=Users,DC=bmo,DC=co,DC=ca";
X509CertSelector xcs = new X509CertSelector();
xcs.setSubject(referenceID);
CertStore cs = CertStore.getInstance("LDAP", lcsp);
Collection certificates = cs.getCertificates((CertSelector)xcs);
System.out.println("size: "+ certificates.size());
Iterator certificate = certificates.iterator();
while(certificate.hasNext()) {
System.out.println(certificate.next());
}
} catch(Exception e) {
e.printStackTrace();
}
}
public static void main(String[] args) {
System.out.println("main() called.");
CertStoreTest test = new CertStoreTest();
}
}
When I run this program, I get the size as 0 where I am expecting as 1.
main() called.
size: 0
I also have openldap running on a linux system, and in the above java program if I point to that server and with appropriate domain name information, java is able to pull the certificate associated with that domain name.
Not sure what I am missing when I try to retrive certificate from Windows Active Directory.
Can anyone shed some light on this as I have been stuck for few days now.
I am trying to understand Akka Clustering for parallel computation using nodes. So, I wrote one factorial program and want to run that on a cluster of 3 nodes (inclusive master).
I am using a configuration file to provide seed nodes and cluster provider. And reading file in my code.
cluster {
akka {
actor {
provider = "cluster"
}
remote {
log-remote-lifecycle-events = off
netty.tcp {
hostname = "127.0.0.1"
port = 0
}
}
cluster {
seed-nodes = [
"akka.tcp://ClusterSystem#127.0.0.1:9876",
"akka.tcp://ClusterSystem#127.0.0.1:6789"]
# auto downing is NOT safe for production deployments.
# you may want to use it during development, read more about it in the docs.
#
# auto-down-unreachable-after = 10s
}
}
}
Following is the java code:
package test
import java.io.File
import akka.actor.{Actor,ActorSystem, Props}
import akka.stream.ActorMaterializer
import com.typesafe.config.ConfigFactory
import scala.concurrent.ExecutionContextExecutor
class Factorial extends Actor {
override def receive = {
case (n: Int) => fact(n)
}
def fact(n:Int): Int ={
if (n<=1){
return 1
}
else {
return n * fact(n - 1)
}
}
}
object ClusterActor {
def main(args: Array[String]): Unit = {
val configFile = "E:/Scala/StatsRuleEngine/Resources/local_configuration.conf"
val config = ConfigFactory.parseFile(new File(configFile))
implicit val system:ActorSystem = ActorSystem("ClusterSystem" ,config.getConfig("cluster"))
implicit val materializer:ActorMaterializer = ActorMaterializer()
implicit val executionContext: ExecutionContextExecutor = system.dispatcher
val FacActor = system.actorOf(Props[Factorial],"Factorial")
FacActor ! (5)
}
}
On running the program, I am getting below error
Remote connection to [null] failed with java.net.ConnectException: Connection refused: no further information: /127.0.0.1:6789
[WARN] [01/21/2019 16:31:15.979] [New I/O boss #3] [NettyTransport(akka://ClusterSystem)] Remote connection to [null] failed with java.net.ConnectException: Connection refused: no further information: /127.0.0.1:9876
I tried to search, but I don't why this error is coming.
When you boot your nodes, you need to specify the exact ports that will be open in config
netty.tcp {
hostname = "127.0.0.1"
port = 0 // THE EXACT PORT
}
So, if your seed nodes say 9876 and 6789. Two of nodes have to specify
netty.tcp {
hostname = "127.0.0.1"
port = 9876
}
and
netty.tcp {
hostname = "127.0.0.1"
port = 6789
}
Note, that the node that is listed first in seed nodes list must start first.
enter image description here public class GemfireTest {
public static void main(String[] args) throws NameResolutionException, TypeMismatchException, QueryInvocationTargetException, FunctionDomainException {
ServerLauncher serverLauncher = new ServerLauncher.Builder()
.setMemberName("server1")
.setServerPort(40404)
.set("start-locator", "127.0.0.1[9090]")
.build();
serverLauncher.start();
String queryString = "SELECT * FROM /gemregion";
ClientCache cache = new ClientCacheFactory().create();
QueryService queryService = cache.getQueryService();
Query query = queryService.newQuery(queryString);
SelectResults results = (SelectResults)query.execute();
int size = results.size();
System.out.println(size);
}
}
trying to run a locator and a server inside my java application getting an exception below:
Exception in thread "main" java.lang.IllegalStateException: A
connection to a distributed system already exists in this VM. It has
the following configuration: ack-severe-alert-threshold="0"
ack-wait-threshold="15" archive-disk-space-limit="0"
archive-file-size-limit="0" async-distribution-timeout="0"
async-max-queue-size="8" async-queue-timeout="60000"
bind-address="" cache-xml-file="cache.xml"
cluster-configuration-dir="" cluster-ssl-ciphers="any"
cluster-ssl-enabled="false" cluster-ssl-keystore=""
cluster-ssl-keystore-password="" cluster-ssl-keystore-type=""
cluster-ssl-protocols="any"
cluster-ssl-require-authentication="true" cluster-ssl-truststore=""
cluster-ssl-truststore-password="" conflate-events="server"
conserve-sockets="true" delta-propagation="true"
deploy-working-dir="C:\Users\Saranya\IdeaProjects\Gemfire"
disable-auto-reconnect="false" disable-tcp="false"
distributed-system-id="-1" distributed-transactions="false"
durable-client-id="" durable-client-timeout="300"
enable-cluster-configuration="true"
enable-network-partition-detection="true"
enable-time-statistics="false" enforce-unique-host="false"
gateway-ssl-ciphers="any" gateway-ssl-enabled="false"
gateway-ssl-keystore="" gateway-ssl-keystore-password=""
gateway-ssl-keystore-type="" gateway-ssl-protocols="any"
gateway-ssl-require-authentication="true" gateway-ssl-truststore=""
gateway-ssl-truststore-password="" groups=""
http-service-bind-address="" http-service-port="7070"
http-service-ssl-ciphers="any" http-service-ssl-enabled="false"
http-service-ssl-keystore="" http-service-ssl-keystore-password=""
http-service-ssl-keystore-type="" http-service-ssl-protocols="any"
http-service-ssl-require-authentication="false"
http-service-ssl-truststore=""
http-service-ssl-truststore-password="" jmx-manager="false"
jmx-manager-access-file="" jmx-manager-bind-address=""
jmx-manager-hostname-for-clients="" jmx-manager-http-port="7070"
jmx-manager-password-file="" jmx-manager-port="1099"
jmx-manager-ssl-ciphers="any" jmx-manager-ssl-enabled="false"
jmx-manager-ssl-keystore="" jmx-manager-ssl-keystore-password=""
jmx-manager-ssl-keystore-type="" jmx-manager-ssl-protocols="any"
jmx-manager-ssl-require-authentication="true"
jmx-manager-ssl-truststore="" jmx-manager-ssl-truststore-password=""
jmx-manager-start="false" jmx-manager-update-rate="2000"
load-cluster-configuration-from-dir="false" locator-wait-time="0"
locators="127.0.0.1[9090]" (wanted "") lock-memory="false"
log-disk-space-limit="0"
log-file="C:\Users\Saranya\IdeaProjects\Gemfire\server1.log"
(wanted "") log-file-size-limit="0" log-level="config" max-num-reconnect-tries="3" max-wait-time-reconnect="60000"
mcast-address="/239.192.81.1" mcast-flow-control="1048576, 0.25,
5000" mcast-port="0" mcast-recv-buffer-size="1048576"
mcast-send-buffer-size="65535" mcast-ttl="32"
member-timeout="5000" membership-port-range="[1024,65535]"
memcached-bind-address="" memcached-port="0"
memcached-protocol="ASCII" name="server1" (wanted "")
off-heap-memory-size="" redis-bind-address="" redis-password=""
redis-port="0" redundancy-zone="" remote-locators=""
remove-unresponsive-client="false" roles=""
security-client-accessor="" security-client-accessor-pp=""
security-client-auth-init="" security-client-authenticator=""
security-client-dhalgo="" security-log-file=""
security-log-level="config" security-manager=""
security-peer-auth-init="" security-peer-authenticator=""
security-peer-verifymember-timeout="1000" security-post-processor=""
security-shiro-init="" security-udp-dhalgo=""
serializable-object-filter="!" server-bind-address=""
server-ssl-ciphers="any" server-ssl-enabled="false"
server-ssl-keystore="" server-ssl-keystore-password=""
server-ssl-keystore-type="" server-ssl-protocols="any"
server-ssl-require-authentication="true" server-ssl-truststore=""
server-ssl-truststore-password="" socket-buffer-size="32768"
socket-lease-time="60000" ssl-ciphers="any" ssl-cluster-alias=""
ssl-default-alias="" ssl-enabled-components="[]"
ssl-gateway-alias="" ssl-jmx-alias="" ssl-keystore=""
ssl-keystore-password="" ssl-keystore-type="" ssl-locator-alias=""
ssl-protocols="any" ssl-require-authentication="true"
ssl-server-alias="" ssl-truststore="" ssl-truststore-password=""
ssl-truststore-type="" ssl-web-alias=""
ssl-web-require-authentication="false" start-dev-rest-api="false"
start-locator="127.0.0.1[9090]" (wanted "")*
statistic-archive-file="" statistic-sample-rate="1000"
statistic-sampling-enabled="true" tcp-port="0"
udp-fragment-size="60000" udp-recv-buffer-size="1048576"
udp-send-buffer-size="65535" use-cluster-configuration="true"
user-command-packages="" validate-serializable-objects="false"
at
org.apache.geode.distributed.internal.InternalDistributedSystem.validateSameProperties(InternalDistributedSystem.java:2959)
at
org.apache.geode.distributed.DistributedSystem.connect(DistributedSystem.java:199)
at
org.apache.geode.cache.client.ClientCacheFactory.basicCreate(ClientCacheFactory.java:243)
at
org.apache.geode.cache.client.ClientCacheFactory.create(ClientCacheFactory.java:214)
at GemfireTest.main(GemfireTest.java:61)
How to solve this exception?
The error here is pretty self explanatory: you can’t have more than one connection to a distributed system within a single JVM. In this particular case you’re starting both a server cache (ServerLauncher) and a client cache (ClientCacheFactory) within the same JVM, which is not supported.
To solve the issue, use two different applications or JVMs, one for the server and another one for the client executing the query.
Cheers.
I'm trying to run the following code to get twitter information live:
import org.apache.spark._
import org.apache.spark.streaming._
import org.apache.spark.streaming.twitter._
import org.apache.spark.streaming.StreamingContext._
import twitter4j.auth.Authorization
import twitter4j.Status
import twitter4j.auth.AuthorizationFactory
import twitter4j.conf.ConfigurationBuilder
import org.apache.spark.streaming.api.java.JavaStreamingContext
import org.apache.spark.rdd.RDD
import org.apache.spark.SparkContext
import org.apache.spark.mllib.feature.HashingTF
import org.apache.spark.mllib.linalg.Vector
import org.apache.spark.SparkConf
import org.apache.spark.api.java.JavaSparkContext
import org.apache.spark.api.java.function.Function
import org.apache.spark.streaming.Duration
import org.apache.spark.streaming.api.java.JavaDStream
import org.apache.spark.streaming.api.java.JavaReceiverInputDStream
val consumerKey = "xxx"
val consumerSecret = "xxx"
val accessToken = "xxx"
val accessTokenSecret = "xxx"
val url = "https://stream.twitter.com/1.1/statuses/filter.json"
val sparkConf = new SparkConf().setAppName("Twitter Streaming")
val sc = new SparkContext(sparkConf)
val documents: RDD[Seq[String]] = sc.textFile("").map(_.split(" ").toSeq)
// Twitter Streaming
val ssc = new JavaStreamingContext(sc,Seconds(2))
val conf = new ConfigurationBuilder()
conf.setOAuthAccessToken(accessToken)
conf.setOAuthAccessTokenSecret(accessTokenSecret)
conf.setOAuthConsumerKey(consumerKey)
conf.setOAuthConsumerSecret(consumerSecret)
conf.setStreamBaseURL(url)
conf.setSiteStreamBaseURL(url)
val filter = Array("Twitter", "Hadoop", "Big Data")
val auth = AuthorizationFactory.getInstance(conf.build())
val tweets : JavaReceiverInputDStream[twitter4j.Status] = TwitterUtils.createStream(ssc, auth, filter)
val statuses = tweets.dstream.map(status => status.getText)
statuses.print()
ssc.start()
But when it arrives at this command: val sc = new SparkContext(sparkConf), the following error appears:
17/05/09 09:08:35 WARN SparkContext: Multiple running SparkContexts
detected in the same JVM! org.apache.spark.SparkException: Only one
SparkContext may be running in this JVM (see SPARK-2243). To ignore
this error, set spark.driver.allowMultipleContexts = true.
I have tried to add the following parameters to the sparkConf value, but the error still appears:
val sparkConf = new SparkConf().setAppName("Twitter Streaming").setMaster("local[4]").set("spark.driver.allowMultipleContexts", "true")
If I ignore the error and continue running commands I get this other error:
17/05/09 09:15:44 WARN ReceiverSupervisorImpl: Restarting receiver
with delay 2000 ms: Error receiving tweets 401:Authentication
credentials (https://dev.twitter.com/pages/auth) were missing or
incorrect. Ensure that you have set valid consumer key/secret, access
token/secret, and the system clock is in sync. \n\n\nError 401 Unauthorized
HTTP ERROR: 401 Problem accessing
'/1.1/statuses/filter.json'. Reason:Unauthorized
Any kind of contribution is appreciated. A greeting and have a good day.
A Spark-shell already prepares a spark-session or spark-context for you to use - so you don't have to / can't initialize a new one. Usually you will have a line telling you under what variable it is available to you a the end of the spark-shell launch process.
allowMultipleContexts exists only for testing some functionalities of Spark, and shouldn't be used in most cases.
I was trying to run a simple jdbc code [ using jdk 1.6, oracle 10g] as,
package javaapplication2;
import java.text.*;
import java.sql.*;
import java.io.FileWriter;
import java.io.PrintWriter;
import java.io.BufferedWriter;
/**
* #author animark
*/
public class CallableStatementEx1 {
public CallableStatementEx1(){;}
public static void main(String s[]) throws Exception {
try
{
Class.forName("oracle.jdbc.driver.OracleDriver").newInstance();
Connection con=null;
String url= "jdbc:oracle:thin:#//localhost:1521:orcl" ;
con = DriverManager.getConnection(url,"scott","password");
String query="update emp set HIREDATE=?,ENAME=? where empno=?";
//Step1: Get PreparedStatement
PreparedStatement ps=con.prepareStatement(query);
//Prepare java.sql.Date object
/*
This logic shows how to convert simple String that is in
dd-MM-yyyy format into Date object
*/
SimpleDateFormat sdf=new SimpleDateFormat("dd-MM-yyyy");
java.util.Date d=sdf.parse("26-12-2001");
java.sql.Date newdate=new java.sql.Date(d.getTime());
//Step2: set parameters
ps.setDate(1,newdate);
ps.setString(2,"animark");
ps.setInt(3,7839);
//Step3: execute the query
int i=ps.executeUpdate();
System.out.println("record updated count: "+i);
con.close();
}
catch(Exception e)
{
e.printStackTrace();
}
}//main
}//class
The code is getting compiled properly. But when I'm trying to run it, i'm getting the following exception..
java.sql.SQLException: Io exception: The Network Adapter could not establish the connection
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:112)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:146)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:255)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:387)
at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:414)
at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:165)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:35)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:801)
at java.sql.DriverManager.getConnection(Unknown Source)
at java.sql.DriverManager.getConnection(Unknown Source)
at javaapplication2.CallableStatementEx1.main(CallableStatementEx1.java:19)
I've checked the oracle services and all of them are up and running.
Also, please find the contents of other files as
=================================================================================
tnsnames.ora:
# tnsnames.ora Network Configuration File: C:\oraclexe\app\oracle\product\10.2.0\server\BIN\network\admin\tnsnames.ora
# Generated by Oracle configuration tools.
ORCL =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = localhost)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = orcl)
)
)
EXTPROC_CONNECTION_DATA =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1))
)
(CONNECT_DATA =
(SID = PLSExtProc)
(PRESENTATION = RO)
)
)
====================================================================================
sqlnet.ora
# sqlnet.ora Network Configuration File: C:\oraclexe\app\oracle\product\10.2.0\server\BIN\network\admin\sqlnet.ora
# Generated by Oracle configuration tools.
# This file is actually generated by netca. But if customers choose to
# install "Software Only", this file wont exist and without the native
# authentication, they will not be able to connect to the database on NT.
SQLNET.AUTHENTICATION_SERVICES= (NTS)
NAMES.DIRECTORY_PATH= (TNSNAMES, EZCONNECT)
================================================================================
# listener.ora Network Configuration File: C:\oraclexe\app\oracle\product\10.2.0\server\BIN\network\admin\listener.ora
# Generated by Oracle configuration tools.
SID_LIST_LISTENER =
(SID_LIST =
(SID_DESC =
(SID_NAME = PLSExtProc)
(ORACLE_HOME = C:\oraclexe\app\oracle\product\10.2.0\server\BIN)
(PROGRAM = extproc)
)
)
LISTENER =
(DESCRIPTION_LIST =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1))
(ADDRESS = (PROTOCOL = TCP)(HOST = localhost)(PORT = 1521))
)
)
===============================================================================
I'm being able to connect to the instance "orcl" using credentials "scott/password", but when i'm trying to connect using statement
SQL> connect sys/password#orcl as sysdba
I'm getting the following error..
ORA-12514: TNS:listener does not currently know of service requested in connect
descriptor.
I've also run LSNRCTL for orcl and found
LSNRCTL for 32-bit Windows: Version 10.2.0.1.0 - Production on 22-JUL-2012 13:42:30
Copyright (c) 1991, 2005, Oracle. All rights reserved.
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=localhost)(PORT=1521))(CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=orcl)))
Services Summary...
Service "PLSExtProc" has 1 instance(s).
Instance "PLSExtProc", status UNKNOWN, has 1 handler(s) for this service...
Handler(s):
"DEDICATED" established:0 refused:0
LOCAL SERVER
The command completed successfully
Please help me if I'm doing anything wrong here.
Try taking the // out of the connection URL. Instead of
String url= "jdbc:oracle:thin:#//localhost:1521:orcl" ;
try
String url= "jdbc:oracle:thin:#localhost:1521:orcl" ;