I need to know how create partitioned table in Hbase from java. Under is the command I use in shell but i need the Java equivalent instructions because I want to create tables in dynamically mode.
create 'DATABASE_NAMEB:TABLE_NAME','FAMILY_NAME', SPLITS => ['1','2','3','4','5','6','7','8','9','0','A','B','C','D','E','F','01','02','03','04','05','06','07','08','09','00','0A','0B','0C','0D','0E','0F' ]
I found the solution in this site
https://www.programmersought.com/article/55225122809/
HBaseAdmin admin = new HBaseAdmin(config);
HTableDescriptor tableDesc = new HTableDescriptor(tablename);
tableDesc.addFamily(new HColumnDescriptor("cf1"));
byte[][] splitKeys = {
Bytes.toBytes("10"),
Bytes.toBytes("20"),
Bytes.toBytes("30")
};
admin.createTable(tableDesc, splitKeys);
admin.close();
Related
I am trying to start a Kafka server form Java
Specifically, how can I translate this line of Scala into a line of Java?
private val server = new KafkaServer(serverConfig, kafkaMetricsReporters = reporters)
I can create the serverConfig easily, but I can't seem to be able to create the kafkaMetricsReporters parameter.
Note: I can create a KafkaServerStartable but I would like to create a normal KafkaServer to avoid the JVM exiting in case of error.
Apache Kafka version 0.11.0.1
The kafkaMetricsReporters parameter is a scala Seq.
You can either:
Create a Java collection and convert it into a Seq:
You need to import scala.collection.JavaConverters:
List<KafkaMetricsReporter> reportersList = new ArrayList<>();
...
Seq<KafkaMetricsReporter> reportersSeq = JavaConverters.asScalaBufferConverter(reportersList).asScala();
Use KafkaMetricsReporter.startReporters() method to create them for you from your configuration:
As KafkaMetricsReporter is a singleton, you need to use the MODULE notation to use it:
Seq<KafkaMetricsReporter> reporters = KafkaMetricsReporter$.MODULE$.startReporters(new VerifiableProperties(props));
Also the KafkaServer constructor has 2 other arguments that are required when calling it from Java:
time can easily be created using new org.apache.kafka.common.utils.SystemTime()
threadNamePrefix is an Option. If you import scala.Option, you'll be able to call Option.apply("prefix")
Putting it all together:
Properties props = new Properties();
props.put(...);
KafkaConfig config = KafkaConfig.fromProps(props);
Seq<KafkaMetricsReporter> reporters = KafkaMetricsReporter$.MODULE$.startReporters(new VerifiableProperties(props));
KafkaServer server = new KafkaServer(config, new SystemTime(), Option.apply("prefix"), reporters);
server.startup();
I have the following code that runs a windows service from inside Java.The code uses JInterop Java library, JInterop is a pure Java COM client for windows COM server. More details of JIntop are available here [http://fishi.devtail.io/weblog/2015/01/21/pure-java-dcom-bridge-j-interop/]
String cmdFile = "service.bat";
results = wbemServices_dispatch.callMethodA(
"Get", new Object[]{ new JIString("Win32_Process"),
new Integer(0), JIVariant.OPTIONAL_PARAM()});
IJIDispatch wbemObjectSet_dispatch = (IJIDispatch)JIObjectFactory.narrowObject(
(results[0]).getObjectAsComObject());
results = wbemObjectSet_dispatch.callMethodA("Create",
new Object[]{ new JIString(targetFilePrefix + cmdFile),
JIVariant.OPTIONAL_PARAM(),
JIVariant.OPTIONAL_PARAM()});
Is it possible to run a powershell file(.ps1) as a service in the same manner as above using the same library, or in some other way.
You can create a batch file which, in-turns, can trigger a powershell script like this:
#echo off
Powershell.exe set-executionpolicy remotesigned -File C:\folder\MyScript.ps1
pause
Save it as "Trigger_ps.bat"
Then you can use the sc command to create a windows service by mentioning this batch file path like this:
SC CREATE PS_Trigger_Service Displayname= "PS_Trigger_Service" binpath= "C:\folder\Trigger_ps.bat" start= auto
This should solve your purpose.
So here is the setup.
Currently I have two Spark Applications initialized. I need to pass data between them (preferably through shared sparkcontext/sqlcontext so I can just query a temp table). I currently use Parquet Files to dataframe transfer, but is it possible any other way?
MasterURL points to the same SparkMaster
Start Spark via Terminal:
/opt/spark/sbin/start-master.sh;
/opt/spark/sbin/start-slave.sh spark://`hostname`:7077
Java App Setup:
JavaSparkContext context = new JavaSparkContext(conf);
//conf = setMaster(MasterURL), 6G memory, and 4 cores.
SQLContext sqlContext = new SQLContext(parentContext.sc());
Then I register an existing frame later on
//existing dataframe to temptable
df.registerTempTable("table");
and
SparkR
sc <- sparkR.init(master='MasterURL', sparkEnvir=list(spark.executor.memory='6G', spark.cores.max='4')
sqlContext <- sparkRSQL.init(sc)
# attempt to get temptable
df <- sql(sqlContext, "SELECT * FROM table"); # throws the error
As far as I know it is not possible given your current configuration. Tables created using registerTempTable are bound to the specific SQLContext which has been used to create corresponding DataFrame. Even if your Java and SparkR applications use the same master their drivers run on separate JVMs and cannot share single SQLContext.
There are tools, like Apache Zeppelin, which take a different approach with a single SQLContext (and SparkContext) which is exposed to individual backends. This way you can register table using for example Scala and read it from Python. There is a fork of Zeppelin which provides some support for SparkR and R. You can check how it starts and interacts R backend.
I am using spark to perform some computations but want it to be submitted from java application.It works proper using when submitted using spark-submit script.Has anyone tried to do this?
Thanks.
Don't forget to add the fat JAR containing your code to the context.
val conf = new SparkConf()
.setMaster(...)
.setAppName(...)
.setJars("/path/to/code.jar")
val sc = new SparkContext(conf)
As long as you have a master and available worker started, you should be able to if you have the following in your java application:
String master = "spark://IP:7077"; //set IP address to that of your master
String appName = "Name of your Application Here";
SparkConf conf = new SparkConf().setAppName(appName).setMaster(master);;
JavaSparkContext sc = new JavaSparkContext(conf);
I was able to run junit tests from within IntelliJ that utilized the JavaSparkContext without having to use the spark-submit script. I am running into issues when performing actions on DataFrames though (not sure if that's related).
I am learning Pig jobs and want to run pig script on a remote cluster through java code using PigServer. Can anybody guide me how to achieve this? Thanks in advance.
Can the above code be used to do a remote call i.e. Pig is installed on cluster1 & call is made from the application server outside the cluster?
You have to use the PigServer class to connect to your cluster, register your Pig queries and get results. You can either choose to run a script by passing your filename on your disk, or you can directly write your Pig script lines and pass it as Java strings.
To pass a Pig script from the filename:
PigServer pig = new PigServer(ExecType.MAPREDUCE);
pig.registerScript("/path/to/test.pig");
To pass your Pig program as Strings:
PigServer pig = new PigServer(ExecType.MAPREDUCE);
pig.registerQuery("A = LOAD 'something' USING PigLoader();");
You can get back the results for example this way:
Iterator<Tuple> i = pig.openIterator("A");
HashMap<Integer, Integer> map = new HashMap<Integer, Integer>();
while (i.hasNext()) {
Integer val = DataType.toInteger(i.next().get(0));
map.put(val, val);
}
Note that you need to have some properties in your classpath, namely fs.default.name and mapred.job.tracker or you can just add them to the PigServer constructor.