I am trying to create an EMR Cluster using Java. I have created the jar file and put that into a lambda function. I'm calling the lambda from AWS Step functions. I created the maven package including the AWS JAVA SDK dependencies and also imported all the packages
import java.io.IOException;
import com.amazonaws.auth.*;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.auth.PropertiesCredentials;
import com.amazonaws.services.elasticmapreduce.*;
import com.amazonaws.services.elasticmapreduce.model.AddJobFlowStepsRequest;
import com.amazonaws.services.elasticmapreduce.model.AddJobFlowStepsResult;
import com.amazonaws.services.elasticmapreduce.model.RunJobFlowRequest;
import com.amazonaws.services.elasticmapreduce.model.*;
import com.amazonaws.services.elasticmapreduce.model.HadoopJarStepConfig;
import com.amazonaws.services.elasticmapreduce.model.StepConfig;
import com.amazonaws.services.elasticmapreduce.util.StepFactory;
import com.amazonaws.AmazonServiceException;
import com.amazonaws.auth.AWSCredentials;
import com.amazonaws.auth.BasicAWSCredentials;
public class CreateCluster {
public static void main(String[] args) {
AWSCredentials credentials = new BasicAWSCredentials("access key", "secret key");
// myApp={[Hadoop]};
AmazonElasticMapReduceClient emr = new AmazonElasticMapReduceClient(credentials);
String COMMAND_RUNNER = "command-runner.jar";
String DEBUGGING_COMMAND = "state-pusher-script";
String DEBUGGING_NAME = "Setup Hadoop Debugging";
StepFactory stepFactory = new StepFactory();
StepConfig enabledebugging = new StepConfig()
.withName(DEBUGGING_NAME)
.withActionOnFailure(ActionOnFailure.TERMINATE_CLUSTER)
.withHadoopJarStep(new HadoopJarStepConfig()
.withJar(COMMAND_RUNNER)
.withArgs(DEBUGGING_COMMAND));
RunJobFlowRequest request = new RunJobFlowRequest()
.withName("REMR")
.withReleaseLabel("emr-5.16.0")
.withSteps(enabledebugging)
// .withApplications(myApp)
.withLogUri("s3n://r.base.ihm/emr-log/")
.withServiceRole("service_role")
.withJobFlowRole("jobflow_role")
.withInstances(new JobFlowInstancesConfig()
.withEc2KeyName("emr")
.withEc2SubnetId("subnet-d1fbb8ee")
.withInstanceCount(3)
.withKeepJobFlowAliveWhenNoSteps(false)
.withMasterInstanceType("m4.large")
.withSlaveInstanceType("m4.large"));
RunJobFlowResult result = emr.runJobFlow(request);
}
}
but still I'm getting the error
java.lang.NoClassDefFoundError
{
"errorMessage": "Error loading class com.ihm.base.spark.CreateCluster: com/amazonaws/auth/AWSCredentials",
"errorType": "java.lang.NoClassDefFoundError"
}
Any ideas on what I am missing here?
Related
import java.util.Collections;
import java.util.Map;
import javax.annotation.Nullable;
import org.jetbrains.annotations.NotNull;
import org.json.JSONObject;
import com.google.actions.api.smarthome.DisconnectRequest;
import com.google.actions.api.smarthome.ExecuteRequest;
import com.google.actions.api.smarthome.ExecuteResponse;
import com.google.actions.api.smarthome.QueryRequest;
import com.google.actions.api.smarthome.QueryResponse;
import com.google.actions.api.smarthome.SmartHomeApp;
import com.google.actions.api.smarthome.SyncRequest;
import com.google.actions.api.smarthome.SyncResponse;
import com.google.actions.api.smarthome.SyncResponse.Payload;
public class GoogleDeviceSync extends SmartHomeApp {
#NotNull
#Override
public SyncResponse onSync(#NotNull SyncRequest syncRequest, #Nullable Map<?, ?> map) {
Payload payload = new Payload();
payload.setAgentUserId("1836.15267389");
payload.setDevices(
new Device[] {
new Device.Builder()
.setId("123")
.setType("action.devices.types.OUTLET")
.addTrait("action.devices.traits.OnOff")
.setName(
Collections.singletonList("My Outlet 1234"),
"Night light",
Collections.singletonList("Wall plug"))
.setWillReportState(true)
.setDeviceInfo("lights-out-inc", "hs1234", "3.2", "11.4")
.setCustomData(
new JSONObject()
.put("fooValue", 74)
.put("barValue", true)
.put("bazValue", "foo"))
.build(),
new Device.Builder()
.setId("456")
.setType("action.devices.types.LIGHT")
.addTrait("action.devices.traits.OnOff")
.addTrait("action.devices.traits.Brightness")
.addTrait("action.devices.traits.ColorTemperature")
.addTrait("action.devices.traits.ColorSpectrum")
.setName(
Collections.singletonList("Lights Out Inc. bulb A19 color hyperglow"),
"Lamp",
Collections.singletonList("Reading lamp"))
.setWillReportState(true)
.setDeviceInfo("Lights Out Inc.", "hg11", "1.2", "5.4")
.setCustomData(
new JSONObject()
.put("fooValue", 12)
.put("barValue", false)
.put("bazValue", "bar"))
.build(),
});
return new SyncResponse(syncRequest.getRequestId(), payload);
}
}
Which jar is payload.setDevices(new Device[]) and Device introduced from?
From the https://developers.google.com/assistant/smarthome/develop/process-intents document, there is only code, and the introduction of jar is not shown. So which jar does'device' need to introduce
Adding dependencies (jars)
The dependency (the jar) that needs to be added in the gradle.build should be:
implementation 'com.google.actions:actions-on-google:1.8.0'
The specific version will depend on your configuration and environment.
You can see a full example gradle.build. In the same folder there are also other configuration examples.
Importing Classes
I also add some information on where the Payload and Device classes are.
Payload is nested inside SyncResponse and Device is nested inside Payload.
Payload is imported with the following import statement:
import com.google.actions.api.smarthome.SyncResponse.Payload
Device can be imported with the following import statement:
import com.google.actions.api.smarthome.SyncResponse.Payload.Device
Creating Objects
To create a new Payload from the Payload in the SyncResponse:
new SyncResponse.Payload()
And you can try to create a Device from its builder:
SyncResponse.Payload.Device.Builder deviceBuilder = new syncResponse.Payload.Device.Builder()
See this example for usage of the 'Payload' and 'Device'.
I'm interested in Tekton these days.
However there are some issue when I implement Task with java fabric8.tekton apis.
There exist api which is adding steps in spec in units of container(withContainer) in TaskBuilder class.
However I got error message in rune time like below,
Can I get some advices?
Tekton version - v0.10.1
I used packages like below:
io.fabric8:kubernetes-client:4.7.1
io.fabric8:tekton-client:4.7.1
Here is my complete test code.
package com.example.tekton;
import java.util.ArrayList;
import java.util.List;
import io.fabric8.kubernetes.api.model.Container;
import io.fabric8.kubernetes.api.model.ContainerBuilder;
import io.fabric8.kubernetes.client.BaseClient;
import io.fabric8.kubernetes.client.Config;
import io.fabric8.kubernetes.client.ConfigBuilder;
import io.fabric8.tekton.client.TektonClient;
import io.fabric8.tekton.client.DefaultTektonClient;
import io.fabric8.tekton.client.handlers.TaskHandler;
import io.fabric8.tekton.client.handlers.TaskRunHandler;
import io.fabric8.tekton.pipeline.v1alpha1.ArrayOrString;
import io.fabric8.tekton.pipeline.v1alpha1.Task;
import io.fabric8.tekton.pipeline.v1alpha1.TaskBuilder;
import io.fabric8.tekton.pipeline.v1alpha1.TaskRun;
import io.fabric8.tekton.pipeline.v1alpha1.TaskRunBuilder;
import io.fabric8.tekton.pipeline.v1alpha1.TaskRefBuilder;
public class DefaultKubernetesTest {
public Task getTask() {
Container con = new ContainerBuilder()
.withNewImage("ubuntu")
.withNewName("echo-hello-world")
.addNewCommand("echo")
.addNewArg("hello jinwon world")
.build();
Task task = new TaskBuilder()
.withApiVersion("tekton.dev/v1alpha1")
.withKind("Task")
.withNewMetadata()
.withName("echo-hello-world-test")
.endMetadata()
.withNewSpec()
.addNewStep()
.withContainer(con)
.endStep()
.endSpec()
.build();
return task;
}
public TaskRun getTaskRun() {
TaskRun taskRun = new TaskRunBuilder()
.withNewMetadata()
.withName("taskrun")
.endMetadata()
.withNewSpec()
.withTaskRef(new TaskRefBuilder().withName("echo-hello-world-test").withApiVersion("tekton.dev/v1alpha1").withKind("Task").build())
.endSpec().build();
return taskRun;
}
public static void main(String[] args) {
ConfigBuilder config = new ConfigBuilder();
DefaultKubernetesTest kubeTest = new DefaultKubernetesTest();
String username = "testUser";
String password = "testPwd";
config = config.withMasterUrl("https://192.168.6.236:6443");
config = config.withUsername(username);
config = config.withPassword(password);
Config kubeConfig = config.build();
try (DefaultTektonClient test = new DefaultTektonClient(kubeConfig)) {
Task task = kubeTest.getTask();
TaskRun taskRun = kubeTest.getTaskRun();
test.tasks().inNamespace("test").create(task);
test.taskRuns().inNamespace("test").create(taskRun);
test.close();
}
}
}
Tekton ships with an admission controller, which validates the CRD specs before allowing them into the cluster. Because the project is still in alpha, its moving quite fast. Fabric8 may be templating out K8s objects against a different spec from what has been installed on your cluster. You should be able to validate the spec version used in Fabric8 and remove all the Tekton objects in your cluster and re-apply them at a specific version.
I am writing code to create an Amazon Web Services SNS client in Eclipse, when I get an error saying
The method withRegion(Region) from the type
AwsClientBuilder is not visible
Here is my code
package com.amazonaws.samples;
import java.util.Date;
import com.amazonaws.auth.AWSCredentials;
import com.amazonaws.auth.AWSStaticCredentialsProvider;
import com.amazonaws.auth.AnonymousAWSCredentials;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.client.builder.AwsClientBuilder;
import com.amazonaws.regions.Region;
import com.amazonaws.regions.Regions;
import com.amazonaws.services.sns.AmazonSNS;
import com.amazonaws.services.sns.AmazonSNSClient;
import com.amazonaws.services.sns.AmazonSNSClientBuilder;
import com.amazonaws.services.sns.model.CreateTopicRequest;
import com.amazonaws.services.sns.model.CreateTopicResult;
import com.amazonaws.services.sns.model.PublishRequest;
// Example SNS Sender
public class Main {
// AWS credentials -- replace with your credentials
static String ACCESS_KEY = "<Your AWS Access Key>";
static String SECRET_KEY = "<Your AWS Secret Key>";
// Sender loop
public static void main(String... args) throws Exception {
// Create a client
AWSCredentials awsCred = new AnonymousAWSCredentials();
AWSStaticCredentialsProvider cred = new AWSStaticCredentialsProvider(awsCred);
Region region = Region.getRegion(Regions.US_EAST_1);
AmazonSNS service = AmazonSNSClientBuilder.standard().withRegion(region).withCredentials(cred).build(); // Error message: The method withRegion(Region) from the type AwsClientBuilder<AmazonSNSClientBuilder,AmazonSNS> is not visible
// Create a topic
CreateTopicRequest createReq = new CreateTopicRequest()
.withName("MyTopic3");
CreateTopicResult createRes = service.createTopic(createReq);
for (;;) {
// Publish to a topic
PublishRequest publishReq = new PublishRequest()
.withTopicArn(createRes.getTopicArn())
.withMessage("Example notification sent at " + new Date());
service.publish(publishReq);
Thread.sleep(1000);
}
}
}
In the screenshot it shows where the error occurs with the red underline in dotted line:
What should I check to correct this?
You are passing the wrong parameter, withRegion takes either a String or a Regions (note, not Region, singular).
Try passing Regions.EU_WEST_1.
Both AmazonSNSClientBuilder.standard().withRegion(Regions.EU_WEST_1).build();
and AmazonSNSClientBuilder.standard().withRegion("eu-west-1").build();
are working fine for me.
I'm writing a simple Java client code to add values in HBase table. I'm using put.add(byte[] columnFamily, byte[] columnQualifier, byte[] value), but this method is deprecated in new HBase API. Can anyone please help in what is the way of doing it using new Put API?
Using maven I have downloaded jar for HBase version 1.2.0.
I'm using the following code :
package com.NoSQL;
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.Cell;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.KeyValue;
import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.client.Admin;
import org.apache.hadoop.hbase.client.Connection;
import org.apache.hadoop.hbase.client.ConnectionFactory;
import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.client.Table;
import org.apache.hadoop.hbase.util.Bytes;
import org.apache.log4j.Level;
import org.apache.log4j.Logger;
public class PopulatingData {
public static void main(String[] args) throws IOException{
String table = "Employee";
Logger.getRootLogger().setLevel(Level.WARN);
Configuration conf = HBaseConfiguration.create();
Connection con = ConnectionFactory.createConnection(conf);
Admin admin = con.getAdmin();
if(admin.tableExists(TableName.valueOf(table))) {
Table htable = con.getTable(TableName.valueOf(table));
/*********** adding a new row ***********/
// adding a row key
Put p = new Put(Bytes.toBytes("row1"));
p.add(Bytes.toBytes("ContactDetails"), Bytes.toBytes("Mobile"), Bytes.toBytes("9876543210"));
p.add(Bytes.toBytes("ContactDetails"), Bytes.toBytes("Email"), Bytes.toBytes("abhc#gmail.com"));
p.add(Bytes.toBytes("Personal"), Bytes.toBytes("Name"), Bytes.toBytes("Abhinav Rawat"));
p.add(Bytes.toBytes("Personal"), Bytes.toBytes("Age"), Bytes.toBytes("21"));
p.add(Bytes.toBytes("Personal"), Bytes.toBytes("Gender"), Bytes.toBytes("M"));
p.add(Bytes.toBytes("Employement"), Bytes.toBytes("Company"), Bytes.toBytes("UpGrad"));
p.add(Bytes.toBytes("Employement"), Bytes.toBytes("DOJ"), Bytes.toBytes("11:06:2018"));
p.add(Bytes.toBytes("Employement"), Bytes.toBytes("Designation"), Bytes.toBytes("ContentStrategist"));
htable.put(p);
/**********************/
System.out.print("Table is Populated");`enter code here`
}else {
System.out.println("The HBase Table named "+table+" doesn't exists.");
}
System.out.println("Returnning Main");
}
}
Use addColumn() method :
Put put = new Put(Bytes.toBytes(rowKey));
put.addColumn(NAME_FAMILY, NAME_COL_QUALIFIER, name);
Please refer more details in below javadoc :
https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Put.html
I am trying to make a java app which makes connection with facebook.I am using facebok4j to achieve this,I made an app in fb devolopers and got the key annd id for it.But when i am passing it to get an access token its returning an exception error.Please help me
java code
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileWriter;
import java.io.IOException;
import net.sf.json.JSONArray;
import net.sf.json.JSONObject;
import facebook4j.Facebook;
import facebook4j.FacebookException;
import facebook4j.FacebookFactory;
import facebook4j.Post;
import facebook4j.ResponseList;
import facebook4j.auth.AccessToken;
import facebook4j.auth.OAuthAuthorization;
import facebook4j.auth.OAuthSupport;
import facebook4j.conf.Configuration;
import facebook4j.conf.ConfigurationBuilder;
public class Fbsample {
public static Configuration createConfiguration()
{
ConfigurationBuilder confBuilder = new ConfigurationBuilder();
confBuilder.setDebugEnabled(true);
confBuilder.setOAuthAppId("*****");
confBuilder.setOAuthAppSecret("*****");
confBuilder.setUseSSL(true);
confBuilder.setJSONStoreEnabled(true);
Configuration configuration = confBuilder.build();
return configuration;
}
public static void main(String[] argv) throws FacebookException {
Configuration configuration = createConfiguration();
FacebookFactory facebookFactory = new FacebookFactory(configuration );
Facebook facebookClient = facebookFactory.getInstance();
AccessToken accessToken = null;
try{
OAuthSupport oAuthSupport = new OAuthAuthorization(configuration );
accessToken = oAuthSupport.getOAuthAppAccessToken();
}catch (FacebookException e) {
System.err.println("Error while creating access token " + e.getLocalizedMessage());
}
facebookClient.setOAuthAccessToken(accessToken);
//results in an error says {An active access token must be used to query information about the current user}
}
}
For now i specified my token and id as *.When running its returning 'Error while creating access token graph.facebook.com'.Thanks in advance.