i am a new bee to GCP and trying to develop springboot rest api that connects to GCP Bigtable, do we have a quickstart guide that helps in development.
Yup, there is. Check out this link to get started. You need the Java client library.
https://cloud.google.com/bigtable/docs/reference/libraries#client-libraries-install-java
for Maven:
<dependencyManagement>
<dependencies>
<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>libraries-bom</artifactId>
<version>26.1.4</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
<dependencies>
<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>google-cloud-bigtable</artifactId>
</dependency>
Then check this for usage:
https://cloud.google.com/bigtable/docs/reference/libraries#client-libraries-usage-java
import com.google.api.gax.rpc.NotFoundException;
import com.google.cloud.bigtable.data.v2.BigtableDataClient;
import com.google.cloud.bigtable.data.v2.BigtableDataSettings;
import com.google.cloud.bigtable.data.v2.models.Row;
import com.google.cloud.bigtable.data.v2.models.RowCell;
public class Quickstart {
public static void main(String... args) {
String projectId = args[0]; // my-gcp-project-id
String instanceId = args[1]; // my-bigtable-instance-id
String tableId = args[2]; // my-bigtable-table-id
quickstart(projectId, instanceId, tableId);
}
public static void quickstart(String projectId, String instanceId, String tableId) {
BigtableDataSettings settings =
BigtableDataSettings.newBuilder().setProjectId(projectId).setInstanceId(instanceId).build();
// Initialize client that will be used to send requests. This client only needs to be created
// once, and can be reused for multiple requests. After completing all of your requests, call
// the "close" method on the client to safely clean up any remaining background resources.
try (BigtableDataClient dataClient = BigtableDataClient.create(settings)) {
System.out.println("\nReading a single row by row key");
Row row = dataClient.readRow(tableId, "r1");
System.out.println("Row: " + row.getKey().toStringUtf8());
for (RowCell cell : row.getCells()) {
System.out.printf(
"Family: %s Qualifier: %s Value: %s%n",
cell.getFamily(), cell.getQualifier().toStringUtf8(), cell.getValue().toStringUtf8());
}
} catch (NotFoundException e) {
System.err.println("Failed to read from a non-existent table: " + e.getMessage());
} catch (Exception e) {
System.out.println("Error during quickstart: \n" + e.toString());
}
}
}
Related
I am using spring.cloud to connect to Azure Service Bus in Java. Here is maven dependency I am using:
<dependency>
<groupId>com.azure.spring</groupId>
<artifactId>spring-cloud-azure-starter-integration-servicebus</artifactId>
<version>4.5.0</version>
</dependency>
I am able to consume the message from the queue as byte array and it converts the message to string. Here is my main code after receiving a message from queue:
#ServiceActivator(inputChannel = INPUT_CHANNEL)
public void messageReceiver(byte[] payload, #Header(AzureHeaders.CHECKPOINTER) Checkpointer checkpointer) {
String message = new String(payload);
LOGGER.info("New message received: '{}'", message);
checkpointer.success()
.doOnSuccess(s -> LOGGER.info("Message '{}' successfully checkpointed", message))
.doOnError(e -> LOGGER.error("Error found", e))
.block();
}
And here is my example data in JSON as short version:
{
"serverId": 123,
"message": "{some message}"
}
What I would like to do is to create a Java object like this:
public class ExampleMessage {
private final Integer serverId;
private final String message;
and when a message from queue is consumed, it will convert the message to my Java object. I am used to using DataTypeProvider in order to use custom Java object for AMQP message consumption which will convert and validate the conversion behind the scene. Does spring.cloud.azure has built-in method/functionality for deserialization? or Do I manually deserialize and do validation for a consumed message?
Here I was able to convert the Json object to java object using Gson class.
I am just reading a message from the service bus and converting it to java object.
my pom.xml (dependencies)
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId>
<version>2.8.6</version>
</dependency>
<dependency>
<groupId>com.azure</groupId>
<artifactId>azure-messaging-servicebus</artifactId>
<version>7.0.0</version>
</dependency>
<!-- https://mvnrepository.com/artifact/io.projectreactor/reactor-core -->
<dependency>
<groupId>io.projectreactor</groupId>
<artifactId>reactor-core</artifactId>
<version>3.3.11.RELEASE</version>
</dependency>
My test class :
public class TestClass {
public String name ;
public int version;
TestClass(String n , int v)
{
this.name = n ;
this.version = v ;
}
}
The main class :
#SpringBootApplication
public class ServicebustestApplication {
public static void main(String[] args) throws InterruptedException {
SpringApplication.run(ServicebustestApplication.class, args);
String conn = " Endpoint=sb://v-mganorkarjsonobject.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=HglLVGlgMsYZGQMOtUfp4g2oka1CpCbVR0YEHgly7jA= ";
CountDownLatch countdownLatch1 = new CountDownLatch(1);
ServiceBusProcessorClient processorClient = new ServiceBusClientBuilder()
.connectionString("<Your COnnection String >")
.processor()
.queueName("test")
.processMessage(ServicebustestApplication::processMessage)
.processError(context -> processError(context,countdownLatch1))
.buildProcessorClient();
processorClient.start();
TimeUnit.SECONDS.sleep(10);
processorClient.close();
}
private static void processMessage(ServiceBusReceivedMessageContext context) {
ServiceBusReceivedMessage message = context.getMessage();
System.out.printf("Processing message. Session: %s, Sequence #: %s. Contents: %s%n", message.getMessageId(),
message.getSequenceNumber(), message.getBody());
Gson gson = new Gson();
TestClass testobject = gson.fromJson(String.valueOf(message.getBody()),TestClass.class);
System.out.println("name: "+testobject.name +" version: "+ testobject.version+"");
}
private static void processError(ServiceBusErrorContext context, CountDownLatch countdownLatch) {
}
}
Here the callback to process message will process the message and then we can use the GSON to convert the json string to java object.
Gson gson = new Gson();
TestClass testobject = gson.fromJson(String.valueOf(message.getBody()),TestClass.class);
output of the code :
When trying to connect to the secret manager my code is throwing this exception. I am trying to Create a Secrets Manager client.
AWSSecretsManager client =
AWSSecretsManagerClientBuilder.standard()
.withRegion(region)
.build();
In Pom.xml have added the following dependencies.
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-secretsmanager</artifactId>
<version>1.11.965</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk</artifactId>
<version>1.11.965</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-core</artifactId>
<version>1.11.965</version>
</dependency>
The solution provided by #smac2020 was not helpful for me, as I was not dealing with secrets, but AWS Cognito.
And this is what I found out - might be helpful for someone in the future
I got this error while I updated to the latest version of cognitoip 1.12.167
implementation group: 'io.awspring.cloud', name: 'spring-cloud-starter-aws-parameter-store-config', version: 1.12.167
This automatically downloads/is bundled with an OLD VERSION of aws-java-sdk-core and jmespath-java dependenies.
The EnhancedJsonErrorUnmarshaller is a new class with the latest version of aws-java-sdk-core, which was not in the default version that cognitoip was bundled with.
Solution: Update the aws-java-sdk-core manually to match the version of cognitoip.
implementation group: 'com.amazonaws', name: 'aws-java-sdk-core', version: '1.12.167'
This has the EnhancedJsonErrorUnmarshaller in it.
Amazon suggests moving to the AWS SDK for Java V2. You can find Secret Manager V2 code in Github here.
The POM file that contains the dependencies is located in Github in the SecretManager folder.
V2 code has been tested many times and this code works:
package com.example.secrets;
//snippet-start:[secretsmanager.java2.create_secret.import]
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.secretsmanager.SecretsManagerClient;
import software.amazon.awssdk.services.secretsmanager.model.CreateSecretRequest;
import software.amazon.awssdk.services.secretsmanager.model.CreateSecretResponse;
import software.amazon.awssdk.services.secretsmanager.model.SecretsManagerException;
//snippet-end:[secretsmanager.java2.create_secret.import]
/**
* To run this AWS code example, ensure that you have setup your development environment, including your AWS credentials.
*
* For information, see this documentation topic:
*
*https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/get-started.html
*/
public class CreateSecret {
public static void main(String[] args) {
final String USAGE = "\n" +
"Usage:\n" +
" CreateSecret <secretName> <secretValue> \n\n" +
"Where:\n" +
" secretName - the name of the secret (for example, tutorials/MyFirstSecret). \n"+
" secretValue - the secret value. \n";
if (args.length != 2) {
System.out.println(USAGE);
System.exit(1);
}
String secretName = args[0];
String secretValue= args[1];
Region region = Region.US_EAST_1;
SecretsManagerClient secretsClient = SecretsManagerClient.builder()
.region(region)
.build();
String secretARN = createNewSecret(secretsClient, secretName, secretValue);
System.out.println("The secret ARN is "+ secretARN);
secretsClient.close();
}
//snippet-start:[secretsmanager.java2.create_secret.main]
public static String createNewSecret( SecretsManagerClient secretsClient, String secretName, String secretValue) {
try {
CreateSecretRequest secretRequest = CreateSecretRequest.builder()
.name(secretName)
.description("This secret was created by the AWS Secret Manager Java API")
.secretString(secretValue)
.build();
CreateSecretResponse secretResponse = secretsClient.createSecret(secretRequest);
return secretResponse.arn();
} catch (SecretsManagerException e) {
System.err.println(e.awsErrorDetails().errorMessage());
System.exit(1);
}
return "";
}
//snippet-end:[secretsmanager.java2.create_secret.main]
}
I have already successfully deployed my app on Heroku, but my app is crashing while running.
I`m getting am error:
Error R10 (Boot timeout) -> Web process failed to bind to $PORT within
90 seconds of launch
I found in internet this code, which pasted in main class - no result:
public static String PORT = System.getenv("PORT");
public static String SERVER_URL = System.getenv("SERVER_URL");
Procfile:
web: java $JAVA_OPTS -Dserver.port=$PORT -cp
target/classes:target/dependency/* Bot
Pom.xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.example</groupId>
<version>1.0-SNAPSHOT</version>
<artifactId>tgBot</artifactId>
<dependencies>
<dependency>
<groupId>org.telegram</groupId>
<artifactId>telegrambots</artifactId>
<version>4.1.2</version>
</dependency>
</dependencies>
<properties>
<maven.compiler.source>1.6</maven.compiler.source>
<maven.compiler.target>1.6</maven.compiler.target>
</properties>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<executions>
<execution>
<id>copy-dependencies</id>
<phase>package</phase>
<goals><goal>copy-dependencies</goal></goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
Main class:
import org.telegram.telegrambots.ApiContextInitializer;
import org.telegram.telegrambots.bots.TelegramLongPollingBot;
import org.telegram.telegrambots.meta.TelegramBotsApi;
import org.telegram.telegrambots.meta.api.objects.Message;
import org.telegram.telegrambots.meta.api.objects.Update;
import org.telegram.telegrambots.meta.exceptions.TelegramApiRequestException;
import java.io.IOException;
public class Bot extends TelegramLongPollingBot {
public static String PORT = System.getenv("PORT");
public static String SERVER_URL = System.getenv("SERVER_URL");
public static void main(String[] args) {
ApiContextInitializer.init();
TelegramBotsApi bot = new TelegramBotsApi();
try {
bot.registerBot(new Bot());
} catch (TelegramApiRequestException e) {
e.printStackTrace();
}
}
public void onUpdateReceived(Update update) {
Message message = update.getMessage();
Methods method = new Methods();
Answers answer = new Answers();
Model model = new Model();
if (message != null && message.hasText()) {
if (message.getText() == answer.row1Button) {
method.sendMsg(message, answer.faq);
}
String s = message.getText();
if ("/start".equals(s) || "Справка/помощь по боту".equals(s) || "/help".equals(s)) {
method.sendMsg(message, answer.faq);
} else if ("/api".equals(s)) {
method.sendMsg(message, answer.api);
} else {
try {
method.sendMsg(message, Weather.getWeather(message.getText(), model));
} catch (IOException e) {
method.sendMsg(message, answer.fail);
}
}
}
}
public String getBotUsername() {
return "Weather";
}
public String getBotToken() {
return "my bot token :D";
}
}
This could help you https://github.com/pengrad/telegram-bot-heroku, but it uses other library to work with Telegram Bot API – java-telegram-bot-api
There is a Procfile (need to update main class there) and build.gradle files for deploy.
By default it sets Webhook:
public class Main {
public static void main(String[] args) {
final String portNumber = System.getenv("PORT");
if (portNumber != null) {
port(Integer.parseInt(portNumber));
}
// current app url to set webhook
// should be set via heroku config vars
// https://devcenter.heroku.com/articles/config-vars
// heroku config:set APP_URL=https://app-for-my-bot.herokuapp.com
final String appUrl = System.getenv("APP_URL");
// define list of bots
BotHandler[] bots = new BotHandler[]{
new TestTelegramBot()
};
// set bot to listen https://my-app.heroku.com/BOTTOKEN
// register this URL as Telegram Webhook
for (BotHandler bot : bots) {
String token = bot.getToken();
post("/" + token, bot);
if (appUrl != null) {
bot.getBot().execute(new SetWebhook().url(appUrl + "/" + token));
}
}
}
}
Can easily change to long polling:
bot.setUpdatesListener(updates -> {
for (Update update : updates) {
onUpdateReceived(update);
}
return UpdatesListener.CONFIRMED_UPDATES_ALL;
});
I am trying to add swagger support to javax rest apis and ended up having the following exception. Any insights will be helpful.
Exception in thread "PortForwarder" java.lang.NoSuchMethodError: org.apache.commons.lang3.Validate.inclusiveBetween(JJJ)V
at com.offbynull.portmapper.mappers.upnpigd.externalmessages.ServiceDiscoveryUpnpIgdRequest$ProbeDeviceType.<init>(ServiceDiscoveryUpnpIgdRequest.java:128)
at com.offbynull.portmapper.mappers.upnpigd.externalmessages.ServiceDiscoveryUpnpIgdRequest$ProbeDeviceType.<clinit>(ServiceDiscoveryUpnpIgdRequest.java:104)
at com.offbynull.portmapper.mappers.upnpigd.UpnpIgdPortMapper.identify(UpnpIgdPortMapper.java:202)
at com.offbynull.portmapper.PortMapperFactory.discover(PortMapperFactory.java:58)
at com.swirlds.p2p.portforwarding.portmapper.PortMapperPortForwarder.execute(PortMapperPortForwarder.java:60)
at com.swirlds.p2p.portforwarding.portmapper.PortMapperPortForwarder.run(PortMapperPortForwarder.java:172)
at java.lang.Thread.run(Thread.java:748)
Code to initialise the server
public static void initREST(int port, String[] packages) {
URI baseUri = UriBuilder.fromUri("http://0.0.0.0").port(port).build();
ResourceConfig config = new ResourceConfig()
.register(new CORSFilter())
.register(JacksonFeature.class);
configureSwagger(config);
for (String pkg : packages) {
config.packages(pkg);
}
System.out.println("Attempting to start Grizzly on " + baseUri);
GrizzlyHttpServerFactory.createHttpServer(baseUri, config);
} catch (IOException e1) {
e1.printStackTrace();
}
}
Code to configure swagger:
private static void configureSwagger(ResourceConfig resourceConfig) {
resourceConfig.register(ApiListingResource.class);
resourceConfig.register(SwaggerSerializers.class);
BeanConfig config = new BeanConfig();
config.setVersion("v1");
config.setBasePath("/");
config.setResourcePackage("ipos.hashgraph.rest");
config.setPrettyPrint(true);
config.setScan(true);
}
Maven Dependencies:
<dependency>
<groupId>io.swagger</groupId>
<artifactId>swagger-jaxrs</artifactId>
<version>1.5.0</version>
</dependency>
<dependency>
<groupId>org.glassfish.jersey.containers</groupId>
<artifactId>jersey-container-grizzly2-servlet</artifactId>
</dependency>
I'm running spark job from EMR cluster which connects to Cassandra on EC2
The following are the dependencies which I'm using for my project.
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
<version>1.6.0</version>
</dependency>
<dependency>
<groupId>com.datastax.spark</groupId>
<artifactId>spark-cassandra-connector_2.10</artifactId>
<version>1.5.0-M1</version>
</dependency>
<dependency>
<groupId>com.datastax.cassandra</groupId>
<artifactId>cassandra-driver-core</artifactId>
<version>2.1.6</version>
</dependency>
<dependency>
<groupId>com.datastax.spark</groupId>
<artifactId>spark-cassandra-connector-java_2.10</artifactId>
<version>1.5.0-M3</version>
</dependency>
The issue that Im facing here is if I use the cassandra-driver-core 3.0.0 , I get the following error
java.lang.ExceptionInInitializerError
at mobi.vserv.SparkAutomation.DriverTester.doTest(DriverTester.java:28)
at mobi.vserv.SparkAutomation.DriverTester.main(DriverTester.java:16)
Caused by: java.lang.IllegalStateException: Detected Guava issue #1635 which indicates that a version of Guava less than 16.01 is in use. This introduces codec resolution issues and potentially other incompatibility issues in the driver. Please upgrade to Guava 16.01 or later.
at com.datastax.driver.core.SanityChecks.checkGuava(SanityChecks.java:62)
at com.datastax.driver.core.SanityChecks.check(SanityChecks.java:36)
at com.datastax.driver.core.Cluster.<clinit>(Cluster.java:67)
... 2 more
I have tried including the guaua version 19.0.0 also but still I'm unable to run the job
and when I degrate the cassandra-driver-core 2.1.6 I get the following error.
com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /EMR PUBLIC IP:9042 (com.datastax.driver.core.TransportException: [/EMR PUBLIC IP:9042] Cannot connect))
at com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:223)
at com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:78)
at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1272)
at com.datastax.driver.core.Cluster.init(Cluster.java:158)
at com.datastax.driver.core.Cluster.connect(Cluster.java:248)
Please note that I have tested my code locally and it runs absolutely fine and I have followed the different combinations of dependencies as mentioned here https://github.com/datastax/spark-cassandra-connector
Code :
public class App1 {
private static Logger logger = LoggerFactory.getLogger(App1.class);
static SparkConf conf = new SparkConf().setAppName("SparkAutomation").setMaster("yarn-cluster");
static JavaSparkContext sc = null;
static
{
sc = new JavaSparkContext(conf);
}
public static void main(String[] args) throws Exception {
JavaRDD<String> Data = sc.textFile("S3 PATH TO GZ FILE/*.gz");
JavaRDD<UserSetGet> usgRDD1=Data.map(new ConverLineToUSerProfile());
List<UserSetGet> t3 = usgRDD1.collect();
for(int i =0 ; i <=t3.size();i++){
try{
phpcallone php = new phpcallone();
php.sendRequest(t3.get(i));
}
catch(Exception e){
logger.error("This Has reached ====> " + e);
}
}
}
}
public class phpcallone{
private static Logger logger = LoggerFactory.getLogger(phpcallone.class);
static String pid;
public void sendRequest(UserSetGet usg) throws JSONException, IOException, InterruptedException {
UpdateCassandra uc= new UpdateCassandra();
try {
uc.UpdateCsrd();
}
catch (ClassNotFoundException e) {
e.printStackTrace(); }
}
}
}
public class UpdateCassandra{
public void UpdateCsrd() throws ClassNotFoundException {
Cluster.Builder clusterBuilder = Cluster.builder()
.addContactPoint("PUBLIC IP ").withPort(9042)
.withCredentials("username", "password");
clusterBuilder.getConfiguration().getSocketOptions().setConnectTimeoutMillis(10000);
try {
Session session = clusterBuilder.build().connect("dmp");
session.execute("USE dmp");
System.out.println("Connection established");
} catch (Exception e) {
e.printStackTrace();
}
}
}
Assuming that you are using EMR 4.1+, you can pass in the guava jar into the --jars option for spark submit. Then supply a configuration file to EMR to use user class paths first.
For example, in a file setup.json
[
{
"Classification": "spark-defaults",
"Properties": {
"spark.driver.userClassPathFirst": "true",
"spark.executor.userClassPathFirst": "true"
}
}
]
You would supply the --configurations file://setup.json option into the create-cluster aws cli command.