Following google's Getting Started I use following code to get the list of all files in a remote directory
class GCSFileStorage {
String bucket = "bucket_name";
String remoteDirectoryPath = "remote/path";
int fetchBlockSize = 1024 * 1024;
GcsService gcsService =
GcsServiceFactory.createGcsService(RetryParams.getDefaultInstance());
List<String> list() {
List<String> filenames = new List();
ListResult listResult = gcsService.list(bucket, ListOptions.DEFAULT);
while (listResult.hasNext()) {
ListItem listItem = listResult.next();
filenames += listItem.getName();
}
return filenames;
}
}
GCSFileStorage gcs = new GCSFileStorage();
gcs.list();
But this code fails with an exception:
java.io.IOException: com.google.appengine.tools.cloudstorage.RetriesExhaustedException:
...
Caused by: java.io.IOException: java.lang.NullPointerException
...
Caused by: java.lang.NullPointerException
at com.google.appengine.tools.cloudstorage.dev.LocalRawGcsService$BlobStorageAdapter.<init>(LocalRawGcsService.java:123)
at com.google.appengine.tools.cloudstorage.dev.LocalRawGcsService$BlobStorageAdapter.getInstance(LocalRawGcsService.java:184)
I suspect that I somehow should authorize in gcs and this may be the reason of failure. However I haven't found proper way to init everything that gcs needs for work.
As #ozarov mentioned the client I was using is specific for App Engine. It was added through dependency
com.google.appengine.tools:appengine-gcs-client:0.5
Instead REST API client should be used. Its dependency is
com.google.apis:google-api-services-storage:v1-rev44-1.20.0
Then the code to fetch files list may look as follows
import com.google.api.client.googleapis.auth.oauth2.GoogleCredential;
import com.google.api.client.http.javanet.NetHttpTransport;
import com.google.api.client.json.jackson2.JacksonFactory;
import com.google.api.services.storage.Storage;
import com.google.api.services.storage.StorageScopes;
import com.google.api.services.storage.model.Objects;
import com.google.api.services.storage.model.StorageObject;
import com.google.common.collect.Lists;
import java.io.File;
import java.io.IOException;
import java.security.GeneralSecurityException;
import java.util.LinkedList;
import java.util.List;
class GCSFileStorage {
String bucket = "bucket_name";
String remoteDirectoryPath = "remote/path";
Storage storage
public GCSFileStorage() throws GeneralSecurityException, IOException {
storage = setupStorage();
}
List<String> list() throws IOException {
List<String> allItems = new LinkedList<String>();
Objects response = storage.objects().list(bucket).
setPrefix(remoteDirectoryPath).execute();
for (StorageObject obj: response.getItems()) {
allItems.add(obj.getName());
}
while (response.getNextPageToken() != null) {
String pageToken = response.getNextPageToken();
response = storage.objects().list(bucket).
setPrefix(remoteDirectoryPath).setPageToken(pageToken).execute();
for (StorageObject obj: response.getItems()) {
allItems.add(obj.getName());
}
}
return allItems;
}
Storage setupStorage() throws GeneralSecurityException, IOException {
GoogleCredential credential = new GoogleCredential.Builder().
setTransport(new NetHttpTransport()).
setJsonFactory(new JacksonFactory()).
setServiceAccountId("your_account_id").
setServiceAccountScopes(
Lists.newArrayList(StorageScopes.DEVSTORAGE_FULL_CONTROL)).
setServiceAccountPrivateKeyFromP12File(
new File("/local/path/to/private/key.p12")).
build();
return new Storage.
Builder(new NetHttpTransport(),
new JacksonFactory(), credential).
setApplicationName("foo").build();
}
}
How do you run this code? This GCS client is specific for App Engine and should run by either a deployed app or locally using the AE dev appserver or unit-tests (which should configure the AE runtime environment using LocalServiceTestHelper).
Related
Hi I am trying to connect to Big Query and I am using a google service account with a JSON key. I am getting the below error. This is in my java batch program.
Insert operation not performed
com.google.cloud.bigquery.BigQueryException: Unexpected error refreshing access token
This fixed my issue. more here https://cloud.google.com/bigquery/docs/authorization
import com.google.auth.Credentials;
import com.google.auth.oauth2.ServiceAccountJwtAccessCredentials;
import com.google.cloud.bigquery.BigQuery;
import com.google.cloud.bigquery.BigQueryOptions;
import com.google.cloud.bigquery.Dataset;
import java.io.FileInputStream;
import java.net.URI;
public class Example {
public static void main(String... args) throws Exception {
String projectId = "myproject";
// Load JSON file that contains service account keys and create ServiceAccountJwtAccessCredentials object.
String credentialsPath = "/path/to/key.json";
URI audience = URI.create("https://bigquery.googleapis.com/");
Credentials credentials = null;
try (FileInputStream is = new FileInputStream(credentialsPath)) {
credentials = ServiceAccountJwtAccessCredentials.fromStream(is, audience);
}
// Instantiate BigQuery client with the credentials object.
BigQuery bigquery =
BigQueryOptions.newBuilder().setCredentials(credentials).build().getService();
// Use the client to list BigQuery datasets.
System.out.println("Datasets:");
bigquery
.listDatasets(projectId)
.iterateAll()
.forEach(dataset -> System.out.printf("%s%n", dataset.getDatasetId().getDataset()));
}
}
When I tried to give the source URI of a folder inside my bucket (which has around 400 CSV files) in the Java program, it has not moved any files to BQ table. If I try with a single csv file , it moves.
package com.example.bigquerydatatransfer;
import com.google.api.gax.rpc.ApiException;
import com.google.cloud.bigquery.datatransfer.v1.CreateTransferConfigRequest;
import com.google.cloud.bigquery.datatransfer.v1.DataTransferServiceClient;
import com.google.cloud.bigquery.datatransfer.v1.ProjectName;
import com.google.cloud.bigquery.datatransfer.v1.TransferConfig;
import com.google.protobuf.Struct;
import com.google.protobuf.Value;
import java.io.IOException;
import java.util.HashMap;
import java.util.Map;
// Sample to create google cloud storage transfer config
public class Cloud_to_BQ {
public static void main(String[] args) throws IOException {
final String projectId = "dfp-bq";
String datasetId = "mytest1";
String tableId = "PROG_DATA";
String sourceUri = "gs://dfp-bq/C:\\PROG_Reports";
String fileFormat = "CSV";
String fieldDelimiter = ",";
String skipLeadingRows = "1";
Map<String, Value> params = new HashMap<>();
params.put(
"destination_table_name_template", Value.newBuilder().setStringValue(tableId).build());
params.put("data_path_template", Value.newBuilder().setStringValue(sourceUri).build());
params.put("write_disposition", Value.newBuilder().setStringValue("APPEND").build());
params.put("file_format", Value.newBuilder().setStringValue(fileFormat).build());
params.put("field_delimiter", Value.newBuilder().setStringValue(fieldDelimiter).build());
params.put("skip_leading_rows", Value.newBuilder().setStringValue(skipLeadingRows).build());
TransferConfig transferConfig =
TransferConfig.newBuilder()
.setDestinationDatasetId(datasetId)
.setDisplayName("Trial_Run_PROG_DataTransfer")
.setDataSourceId("google_cloud_storage")
.setParams(Struct.newBuilder().putAllFields(params).build())
.setSchedule("every 24 hours")
.build();
createCloudStorageTransfer(projectId, transferConfig);
}
public static void createCloudStorageTransfer(String projectId, TransferConfig transferConfig)
throws IOException {
try (DataTransferServiceClient client = DataTransferServiceClient.create()) {
ProjectName parent = ProjectName.of(projectId);
CreateTransferConfigRequest request =
CreateTransferConfigRequest.newBuilder()
.setParent(parent.toString())
.setTransferConfig(transferConfig)
.build();
TransferConfig config = client.createTransferConfig(request);
System.out.println("Cloud storage transfer created successfully :" + config.getName());
} catch (ApiException ex) {
System.out.print("Cloud storage transfer was not created." + ex.toString());
}
}
}
Is there any way I can move all the files to the BQ table at a stretch?
2022-08-04T07:27:50.185847509ZNo files found matching: "gs://dfp-bq/C:\PROG_Reports" - This is the BQ logs for the Run.
I wrote a Java program to list all the buckets and to upload a file in S3 compatible Object storage service.
The program is working fine in Windows my local machine but when I (after changing the path of the file to be uploaded of course ) transfer the runnable jar in the remote linux server and execute it I'm getting the following error-
> Exception in thread "main"
> com.amazonaws.services.s3.model.AmazonS3Exception: The request
> signature we calculated does not match the signature you provided.
> Check your AWS Secret Access Key and signing method. For more
> information, see REST Authentication and SOAP Authentication for
> details. (Service: Amazon S3; Status Code: 403; Error Code:
> SignatureDoesNotMatch; Request ID:
> 4e271b5b-d7f5-42b3-a4ad-886988bcb785; S3 Extended Request ID: null),
> S3 Extended Request ID: null
The issue seems to be in the 2nd half of the program as the list of buckets are returning in linux env. as well but during the file upload it is throwing error.
import java.io.File;
import java.text.DateFormat;
import java.text.SimpleDateFormat;
import java.util.ArrayList;
import java.util.Date;
import java.util.List;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3Client;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import com.amazonaws.services.s3.S3ClientOptions;
import com.amazonaws.auth.AWSCredentialsProvider;
import com.amazonaws.services.s3.model.Bucket;
/**
* List your Amazon S3 buckets.
*/
public class ListBuckets
{
private static void listObjects(AmazonS3 s3) {
List<Bucket> buckets = s3.listBuckets();
System.out.println("Your Amazon S3 buckets are:");
for (Bucket b : buckets) {
System.out.println("* " + b.getName());
}
}
private static void putObject(AmazonS3 s3, String bucketName, String objectName, String pathName) throws Exception
{
s3.putObject(bucketName, objectName, new File(pathName));
}
private static void time(String t) {
DateFormat dateFormat = new SimpleDateFormat("yyyy/MM/dd HH:mm:ss");
Date date = new Date();
System.out.println(t+"-->"+dateFormat.format(date));
}
public static void main(String[] args) throws Exception
{
final String accessKey = "XXXXXXXXXXXXXX";
final String secretKey = "XXXXXXXXXXXXXXXXXXXXXXXX";
BasicAWSCredentials credentials = new BasicAWSCredentials(accessKey, secretKey);
#SuppressWarnings("deprecation")
final AmazonS3 s3 = new AmazonS3Client(credentials);
S3ClientOptions opts = new S3ClientOptions().withPathStyleAccess(true);
s3.setS3ClientOptions(opts);
s3.setEndpoint("https://XXXXXX.com");
ListBuckets.time("startTime");
ListBuckets.listObjects(s3);
//String pathName = "C:\\Users\\XXXXXX\\Documents\\New folder\\New Text Document - Copy.txt";
String pathName = "/home/abcd/XXXXX/objectStorage/CHANGELOG.mdown";
ListBuckets.putObject(s3, "snap-shot/sample-aws-ex", pathName, pathName);
ListBuckets.time("end time");
}
}`
Unbelievable!
You know what the issue was in linux? The object name and path name are two different things.
putObject(AmazonS3 s3, String bucketName, String objectName, String pathName)
where the pathName is the path of your file i.e.
String pathName = "/home/abc/xxxxx/objectStorage/errorlog.txt";
Notice it starts with forward slash, whereas object name should not start with / i.e.
String objectName = "home/abc/xxxxxx/objectStorage/errorlog.txt";
I wish the exception thrown would have given better clarity on what was wrong. The exception thrown only made me deviate from the root cause.
Can some one give me a direction how can I implement this aws email template tutorial by a java code? Through java code I want to set this AWS Email Template and through java only I want to set the parameter values to the template and through java only I want to send the email.
I cant find any tutorial or direction from which I can translate above requests in java code.
The "code" in your link is actually just some JSON templates for sending and formatting email, and a few calls to an AWS command line tool. If you need to make AWS send email calls from a Java process then you need to take a look at:
The SES API
The Javadoc for the Java client lib
I am able to code it successfully. Pasting the example code here.
import java.io.IOException;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Calendar;
import java.util.List;
import com.amazonaws.auth.AWSStaticCredentialsProvider;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.services.simpleemail.AmazonSimpleEmailService;
import com.amazonaws.services.simpleemail.AmazonSimpleEmailServiceClientBuilder;
import com.amazonaws.services.simpleemail.model.BulkEmailDestination;
import com.amazonaws.services.simpleemail.model.BulkEmailDestinationStatus;
import com.amazonaws.services.simpleemail.model.Destination;
import com.amazonaws.services.simpleemail.model.SendBulkTemplatedEmailRequest;
import com.amazonaws.services.simpleemail.model.SendBulkTemplatedEmailResult;
public class AmazonSESSample2 {
public static void main(String[] args) throws IOException {
String accessKeyId = "accessKeyId";
String secretKeyId = "secretKeyId";
String region = "us-east-1";
List<BulkEmailDestination> listBulkEmailDestination = null;
SendBulkTemplatedEmailRequest sendBulkTemplatedEmailRequest = null;
try {
AmazonSimpleEmailService client = getAmazonSESClient(accessKeyId, secretKeyId, region);
listBulkEmailDestination = new ArrayList<>();
for(String email : getRecievers()) {
String replacementData="{"
+ "\"FULL_NAME\":\"AAA BBB\","
+ "\"USERNAME\":\""+email+"\","
+ "}";
BulkEmailDestination bulkEmailDestination = new BulkEmailDestination();
bulkEmailDestination.setDestination(new Destination(Arrays.asList(email)));
bulkEmailDestination.setReplacementTemplateData(replacementData);
listBulkEmailDestination.add(bulkEmailDestination);
}
sendBulkTemplatedEmailRequest = new SendBulkTemplatedEmailRequest();
sendBulkTemplatedEmailRequest.setSource("noreply#mydomain.com");
sendBulkTemplatedEmailRequest.setTemplate("welcome-email-en_GB-v1");
sendBulkTemplatedEmailRequest.setDefaultTemplateData("{\"FULL_NAME\":\"friend\", \"USERNAME\":\"unknown\"}");
sendBulkTemplatedEmailRequest.setDestinations(listBulkEmailDestination);
SendBulkTemplatedEmailResult res = client.sendBulkTemplatedEmail(sendBulkTemplatedEmailRequest);
System.out.println("======================================");
System.out.println(res.getSdkResponseMetadata());
System.out.println("======================================");
for(BulkEmailDestinationStatus status : res.getStatus()) {
System.out.println(status.getStatus());
System.out.println(status.getError());
System.out.println(status.getMessageId());
}
} catch (Exception ex) {
System.out.println("The email was not sent. Error message: " + ex.getMessage());
ex.printStackTrace();
}
}
public static List<String> getRecievers() {
ArrayList<String> list = new ArrayList<>();
list.add("aaa+1#gmail.com");
list.add("aaa+2#gmail.com");
list.add("aaa+3#gmail.com");
list.add("aaa+4#gmail.com");
return list;
}
public static AmazonSimpleEmailService getAmazonSESClient(String accessKeyId, String secretKeyId, String region) {
BasicAWSCredentials awsCreds = new BasicAWSCredentials(accessKeyId, secretKeyId);
AmazonSimpleEmailService client = AmazonSimpleEmailServiceClientBuilder.standard()
.withCredentials(new AWSStaticCredentialsProvider(awsCreds))
.withRegion(region)
.build();
return client;
}
}
I have been playing with Amazon's Product Advertising API, and I cannot get a request to go through and give me data. I have been working off of this: http://docs.amazonwebservices.com/AWSECommerceService/2011-08-01/GSG/ and this: Amazon Product Advertising API signed request with Java
Here is my code. I generated the SOAP bindings using this: http://docs.amazonwebservices.com/AWSECommerceService/2011-08-01/GSG/YourDevelopmentEnvironment.html#Java
On the Classpath, I only have: commons-codec.1.5.jar
import com.ECS.client.jax.AWSECommerceService;
import com.ECS.client.jax.AWSECommerceServicePortType;
import com.ECS.client.jax.Item;
import com.ECS.client.jax.ItemLookup;
import com.ECS.client.jax.ItemLookupRequest;
import com.ECS.client.jax.ItemLookupResponse;
import com.ECS.client.jax.ItemSearchResponse;
import com.ECS.client.jax.Items;
public class Client {
public static void main(String[] args) {
String secretKey = <my-secret-key>;
String awsKey = <my-aws-key>;
System.out.println("API Test started");
AWSECommerceService service = new AWSECommerceService();
service.setHandlerResolver(new AwsHandlerResolver(
secretKey)); // important
AWSECommerceServicePortType port = service.getAWSECommerceServicePort();
// Get the operation object:
com.ECS.client.jax.ItemSearchRequest itemRequest = new com.ECS.client.jax.ItemSearchRequest();
// Fill in the request object:
itemRequest.setSearchIndex("Books");
itemRequest.setKeywords("Star Wars");
// itemRequest.setVersion("2011-08-01");
com.ECS.client.jax.ItemSearch ItemElement = new com.ECS.client.jax.ItemSearch();
ItemElement.setAWSAccessKeyId(awsKey);
ItemElement.getRequest().add(itemRequest);
// Call the Web service operation and store the response
// in the response object:
com.ECS.client.jax.ItemSearchResponse response = port
.itemSearch(ItemElement);
String r = response.toString();
System.out.println("response: " + r);
for (Items itemList : response.getItems()) {
System.out.println(itemList);
for (Item item : itemList.getItem()) {
System.out.println(item);
}
}
System.out.println("API Test stopped");
}
}
Here is what I get back.. I was hoping to see some Star Wars books available on Amazon dumped out to my console :-/:
API Test started
response: com.ECS.client.jax.ItemSearchResponse#7a6769ea
com.ECS.client.jax.Items#1b5ac06e
API Test stopped
What am I doing wrong (Note that no "item" in the second for loop is being printed out, because its empty)? How can I troubleshoot this or get relevant error information?
I don't use the SOAP API but your Bounty requirements didn't state that it had to use SOAP only that you wanted to call Amazon and get results. So, I'll post this working example using the REST API which will at least fulfill your stated requirements:
I would like some working example code that hits the amazon server and returns results
You'll need to download the following to fulfill the signature requirements:
http://associates-amazon.s3.amazonaws.com/signed-requests/samples/amazon-product-advt-api-sample-java-query.zip
Unzip it and grab the com.amazon.advertising.api.sample.SignedRequestsHelper.java file and put it directly into your project. This code is used to sign the request.
You'll also need to download Apache Commons Codec 1.3 from the following and add it to your classpath i.e. add it to your project's library. Note that this is the only version of Codec that will work with the above class (SignedRequestsHelper)
http://archive.apache.org/dist/commons/codec/binaries/commons-codec-1.3.zip
Now you can copy and paste the following making sure to replace your.pkg.here with the proper package name and replace the SECRET and the KEY properties:
package your.pkg.here;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.StringWriter;
import java.util.HashMap;
import java.util.Map;
import java.util.Properties;
import java.util.logging.Level;
import java.util.logging.Logger;
import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.parsers.ParserConfigurationException;
import javax.xml.transform.OutputKeys;
import javax.xml.transform.Transformer;
import javax.xml.transform.TransformerException;
import javax.xml.transform.TransformerFactory;
import javax.xml.transform.dom.DOMSource;
import javax.xml.transform.stream.StreamResult;
import org.w3c.dom.Document;
import org.xml.sax.SAXException;
public class Main {
private static final String SECRET_KEY = "<YOUR_SECRET_KEY>";
private static final String AWS_KEY = "<YOUR_KEY>";
public static void main(String[] args) {
SignedRequestsHelper helper = SignedRequestsHelper.getInstance("ecs.amazonaws.com", AWS_KEY, SECRET_KEY);
Map<String, String> params = new HashMap<String, String>();
params.put("Service", "AWSECommerceService");
params.put("Version", "2009-03-31");
params.put("Operation", "ItemLookup");
params.put("ItemId", "1451648537");
params.put("ResponseGroup", "Large");
String url = helper.sign(params);
try {
Document response = getResponse(url);
printResponse(response);
} catch (Exception ex) {
Logger.getLogger(Main.class.getName()).log(Level.SEVERE, null, ex);
}
}
private static Document getResponse(String url) throws ParserConfigurationException, IOException, SAXException {
DocumentBuilder builder = DocumentBuilderFactory.newInstance().newDocumentBuilder();
Document doc = builder.parse(url);
return doc;
}
private static void printResponse(Document doc) throws TransformerException, FileNotFoundException {
Transformer trans = TransformerFactory.newInstance().newTransformer();
Properties props = new Properties();
props.put(OutputKeys.INDENT, "yes");
trans.setOutputProperties(props);
StreamResult res = new StreamResult(new StringWriter());
DOMSource src = new DOMSource(doc);
trans.transform(src, res);
String toString = res.getWriter().toString();
System.out.println(toString);
}
}
As you can see this is much simpler to setup and use than the SOAP API. If you don't have a specific requirement for using the SOAP API then I would highly recommend that you use the REST API instead.
One of the drawbacks of using the REST API is that the results aren't unmarshaled into objects for you. This could be remedied by creating the required classes based on the wsdl.
This ended up working (I had to add my associateTag to the request):
public class Client {
public static void main(String[] args) {
String secretKey = "<MY_SECRET_KEY>";
String awsKey = "<MY AWS KEY>";
System.out.println("API Test started");
AWSECommerceService service = new AWSECommerceService();
service.setHandlerResolver(new AwsHandlerResolver(secretKey)); // important
AWSECommerceServicePortType port = service.getAWSECommerceServicePort();
// Get the operation object:
com.ECS.client.jax.ItemSearchRequest itemRequest = new com.ECS.client.jax.ItemSearchRequest();
// Fill in the request object:
itemRequest.setSearchIndex("Books");
itemRequest.setKeywords("Star Wars");
itemRequest.getResponseGroup().add("Large");
// itemRequest.getResponseGroup().add("Images");
// itemRequest.setVersion("2011-08-01");
com.ECS.client.jax.ItemSearch ItemElement = new com.ECS.client.jax.ItemSearch();
ItemElement.setAWSAccessKeyId(awsKey);
ItemElement.setAssociateTag("th0426-20");
ItemElement.getRequest().add(itemRequest);
// Call the Web service operation and store the response
// in the response object:
com.ECS.client.jax.ItemSearchResponse response = port
.itemSearch(ItemElement);
String r = response.toString();
System.out.println("response: " + r);
for (Items itemList : response.getItems()) {
System.out.println(itemList);
for (Item itemObj : itemList.getItem()) {
System.out.println(itemObj.getItemAttributes().getTitle()); // Title
System.out.println(itemObj.getDetailPageURL()); // Amazon URL
}
}
System.out.println("API Test stopped");
}
}
It looks like the response object does not override toString(), so if it contains some sort of error response, simply printing it will not tell you what the error response is. You'll need to look at the api for what fields are returned in the response object and individually print those. Either you'll get an obvious error message or you'll have to go back to their documentation to try to figure out what is wrong.
You need to call the get methods on the Item object to retrieve its details, e.g.:
for (Item item : itemList.getItem()) {
System.out.println(item.getItemAttributes().getTitle()); //Title of item
System.out.println(item.getDetailPageURL()); // Amazon URL
//etc
}
If there are any errors you can get them by calling getErrors()
if (response.getOperationRequest().getErrors() != null) {
System.out.println(response.getOperationRequest().getErrors().getError().get(0).getMessage());
}