I created a csv file with three columns in a row..in google bigquery in created a dataset with one table with csv file ....for this i completed my java code...but now i have to add a new column to existed row dynamically in java code..?
// Main method for data print.
#SuppressWarnings({ "rawtypes", "unchecked" })
public static void main(String[] args) throws IOException, InterruptedException {
// Create a new BigQuery client authorized via OAuth 2.0 protocol
Bigquery bigquery = createAuthorizedClient();
TableRow row = new TableRow();
row.set("Column1", "Sample23");
row.set("Column2", 7);
row.set("Column3", "Sample25");
TableDataInsertAllRequest.Rows rows = new TableDataInsertAllRequest.Rows();
rows.setJson(row);
List rowList = new ArrayList();
rowList.add(rows);
TableDataInsertAllRequest content =
new TableDataInsertAllRequest().setRows(rowList);
TableDataInsertAllResponse response = bigquery.tabledata().insertAll(PROJECT_ID, DATASET_ID, TABLE_ID, content).execute();
System.out.println("Final response = " + response);
}
There are two table operations Update and Patch.
You need to use the Update command, to add new columns to your schema.
Important side notes:
order is important. If you change the ordering, it will look like an incompatible schema.
you can only add new fields at the end of the table. On the old columns, you have the option to change required to nullable.
you cannot add a required field to an existing schema.
you cannot remove old fields, once a table's schema has been specified you cannot change it without first deleting all the of the data associated with it. If you want to change a table's schema, you must specify a writeDisposition of WRITE_TRUNCATE. For more information, see the Jobs resource.
Here is an example of a curl session that adds fields to a schema. It should be relatively easy to adapt to Java. It uses auth.py from here
When using Table.Update(), you must include the full table schema again. If you don't provide an exact matching schema you can get: Provided Schema does not match Table. For example I didn't paid attention to details and in one of my update calls I didn't include an old field like created and it failed.
Actually I didn't use any jobs in my java code. I simply created a dataset with one table with a row in three columns. Now I have to add new column at java code not in csv file. I am posting my complete source code:
public class BigQueryJavaGettingStarted {
// Define required variables.
private static final String PROJECT_ID = "nvjsnsb";
private static final String DATASET_ID = "nvjjvv";
private static final String TABLE_ID = "sampledata";
private static final String CLIENTSECRETS_LOCATION = "client_secrets.json";
static GoogleClientSecrets clientSecrets = loadClientSecrets(CLIENTSECRETS_LOCATION);
// Static variables for API scope, callback URI, and HTTP/JSON functions
private static final List<String> SCOPES = Arrays.asList(BigqueryScopes.BIGQUERY);
private static final String REDIRECT_URI = "https://www.example.com/oauth2callback";
// Global instances of HTTP transport and JSON factory objects.
private static final HttpTransport TRANSPORT = new NetHttpTransport();
private static final JsonFactory JSON_FACTORY = new JacksonFactory();
private static GoogleAuthorizationCodeFlow flow = null;
// Main method for data print.
#SuppressWarnings({ "rawtypes", "unchecked" })
public static void main(String[] args) throws IOException, InterruptedException {
// Create a new BigQuery client authorized via OAuth 2.0 protocol
Bigquery bigquery = createAuthorizedClient();
TableRow row = new TableRow();
row.set("Column1", "OneMoreCol1");
row.set("Column2", 79);
row.set("Column3", "OneMoreCol2");
TableDataInsertAllRequest.Rows rows = new TableDataInsertAllRequest.Rows();
rows.setJson(row);
List rowList = new ArrayList();
rowList.add(rows);
TableDataInsertAllRequest content = new TableDataInsertAllRequest().setRows(rowList);
TableDataInsertAllResponse response = bigquery.tabledata().insertAll(PROJECT_ID, DATASET_ID, TABLE_ID, content).execute();
System.out.println("Final response = " + response);
}
// Create Authorized client.
public static Bigquery createAuthorizedClient() throws IOException {
String authorizeUrl = new GoogleAuthorizationCodeRequestUrl(
clientSecrets,
REDIRECT_URI,
SCOPES).setState("").build();
System.out.println("Paste this URL into a web browser to authorize BigQuery Access:\n" + authorizeUrl);
System.out.println("... and type the code you received here: ");
BufferedReader in = new BufferedReader(new InputStreamReader(System.in));
String authorizationCode = in.readLine();
// Exchange the auth code for an access token and refresh token
Credential credential = exchangeCode(authorizationCode);
return new Bigquery(TRANSPORT, JSON_FACTORY, credential);
}
// Exchange code method.
static Credential exchangeCode(String authorizationCode) throws IOException {
GoogleAuthorizationCodeFlow flow = getFlow();
GoogleTokenResponse response =
flow.newTokenRequest(authorizationCode).setRedirectUri(REDIRECT_URI).execute();
return flow.createAndStoreCredential(response, null);
}
// Get flow.
static GoogleAuthorizationCodeFlow getFlow() {
if (flow == null) {
HttpTransport httpTransport = new NetHttpTransport();
JacksonFactory jsonFactory = new JacksonFactory();
flow = new GoogleAuthorizationCodeFlow.Builder(httpTransport,
jsonFactory,
clientSecrets,
SCOPES)
.setAccessType("offline").setApprovalPrompt("force").build();
}
return flow;
}
// Load client secrets.
private static GoogleClientSecrets loadClientSecrets(String clientSecretsLocation) {
try {
clientSecrets = GoogleClientSecrets.load(new JacksonFactory(),
new InputStreamReader(BigQueryJavaGettingStarted.class.getResourceAsStream(clientSecretsLocation)));
} catch (Exception e) {
System.out.println("Could not load client_secrets.json");
e.printStackTrace();
}
return clientSecrets;
}
}
Related
I need to connect to SharePoint in order to upload files to https://xxxxxxgroup.sharepoint.com/sites/xxx-xxxxxDepartment from a webapp I developed.
I registered my app and all I have is:
tennantId
clientId
pemCertificate
so I'm using msgraph-sdk-java and as authentication OnBehalfOf Provider. below is my code:
public GraphServiceClient sharepointAuth() {
private final static String CLIENT_ID = "xxxxxxxxx-xxxx-xxxxx-xxxx-xxxxxxxx";
private final static String TENANT_ID = "xxxxxxxxx-xxxx-xxxxx-xxxx-xxxxxxxx";
private final static List<String> SCOPES = Arrays.asList("xxxxx");
private final static String CERTIFICATE_PATH = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx";
try {
final OnBehalfOfCredential onBehalfOfCredential = new OnBehalfOfCredentialBuilder()
.clientId(CLIENT_ID)
.tenantId(TENANT_ID)
.pemCertificate(CERTIFICATE_PATH)
//.userAssertion()
.build();
final TokenCredentialAuthProvider tokenCredentialAuthProvider = new TokenCredentialAuthProvider(SCOPES, credential1);
final GraphServiceClient graphClient = GraphServiceClient
.builder()
.authenticationProvider(tokenCredentialAuthProvider)
.buildClient();
System.out.println("------after auth------");
final Drive result = graphClient
.me()
.drive()
.buildRequest()
.get();
System.out.println("Found Drive " + result.id);
} catch(Exception e) {
System.out.println("exception");
System.out.println(e);
}
return null;
}
I'm not facing any error but I'm not able to get the drive, in logs:
------after auth------
I didn't find any example online, this is my 3rd full day working on it, I have many question:
pemCertificate: must be absolute/root path? (I tried both and nothing succeded)
userAssertion: is it mandatory? and how to fill it?
any example on how to connect to sharepoint? (if I only have tenantId, clienId and pemCertificate)
I appreciate your help!
Im using a simple example from amazon aws site to connect to opensearch index.
This is the example source https://docs.aws.amazon.com/opensearch-service/latest/developerguide/request-signing.html#request-signing-java.
The health status of my node is yellow and its open
yellow open my-index
The error message
Exception in thread "main" java.net.ConnectException
at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:943)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:227)
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1256)
at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1231)
at org.elasticsearch.client.RestHighLevelClient.index(RestHighLevelClient.java:587)
at com.amazonaws.lambda.demo.AWSElasticsearchServiceClient.main(AWSElasticsearchServiceClient.java:41)
Caused by: java.net.ConnectException
at org.apache.http.nio.pool.RouteSpecificPool.timeout(RouteSpecificPool.java:168)
at org.apache.http.nio.pool.AbstractNIOConnPool.requestTimeout(AbstractNIOConnPool.java:561)
at org.apache.http.nio.pool.AbstractNIOConnPool$InternalSessionRequestCallback.timeout(AbstractNIOConnPool.java:822)
at org.apache.http.impl.nio.reactor.SessionRequestImpl.timeout(SessionRequestImpl.java:183)
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processTimeouts(DefaultConnectingIOReactor.java:210)
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:155)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:348)
at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:192)
at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
at java.lang.Thread.run(Unknown Source) ```
private static String region = "us-west-1";
private static String domainEndpoint = "<my-index...amazon.com>"; // e.g. https://search-mydomain.us-west-1.es.amazonaws.com
private static String index = "my-index";
private static String type = "_doc";
private static String id = "1";
static final AWSCredentialsProvider credentialsProvider = new DefaultAWSCredentialsProviderChain();
``` public static void main(String[] args) throws IOException {
RestHighLevelClient searchClient = searchClient(serviceName, region);
// Create the document as a hash map
Map<String, Object> document = new HashMap<>();
document.put("title", "Walk the Line");
document.put("director", "James Mangold");
document.put("year", "2005");
// Form the indexing request, send it, and print the response
IndexRequest request = new IndexRequest(index, type, id).source(document);
IndexResponse response = searchClient.index(request, RequestOptions.DEFAULT);
System.out.println(response.toString());
}
// Adds the interceptor to the OpenSearch REST client
public static RestHighLevelClient searchClient(String serviceName, String region) {
AWS4Signer signer = new AWS4Signer();
signer.setServiceName(serviceName);
signer.setRegionName(region);
HttpRequestInterceptor interceptor = new AWSRequestSigningApacheInterceptor(serviceName, signer, credentialsProvider);
return new RestHighLevelClient(RestClient.builder(HttpHost.create(domainEndpoint)).setHttpClientConfigCallback(hacb -> hacb.addInterceptorLast(interceptor)));
}
Try this example. I tried the same and it did work well for me. I did not bother doing anything in regards to the cert as I had followed AWS demo examples to create the domain.
Hopefully this is what you are looking for...
So I have a json file that has some request and response data, and what I want to accomplish is iterate through this data and create a pact file that uses each request and response.
So at the moment I am using a parameterized test in junit to kinda iterate through our json data, and this basically works except for because the producer name is the same for all pacts, it creates the same file and is overwriting the previous.
private JsonObject requestObject;
private static Gson gson = new Gson();
private static File jsonFile = readJsonFile();
private static int randValue = new Random().nextInt(500);
private static String consmerName = "phx-ev-consumer" + randValue;
#Rule
public PactProviderRuleMk2 provider = new PactProviderRuleMk2("phx-ev-svc-provider", "localhost", 8080, this);
final RestTemplate restTemplate = new RestTemplate();
public EligibilityApiConsumerPactTest(JsonObject requestObject) {
this.requestObject = requestObject;
}
#Parameterized.Parameters
public static Collection primeNumbers() throws JsonSyntaxException, JsonIOException, FileNotFoundException {
return getJson();
}
#Pact(state = "provider accets submit contact form", provider = "phx-ev-svc-provider" , consumer = "phx-ev-consumer")
public RequestResponsePact createFragment(PactDslWithProvider builder) {
Map<String, String> requestHeaders = new HashMap<>();
requestHeaders.put("Content-Type", "application/json");
requestHeaders.put("SM_USER", "wtadmin");
requestHeaders.put("Cookie", "SMCHALLENGE=YES");
// Auth headers
String authString = "wtadmin:labcorp1";
String authEncoded = Base64.getEncoder().encodeToString(authString.getBytes());
requestHeaders.put("Authorization", "Basic " + authEncoded);
Map<String, String> responseHeaders = new HashMap<>();
responseHeaders.put("Content-Type", "application/json");
String jsonRequest = requestObject.get("request").toString();
String jsonResponse = requestObject.get("response").toString();
RequestResponsePact pact = builder.given("phx-eligibility").uponReceiving("Phoenix Eligibility Request")
.method("POST").headers(requestHeaders).body(jsonRequest).path("/phx-rest/eligibility")
.willRespondWith().status(200).headers(responseHeaders).body(jsonResponse).toPact();
return pact;
}
#Test
#PactVerification("phx-ev-svc-provider")
public void runTest() throws IOException {
MultiValueMap<String, String> requestHeaders = new LinkedMultiValueMap<>();
requestHeaders.add("Content-Type", "application/json");
requestHeaders.add("SM_USER", "wtadmin");
requestHeaders.add("Cookie", "SMCHALLENGE=YES");
// Auth headers
String authString = "wtadmin:labcorp1";
String authEncoded = Base64.getEncoder().encodeToString(authString.getBytes());
requestHeaders.add("Authorization", "Basic " + authEncoded);
String jsonRequest = requestObject.get("request").toString();
restTemplate.exchange(provider.getConfig().url() + "/phx-rest/eligibility", HttpMethod.POST,
new HttpEntity<>(jsonRequest, requestHeaders), String.class);
}
public static List<JsonObject> getJson() throws JsonSyntaxException, JsonIOException, FileNotFoundException {
List<JsonObject> results = new ArrayList<JsonObject>();
JsonObject jsonObject = gson.fromJson(new FileReader(jsonFile), JsonObject.class);
JsonArray input = jsonObject.getAsJsonArray("input");
Iterator<JsonElement> iter = input.iterator();
while (iter.hasNext()) {
JsonObject obj = (JsonObject) iter.next();
results.add(obj);
}
return results;
}
public static File readJsonFile() {
File base = new File("");
File inputFile = new File(base.getAbsolutePath() + "/pact/input/eligibility.json");
return inputFile;
}
Not sure if there is a better way to accomplish this, I have looked at the Github for Pact Jvm and looked through stack overflow but have not been able to find someone creating pact files, without statically specifying all of the data.
A Pact file is essentially a JSON document that contains details about a consumer, a provider and a list of interactions. In your case, you seems to have the same consumer and provider, but a JSON file with the requests and responses that make up the interactions.
So you need to create a single pact file, but with an interaction added for each item in your JSON file.
There are a number of ways you can do that, but if you modify your example test, you can chain the calls using the DSL builder by calling .uponReceiving again after the last .body. You can do this in a loop, each additional call to .uponReceiving will start adding a new interaction to the pact. You will have to give each interaction a unique description.
Then call .toPact() at the end to create the final pact.
I'm trying to programmatically update a public spreadsheet (set to anyone can edit) via the API but it fails with
401 - "The request does not have valid authentication credentials."
I would expect to not need "valid authentication credentials" since it's a publicly editable spreadsheet. I can GET data from the sheet just fine, although I had to generate a "browser" API Key since apparently using an Android Key doesn't work.
Anyone know if there is a trick to getting an update to work, or is this not possible with the API?
Sample code I'm hacking together:
// Don't think I even need this?
GoogleCredential credential = new GoogleCredential();
credential.createScoped(Collections.singleton(SheetsScopes.SPREADSHEETS));
HttpTransport transport = AndroidHttp.newCompatibleTransport();
JsonFactory factory = JacksonFactory.getDefaultInstance();
final Sheets sheets = new Sheets.Builder(transport, factory, credential)
.setApplicationName("My Awesome App")
.build();
final String sheetID = "[ID Of Valid Public Spreadsheet Here]";
final String range = "A:S";
final ValueRange content = new ValueRange();
content.set("Column A Name", "Some Value to Set");
new Thread() {
#Override
public void run() {
try {
UpdateValuesResponse valueRange = sheets.spreadsheets().values()
.update(sheetID, range, content)
.setKey("My-Valid-Browser-Api-Key")
.execute();
mLog.D("Got values: " + valueRange);
}
catch (IOException e) {
mLog.E("Sheets failed", e);
}
}
}.start();
The Sheets V4 API today does not allow anonymous edits, even for sheets that allow it. (It does allow anonymous reads for sheets that allow it.)
I want to use google task api and want to get tasklist,update,delete,add etc.. and I found this link https://developers.google.com/google-apps/tasks/oauth-and-tasks-on-android where step by step procedure is given on that link the library which are given are deprecated.
That's why I have downloaded latetst library google-api-java-client-1.12.0-beta from here http://code.google.com/p/google-api-java-client/downloads/detail?name=google-api-java-client-1.12.0-beta.zip&can=2&q= and google-api-services-tasks-v1-rev5-java-1.12.0-beta from here http://code.google.com/p/google-api-java-client/wiki/APIs#Tasks_API and try the code given and similar to it but no luck not get anything i am successfully get accesstoken but not get anything and in the latest libs most of method are changes so how to inialize the Tasks and get TaskList,create,delete etc...... Not a single document i found related to updated library.
Hope for your regards.
Thanks.
This solution is for Server to server communication using OAuth 2.0
It is a three step process
Authenticate using OAuth 2.0
Get the com.google.api.services.tasks.Tasks service object
Get the required Task or TaskList
In this sample code it uses the domain id "abc.com" and the user is "user1#abc.com". For gmail users, please provide the gmailid (abc#gmail.com) as consumerkey and leave "xoauth_requestor_id" as gmailid
import com.google.api.client.http.*;
import com.google.api.client.http.javanet.NetHttpTransport;
import com.google.api.client.json.jackson.JacksonFactory;
import com.google.api.services.tasks.*;
import com.google.api.client.auth.oauth.OAuthHmacSigner;
import com.google.api.client.auth.oauth.OAuthParameters;
public class GoogleConnection {
public Tasks setup() throws Exception {
com.google.api.services.tasks.Tasks tasks = null;
HttpRequestFactory httpRequestFactory = null;
HttpRequestInitializer httpRequestInitializer = null;
OAuthHmacSigner signer = new OAuthHmacSigner();
HttpTransport httpTransport = new NetHttpTransport();
OAuthParameters oauthParameters = new OAuthParameters();
final ArrayMap<String, Object> customKeys = new ArrayMap<String, Object>();
customKeys.add("xoauth_requestor_id", "user1#abc.com");
signer.clientSharedSecret = "secret_key_received_from_google";
oauthParameters.version = "2.0";
oauthParameters.consumerKey = "abc.com";
oauthParameters.signer = signer;
httpRequestFactory = createRequestFactory(httpTransport, oauthParameters, "20000", "20000");
httpRequestInitializer = httpRequestFactory.getInitializer();
tasks = new com.google.api.services.tasks.Tasks.Builder(httpTransport, new JacksonFactory(), httpRequestInitializer)
.setTasksRequestInitializer(new TasksRequestInitializer() {
#Override
public void initializeTasksRequest(TasksRequest<?> request) throws IOException {
#SuppressWarnings("rawtypes")
TasksRequest tasksRequest = (TasksRequest) request;
tasksRequest.setUnknownKeys(customKeys);
tasksRequest.setKey("keyapi_received_from_google_by_registering_your_app");
}
})
.setApplicationName("")
.build();
return tasks;
}
}
Getting Tasks from a Task List
Instantiate GoogleConnection class
public List<com.google.api.services.tasks.model.Task> getTasksFromTaskList(String taskListId) throws Exception {
com.google.api.services.tasks.Tasks tasksService = googleConnection.setup();
com.google.api.services.tasks.model.Tasks result = tasksService .tasks().list(taskListId).execute();
return result.getItems();
}