Develop Reference

How get all log events in amazon via loop?

Every amazon API has it's own token which you have to set to next request. But with aws log api I got infinity loop:
public class Some {
public static void main (String[] args) {
final GetLogEventsRequest request = new GetLogEventsRequest()
.withLogGroupName("myGroup")
.withLogStreamName("myStrean");
final AWSLogs awsLogs = AWSLogsClientBuilder.defaultClient();
Collection<OutputLogEvent> result = new ArrayList<>();
GetLogEventsResult response = null;
do {
response = awsLogs.getLogEvents(request);
result.addAll(response.getEvents());
request.withNextToken(response.getNextBackwardToken());
} while (response.getNextBackwardToken() != null);
}
}
From documentation:
nextBackwardToken
The token for the next set of items in the backward direction. The token expires after 24 hours. This token will never be null. If you have reached the end of the stream, it will return the same token you passed in.
So it can not be null like LastEvaluatedKey when you scan dynamodb:
Map<String, AttributeValue> lastKeyEvaluated = null;
do {
ScanRequest scanRequest = new ScanRequest()
.withTableName("ProductCatalog")
.withLimit(10)
.withExclusiveStartKey(lastKeyEvaluated);
ScanResult result = client.scan(scanRequest);
for (Map<String, AttributeValue> item : result.getItems()){
printItem(item);
}
lastKeyEvaluated = result.getLastEvaluatedKey();
} while (lastKeyEvaluated != null);
So and what I should pass to request.withNextToken if we speak about log api??? And if nextBackwardToken (and nextForwardToken too) can not be null - how to detect that I receive the last response from amazon???

The documentation which you've cited is quite straightforward. I think you need something like this:
final AWSLogs awsLogs = AWSLogsClientBuilder.defaultClient();
Collection<OutputLogEvent> result = new ArrayList<>();
String nextToken = null;
GetLogEventsResult response;
do {
GetLogEventsRequest request = new GetLogEventsRequest()
.withLogGroupName("myGroup")
.withLogStreamName("myStrean");
if (nextToken != null) request = request.withNextToken(nextToken);
response = awsLogs.getLogEvents(request);
result.addAll(response.getEvents());
// check if token is the same
if (response.getNextForwardToken().equals(nextToken)) break;
// save new token
nextToken = response.getNextForwardToken();
} while (true);
So, you simply need to create request each time with the new token, until it becomes equals to the old one.

Twilio - Validating Incoming Callback Request - Java

When Twilio invokes a callback method to fetch the TwiML <Say> for Voice, I see that Twilio sets "x-twilio-signature" in the HTTP header.
I need to verify that the actual request came from Twilio.
I have a simple war file running on Tomcat and the app is built using Spring.
I did something like the following:
//Get the TwilioUtils object initialized
TwilioUtils twilioUtils = new TwilioUtils("******myAuthToken");
//Get the URL from HttpRequest
String url = httpRequest.getRequestURL().toString();
Map<String, String> allRequestParams = getAllRequestParams(httpRequest);
Map<String, String> headers = getAllRequestHeaders(httpRequest);
//Get the signature generated for the Url and request parameters
//allRequestParams is a map of all request values posted to my service by Twilio
String validSig = twilioUtils.getValidationSignature(url, allRequestParams);
//Get the x-twilio-signature value from the http header map
String xTwilioSignature = headers.get("x-twilio-signature”);
//This is different from what I get below
logger.info("validSig = " + validSig);
logger.info("xTwilioSignature = " + xTwilioSignature );
//This is always false
logger.info("Signature matched : " + twilioUtils.validateRequest(xTwilioSignature, url,
allRequestParams));
I would like to know what am I doing wrong. Is my approach to validate "x-twilio-signature" incorrect?
If it is incorrect, what's the right way to do it?
I am using the helper library class TwilioUtils provided by Twilio to validate it.
All the time the signature from Twilio is different from what I get from the TwilioUtils object.

Megan from Twilio here.
Are you following the steps suggested in the security documentation?
validateRequest expects three arguments. I believe you're missing the url there.
Consider this example:
public class TwilioUtilsExample {
public static void main(String[] args) {
// Account details
String accountSid = "ACXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX";
String authToken = "YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY";
//This is the signature we expect
String expected_sig = "SSSSSSSSSSSSSSSSSSSSSSSSSSSS";
//This is the url that twilio requested
String url = "http://UUUUUUUUUUUUUUU";
//These are the post params twilio sent in its request
Map<String,String> params = new HashMap<String,String>();
// Be sure to see the signing notes at twilio.com/docs/security
TwilioUtils util = new TwilioUtils(authToken, accountSid);
boolean result = util.validateRequest(expected_sig, url, params);
if (result) {
System.out.print( "The signature is valid!\n" );
} else {
System.out.print( "The signature was NOT VALID. It might have been spoofed!\n" );
}
}
}
Hope this is helpful!

cant convert json string to regular string array in java

iam trying to convert this json string back to an array but i can't seem to do it
["\"abba\"","\"repaper\"","\"minim\"","\"radar\"","\"murdrum\"","\"malayalam
\"","\"turrut\"","\"navan\""]
can anyone help, or point me in the right direction of some tutorials. Ive tried split(",") etc but im really not too sure how to extract the words themselves.
client code:
Gson gson;
String[] words = { "hello", "Abba", "repaper", "Minim", "radar",
"murdrum", "malayalam", "cheese", "turrut","Navan" };
gson = new Gson();
String json = gson.toJson(words);
ClientConfig config = new DefaultClientConfig();
Client client = Client.create(config);
WebResource service = client
.resource("http://localhost:8090/RestSampleApp/rest/webservice/returnarray");
ClientResponse response = service.type(MediaType.APPLICATION_JSON)
.post(ClientResponse.class, json);
String output = response.getEntity(String.class);
//String target2 = gson.fromJson(json, String.class);
System.out.println(output);
webservice code:
#POST
#Path("returnarray")
#Consumes(MediaType.APPLICATION_JSON)
#Produces(MediaType.APPLICATION_JSON)
public String returnstuff(String list) {
list2 = list.substring(1, list.length() - 1); //gets rid of "[]"
temp = new ArrayList<String>(Arrays.asList(list2.split(",")));
Algorithim algo = new Algorithim(temp); // instance of algorithim class takes in arrayList
algo.getpalindromesarray(); //creates plaindrome arraylist
newlist = algo.getnewlist();
String details = gson.toJson(newlist);
return details;
}

EDIT: My previous answer wasn't correct, see this new one...
You are not using Gson correctly... You are serializing the objects well, but you're not doing a correct deserialization... I suggest you to take a brief look to Gson documentation, it's few lines and you'll understand it better...
First you serialize your array correctly in your client, with:
String json = gson.toJson(words);
Then you send it using Jersey API, I think that it's correct (although I'm not an expert in Jersey...)
Then your problem is that you are not deserializing the JSON correctly in your web service. You should parse the JSON string passed as a parameter, and you can do it with Gson as well, like this:
Gson gson = new Gson();
Type listOfStringsType = new TypeToken<List<String>>() {}.getType();
List<String> parsedList = gson.fromJson(list, listOfStringsType);
Now you can do whatever you want with your list of words working with a proper Java List.
Once you finish your work, you serialize again the List to return it, with:
String details = gson.toJson(parsedList);
Now you have to repeat the deserializing operation in your client to parse the response and get again a List<String>...
Note: You should never try to do things like serialize/deserialize JSON (or XML...) manually. A manual solution may work fine in a particular situation, but it can't be easily adapted to changes, thus, if your JSON responses change, even only slightly, you'll have to change a lot of code... Always use libraries for this kind of things!

You'd better use json library, e.g. Jackson. If code yourself, you can do like below:
// 1st: remove [ and ]
s=s.substring(1, s.length()-1);
// 2nd: remove "
s=s.replaceAll("\"\"", "");
// 3rd: split
String[] parts = s.split(",");

You can try to use String.format to specify the format you waant to pass as part of your request.
For Ex :
WebResource webResource = client.resource("http://localhost:8090/RestSampleApp/rest/webservice/returnarray");
String input = String.format("{\manual format "}",parameter);
ClientResponse response = webResource.type("application/json").post(ClientResponse.class, input);
This is how I have acheived my goal

How to save bulk documents in couchdb using lightcouch api in java

I am using the lightcouch API to connect to couchdb through Java. I am able to save a single document using dbclient.save(object) method. However, my requirement is to save bulk documents at a time. I am not able to find any methods related to saving bulk documents using the Lightcouch api. Please suggest any possible solution.
Thanks in advance!

I decided to give it a go. I have a database holding documents that describe a person.
Here is my Person class which extends Document LightCouch:
public class Person extends Document {
private String firstname = "";
private String lastname = "";
private int age = -1;
public Person(String firstname, String lastname, int age) {
super();
this.setFirstname(firstname);
this.setLastname(lastname);
this.setAge(age);
}
// setters and getters omitted for brevity
}
The algorithm is simple.
Create an array of type Document
Put your documents into the array
Create a HTTP POST request
Put the JSON converted array into the request body
Send it
Here is roughly what the code could look like.
Note: try/catch omitted for brevity! Of course you are expected to use them.
public static void main(String[] args) {
// You could also use a List and then convert it to an array
Document[] docs = new Document[2];
docs[0] = new Person("John", "Smith", 34);
docs[1] = new Person("Jane", "Smith", 30);
DefaultHttpClient httpClient = new DefaultHttpClient();
// Note the _bulk_docs
HttpPost post = new HttpPost("http://127.0.0.1:5984/persons/_bulk_docs");
Gson gson = new Gson();
StringEntity data =
new StringEntity("{ \"docs\": " + gson.toJson(docs) + "}");
data.setContentType("application/json");
post.setEntity(data);
HttpResponse response = httpClient.execute(post);
if (response.getStatusLine().getStatusCode() != 201) {
throw new RuntimeException("Failed. HTTP error code: "
+ response.getStatusLine().getStatusCode());
}
BufferedReader br = new BufferedReader(
new InputStreamReader((response.getEntity().getContent())));
String output;
while ((output = br.readLine()) != null) {
System.out.println(output);
}
httpClient.getConnectionManager().shutdown();
}
I'll describe the two noteworthy parts in this example.
First one is the collection of documents. In this case I used an array instead of a List for the example.
Document[] docs = new Document[2];
docs[0] = new Person("John", "Smith", 34);
docs[1] = new Person("Jane", "Smith", 30);
You could use a List as well and later convert it to an array using Java's utility methods.
Second one is the StringEntity. As per CouchDB's documentation on the HTTP Bulk Document API on modify multiple documents with a single request the JSON structure of your request body should look like this.
{
"docs": [
DOCUMENT,
DOCUMENT,
DOCUMENT
]
}
This is the reason for the somewhat ugly StringEntity definition.
StringEntity data = new StringEntity("{ \"docs\": " + gson.toJson(docs) + "}");
As a response you'll get a JSON array containing objects whose fields represent the *_id* and *_rev* of the inserted document along with a transaction status indicator.

I did the same thing but with spring Rest Template
I created a class which would hold the documents to be updated int he following way.
public class BulkUpdateDocument {
private List<Object> docs;
}
My Rest code looks like this.
BulkUpdateDocument doc = new BulkUpdateDocument(ListOfObjects);
Gson gson = new Gson();
RestTemplate restTemplate = new RestTemplate();
HttpHeaders header = new HttpHeaders();
header.setContentType(MediaType.APPLICATION_JSON_UTF8);
HttpEntity<?> requestObject = new HttpEntity<Object>(gson.toJson(doc), header);
ResponseEntity<Object> postForEntity = restTemplate.postForEntity(path + "/_bulk_docs", requestObject, Object.class);

How to get the Index Size in Solr using Java

I need to get total size of an index in Apache Solr using Java. The following code gets the total number of documents but I am looking for the size. And with the use of ReplicationHandler I was thinking that I can get the index size as told by someone here on this link.. http://lucene.472066.n3.nabble.com/cheking-the-size-of-the-index-using-solrj-API-s-td692686.html but I am not getting the index size.
BufferedWriter out1 = null;
FileWriter fstream1 = new FileWriter("src/test/resources/solr-document-id-desc.txt");
out1 = new BufferedWriter(fstream1);
ApplicationContext context = null;
context = new ClassPathXmlApplicationContext("application-context.xml");
CommonsHttpSolrServer solrServer = (CommonsHttpSolrServer) context.getBean("solrServer");
SolrQuery solrQuery = new SolrQuery().setQuery("*:*");
QueryResponse rsp = solrServer.query(solrQuery);
//I am trying to use replicationhandler but I am not able to get the index size using statistics. Is there any way to get the index size..?
ReplicationHandler handler2 = new ReplicationHandler();
System.out.println( handler2.getDescription());
NamedList statistics = handler2.getStatistics();
System.out.println("Statistics "+ statistics);
System.out.println(rsp.getResults().getNumFound());
Iterator<SolrDocument> iter = rsp.getResults().iterator();
while (iter.hasNext()) {
SolrDocument resultDoc = iter.next();
System.out.println(resultDoc.getFieldNames());
String id = (String) resultDoc.getFieldValue("numFound");
String description = (String) resultDoc.getFieldValue("description");
System.out.println(id+"~~"+description);
out1.write(id+"~~"+description);
out1.newLine();
}
out1.close();
Any suggestions will be appreciated..
Update Code:-
ReplicationHandler handler2 = new ReplicationHandler();
System.out.println( handler2.getDescription());
NamedList statistics = handler2.getStatistics();
System.out.println("Statistics "+ statistics.get("indexSize"));

The indexsize is available with the statistics in ReplicationHandler
org.apache.solr.handler.ReplicationHandler
code
public NamedList getStatistics() {
NamedList list = super.getStatistics();
if (core != null) {
list.add("indexSize", NumberUtils.readableSize(getIndexSize()));
}
}
You can use the URL http://localhost:8983/solr/replication?command=details , which returns the index size.
<lst name="details">
<str name="indexSize">26.13 KB</str>
.....
</lst>
Not sure if it works with the instantiation of ReplicationHandler, as it would need the reference of the core and the index.

You can use the command in the data directory
- du -kx

as said in this post you can use MAT tool in order to see the memory consumption. I think that you could use in your code. Enjoy solr!