Develop Reference

Parsing curl response with Java

Before writing something like "why don't you use Java HTTP client such as apache, etc", I need you to know that the reason is SSL. I wish I could, they are very convenient, but I can't.
None of the available HTTP clients support GOST cipher suite, and I get handshake exception all the time. The ones which do support the suite, doesn't support SNI (they are also proprietary) - I'm returned with a wrong cert and get handshake exception over and over again.
The only solution was to configure openssl (with gost engine) and curl and finally execute the command with Java.
Having said that, I wrote a simple snippet for executing a command and getting input stream response:
public static InputStream executeCurlCommand(String finalCurlCommand) throws IOException
{
return Runtime.getRuntime().exec(finalCurlCommand).getInputStream();
}
Additionally, I can convert the returned IS to a string like that:
public static String convertResponseToString(InputStream isToConvertToString) throws IOException
{
StringWriter writer = new StringWriter();
IOUtils.copy(isToConvertToString, writer, "UTF-8");
return writer.toString();
}
However, I can't see a pattern according to which I could get a good response or a desired response header:
Here's what I mean
After executing a command (with -i flag), there might be lots and lots of information like in the screen below:
At first, I thought that I could just split it with '\n', but the thing is that a required response's header or a response itself may not satisfy the criteria (prettified JSON or long redirect URL break the rule).
Also, the static line GOST engine already loaded is a bit annoying (but I hope that I'll be able to get rid of it and nothing unrelated info like that will emerge)
I do believe that there's a pattern which I can use.
For now I can only do that:
public static String getLocationRedirectHeaderValue(String curlResponse)
{
String locationHeaderValue = curlResponse.substring(curlResponse.indexOf("Location: "));
locationHeaderValue = locationHeaderValue.substring(0, locationHeaderValue.indexOf("\n")).replace("Location: ", "");
return locationHeaderValue;
}
Which is not nice, obviosuly
Thanks in advance.

Instead of reading the whole result as a single string you might want to consider reading it line by line using a scanner.
Then keep a few status variables around. The main task would be to separate header from body. In the body you might have a payload you want to treat differently (e.g. use GSON to make a JSON object).
The nice thing: Header and Body are separated by an empty line. So your code would be along these lines:
boolean inHeader = true;
StringBuilder b = new StringBuilder;
String lastLine = "";
// Technically you would need Multimap
Map<String,String> headers = new HashMap<>();
Scanner scanner = new Scanner(yourInputStream);
while scanner.hasNextLine() {
String line = scanner.nextLine();
if (line.length() == 0) {
inHeader = false;
} else {
if (inHeader) {
// if line starts with space it is
// continuation of previous header
treatHeader(line, lastLine);
} else {
b.append(line);
b.appen(System.lineSeparator());
}
}
}
String body = b.toString();

How to scrape a website, http get vs http post?

I am new to programming and know very little about http, but I wrote a code to scrape a website in Java, and have been running into the issue that my code scrapes "get" http calls (based on typing in a URL) but I do not know how to go about scraping data for a "post" http call.
After a brief overview on http, I believe I will need to simulate the browser, but do not know how to do this in Java. The website I have been trying to use.
As I need to scrape that source code for all the pages, the URL does not change as each next button is clicked. I have used Firefox firebug to look at what is going on when the button is clicked, but I do not know all that I am looking for.
My code to scrape the data as of now is:
public class Scraper {
private static String month = "11";
private static String day = "4";
private static String url = "http://cpdocket.cp.cuyahogacounty.us/SheriffSearch/results.aspx?q=searchType%3dSaleDate%26searchString%3d"+month+"%2f"+day+"%2f2013%26foreclosureType%3d%27NONT%27%2c+%27PAR%27%2c+%27COMM%27%2c+%27TXLN%27"; // the input website to be scraped
public static String sourcetext; //The source code that has been scraped
//scrapeWebsite runs the method to scrape the input URL and returns a string to be parsed.
public static void scrapeWebsite() throws IOException {
URL urlconnect = new URL(url); //creates the url from the variable
URLConnection connection = urlconnect.openConnection();
BufferedReader in = new BufferedReader(new InputStreamReader(
connection.getInputStream(), "UTF-8"));
String inputLine;
StringBuilder sourcecode = new StringBuilder(); // creates a stringbuilder which contains the sourcecode
while ((inputLine = in.readLine()) != null)
sourcecode.append(inputLine);
in.close();
sourcetext = sourcecode.toString();
}
What would be the best way to go about scraping all the pages for each "post" call?

Take a look at the jersey client interface
View the source of each page and determine the pattern of the url for next an previous pages then loop through.

how to post in user's Streams using google plus api in java

I want to share some information in google plus wall from my application.and I am trying for moment.insert, But getting 400 error . Can somebody help me
#Override
public JSONObject getGooglePlusAddUseractivities(Object token) {
Token accessToken = (Token) token;
OAuthService service = createOAuthServiceForGooglePlus();
OAuthRequest request = new OAuthRequest(Method.POST,"https://www.googleapis.com/plus/v1/people/me/moments/vault");
request.addQuerystringParameter("alt", "json");
service.signRequest(accessToken, request);
JSONObject object=new JSONObject();
try {
object.put("kind","plus#moment");
object.put("type","http://schemas.google.com/AddActivity");
JSONObject obj1=new JSONObject();
obj1.put("kind", "plus#itemScope");
obj1.put("url","https://plus.google.com/me");
obj1.put("description","Sign up now to claim and redeem your credits while shopping! ");
obj1.put("image","http://invite.png");
obj1.put("contentUrl", "www.abcd.com");
obj1.put("thumbnailUrl", "http://logo1_favicon.png");
object.putOpt("target", obj1);;
}catch(Exception ex) {
ex.printStackTrace();
}
request.addPayload(object.toString());
request.addHeader("Content-Type", "application/json");
System.out.println("request : "+request.getBodyContents());
Response response = request.send();
String responseBody = response.getBody();
JSONObject googleJSON = null;
try {
googleJSON = new JSONObject(responseBody);
}
catch (JSONException e) {
System.out.println("can not create JSON Object");
}
getting 400 error ?? anyone can tell me..... where am wrong ..!!`

It isn't clear from the documentation, but you can't provide both the target.url and most other target metadata. This is currently opened as bug 485 in the issue tracking system - please go there and star the issue to make sure they properly prioritize a fix.
If you remove the target.url value and add a target.id value, it should work.
(As an aside, this does not post in the user's stream, but will post an App Activity in their app moment vault. They must manually share the activity if they choose.)

At this time, it is not possible to programmatically write to a user's Stream. As a developer, you have two options:
Write an AppActivity (formerly known as a Moment), which writes information to Google, but not to a Google+ Stream. These activities are visible at plus.google.com/apps, and will be used by Google in additional ways over time.
Create an Interactive Post Share button, which a user must initiate. However, you can pre-fill both the text of the post and up to 10 intended recipients. The user can make changes if they want and then perform the actual share. You can learn more at https://developers.google.com/+/web/share/interactive or by watching this Google+ Developers Live episode: http://www.youtube.com/watch?v=U4Iw28jWtAY.

Java example of how to log in to Google App Engine with a Facebook account using OAuth

I searched a lot, read many blogs, articles, tutorials, but until now did not get a working example of using a Facebook account to log in to my application.
I know that I have to use OAuth, get tokens, authorizations, etc...
Can anyone share an example?

Here is how I do it on App Engine:
Step 1) Register an "app" on Facebook (cf. https://developers.facebook.com/ ). You give Facebook a name for the app and a url. The url you register is the url to the page (jsp or servlet) that you want to handle the login. From the registration you get two strings, an "app ID" and an "app secret" (the latter being your password, do not give this out or write it in html).
For this example, let's say the url I register is "http://myappengineappid.appspot.com/signin_fb.do".
2) From a webpage, say with a button, you redirect the user to the following url on Facebook, substituting your app id for "myfacebookappid" in the below example. You also have to choose which permissions (or "scopes") you want the ask the user (cf. https://developers.facebook.com/docs/reference/api/permissions/ ). In the example I ask for access to the user's email only.
(A useful thing to know is that you can also pass along an optional string that will be returned unchanged in the "state" parameter. For instance, I pass the user's datastore key, so I can retrieve the user when Facebook passes the key back to me. I do not do this in the example.)
Here is a jsp snippet:
<%#page import="java.net.URLEncoder" %>
<%
String fbURL = "http://www.facebook.com/dialog/oauth?client_id=myfacebookappid&redirect_uri=" + URLEncoder.encode("http://myappengineappid.appspot.com/signin_fb.do") + "&scope=email";
%>
<img src="/img/facebook.png" border="0" />
3) Your user will be forwarded to Facebook, and asked to approve the permissions you ask for. Then, the user will be redirected back to the url you have registered. In this example, this is "http://myappengineappid.appspot.com/signin_fb.do" which in my web.xml maps to the following servlet:
import org.json.JSONObject;
import org.json.JSONException;
import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.io.IOException;
import java.net.URL;
import java.net.URLConnection;
import java.net.URLEncoder;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
public class SignInFB extends HttpServlet {
public void service(HttpServletRequest req, HttpServletResponse res) throws ServletException, IOException {
String code = req.getParameter("code");
if (code == null || code.equals("")) {
// an error occurred, handle this
}
String token = null;
try {
String g = "https://graph.facebook.com/oauth/access_token?client_id=myfacebookappid&redirect_uri=" + URLEncoder.encode("http://myappengineappid.appspot.com/signin_fb.do", "UTF-8") + "&client_secret=myfacebookappsecret&code=" + code;
URL u = new URL(g);
URLConnection c = u.openConnection();
BufferedReader in = new BufferedReader(new InputStreamReader(c.getInputStream()));
String inputLine;
StringBuffer b = new StringBuffer();
while ((inputLine = in.readLine()) != null)
b.append(inputLine + "\n");
in.close();
token = b.toString();
if (token.startsWith("{"))
throw new Exception("error on requesting token: " + token + " with code: " + code);
} catch (Exception e) {
// an error occurred, handle this
}
String graph = null;
try {
String g = "https://graph.facebook.com/me?" + token;
URL u = new URL(g);
URLConnection c = u.openConnection();
BufferedReader in = new BufferedReader(new InputStreamReader(c.getInputStream()));
String inputLine;
StringBuffer b = new StringBuffer();
while ((inputLine = in.readLine()) != null)
b.append(inputLine + "\n");
in.close();
graph = b.toString();
} catch (Exception e) {
// an error occurred, handle this
}
String facebookId;
String firstName;
String middleNames;
String lastName;
String email;
Gender gender;
try {
JSONObject json = new JSONObject(graph);
facebookId = json.getString("id");
firstName = json.getString("first_name");
if (json.has("middle_name"))
middleNames = json.getString("middle_name");
else
middleNames = null;
if (middleNames != null && middleNames.equals(""))
middleNames = null;
lastName = json.getString("last_name");
email = json.getString("email");
if (json.has("gender")) {
String g = json.getString("gender");
if (g.equalsIgnoreCase("female"))
gender = Gender.FEMALE;
else if (g.equalsIgnoreCase("male"))
gender = Gender.MALE;
else
gender = Gender.UNKNOWN;
} else {
gender = Gender.UNKNOWN;
}
} catch (JSONException e) {
// an error occurred, handle this
}
...
I have removed error handling code, as you may want to handle it differently than I do. (Also, "Gender" is of course a class that I have defined.) At this point, you can use the data for whatever you want, like registering a new user or look for an existing user to log in. Note that the "myfacebookappsecret" string should of course be your app secret from Facebook.
You will need the "org.json" package to use this code, which you can find at: http://json.org/java/ (just take the .java files and add them to your code in an org/json folder structure).
I hope this helps. If anything is unclear, please do comment, and I will update the answer.
Ex animo, - Alexander.
****UPDATE****
I want to add a few tidbits of information, my apologies if some of this seems a bit excessive.
To be able to log in a user by his/her Facebook account, you need to know which user in the datastore we are talking about. If it's a new user, easy, create a new user object (with a field called "facebookId", or whatever you want to call it, whose value you get from Facebook), persist it in the datastore and log the user in.
If the user exist, you need to have the field with the facebookId. When the user is redirected from Facebook, you can grab the facebookId, and look in the datastore to find the user you want to log in.
If you already have users, you will need to let them log in the way you usually do, so you know who they are, then send them to Facebook, get the facebookId back and update their user object. This way, they can log in using Facebook the next time.
Another small note: The user will be presented with a screen on Facebook asking to allow your app access to whatever scopes you ask for, there is no way around this (the less scopes you ask for, the less intrusive it seems, though). However, this only happens the first time a user is redirected (unless you ask for more scopes later, then it'll ask again).

You can try face4j https://github.com/nischal/face4j/wiki . We've used it on our product http://grabinbox.com and have open sourced it for anyone to use. It works well on GAE.
There is an example on the wiki which should help you integrate login with facebook in a few minutes.
face4j makes use of oAuth 2.0 and the facebook graph API.

I had a lot of difficulty when trying to implement the OAuth signing myself. I spent a lot of time trying to debug an issue with my tokens not actually getting authorized - a common problem apparently. Unfortunately, none of the solutions worked for me so I ended up just using Scribe, a nifty Java OAuth library that has the added benefit of supporting other providers besides for Facebook (e.g. Google, Twitter, etc.)

You can take a look at LeanEngine, the server part: https://github.com/leanengine/LeanEngine-Server/tree/master/lean-server-lib/src/main/java/com/leanengine/server/auth

Check facebook's java APIs.
Other examples: http://code.google.com/p/facebook-java-api/wiki/Examples

App engine Url request utf-8 characters becoming '??' or '???'

I have an error where I am loading data from a web-service into the datastore. The problem is that the XML returned from the web-service has UTF-8 characters and app engine is not interpreting them correctly. It renders them as ??.
I'm fairly sure I've tracked this down to the URL Fetch request. The basic flow is: Task queue -> fetch the web-service data -> put data into datastore so it definitely has nothing to do with request or response encoding of the main site.
I put log messages before and after Apache Digester to see if that was the cause, but determined it was not. This is what I saw in logs:
string from the XML: "Doppelg��nger"
After digester processed: "Doppelg??nger"
Here is my url fetching code:
public static String getUrl(String pageUrl) {
StringBuilder data = new StringBuilder();
log.info("Requesting: " + pageUrl);
for(int i = 0; i < 5; i++) {
try {
URL url = new URL(pageUrl);
URLConnection connection = url.openConnection();
connection.connect();
BufferedReader reader = new BufferedReader(new InputStreamReader(connection.getInputStream()));
String line;
while ((line = reader.readLine()) != null) {
data.append(line);
}
reader.close();
break;
} catch (Exception e) {
log.warn("Failed to load page: " + pageUrl, e);
}
}
String resp = data.toString();
if(resp.isEmpty()) {
return null;
}
return resp;
Is there a way I can force this to recognize the input as UTF-8. I tested the page I am loading and the W3c validator recognized it as valid utf-8.
The issue is only on app engine servers, it works fine in the development server.
Thanks

try
BufferedReader reader = new BufferedReader(new InputStreamReader(connection.getInputStream(), "UTF-8"));

I was drawn into the same issue 3 months back Mike. It does look like and I would assume your problems are same.
Let me recollect and put it down here. Feel free to add if I miss something.
My set up was Tomcat and struts.
And the way I resolved it was through correct configs in Tomcat.
Basically it has to support the UTF-8 character there itself. useBodyEncodingForURI in the connector. this is for GET params
Plus you can use a filter for POST params.
A good resource where yu can find all this in one roof is Click here!
I had a problem in the production thereafter where I had apache webserver redirecting request to tomcat :). Similarly have to enable UTF-8 there too. The moral of the story resolve the problem as it comes :)