Develop Reference

Parsing JSON response(includes a list) in java

I did a http client and I'm getting a response. I am using a JSONObject to parse the data and when I execute the code below it prints out all of the JSON just fine
HttpResponse response = httpclient.execute(httppost);
HttpEntity entity = response.getEntity();
if(entity!=null){
try(InputStream instream = entity.getContent()) {
String responseString = readInputStream(instream);
JSONObject job = new JSONObject(responseString);
statusLabel.setText("Command Result: " + job.toString());
Here is the readInputSream function:
static private String readInputStream(InputStream inputStream) throws IOException {
BufferedReader reader = new BufferedReader(new InputStreamReader(
inputStream, "UTF-8"));
String tmp;
StringBuilder sb = new StringBuilder();
while ((tmp = reader.readLine()) != null) {
sb.append(tmp).append("\n");
}
if (sb.length() > 0 && sb.charAt(sb.length() - 1) == '\n') {
sb.setLength(sb.length() - 1);
}
reader.close();
return sb.toString();
}
If I change it from job.toString() to:
statusLabel.setText("Command Result: " + job.get("result"));
it prints a 1 which is correct, it works all the way up to my_list. I'm not sure how to parse the list. I put a snippet of the response below. Ive tried "my_list", "my_list[]", my_list[0]" which none have worked. I get JSONObject "blank" not found
{"result":1, "ver":1,"total":2,"catch":true,"my_list":[{"id":3,"mid":0,"format":3,"user":4,"property":1,"type":0,"title":"hello","start":146,"end":1464,"hid":3,"bid":1,"reason":1,"time":0,"creator":"1","hello":0,"my":0,"year":"0","ggg":614,"name":"","ch":"0","attr":0,"type":1,"vtype":1,"tm_log": {"fr":4,"action":0,"vr":"82","started":1,"av_ended":2,"tr":1}}

The element you trying to retrieve is parsed into a JSONArray, not a JSONObject. Try:
JSONArray my_list = job.getJSONArray("my_list");

Assuming that you are using json parser project JSON-java to parse your JSON you need to retrieve a JSONArray instance - this is how arrays are storred in JSONObject. so do the following: JSONArray my_list = job.getJSONArray("my_list"); and then use methods of JSONArray class to access your array. The Javadoc to JSON-java package can be found here: http://stleary.github.io/JSON-java/index.html. Also note that JSON-java is very simple and easy to use JSON parser project but it is not very efficient for any serious project. Common recommendation for commercial use is Jackson JSON Processor which is one of the fastest and powerful JSON parsers. Here are some links to read about it: https://github.com/FasterXML/jackson, http://wiki.fasterxml.com/JacksonHome

How to Properly Check for JSON Updates - Java

I'm trying to make a program to check for announcements via a web API - This connects to a remote server and reads the JSON on the page - I cannot test my code as the server is not live yet. Would this work & be the correct way to go about this?
public class AnnouncementChecker implements Runnable{
private final String announcementsURL = "REDACTED";
private String lastAnnouncement = "";
#Override
public void run(){
try {
URL url = new URL(announcementsURL);
HttpURLConnection http = (HttpURLConnection) url.openConnection();
http.setRequestMethod("conditional GET");
http.setRequestProperty("Connection", "keep-alive");
http.setUseCaches(true);
http.setAllowUserInteraction(false);
if (lastAnnouncement != ""){
http.setRequestProperty("If-Modified-Since", lastAnnouncement);
}
http.setConnectTimeout(10);
http.connect();
int status = http.getResponseCode();
if (status == 304 || (status == 200 && lastAnnouncement == "")){
lastAnnouncement = http.getHeaderField("Last-Modified");
BufferedReader br = new BufferedReader(new InputStreamReader(http.getInputStream()));
StringBuilder sb = new StringBuilder();
String line;
while ((line = br.readLine()) != null) {
sb.append(line+"\n");
}
br.close();
String json = sb.toString();
JSONParser parser = new JSONParser();
JSONObject jsonResponse = (JSONObject) parser.parse(json);
//String announcement = (String) jsonResponse.get("message");
//TODO What to do with announcement...
}
http.getInputStream().close();
http.disconnect();
} catch (IOException | ParseException e) {
e.printStackTrace();
}
}
}

I would recommend setting up a test of some kind, despite the server not being available. This would give you the answer to your question and the test would be there forever going forward to protect you when you make changes to the code and the business requirements change.
To help you with that I would recommend splitting up the code that returns the response and the code that does the parsing. That way you can test the parsing independent of the part that makes the HTTP connection.
If you have no idea how to do that then I'd be happy to post an example for you.

Scraping cricinfo only two pages are being loaded

I am scraping espncricinfo.
Goal is to extract batting statistics for each player of specified country. Right now I am extracting all players in Input Country and then for each player I am parsing another link that will give me batting stats of that player.
I am using Apache HttpComponents for http requests and JSoup for parsing DOM elements.
Everything is going perfect but the only problem I am facing is when I start scraping two players are being scraped perfectly and then my application hangs.
I've narrowed the problem to a method which grabs a single page, If I provide any link of espncricinfo to this method it is only able to process two requests and no more.
I imagine the problem might be some kind of bot prevention mechanism implemented by espncricinfo. Can anybody help how to bypass this?
Here is code of grab method
public Document scrapSinglePage(String method, String url) {
try {
HttpGet httpGet = new HttpGet(url);
String htmlResponse = "";
HttpResponse httpResponse = httpClient.execute(httpGet, getLocalContext());
BufferedReader rd = new BufferedReader(new InputStreamReader(httpResponse.getEntity().getContent()));
String line = "";
while ((line = rd.readLine()) != null) {
htmlResponse += "\r\n" + line;
}
//Parse response
document = Jsoup.parse(htmlResponse);
return document;
} catch (IOException ex) {
Logger.getLogger(Scrapper.class.getName()).log(Level.SEVERE, null, ex);
return null;
}
}
I will appreciate your help on this.

"StringBuffer sb = new StringBuffer()" get a null value in Android

I use the code below which in my http get request,but what I get from return is a null.I don't know why.
public static String getResponseFromGetUrl(String url) throws Exception {
StringBuffer sb = new StringBuffer();
try {
HttpResponse httpResponse = httpclient.execute(httpRequest);
String inputLine = "";
if (httpResponse.getStatusLine().getStatusCode() == HttpStatus.SC_OK) {
InputStreamReader is = new InputStreamReader(httpResponse
.getEntity().getContent());
BufferedReader in = new BufferedReader(is);
while ((inputLine = in.readLine()) != null) {
sb.append(inputLine);
}
in.close();
}
} catch (Exception e) {
e.printStackTrace();
return "net_error";
} finally {
httpclient.getConnectionManager().shutdown();
}
return sb.toString();
}
And what I have use the function is
String json_str = HttpUtils.getResponseFromGetUrl("www.xxx.com/start");
if ((json_str == null)) Log.d("Chen", "lastestTimestap----" + "json_str == null");
And sometimes the Log will be printed.Not always,in fact like 1%.But I don't know why it caused.

This code will not produce a "null". There must be more code you are not showing.
If this is all the code you have I suggest you remove the StringBuffer and replace it with
return "";
More likely you have forgetten to mention some code which is doing something like
Object o = null;
sb.append(o); // appears as "null"
EDIT: Based on your update, I would have to assume you are reading a line like "null"
It is highly unlikely you want to discard the newline between each line. I suggest either you append("\n") as well or just record all the text you get without regard for new lines.
BTW Please don't use StringBuffer as its replacement StringBuilder has been around for almost ten years. There is a common misconception that using StringBuffer helps with multi-threading but more often it results in incorrect code because it is very harder, if not impossible to use StringBuffer correctly in a multi-threaded context

How can I retrieve a feed in JSON from a Java Servlet?

I want to make an Http request and store the result in a JSONObject. I haven't worked much with servlets, so I am unsure as to whether I am 1) Making the request properly, and 2) supposed to create the JSONObject. I have imported the JSONObject and JSONArray classes, but I don't know where I ought to use them. Here's what I have:
public void doGet(HttpServletRequest req, HttpServletResponse resp)
throws IOException {
//create URL
try {
// With a single string.
URL url = new URL(FEED_URL);
// Read all the text returned by the server
BufferedReader in = new BufferedReader(new InputStreamReader(url.openStream()));
String str;
while ((str = in.readLine()) != null) {
// str is one line of text; readLine() strips the newline character(s)
}
in.close();
} catch (MalformedURLException e) {
}
catch (IOException e) {
}
My FEED_URL is already written so that it will return a feed formatted for JSON.
This has been getting to me for hours. Thank you very much, you guys are an invaluable resource!

First gather the response into a String:
BufferedReader in = new BufferedReader(new InputStreamReader(url.openStream()));
StringBuilder fullResponse = new StringBuilder();
String str;
while ((str = in.readLine()) != null) {
fullResponse.append(str);
}
Then, if the string starts with "{", you can use:
JSONObject obj = new JSONObject(fullResponse.toString()); //[1]
and if it starts with "[", you can use:
JSONArray arr = new JSONArray(fullResponse.toStrin()); //[2]
[1] http://json.org/javadoc/org/json/JSONObject.html#JSONObject%28java.lang.String%29
[2] http://json.org/javadoc/org/json/JSONArray.html#JSONArray%28java.lang.String%29

Firstly, this is actually not a servlet problem. You don't have any problems with javax.servlet API. You just have problems with java.net API and the JSON API.
For parsing and formatting JSON strings, I would recommend to use Gson (Google JSON) instead of the legacy JSON API's. It has much better support for generics and nested properties and can convert a JSON string to a fullworthy javabean in a single call.
I've posted a complete code example before here. Hope you find it useful.