I am reading from an API provided by a company, but the problem is that one of the accounts from which I am getting the data has around 22000 json objects, it reads fine with small amounts of data, i would say up to 8000 records, but then I get issues like the json is not well formatted besides the problem of being able to read the response.
The response comes this way:
<?xml version="1.0" encoding="utf-8"?>
<string xmlns="http://ywers.com">
[{"Name":"Edward", "LastName":"Jones", "Address":"{accepted}"}
,{"Name":"Carlos", "LastName":"Ramirez", "Address":"{Rejected}"}, ....... 22k more records here]</string>
I asked for some help earlier on here for the best way to do this, and i got a response about reading it using the xml parser and then a json parser, i am using GSON.
String XML = "<Your XML Response>";
XPathExpression xpath = XPathFactory.newInstance()
.newXPath().compile("/*[local-name()='string']");
String json = xpath.evaluate(new InputSource(new StringReader(XML)));
and then
JSONArray jsonRoot = new JSONArray(json.trim());
System.out.println(jsonRoot.getJSONObject(0).getString("Address")); // {accepted}
The problem with this is approach i am having is that it throws errors when reading the XML, it starts reading but after a while it stops with errors like:
java.lang.OutOfMemoryError
at java.lang.AbstractStringBuilder.enlargeBuffer(AbstractBuilder.java:94)
at java.lang.StringBuffer.append(StringBuffer.java:219)
at org.apache.harmony.xml.dom.CharacterDataImpl.appendData(CharacterDataImpl.java:43)
......
I would appreciate any advise on how to proceed with this, I am kind of new to android.
I don't know who would wrap 22k objects inside a xml string, but apparently someone is doing that. From my experience, your out of memory is because the you try to convert all the response to string but the response is too big to be handled. I recommend you to stream the JSON data. You can do stream the JSON data from the inputstream response that you get from the your HTTP post, but you need to skip the XML part by creating another input stream from the original response input stream and skip the XML part
Before I use the streaming API from google GSON I also got OOM error because the JSON data I got is very big data (many images and sounds in Base64 encoding) but with GSON streaming I can overcome that error because it reads the data per token not all at once. And for alternative you can also use Jackson JSON library I think it also have streaming API and how to use it almost same with my implementation with google GSON. I hope my answer can help you and if you have another question about my answer feel free to ask in the comment :)
Related
I'm calling a soap webservice from my java application.
I get response and I want to parse it and get data.
The problem is that field <tranData>, contains structure with >< instead of <>. How can I parse this document to get data from field <tranData>?
This is response structure:
<response>
<Portfolio>
<ID>1</ID>
<holder>2</holder>
</Portfolio>
<tranData> <responseOne><header><code>1</code></header></responseOne></tranData>
Please remember that, this is only a example of response, and the amount of data will be much bigger, so the solution should be fast.
What you show us is the actual document as it is received over the wire, right? So <tranData> contains an XML string that has been escaped to not interfere with the markup of the rest of the containing document.
When you read the content of the <tranData> element, the XML processor will 'unescape' the string and give you the 'original' value:
<responseOne><header><code>1</code></header></responseOne>
What you do with that value is a different story. You can parse it as yet another XML document and retrieve the value of the <code> element, or just pass the string along to some other processing step.
We have used org.json library to convert xml data to json data
the code is very simple like below
JSONObject obj = XML.toJSONObject(xml);
In some cases it throws exception as
org.json.JSONException: JSON does not allow non-finite numbers.
We are processing huge amount of data at a time. there is more above ten thousand xml data are converting to json data. But a 10 or 15 are failing and throws the exception I mentioned above.
But when I took that xml file which throws the exception and run it as a single data it get succeed. We cannot figure out why is this happenning. If anyone used this library please give some advices.
I am using Spring security oauth2. By default oauth2 returns it's own error format like {error : "Invalid_grant", error_description : "something"}. I want to change it my own custom format so in my application, it remains consistent. Can anyone please help me? I have gone through lots of links but didn't find any suitable solution till now.
What you get as a result is a JSON document.
Look at Jackson or Gson libraries for example to parse(deserialize) JSON documents. You can get data values 1 by 1 or deserialize into a class instance.
Once you parse modify the data as you wish
Use the same library Jackson or Gson to write(serialize) a new JSON document.
Jackson may also produce as output XML, YAML and CSV documents
https://github.com/FasterXML/jackson-core
https://github.com/google/gson
I really need help to extract Mircodata which is embedded in HTML5. My purpose is to get structured data from a webpage just like this tool of google: http://www.google.com/webmasters/tools/richsnippets. I have searched a lot but there is no possible solution.
Currently, I use the any23 library but I can’t find any documentation, just only javadocs which dont provide enough information for me.
I use any23's Microdata Extractor but getting stuck at the third parameter: "org.w3c.dom.Document in". I can't parse a HTML content to be a w3cDom. I have used JTidy as well as JSoup but the DOM objects in these library are not fixed with the Extractor constructor. In addition, I also doubt about the 2nd parameter of the Microdata Extractor.
I hope that anyone can help me to do with any23 or suggest another library can solve this extraction issues.
Edit: I found solution myself by using the same way as any23 command line tool did. Here is the snippet of code:
HTTPDocumentSource doc = new HTTPDocumentSource(DefaultHTTPClient.createInitializedHTTPClient(), value);
InputStream documentInputInputStream = doc.openInputStream();
TagSoupParser tagSoupParser = new TagSoupParser(documentInputInputStream, doc.getDocumentURI());
Document document = tagSoupParser.getDOM();
ByteArrayOutputStream byteArrayOutput = new ByteArrayOutputStream();
MicrodataParser.getMicrodataAsJSON(tagSoupParser.getDOM(),new PrintStream(byteArrayOutput));
String result = byteArrayOutput.toString("UTF-8");
These line of code only extract microdata from HTML and write them in JSON format. I tried to use MicrodataExtractor which can change the output format to others(Rdf, turtle, ...) but the input document seems to only accept XML format. It throws "Document didn't start" when I put in a HTML document.
If anyone found the way to use MicrodataExtractor, please leave the answer here.
Thank you.
xpath is generally the way to consume html or xml.
have a look at: How to read XML using XPath in Java
I am having a really challenging time parsing some XML data returned to my Android app.
The data is sent as XML but printing it on my mobile screen, it comes out as the following:
{"sessid":"5eed0b52c6953b52e262b559b5557be4","session_name":"SESS6cbf091341a26e4687fa7850b465755a,"user":{"uid":"15","name":"guest","pass":"084e0343a0486ff05530df6c705c8bb4","mail":"adeoduye#hotmail.com", "mode":"0","sort":"0","threshold":"0","theme":"","signature":"","signature_format":"0","created":"1306008217","access":"1306094503","login":"1306134979","status":"1","timezone":"3600","language":"","picture":"","init":"adeoduye#hotmail.com","data":a:1:{s:13:\"form_build_id\";s:37:\"form-49ea7a4ef10a8a2b31478696f17e8dee\";","form_build_id":"form-49ea7a4ef10a8a2b31478696f17e8dee","roles":{"2":"authenticated user","3":"guest"}}}
Can anyone please help a newbie and give me some ideas on how to parse this type of output and/or plain XML?
This isn't XML but JSON. You have to parse that string using the JSON API.
Basically you create a JSONObject by feeding the string into a JSONTokenizer. You can now query the values from the JSONObject as described in the API reference example.
The String you're seeing here is in JSON format. You can parse this in Andriod using the following library : http://code.google.com/p/google-gson/
For more info on json, checkout http://json.org.