Develop Reference

Get Google Search Result with Java using Jsoup

first of all i search this problem in stackoverflow database and google. Unfortunately i couldn't find a solution.
I am trying to get Google Search Result for a keyword. Heres my code :
public static void main(String[] args) throws Exception {
Document doc;
try{
doc = Jsoup.connect("https://www.google.com/search?as_q=&as_epq=%22Yorkshire+Capital%22+&as_oq=fraud+OR+allegations+OR+scam&as_eq=&as_nlo=&as_nhi=&lr=lang_en&cr=countryCA&as_qdr=all&as_sitesearch=&as_occt=any&safe=images&tbs=&as_filetype=&as_rights=").userAgent("Mozilla").ignoreHttpErrors(true).timeout(0).get();
Elements links = (Elements) doc.select("li[class=g]");
for (Element link : links) {
Elements titles = link.select("h3[class=r]");
String title = titles.text();
Elements bodies = link.select("span[class=st]");
String body = bodies.text();
System.out.println("Title: "+title);
System.out.println("Body: "+body+"\n");
}
}
catch (IOException e) {
e.printStackTrace();
}
}
And heres the errors : https://prnt.sc/ro4ooi
It says : can only iterate over an array or an instance of java.lang.iterable ( at links )..
When i delete the (Elements) : https://prnt.sc/ro4pa9
Thank you.

How to get unique value of repeated nodes using dom parser

I have a XML having repeated nodes and I have top parse it using DOM parser. After a lot R&D I could find anything on internet which can help me. My xml looks like
<nos1>
<Name>aqwer</Name>
<class>sas</class>
<class>xcd</class>
<class>asd</class>
<Name>cfg</Name>
<Name>cfg</Name>
<nos1>
Any suggestion How can I parse this xml for repeated values.

You can use w3c dom document to parse your XML as follows:
DocumentBuilderFactory df = DocumentBuilderFactory.newInstance();
try
{
DocumentBuilder db = df.newDocumentBuilder();
InputStream is = new ByteArrayInputStream(response.getContent().getBytes("UTF-8"));
org.w3c.dom.Document doc = db.parse(is);
NodeList links = doc.getElementsByTagName("class");
for(int i=0; i< links.getLength(); i++)
{
Node link = links.item(i);
System.out.println(link.getTextContent());
}
}
catch(Exception ex)
{
}
Hope this helps you.

You should read all elements and after reading eliminate the duplicates via a Set. Here is an example using XMLBeam, but any other library will do.
public class TestMultipleElements {
#XBDocURL("resource://test.xml")
public interface Projection {
#XBRead("/nos1/Name")
List<String> getNames();
#XBRead("/nos1/class")
List<String> getClasses();
}
#Test
public void uniqueElements() throws IOException {
Projection projection = new XBProjector().io().fromURLAnnotation(Projection.class);
for (String name : new HashSet<String>(projection.getNames())) {
System.out.println("Found Name:" + name);
}
for (String clazz : new HashSet<String>(projection.getClasses())) {
System.out.println("Found Name:" + clazz);
}
}
}
This prints out:
Found Name:aqwer
Found Name:cfg
Found Name:xcd
Found Name:sas
Found Name:asd

Java XML creates closing xml statement but no opening

I have a program that creates an xml doc.
the filename is unimportant here because the file does get created successfully
the arraylist of entries contains a Unique identifier and a hashmap of
elements + values. the elements are as follows: world, name, location, type and data
all these values are strings and the only one that would ever be blank/null is data
my problem is that the xml file adds all the fields as required with the exception
of the data field. it leaves me with an unopened node . actual result:
<NPC>
<NPC:0>
<name>
the_name
</name>
<data/> <---- this line should have the string "null"
<loc>
2529.1294962948955:
69.0:
951.2612160649056
</loc>
<type>
Quest
</type>
<world>
world
</world>
</NPC:0>
</NPC>
My method for creating the xml file.
public void updateXML(String fileName, ArrayList<XMLEntry> entries)
{
File file = getFileByName(fileName);
try {
DocumentBuilderFactory bFac = DocumentBuilderFactory.newInstance();
DocumentBuilder b = bFac.newDocumentBuilder();
Document doc = b.parse(file);
for(int i = 0; i < entries.size(); i++)
{
XMLEntry entry = entries.get(i);
Node entry_node = doc.getElementsByTagName(entry.getName()).item(0);
if(entry_node == null)
{
Element node = doc.createElement(entry.getName());
doc.getFirstChild().appendChild(node);
entry_node = doc.getElementsByTagName(entry.getName()).item(0);
}
for (Map.Entry<String, String> attributes : entry.getAttributes().entrySet())
{
NamedNodeMap xml_attributes = entry_node.getAttributes();
Node attribute = xml_attributes.getNamedItem(attributes.getKey());
if(attribute == null)
{
if(attributes.getValue() != "" || attributes.getValue() != null)
{
Element new_xml_attribute = doc.createElement(attributes.getKey());
new_xml_attribute.appendChild(doc.createTextNode(attributes.getValue()));
entry_node.appendChild(new_xml_attribute);
} else {
Element new_xml_attribute = doc.createElement(attributes.getKey());
new_xml_attribute.appendChild(doc.createTextNode("null"));
entry_node.appendChild(new_xml_attribute);
}
} else {
attribute.setTextContent(attributes.getValue());
}
TransformerFactory tFac = TransformerFactory.newInstance();
Transformer ts = tFac.newTransformer();
DOMSource src = new DOMSource(doc);
StreamResult result = new StreamResult(file);
ts.transform(src, result);
}
}
} catch (ParserConfigurationException e) {
} catch (TransformerException e1) {
} catch (IOException e2) {
} catch (SAXException e3) {
}
}

<data/> <---- this line should have the string "null"
That isn't an XML close-element tag (which would be </data>). It's an XML empty-element tag, which combines open and close into a single piece of markup. It is semantically identical to <data></data>.
Despite your expectations, it would appear that the empty <data/> element is not being created by the path with the literal "null". Drop a printout into that code, or run it in the debugger, to confirm this. Then use the debugger, or drop in additional printouts as necessary, to figure out why.

Parse some elements from a xml

i want to know if is possible to me to parse some atributes from a xml file, to be a object in java
I don´t wanna to create all fields that are in xml.
So, how can i do this?
For exemple below there is a xml file, and i want only the data inside the tag .
<emit>
<CNPJ>1109</CNPJ>
<xNome>OESTE</xNome>
<xFant>ABATEDOURO</xFant>
<enderEmit>
<xLgr>RODOVIA</xLgr>
<nro>S/N</nro>
<xCpl>402</xCpl>
<xBairro>GOMES</xBairro>
<cMun>314</cMun>
<xMun>MINAS</xMun>
<UF>MG</UF>
<CEP>35661470</CEP>
<cPais>58</cPais>
<xPais>Brasil</xPais>
<fone>03</fone>
</enderEmit>
<IE>20659</IE>
<CRT>3</CRT>

For Java XML parsing where you don't have the XSD and don't want to create a complete object graph to represent the XML, JDOM is a great tool. It allows you to easily walk the XML tree and pick the elements you are interested in.
Here's some sample code that uses JDOM to pick arbitrary values from the XML doc:
// reading can be done using any of the two 'DOM' or 'SAX' parser
// we have used saxBuilder object here
// please note that this saxBuilder is not internal sax from jdk
SAXBuilder saxBuilder = new SAXBuilder();
// obtain file object
File file = new File("/tmp/emit.xml");
try {
// converted file to document object
Document document = saxBuilder.build(file);
//You don't need this or the ns parameters in getChild()
//if your XML document has no namespace
Namespace ns = Namespace.getNamespace("http://www.example.com/namespace");
// get root node from xml. emit in your sample doc?
Element rootNode = document.getRootElement();
//getChild() assumes one and only one, enderEmit element. Use a lib and error
//checking as needed for your document
Element enderEmitElement = rootNode.getChild("enderEmit", ns);
//now we get two of the child from
Element xCplElement = enderEmitElement.getChild("xCpl", ns);
//should be 402 in your example
String xCplValue = xCplElement.getText();
System.out.println("xCpl: " + xCplValue);
Element cMunElement = enderEmitElement.getChild("cMun", ns);
//should be 314 in your example
String cMunValue = cMunElement.getText();
System.out.println("cMun: " + cMunValue);
} catch (JDOMException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}

You can use JAXB to unmarshal the xml into Java object, with which you can read selective elements easily. With JAXB, the given XML can be represented in Java as follows :
enderEmit element :
#XmlRootElement
public class EnderEmit{
private String xLgr;
//Other elements.Here you can define properties for only those elements that you want to load
}
emit element (This represents your XML file):
#XmlRootElement
public class Emit{
private String cnpj;
private String xnom;
private EnderEmit enderEmit;
..
//Add elements that you want to load
}
Now by using the below lines of code, you can read your xml to an object :
String filePath="filePath";
File file = new File(filePath);
JAXBContext jaxbContext = JAXBContext.newInstance(Emit.class);
jaxbUnmarshaller = jaxbContext.createUnmarshaller();
Emit emit = (Emit) jaxbUnmarshaller.unmarshal(file);
The line will give you an emit object for the given xml.

Try to use StringUtils.subStringBetween
try
{
String input = "";
br = new BufferedReader(new FileReader(FILEPATH));
String result = null;
while ((input = br.readLine()) != null) // here we read the file line by line
{
result = StringUtils.substringBetween(input, ">", "<"); // using StringUtils.subStringBetween to get the data what you want
if(result != null) // if the result should not be null because some of the line not having the tags
{
System.out.println(""+result);
}
}
}
catch (IOException e)
{
e.printStackTrace();
}
finally
{
try
{
if (br != null)
{
br.close();
}
}
catch (IOException ex)
{
ex.printStackTrace();
}
}

Android: Parsing XML DOM parser. Converting childnodes to string

Again a question. This time I'm parsing XML messages I receive from a server.
Someone thought to be smart and decided to place HTML pages in a XML message. Now I'm kind of facing problems because I want to extract that HTML page as a string from this XML message.
Ok this is the XML message I'm parsing:
<AmigoRequest>
<From></From>
<To></To>
<MessageType>showMessage</MessageType>
<Param0>general message</Param0>
<Param1><html><head>test</head><body>Testhtml</body></html></Param1>
</AmigoRequest>
You see that in Param1 a HTML page is specified. I've tried to extract the message the following way:
public String getParam1(Document d) {
if (d.getDocumentElement().getTagName().equals("AmigoRequest")) {
NodeList results = d.getElementsByTagName("Param1");
// Messagetype depends on what message we are reading.
if (results.getLength() > 0 && results != null) {
return results.item(0).getFirstChild().getNodeValue();
}
}
return "";
}
Where d is the XML message in document form.
It always returns me a null value, because getNodeValue() returns null.
When i try results.item(0).getFirstChild().hasChildNodes() it will return true because he sees there is a tag in the message.
How can i extract the html message <html><head>test</head><body>Testhtml</body></html> from Param0 in a string?
I'm using Android sdk 1.5 (well almost java) and a DOM Parser.
Thanks for your time and replies.
Antek

You could take the content of param1, like this:
public String getParam1(Document d) {
if (d.getDocumentElement().getTagName().equals("AmigoRequest")) {
NodeList results = d.getElementsByTagName("Param1");
// Messagetype depends on what message we are reading.
if (results.getLength() > 0 && results != null) {
// String extractHTMLTags(String s) is a function that you have
// to implement in a way that will extract all the HTML tags inside a string.
return extractHTMLTags(results.item(0).getTextContent());
}
}
return "";
}
All you have to do is to implement a function:
String extractHTMLTags(String s)
that will remove all HTML tag occurrences from a string.
For that you can take a look at this post: Remove HTML tags from a String

after checking a lot and scratching my head thousands of times I came up with simple alteration that it needs to change your API level to 8

EDIT: I just saw your comment above about getTextContent() not being supported on Android. I'm going to leave this answer up in case it's useful to someone who's on a different platform.
If your DOM API supports it, you can call getTextContent(), as follows:
public String getParam1(Document d) {
if (d.getDocumentElement().getTagName().equals("AmigoRequest")) {
NodeList results = d.getElementsByTagName("Param1");
// Messagetype depends on what message we are reading.
if (results != null) {
return results.getTextContent();
}
}
return "";
}
However, getTextContent() is a DOM Level 3 API call; not all parsers are guaranteed to support it. Xerces-J does.
By the way, in your original example, your check for null is in the wrong place; it should be:
if (results != null && results.getLength() > 0) {
Otherwise, you'd get a NPE if results really does come back as null.

Since getTextContent() isn't available to you, another option would be to write it -- it isn't hard. In fact, if you're writing this solely for your own use -- or your employer doesn't have overly strict rules about open source -- you could look at Apache's implementation as a starting point; lines 610-646 seem to contain most of what you need. (Please be respectful of Apache's copyright and license.)
Otherwise, some rough pseudocode for the method would be:
String getTextContent(Node node) {
if (node has no children)
return "";
if (node has 1 child)
return getTextContent(node.getFirstChild());
return getTextContent(new StringBuffer()).toString();
}
StringBuffer getTextContent(Node node, StringBuffer sb) {
for each child of node {
if (child is a text node) sb.append(child's text)
else getTextContent(child, sb);
}
return sb;
}

Well i was almost there with the code...
public String getParam1(Document d) {
if (d.getDocumentElement().getTagName().equals("AmigoRequest")) {
NodeList results = d.getElementsByTagName("Param1");
// Messagetype depends on what message we are reading.
if (results.getLength() > 0 && results != null) {
DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();
DocumentBuilder db;
Element node = (Element) results.item(0); // get the value of Param1
Document doc2 = null;
try {
db = dbf.newDocumentBuilder();
doc2 = db.newDocument(); //create new document
doc2.appendChild(doc2.importNode(node, true)); //import the <html>...</html> result in doc2
} catch (ParserConfigurationException e) {
// TODO Auto-generated catch block
Log.d(TAG, " Exception ", e);
} catch (DOMException e) {
// TODO: handle exception
Log.d(TAG, " Exception ", e);
} catch (Exception e) {
// TODO: handle exception
e.printStackTrace(); }
return doc2. .....// All I'm missing is something to convert a Document to a string.
}
}
return "";
}
Like explained in the comment of my code. All I am missing is to make a String out of a Document. You can't use the Transform class in Android... doc2.toString() will give you a serialization of the object..
But my next step is write my own parser if this doesnt work out ;)
Not the best code but a temponary solution.
public String getParam1(String b) {
return b
.substring(b.indexOf("<Param1>") + "<Param1>".length(), b.indexOf("</Param1>"));
}
Where String b is the XML document string.