Develop Reference

Illegal characters in XML - java

I'm creating a program which checks the legitimacy of a given URL. I've already created my own algorithm for this, but now I want to add PhishTank's services into my program.
They provide services where you can directly query a URL from their website, but they have set a certain quota on the number of queries you can make per day. The other option, which I'm going with, is to simply download their database and work with it locally, without restrictions.
The file you get is in XML, and found some code to test with, but it seems like their XML contains illegal characters (such as unicode 0x07 -- the [BEL] character) inside CDATA, and so the parsing throws me an exception.
<url><![CDATA[http://shaghaf-edu.com/sign-in/??msg=InvalidOnlineIdException&id[BEL]da9ca9b23227a572d1fb5ff4ff91e3&lpOlbResetErrorCounter=0l=&request_locale=en-us]]></url>
I've done a bit of searching and all I've found is solutions that seem fine to rather small XML-files. The one I'm working with is close to 2.7 million lines -- I'm not sure how efficiently a regex would work in this case or a char-to-char comparison.
I should note that their database is updated hourly, and has to be redownloaded. So cleaning the file once manually isn't an option.
So I'm wondering if there is any fast and efficient way of solving this problem?
I don't have the exact code with me, but I use is a very slight variation of this which I found here on StackOverflow:
private void start() throws Exception
{
URL url = new URL("http://localhost:8080/AutoLogin/resource/web.xml");
URLConnection connection = url.openConnection();
Document doc = parseXML(connection.getInputStream());
NodeList descNodes = doc.getElementsByTagName("description");
for(int i=0; i<descNodes.getLength();i++)
{
System.out.println(descNodes.item(i).getTextContent());
}
}
private Document parseXML(InputStream stream)
throws Exception
{
DocumentBuilderFactory objDocumentBuilderFactory = null;
DocumentBuilder objDocumentBuilder = null;
Document doc = null;
try
{
objDocumentBuilderFactory = DocumentBuilderFactory.newInstance();
objDocumentBuilder = objDocumentBuilderFactory.newDocumentBuilder();
doc = objDocumentBuilder.parse(stream);
}
catch(Exception ex)
{
throw ex;
}
return doc;
}

Answering by asking a question ...
Why not write a simple pre-processing utility?
It could read the XML file as is (line by line); and do whatever is required to turn that content into "correct" XML.
In other words: you should explicitly distinguish between the task of "preparing your input", and "actually working that xml input". This will also make it much easier to do fine tuning. If you find that regular expressions are too expensive; then just change the the "pre-processor" to not use them. And afterwards, easily measure the effects on runtime ...

Extracting elements from an HTTP XML response--HTTP Client & Java

So I've gotten help from here already so I figured why not try it out again!? Any suggestions would be greatly appreciated.
I'm using HTTP client and making a POST request; the response is an XML body that looks like the following:
<?xml version="1.0" encoding="UTF-8"?>
<CartLink
xmlns="http://api.gsicommerce.com/schema/ews/1.0">
<Name>vSisFfYlAPwAAAE_CPBZ3qYh</Name>
<Uri>carts/vSisFfYlAPwAAAE_CPBZ3qYh</Uri>
</CartLink>
Now...
I have an HttpEntity which is
[HttpResponse].getEntity().
Then I get a String representation of the response (which is XML in this case) by saying
String content = EntityUtils.toString(HttpEntity)
I tried following some of the suggestions on this post: How to create a XML object from String in Java? but it did not seem to work for me. When I built up the document it still appeared to be null.
MY END GOAL here is just to get the NAME field.. i.e. the "vSisFfYlAPwAAAE_CPBZ3qYh" part. So do I want to build up a document and then extract it...? Or is there a simpler way? I've been trying different things and I can't seem to get it to work.
Thanks for all of the help guys, it is most appreciated!!

Instead of trying to extract the value with string manipulation, try to use Java's inbuilt ability to parse XML. That's a much better approach. Http Components returns responses in an XML format - there's a reason for that. :)
Here's probably one way to solve your problem:
// Parse the response using DocumentBuilder so you can get at elements easily
DocumentBuilderFactory docBuilderFactory = DocumentBuilderFactory.newInstance();
DocumentBuilder docBuilder = docBuilderFactory.newDocumentBuilder();
Document doc = docBuilder.parse(response);
Element root = doc.getDocumentElement();
// Now let's say you have not one, but 'n' nodes that contain the value
// you're looking for. Use NodeList to get a list of all those nodes and just
// pull out the tag/attribute's value you want.
NodeList nameNodesList = doc.getElementsByTagName("Name");
ArrayList<String> nameValues = null;
// Now iterate through the Nodelist to get the values you want.
for (int i=0; i<nameNodesList.getLength(); i++){
nameValues.add(nameNodesList.item(i).getTextContent());
}
The ArrayList "nameValues" will now hold every single value contained within "Name" tags. You could also create a HashMap to store a key value pair of Nodes and their respective text contents.
Hope this helps.

Android - Parsing XML with XPath

First of all, thanks to all the people who's going to spend a little time on this question.
Second, sorry for my english (not my first language! :D).
Well, here is my problem.
I'm learning Android and I'm making an app which uses a XML file to store some info. I have no problem creating the file, but trying to read de XML tags with XPath (DOM, XMLPullParser, etc. only gave me problems) I've been able to read, at least, the first one.
Let's see the code.
Here is the XML file the app generates:
<dispositivo>
<id>111</id>
<nombre>Name</nombre>
<intervalo>300</intervalo>
</dispositivo>
And here is the function which reads the XML file:
private void leerXML() {
try {
XPathFactory factory=XPathFactory.newInstance();
XPath xPath=factory.newXPath();
// Introducimos XML en memoria
File xmlDocument = new File("/data/data/com.example.gps/files/devloc_cfg.xml");
InputSource inputSource = new InputSource(new FileInputStream(xmlDocument));
// Definimos expresiones para encontrar valor.
XPathExpression tag_id = xPath.compile("/dispositivo/id");
String valor_id = tag_id.evaluate(inputSource);
id=valor_id;
XPathExpression tag_nombre = xPath.compile("/dispositivo/nombre");
String valor_nombre = tag_nombre.evaluate(inputSource);
nombre=valor_nombre;
} catch (Exception e) {
e.printStackTrace();
}
}
The app gets correctly the id value and shows it on the screen ("id" and "nombre" variables are assigned to a TextView each one), but the "nombre" is not working.
What should I change? :)
Thanks for all your time and help. This site is quite helpful!
PD: I've been searching for a response on the whole site but didn't found any.

You're using the same input stream twice, but the second time you use it it's already at the end of file. You have to either open the stream once more or buffer it e.g. in a ByteArrayInputStream and reuse it.
In your case doing this:
inputSource = new InputSource(new FileInputStream(xmlDocument));
before this line
XPathExpression tag_nombre = xPath.compile("/dispositivo/nombre");
should help.
Be aware though that you should properly close your streams.

The problem is that you cannot re-use the stream-input-source multiple times - the first call to tag_id.evaluate(inputSource) already has read the input up to the end.
One solution would be to parse Document in advance:
DocumentBuilderFactory documentBuilderFactory = DocumentBuilderFactory.newInstance();
Document document = documentBuilderFactory.newDocumentBuilder().parse(inputSource);
Source source = new DOMSource(document);
// evalute xpath-expressions on the dom source

string to a dom document in java

I have a string which has an xml inside. I want to write this to a Document in java so i can later retrieve the attributes within it and store them in other strings. Could someone provide me with an example of this please?

Some more details are necessary for a proper answer but you can start with the DocumentBuilder like this:
DocumentBuilder db = DocumentBuilderFactory.newDocumentBuilder()
Document doc = db.parse(yourstring);
You now have a Document from which you can access your XML attributes etc.

Check out http://www.java-tips.org/java-se-tips/javax.xml.parsers/how-to-read-xml-file-in-java.html. Hopefully it will give you a good starting point.

http://www.java-samples.com/showtutorial.php?tutorialid=152
This is a good example of how to parse out some XML. This should be a good start for you.

Document d = DocumentBuilderFactory.newInstance().newDocumentBuilder().parse(new ByteArrayInputStream(myXMLString.getBytes());

how to use Pattern matcher in java?

lets say the string is <title>xyz</title>
I want to extract the xyz out of the string.
I used:
Pattern titlePattern = Pattern.compile("&lttitle&gt\\s*(.+?)\\s*&lt/title&gt");
Matcher titleMatcher = titlePattern.matcher(line);
String title=titleMatcher.group(1));
but I am getting an error for titlePattern.matcher(line);

You say your error occurs earlier (what is the actual error, runs without an error for me), but after solving that you will need to call find() on the matcher once to actually search for the pattern:
if(titleMatcher.find()){
String title = titleMatcher.group(1);
}
Not that if you really match against a string with non-escaped HTML entities like
<title>xyz</title>
Then your regular expression will have to use these, not the escaped entities:
"<title>\\s*(.+?)\\s*</title>"
Also, you should be careful about how far you try to get with this, as you can't really parse HTML or XML with regular expressions. If you are working with XML, it's much easier to use an XML parser, e.g. JDOM.

Not technically an answer but you shouldn't be using regular expressions to parse HTML. You can try and you can get away with it for simple tasks but HTML can get ugly. There are a number of Java libraries that can parse HTML/XML just fine. If you're going to be working a lot with HTML/XML it would be worth your time to learn them.

As others have suggested, it's probably not a good idea to parse HTML/XML with regex. You can parse XML Documents with the standard java API, but I don't recommend it. As Fabian Steeg already answered, it's probably better to use JDOM or a similar open source library for parsing XML.
With javax.xml.parsers you can do the following:
String xml = "<title>abc</title>";
DocumentBuilderFactory docBuilderFactory = DocumentBuilderFactory.newInstance();
DocumentBuilder docBuilder = docBuilderFactory.newDocumentBuilder();
Document doc = docBuilder.parse(new InputSource(new StringReader(xml)));
NodeList nodeList = doc.getElementsByTagName("title");
String title = nodeList.item(0).getTextContent();
This parses your XML string into a Document object which you can use for further lookups. The API is kinda horrible though.
Another way is to use XPath for the lookup:
XPathFactory xpathFactory = XPathFactory.newInstance();
XPath xPath = xpathFactory.newXPath();
String titleByXpath = xPath.evaluate("/title/text()", new InputSource(new StringReader(xml)));
// or use the Document for lookup
String titleFromDomByXpath = xPath.evaluate("/title/text()", doc);