Develop Reference

Get the HTML page using htmlunit

I am trying to get the HTML page of a website (ex http://htmlunit.sourceforge.net) but I get an error of IlleagalArgumentException: Cannot locate declared field class org.apache.http.impl.client.HttpClientBuilder.dnsResolver. My code is as follow:
public class Main1 {
public static void main(String[] args) {
try {
homePage();
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
public static void homePage() throws Exception {
try (final WebClient webClient = new WebClient()) {
final HtmlPage page = webClient.getPage("http://www.google.com");
String text = page.asText();
System.out.println(text);
}
}
}
Is there something wrong with the code? Thanks

It's counter-intuitive but we can use asXml() on HtmlPage or HtmlElement to get it as HTML/XML representation.
page.asXml()
The way you wrote the code, it will return a text representation for what would be shown to a used on browser.
May you need to add this to enable JavaScript:
webClient.options.setJavaScriptEnabled(true)

IlleagalArgumentException: Cannot locate declared field class org.apache.http.impl.client.HttpClientBuilder.dnsResolver
This looks like a wrong version of the HttpClient dependency. Please check your classpath to have only one (and only the correct) version of every dependency.
For the current version you can finde a list of dependencies here http://htmlunit.sourceforge.net/dependencies.html

You can use jsoup parser.
Little code sample
Document doc = Jsoup.connect("http://en.wikipedia.org/").get();
Elements newsHeadlines = doc.select("#mp-itn b a");
Advanced Usage
File input = new File("/tmp/input.html");
Document doc = Jsoup.parse(input, "UTF-8", "http://example.com/");
Element content = doc.getElementById("content");
Elements links = content.getElementsByTag("a");
for (Element link : links) {
String linkHref = link.attr("href");
String linkText = link.text();
}
Helpful URLs
Dom Navigation
Extracting
Working with URLs

PDFBox : Extraction of data from table

How to extract data from a table in a pdf using pdfbox?
In this process, Index of Text and contents can be found using PDContentStream and PageStripper classes.Index of lines in the table have to be found, Can anyone help with which class to extend and which method to implement?
I have tried the following for extracting the start index of texts:
public class Tables {
public static void main(String args[]) throws IOException{
BufferedWriter wr;
File input = new File("test.pdf");
File output = new File("SampleText.txt");
PDDocument pd=new PDDocument();
pd=PDDocument.load(input);
// PDFTextStripper pds=new PDFTextStripper();
// String text=pds.getText(pd);
PDFTextStripper stripper = new PDFTextStripper()
{
#Override
protected void startPage(PDPage page) throws IOException
{
startOfLine = true;
super.startPage(page);
}
#Override
protected void writeLineSeparator() throws IOException
{
startOfLine = true;
super.writeLineSeparator();
}
#Override
protected void writeString(String text, List<TextPosition> textPositions) throws IOException
{
if (startOfLine)
{
TextPosition firstProsition = textPositions.get(0);
writeString(String.format("[%s]", firstProsition.getYDirAdj()));
startOfLine = false;
}
super.writeString(text, textPositions);
}
boolean startOfLine = true;
};
wr = new BufferedWriter(new OutputStreamWriter(new FileOutputStream(output)));
stripper.writeText(pd, wr);
if (pd != null) {
pd.close();
}
// I use close() to flush the stream.
wr.close();
}
}

Recently I did a similar project where I had to extract data from tables.
You have two options here:-
1) You can use tabula (It is an open source tool for extracting tables from pdf). http://tabula.technology/
https://github.com/tabulapdf/tabula
You can use tabula command line tool in your code and extract the data from a specific region.
2) You need to devise your own algorithm for extracting the tabular data.
If you are going to go for the second option then you would need to extract coordinates of the text also. You can override writestring method of pdfTextStripper class (you can google about this). Then you need to think on how to use those information to get the details you need. (Co-ordinates can be very helpful).
If you have the pdf in a standard format then I suggest you to use tabula as there is not much to be do.

fop render to pdf one page out of many

I have a code which runs apache fop against xml content and xsl markup and gives me the apache Intermediate Format output:
StreamSource contentSource = new StreamSource(xmlContentStream);
StreamSource transformSource = new StreamSource(xslMarkupStream);
ByteArrayOutputStream outStream = new ByteArrayOutputStream();
Transformer xslfoTransformer = getTransformer(transformSource);
FOUserAgent foUserAgent = fopFactory.newFOUserAgent();
IFDocumentHandler targetHandler = foUserAgent.getRendererFactory().createDocumentHandler(
foUserAgent, MimeConstants.MIME_PDF);
FPSIFSerializer fpsSerializer = new FPSIFSerializer();
fpsSerializer.setContext(new IFContext(foUserAgent));
fpsSerializer.mimicDocumentHandler(targetHandler);
foUserAgent.setDocumentHandlerOverride(fpsSerializer);
Fop fop = fopFactory.newFop("application/X-fop-intermediate-format", foUserAgent, outStream);
DefaultHandler defaultHandler = fop.getDefaultHandler();
Result res = new SAXResult(defaultHandler);
xslfoTransformer.transform(contentSource, res);
Then I use that Intermediate Format file to render pdf and png files out of it.
I'm able to set up my own serilaizer here (FPSIFSerializer()).
I have several pages reports, but I don't need to process all of them. Is there any way to skip some pages or extract them from IntermediateFormat so I will be able e.g. to render only 1st page as png and then 2nd to pdf, etc ?
There
http://svn.apache.org/viewvc/xmlgraphics/fop/branches/archive/fop-1_1/examples/embedding/java/embedding/intermediate/ExampleConcat.java?view=markup
is an example of how to concatenate files via IFConcatenator, so I wonder about the best way to split the multipage file?
Thank_you!

The way I've done it is using custom document handler.
/**
* Custom Apache FOP Intermediate Format document handler which allows page skipping.
* Not thread safe.
*/
public class IFPageFilter extends IFDocumentHandlerProxy {
private static final Logger LOGGER = LoggerFactory.getLogger(IFPageFilter.class);
private int currentPage;
private final int desiredPage;
/**
* #param delegate The real document handler
* #param desiredPage the page you want to render (1-based). Other pages will be skipped.
*/
public IFPageFilter(final IFDocumentHandler delegate, final int desiredPage) {
super(delegate);
this.desiredPage = desiredPage;
}
#Override
public void startPage(final int index, final String name, final String pageMasterName, final Dimension size) throws IFException {
currentPage = index + 1;
if (currentPage == desiredPage) {
super.startPage(index, name, pageMasterName, size);
} else {
// do nothing
LOGGER.debug("Page skipped");
}
}
#Override
public IFPainter startPageContent() throws IFException {
if (currentPage == desiredPage) {
return super.startPageContent();
} else {
return EmptyPainter.getInstance();
}
}
#Override
public void endPageContent() throws IFException {
if (currentPage == desiredPage) {
super.endPageContent();
}
}
}
Then you can attach your handler like that:
final IFDocumentHandler targetHandler = FOP_FACTORY.getRendererFactory().createDocumentHandler(userAgent, mime);
final IFPageFilter documentHandler = new IFPageFilter(targetHandler, page);
final ByteArrayOutputStream mimeOut = new ByteArrayOutputStream(XSL_STREAM_BUFFER_SIZE);
IFUtil.setupFonts(documentHandler);
// Tell the target handler where to write the PDF to
targetHandler.setResult(new StreamResult(mimeOut));
try (final InputStream is = ifStream.toInputStream()) {
final Source src = new StreamSource(is);
new IFParser().parse(src, documentHandler, userAgent);
}
return mimeOut;
and you will get the only page you need in the output stream.
Class EmptyPainter is a dirty hack. It is empty implementation of apache IFPainter, it used here to skip page content and avoid NPE. I'm not happy about it, but that is the only way I was able to make it work.
Please note that I use FOP 1.1, and if you faced with such problems it worth to look at trunk - some of them already solved there. I guess dirty hack with EmptyPainter will not be necessary in trunk.
Please give tips if something could be done better here.
Thanks

How do I edit a XML node in a file object, using Java

There are a lot of examples on the internet of "reading" files but I can't find anything on "editing" a node value and writing it back out to the original file.
I have a non-working xml writer class that looks like this:
import org.w3c.dom.Document;
public class RunIt {
public static Document xmlDocument;
public static void main(String[] args)
throws TransformerException, IOException {
try {
xmlDocument = DocumentBuilderFactory.newInstance()
.newDocumentBuilder().parse("thor.xml");
} catch (IOException ex) {
ex.printStackTrace();
} catch (SAXException ex) {
ex.printStackTrace();
} catch (ParserConfigurationException ex) {
ex.printStackTrace();
}
addElement("A", "New");
writeDoc();
}
public static void addElement(String path, String val){
Element e = xmlDocument.createElement(path);
e.appendChild(xmlDocument.createTextNode(val));
xmlDocument.getDocumentElement().appendChild(e);
}
public static void writeDoc() throws TransformerException, IOException {
StringWriter writer = new StringWriter();
Transformer tf;
try {
tf = TransformerFactory.newInstance().newTransformer();
tf.transform(new DOMSource(xmlDocument), new StreamResult(writer));
writer.close();
} catch (TransformerConfigurationException e) {
e.printStackTrace();
} catch (TransformerFactoryConfigurationError e) {
e.printStackTrace();
}
}
}
For this example, lets say this is the XML and I want to add a "C" node (inside the A node) that contains the value "New" :
<A>
<B>Original</B>
</A>

You use the Document object to create new nodes. Adding nodes as you suggest involves creating a node, setting its content and then appending it to the root element. In this case your code would look somehting like this:
Element e = xmlDocument.createElement("C");
e.appendChild(xmlDocument.createTextNode("new"));
xmlDocument.getDocumentElement().appendChild(e);
This will add the C node as a new child of A right after the B node.
Additionally, Element has some convenience functions that reduce the amount of required code. The second line above could have been replaced with
e.setTextContent("new");
More complicated efforts involving non root elements will involve you using XPath to fetch the target node to be edited. If you do start to use XPath to target nodes, bear in mind that the JDK XPath performance is abysmal. Avoid using an XPath of "#foo" in favor of constructs like e.getAttribute("foo") whenever you can.
EDIT: Formatting the document back to a string which can be written to a file can be done with the following code.
Document xmlDocument;
StringWriter writer = new StringWriter();
TransformerFactory.newInstance().transform(new DOMSource(xmlDocument), new StreamResult(writer));
writer.close();
String xmlString = writer.toString();
EDIT: Re: updated question with code.
Your code doesn't work because you're conflating 'path' and 'element name'. The parameter to Document.createElement() is the name of the new node, not the location in which to place it. In the example code I wrote I didn't get into locating the appropriate node because you were asking specifically about adding a node to the document parent element. If you want your addElement() to behave the way I think you're expecting it to behave, you'd have to add another parameter for the xpath of the target parent node.
The other problem with your code is that your writeDoc() function doesn't have any output. My example shows writing the XML to a String value. You can write it to any writer you want by adapting the code, but in your example code you use a StringWriter but never extract the written string out of it.
I would suggest rewriting your code something like this
public static void main(String[] args) {
File xmlFile = new File("thor.xml");
Document xmlDocument = DocumentBuilderFactory.newInstance()
.newDocumentBuilder().parse(xmlFile);
// this is effective because we know we're adding to the
// document root element
// if you want to write to an arbitrary node, you must
// include code to find that node
addTextElement(xmlDocument.getDocumentElement(), "C", "New");
writeDoc(new FileWriter(xmlFile);
}
public static Element addTextElement(Node parent, String element, String val){
Element e = addElement(parent, element)
e.appendChild(xmlDocument.createTextNode(val));
return e;
}
public static Element addElement(Node parent, String element){
Element e = xmlDocument.createElement(path);
parent.appendChild(e);
return e;
}
public static void writeDoc(Writer writer) {
try {
Transformer tf = TransformerFactory.newInstance().newTransformer();
tf.transform(new DOMSource(xmlDocument), new StreamResult(writer));
} finally {
writer.close();
}
}

In order to write your document back to a file, you'll need an XML serializer or write your own. If you are using the Xerces library, check out XMLSerializer. For sample usage, you can also check out the DOMWriter samples page.
For more information on Xerces, read this

How can I get Axis 1.4 to not generate several prefixes for the same XML namespace?

I am receiving SOAP requests from a client that uses the Axis 1.4 libraries. The requests have the following form:
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<soapenv:Body>
<PlaceOrderRequest xmlns="http://example.com/schema/order/request">
<order>
<ns1:requestParameter xmlns:ns1="http://example.com/schema/common/request">
<ns1:orderingSystemWithDomain>
<ns1:orderingSystem>Internet</ns1:orderingSystem>
<ns1:domainSign>2</ns1:domainSign>
</ns1:orderingSystemWithDomain>
</ns1:requestParameter>
<ns2:directDeliveryAddress ns2:addressType="0" ns2:index="1"
xmlns:ns2="http://example.com/schema/order/request">
<ns3:address xmlns:ns3="http://example.com/schema/common/request">
<ns4:zipcode xmlns:ns4="http://example.com/schema/common">12345</ns4:zipcode>
<ns5:city xmlns:ns5="http://example.com/schema/common">City</ns5:city>
<ns6:street xmlns:ns6="http://example.com/schema/common">Street</ns6:street>
<ns7:houseNum xmlns:ns7="http://example.com/schema/common">1</ns7:houseNum>
<ns8:country xmlns:ns8="http://example.com/schema/common">XX</ns8:country>
</ns3:address>
[...]
As you can see, several prefixes are defined for the same namespace, e.g. the namespace http://example.com/schema/common has the prefixes ns4, ns5, ns6, ns7 and ns8. Some long requests define several hundred prefixes for the same namespace.
This causes a problem with the Saxon XSLT processor, that I use to transform the requests. Saxon limits the the number of different prefixes for the same namespace to 255 and throws an exception when you define more prefixes.
Can Axis 1.4 be configured to define smarter prefixes, so that there is only one prefix for each namespace?

I have the same issue. For the moment, I've worked around it by writing a BasicHandler extension, and then walking the SOAPPart myself and moving the namespace reference up to a parent node. I don't like this solution, but it does seem to work.
I really hope somebody comes along and tells us what we have to do.
EDIT
This is way too complicated, and like I said, I don't like it at all, but here we go. I actually broke the functionality into a few classes (This wasn't the only manipulation that we needed to do in that project, so there were other implementations) I really hope that somebody can fix this soon. This uses dom4j to process the XML passing through the SOAP process, so you'll need dom4j to make it work.
public class XMLManipulationHandler extends BasicHandler {
private static Log log = LogFactory.getLog(XMLManipulationHandler.class);
private static List processingHandlers;
public static void setProcessingHandlers(List handlers) {
processingHandlers = handlers;
}
protected Document process(Document doc) {
if (processingHandlers == null) {
processingHandlers = new ArrayList();
processingHandlers.add(new EmptyProcessingHandler());
}
log.trace(processingHandlers);
treeWalk(doc.getRootElement());
return doc;
}
protected void treeWalk(Element element) {
for (int i = 0, size = element.nodeCount(); i < size; i++) {
Node node = element.node(i);
for (int handlerIndex = 0; handlerIndex < processingHandlers.size(); handlerIndex++) {
ProcessingHandler handler = (ProcessingHandler) processingHandlers.get(handlerIndex);
handler.process(node);
}
if (node instanceof Element) {
treeWalk((Element) node);
}
}
}
public void invoke(MessageContext context) throws AxisFault {
if (!context.getPastPivot()) {
SOAPMessage message = context.getMessage();
SOAPPart soapPart = message.getSOAPPart();
ByteArrayOutputStream baos = new ByteArrayOutputStream();
try {
message.writeTo(baos);
baos.flush();
baos.close();
ByteArrayInputStream bais = new ByteArrayInputStream(baos.toByteArray());
SAXReader saxReader = new SAXReader();
Document doc = saxReader.read(bais);
doc = process(doc);
DocumentSource ds = new DocumentSource(doc);
soapPart.setContent(ds);
message.saveChanges();
} catch (Exception e) {
throw new AxisFault("Error Caught processing document in XMLManipulationHandler", e);
}
}
}
}
public interface ProcessingHandler {
public Node process(Node node);
}
public class NamespaceRemovalHandler implements ProcessingHandler {
private static Log log = LogFactory.getLog(NamespaceRemovalHandler.class);
private Namespace namespace;
private String targetElement;
private Set ignoreElements;
public NamespaceRemovalHandler() {
ignoreElements = new HashSet();
}
public Node process(Node node) {
if (node instanceof Element) {
Element element = (Element) node;
if (element.isRootElement()) {
// Evidently, we never actually see the root node when we're called from
// SOAP...
} else {
if (element.getName().equals(targetElement)) {
log.trace("Found the target Element. Adding requested namespace");
Namespace already = element.getNamespaceForURI(namespace.getURI());
if (already == null) {
element.add(namespace);
}
} else if (!ignoreElements.contains(element.getName())) {
Namespace target = element.getNamespaceForURI(namespace.getURI());
if (target != null) {
element.remove(target);
element.setQName(new QName(element.getName(), namespace));
}
}
Attribute type = element.attribute("type");
if (type != null) {
log.trace("Replacing type information: " + type.getText());
String typeText = type.getText();
typeText = typeText.replaceAll("ns[0-9]+", namespace.getPrefix());
type.setText(typeText);
}
}
}
return node;
}
public Namespace getNamespace() {
return namespace;
}
public void setNamespace(Namespace namespace) {
this.namespace = namespace;
}
/**
* #return the targetElement
*/
public String getTargetElement() {
return targetElement;
}
/**
* #param targetElement the targetElement to set
*/
public void setTargetElement(String targetElement) {
this.targetElement = targetElement;
}
/**
* #return the ignoreElements
*/
public Set getIgnoreElements() {
return ignoreElements;
}
/**
* #param ignoreElements the ignoreElements to set
*/
public void setIgnoreElements(Set ignoreElements) {
this.ignoreElements = ignoreElements;
}
public void addIgnoreElement(String element) {
this.ignoreElements.add(element);
}
}
No warranty, etc, etc.

For the Request I use this to remove namespaces types:
String endpoint = "http://localhost:5555/yourService";
// Parameter to be send
Integer secuencial = new Integer(11); // 0011
// Make the call
Service service = new Service();
Call call = (Call) service.createCall();
// Disable sending Multirefs
call.setOption( org.apache.axis.AxisEngine.PROP_DOMULTIREFS, new java.lang.Boolean( false) );
// Disable sending xsi:type
call.setOption(org.apache.axis.AxisEngine.PROP_SEND_XSI, new java.lang.Boolean( false));
// XML with new line
call.setOption(org.apache.axis.AxisEngine.PROP_DISABLE_PRETTY_XML, new java.lang.Boolean( false));
// Other Options. You will not need them
call.setOption(org.apache.axis.AxisEngine.PROP_ENABLE_NAMESPACE_PREFIX_OPTIMIZATION, new java.lang.Boolean( true));
call.setOption(org.apache.axis.AxisEngine.PROP_DOTNET_SOAPENC_FIX, new java.lang.Boolean( true));
call.setTargetEndpointAddress(new java.net.URL(endpoint));
call.setSOAPActionURI("http://YourActionUrl");//Optional
// Opertion Name
//call.setOperationName( "YourMethod" );
call.setOperationName(new javax.xml.namespace.QName("http://yourUrl", "YourMethod"));
// Do not send encoding style
call.setEncodingStyle(null);
// Do not send xmlns in the xml nodes
call.setProperty(org.apache.axis.client.Call.SEND_TYPE_ATTR, Boolean.FALSE);
/////// Configuration of namespaces
org.apache.axis.description.OperationDesc oper;
org.apache.axis.description.ParameterDesc param;
oper = new org.apache.axis.description.OperationDesc();
oper.setName("InsertaTran");
param = new org.apache.axis.description.ParameterDesc(new javax.xml.namespace.QName("http://yourUrl", "secuencial"), org.apache.axis.description.ParameterDesc.IN, new javax.xml.namespace.QName("http://www.w3.org/2001/XMLSchema", "int"), int.class, false, false);
oper.addParameter(param);
oper.setReturnType(new javax.xml.namespace.QName("http://www.w3.org/2001/XMLSchema", "int"));
oper.setReturnClass(int.class);
oper.setReturnQName(new javax.xml.namespace.QName("http://yourUrl", "yourReturnMethod"));
oper.setStyle(org.apache.axis.constants.Style.WRAPPED);
oper.setUse(org.apache.axis.constants.Use.LITERAL);
call.setOperation(oper);
Integer ret = (Integer) call.invoke( new java.lang.Object []
{ secuencial });

Alter your client's wsdd to set enableNamespacePrefixOptimization to true
<globalConfiguration >
<parameter name="enableNamespacePrefixOptimization" value="true"/>