This question already has answers here:
How to parse invalid (bad / not well-formed) XML?
(4 answers)
Closed 5 years ago.
I have a process that parses an XML file using JDOM and xpath to parse the file as shown below:
private static SAXBuilder builder = null;
private static Document doc = null;
private static XPath xpathInstance = null;
builder = new SAXBuilder();
Text list = null;
try {
doc = builder.build(new StringReader(xmldocument));
} catch (JDOMException e) {
throw new Exception(e);
}
try {
xpathInstance = XPath.newInstance("//book[author='Neal Stephenson']/title/text()");
list = (Text) xpathInstance.selectSingleNode(doc);
} catch (JDOMException e) {
throw new Exception(e);
}
The above works fine. The xpath expressions are stored in a properties file so these can be changed anytime. Now i have to process some more xml files that come from a legacy system that will only send the xml files in chunks of 4000 bytes. The existing processing reads the 4000 byte chunks and stores them in an Oracle database with each chunk as one row in the database (Making any changes to the legacy system or the processing that stores the chunks as rows in the database is out of the question).
I can build the complete valid XML document by extracting all the rows related to a specific xml document and merging them and then use the existing processing (shown above) to parse the xml document.
The thing is though, the data i need to extract from the XML document will always be on the first 4000 bytes. This chunk ofcourse is not a valid XML document as it will be incomplete but will contain all the data i need. I cant parse just the one chunk as the JDOM builder will reject it.
I am wondering whether i can parse the malformed XML chunk without having to merge all parts (which could get to quite many) in order to get a valid XML document. This will save me several trips to the database to check if a chunk is available and i wont have to merge 100s of chunks only for being able to use the first 4000 bytes.
I know i could probably use java's string functions to extract the relevant data but is this possible using a parser or even xpath? or do they both expect the xml document to be a well formed document before it can parse it?
You could try to use JSoup to parse the invalid XML. By definition XML should be well-formed, otherwise it's invalid and should not be used.
UPDATE - example:
public static void main(String[] args) {
for (Node node : Parser.parseFragment("<test><author name=\"Vlad\"><book name=\"SO\"/>" ,
new Element(Tag.valueOf("p"), ""),
"")) {
print(node, 0);
}
}
public static void print(Node node, int offset) {
for (int i = 0; i < offset; i++) {
System.out.print(" ");
}
System.out.print(node.nodeName());
for (Attribute attribute: node.attributes()) {
System.out.print(", ");
System.out.print(attribute.getKey() + "=" + attribute.getValue());
}
System.out.println();
for (Node child : node.childNodes()) {
print(child, offset + 4);
}
}
Related
I'm using SAX parser and XSSFReader of apache.poi to parse .xlsx file. my sheet contains up to 650 columns and 2000 rows (file size- about 2.5 MB). my code looks like so:
public class MyClass {
public static void main(String path){
try {
OPCPackage pkg = OPCPachage.open(new FileInputStream(path));
XSSFReader reader = new XSSFReader(pkg);
InputStream sheetData = reader.getSheet("rId3"); //the needed sheet
MyHandler handler = new MyHandler();
XMLReader parser = SAXHelper.newXMLReader();
parser.setContentHandler(handler);
parser.parse(new InputSource(sheetData));
}
catch (Exception e){
//or other catches with required exceptions
}
}
}
class MyHandler extends DefaultHandler {
#Override
public void startElement (String uri, String localName, String name, Attributes attributes) {
if("row".equals(name)) {
System.out.pringln("row: " + attributes.getValue("r"));
}
}
}
but unfortunately I saw that it takes 2 or 3 seconds to go over one row, that means that going over the sheet takes over than 30 minutes(!!)
Well, I am sure this is not supposed to be, if it was- noboy was suggesing apache.poi eventaApi for large files, was he?
I want to get to the <mergeCell> values at the end of the XML (after the closing </sheetData>, is there a better way to do it? (I was thinking of handle with the string of the xml, and simply search with some regular expression for the required values, is it possible?)
So- I have two questions:
1. What's wrong with my code/why it takes so long? (when I think about it- it actually sounds as normal situation, 600 cells- why not processing in few seconds?)
2. Is there a way to treat XML as a text file and simply search in it using regex?
I am trying to populate repeated forms with PDFbox. I am using a TreeMap and populating the forms with individual records. The format of the pdf form is such that there are six records listed on page one and a static page inserted on page two. (For a TreeMap larger than six records, the process repeats). The error Im getting is specific to the size of the TreeMap. Therein lies my problem. I can't figure out why when I populate the TreeMap with more than 35 entries I get this warning:
Apr 23, 2018 2:36:25 AM org.apache.pdfbox.cos.COSDocument finalize
WARNING: Warning: You did not close a PDF Document
public class test {
public static void main(String[] args) throws IOException, IOException {
// TODO Auto-generated method stub
File dataFile = new File("dataFile.csv");
File fi = new File("form.pdf");
Scanner fileScanner = new Scanner(dataFile);
fileScanner.nextLine();
TreeMap<String, String[]> assetTable = new TreeMap<String, String[]>();
int x = 0;
while (x <= 36) {
String lineIn = fileScanner.nextLine();
String[] elements = lineIn.split(",");
elements[0] = elements[0].toUpperCase().replaceAll(" ", "");
String key = elements[0];
key = key.replaceAll(" ", "");
assetTable.put(key, elements);
x++;
}
PDDocument newDoc = new PDDocument();
int control = 1;
PDDocument doc = PDDocument.load(fi);
PDDocumentCatalog cat = doc.getDocumentCatalog();
PDAcroForm form = cat.getAcroForm();
for (String s : assetTable.keySet()) {
if (control <= 6) {
PDField IDno1 = (form.getField("IDno" + control));
PDField Locno1 = (form.getField("locNo" + control));
PDField serno1 = (form.getField("serNo" + control));
PDField typeno1 = (form.getField("typeNo" + control));
PDField maintno1 = (form.getField("maintNo" + control));
String IDnoOne = assetTable.get(s)[1];
//System.out.println(IDnoOne);
IDno1.setValue(assetTable.get(s)[0]);
IDno1.setReadOnly(true);
Locno1.setValue(assetTable.get(s)[1]);
Locno1.setReadOnly(true);
serno1.setValue(assetTable.get(s)[2]);
serno1.setReadOnly(true);
typeno1.setValue(assetTable.get(s)[3]);
typeno1.setReadOnly(true);
String type = "";
if (assetTable.get(s)[5].equals("1"))
type += "Hydrotest";
if (assetTable.get(s)[5].equals("6"))
type += "6 Year Maintenance";
String maint = assetTable.get(s)[4] + " - " + type;
maintno1.setValue(maint);
maintno1.setReadOnly(true);
control++;
} else {
PDField dateIn = form.getField("dateIn");
dateIn.setValue("1/2019 Yearlies");
dateIn.setReadOnly(true);
PDField tagDate = form.getField("tagDate");
tagDate.setValue("2019 / 2020");
tagDate.setReadOnly(true);
newDoc.addPage(doc.getPage(0));
newDoc.addPage(doc.getPage(1));
control = 1;
doc = PDDocument.load(fi);
cat = doc.getDocumentCatalog();
form = cat.getAcroForm();
}
}
PDField dateIn = form.getField("dateIn");
dateIn.setValue("1/2019 Yearlies");
dateIn.setReadOnly(true);
PDField tagDate = form.getField("tagDate");
tagDate.setValue("2019 / 2020");
tagDate.setReadOnly(true);
newDoc.addPage(doc.getPage(0));
newDoc.addPage(doc.getPage(1));
newDoc.save("PDFtest.pdf");
Desktop.getDesktop().open(new File("PDFtest.pdf"));
}
I cant figure out for the life of me what I'm doing wrong. This is the first week I've been working with PDFbox so I'm hoping its something simple.
Updated Error Message
WARNING: Warning: You did not close a PDF Document
Exception in thread "main" java.io.IOException: COSStream has been closed and cannot be read. Perhaps its enclosing PDDocument has been closed?
at org.apache.pdfbox.cos.COSStream.checkClosed(COSStream.java:77)
at org.apache.pdfbox.cos.COSStream.createRawInputStream(COSStream.java:125)
at org.apache.pdfbox.pdfwriter.COSWriter.visitFromStream(COSWriter.java:1200)
at org.apache.pdfbox.cos.COSStream.accept(COSStream.java:383)
at org.apache.pdfbox.cos.COSObject.accept(COSObject.java:158)
at org.apache.pdfbox.pdfwriter.COSWriter.doWriteObject(COSWriter.java:522)
at org.apache.pdfbox.pdfwriter.COSWriter.doWriteObjects(COSWriter.java:460)
at org.apache.pdfbox.pdfwriter.COSWriter.doWriteBody(COSWriter.java:444)
at org.apache.pdfbox.pdfwriter.COSWriter.visitFromDocument(COSWriter.java:1096)
at org.apache.pdfbox.cos.COSDocument.accept(COSDocument.java:419)
at org.apache.pdfbox.pdfwriter.COSWriter.write(COSWriter.java:1367)
at org.apache.pdfbox.pdfwriter.COSWriter.write(COSWriter.java:1254)
at org.apache.pdfbox.pdmodel.PDDocument.save(PDDocument.java:1232)
at org.apache.pdfbox.pdmodel.PDDocument.save(PDDocument.java:1204)
at org.apache.pdfbox.pdmodel.PDDocument.save(PDDocument.java:1192)
at test.test.main(test.java:87)
The warning by itself
You appear to get the warning wrong. It says:
Warning: You did not close a PDF Document
So in contrast to what you think, "PDFbox saying PDDocument closed when its not", PDFBox says that you did not close a document!
After your edit one sees that it actually says that a COSStream has been closed and that a possible cause is that the enclosing PDDocument already has been closed. This is a mere possibility!
The warning in your case
That been said, by adding pages from one document to another you probably end up having references to those pages from both documents. In that case in the course of closing both documents (e.g. automatically via garbage collection), the second one closing may indeed stumble across some already closed COSStream instances.
So my first advice to simply do close the documents at the end by
doc.close();
newDoc.close();
probably won't remove the warnings, merely change their timing.
Actually you don't merely create two documents doc and newDoc, you even create new PDDocument instances and assign them to doc again and again, in the process setting the former document objects in that variable free for garbage collection. So you eventually have a big bunch of documents to be closed as soon as not referenced anymore.
I don't think it would be a good idea to close all those documents in doc early, in particular not before saving newDoc.
But if your code will eventually be run as part of a larger application instead of as a small, one-shot test application, you should collect all those PDDocument instances in some Collection and explicitly close them right after saving newDoc and then clear the collection.
Actually your exception looks like one of those lost PDDocument instances has already been closed by garbage collection, so you should collect the documents even in case of a simple one-shot utility to keep them from being GC disposed.
(#Tilman, please correct me if I'm wrong...)
Importing pages
To prevent problems with different documents sharing pages, you can try and import the pages to the target document and thereafter add the imported page to the target document page tree. I.e. replace
newDoc.addPage(doc.getPage(0));
newDoc.addPage(doc.getPage(1));
by
newDoc.addPage(newDoc.importPage(doc.getPage(0)));
newDoc.addPage(newDoc.importPage(doc.getPage(1)));
This should allow you to close each PDDocument instance in doc before losing it.
There are certain drawbacks to this, though, cf. the method JavaDoc and this answer here.
An actual issue in your code
In your combined document you will have many fields with the same name (at least in case of a sufficiently high number of entries in your CSV file) which you initially set to different values. And you access the fields from the PDAcroForm of the respective original document but don't add them to the PDAcroForm of the combined result document.
This is asking for trouble! The PDF format does consider forms to be document-wide with all fields referenced (directly or indirectly) from the AcroForm dictionary of the document, and it expects fields with the same name to effectively be different visualizations of the same field and therefore to all have the same value.
Thus, PDF processors might handle your document fields in unexpected ways, e.g.
by showing the same value in all fields with the same name (as they are expected to have the same value) or
by ignoring your fields (as they are not in the document AcroForm structure).
In particular programmatic reading of your PDF field values will fail because in that context the form is definitively considered document-wide and based in AcroForm. PDF viewers on the other hand might first show your set values and make look things ok.
To prevent this you should rename the fields before merging. You might consider using the PDFMergerUtility which does such a renaming under the hood. For an example usage of that utility class have a look at the PDFMergerExample.
Even though the above answer was marked as the solution to the problem, since the solution is buried in the comments, I wanted to add this answer at this level. I spent several hours searching for the solution.
My code snippets and comments.
// Collection solely for purpose of preventing premature garbage collection
List<PDDocument> sourceDocuments = new ArrayList<>( );
...
// Source document (actually inside a loop)
PDDocument docIn = PDDocument.load( artifactBytes );
// Add document to collection before using it to prevent the problem
sourceDocuments.add( docIn );
// Extract from source document
PDPage extractedPage = docIn.getPage( 0 );
// Add page to destination document
docOut.addPage( extractedPage );
...
// This was failing with "COSStream has been closed and cannot be read."
// Now it works.
docOut.save( bundleStream );
I do have few Word templates, and my requirement is to replace some of the words/place holders in the document based on the user input, using Java. I tried lot of libraries including 2-3 versions of docx4j but nothing work well, they all just didn't do anything!
I know this question has been asked before, but I tried all options I know. So, using what java library I can "really" replace/edit these templates? My preference goes to the "easy to use / Few line of codes" type libraries.
I am using Java 8 and my MS Word templates are in MS Word 2007.
Update
This code is written by using the code sample provided by SO member Joop Eggen
public Main() throws URISyntaxException, IOException, ParserConfigurationException, SAXException
{
URI docxUri = new URI("C:/Users/Yohan/Desktop/yohan.docx");
Map<String, String> zipProperties = new HashMap<>();
zipProperties.put("encoding", "UTF-8");
FileSystem zipFS = FileSystems.newFileSystem(docxUri, zipProperties);
Path documentXmlPath = zipFS.getPath("/word/document.xml");
DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
factory.setNamespaceAware(true);
DocumentBuilder builder = factory.newDocumentBuilder();
Document doc = builder.parse(Files.newInputStream(documentXmlPath));
byte[] content = Files.readAllBytes(documentXmlPath);
String xml = new String(content, StandardCharsets.UTF_8);
//xml = xml.replace("#DATE#", "2014-09-24");
xml = xml.replace("#NAME#", StringEscapeUtils.escapeXml("Sniper"));
content = xml.getBytes(StandardCharsets.UTF_8);
Files.write(documentXmlPath, content);
}
However this returns the below error
java.nio.file.ProviderNotFoundException: Provider "C" Not found
at: java.nio.file.FileSystems.newFileSystem(FileSystems.java:341) at java.nio.file.FileSystems.newFileSystem(FileSystems.java:341)
at java.nio.fileFileSystems.newFileSystem(FileSystems.java:276)
One may use for docx (a zip with XML and other files) a java zip file system and XML or text processing.
URI docxUri = ,,, // "jar:file:/C:/... .docx"
Map<String, String> zipProperties = new HashMap<>();
zipProperties.put("encoding", "UTF-8");
try (FileSystem zipFS = FileSystems.newFileSystem(docxUri, zipProperties)) {
Path documentXmlPath = zipFS.getPath("/word/document.xml");
When using XML:
DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
factory.setNamespaceAware(true);
DocumentBuilder builder = factory.newDocumentBuilder();
Document doc = builder.parse(Files.newInputStream(documentXmlPath));
//Element root = doc.getDocumentElement();
You can then use XPath to find the places, and write the XML back again.
It even might be that you do not need XML but could replace place holders:
byte[] content = Files.readAllBytes(documentXmlPath);
String xml = new String(content, StandardCharsets.UTF_8);
xml = xml.replace("#DATE#", "2014-09-24");
xml = xml.replace("#NAME#", StringEscapeUtils.escapeXml("Sniper")));
...
content = xml.getBytes(StandardCharsets.UTF_8);
Files.delete(documentXmlPath);
Files.write(documentXmlPath, content);
For a fast development, rename a copy of the .docx to a name with the .zip file extension, and inspect the files.
File.write should already apply StandardOpenOption.TRUNCATE_EXISTING, but I have added Files.delete as some error occured. See comments.
Try Apache POI. POI can work with doc and docx, but docx is more documented therefore support of it better.
UPD: You can use XDocReport, which use POI. Also I recomend to use xlsx for templates because it more suitable and more documented
I have spent a few days on this issue, until I found that what makes the difference is the try-with-resources on the FileSystem instance, appearing in Joop Eggen's snippet but not in question snippet:
try (FileSystem zipFS = FileSystems.newFileSystem(docxUri, zipProperties))
Without such try-with-resources block, the FileSystem resource will not be closed (as explained in Java tutorial), and the word document not modified.
Stepping back a bit, there are about 4 different approaches for editing words/placeholders:
MERGEFIELD or DOCPROPERTY fields (if you are having problems with this in docx4j, then you have probably not set up your input docx correctly)
content control databinding
variable replacement on the document surface (either at the DOM/SAX level, or using a library)
do stuff as XHTML, then import that
Before choosing one, you should decide whether you also need to be able to handle:
repeating data (eg adding table rows)
conditional content (eg entire paragraphs which will either be present or absent)
adding images
If you need these, then MERGEFIELD or DOCPROPERTY fields are probably out (though you can also use IF fields, if you can find a library which supports them). And adding images makes DOM/SAX manipulation as advocated in one of the other answers, messier and error prone.
The other things to consider are:
your authors: how technical are they? What does that imply for the authoring UI?
the "user input" you mention for variable replacement, is this given, or is obtaining it part of the problem you are solving?
Please try this to edit or replace the word in document
public class UpdateDocument {
public static void main(String[] args) throws IOException {
UpdateDocument obj = new UpdateDocument();
obj.updateDocument(
"c:\\test\\template.docx",
"c:\\test\\output.docx",
"Piyush");
}
private void updateDocument(String input, String output, String name)
throws IOException {
try (XWPFDocument doc = new XWPFDocument(
Files.newInputStream(Paths.get(input)))
) {
List<XWPFParagraph> xwpfParagraphList = doc.getParagraphs();
//Iterate over paragraph list and check for the replaceable text in each paragraph
for (XWPFParagraph xwpfParagraph : xwpfParagraphList) {
for (XWPFRun xwpfRun : xwpfParagraph.getRuns()) {
String docText = xwpfRun.getText(0);
//replacement and setting position
docText = docText.replace("${name}", name);
xwpfRun.setText(docText, 0);
}
}
// save the docs
try (FileOutputStream out = new FileOutputStream(output)) {
doc.write(out);
}
}
}
}
I want to read the number of characters without spaces in a Word document using Apache POI.
I can get the number of characters with spaces using the SummaryInformation.getCharCount() method as in the following code:
public void countCharacters() throws FileNotFoundException, IOException {
File wordFile = new File(BASE_PATH, "test.doc");
POIFSFileSystem p = new POIFSFileSystem(new FileInputStream(wordFile));
HWPFDocument doc = new HWPFDocument(p);
SummaryInformation props = doc.getSummaryInformation();
int numOfCharsWithSpaces = props.getCharCount();
System.out.println(numOfCharsWithSpaces);
}
However there seems to be no method for returning the number of characters without spaces.
How do I find this value?
If you want to base this on the metadata of the document, all you will get is estimates (according to the Microsoft specs). There are essentially two values which you can play around with:
GKPIDSI_CHARCOUNT (which is what you already accessed in your own code sample)
GKPIDDSI_CCHWITHSPACES
Don't ask me about the exact differences of those two values, though. I haven't designed this stuff...
Below is a code sample to illustrate the access to them (GKPIDDSI_CCHWITHSPACES is a little awkward):
HWPFDocument document = [...];
SummaryInformation summaryInformation = document.getSummaryInformation();
System.out.println("GKPIDSI_CHARCOUNT: " + summaryInformation.getCharCount());
DocumentSummaryInformation documentSummaryInformation = document.getDocumentSummaryInformation();
Integer count = null;
for (Property property : documentSummaryInformation.getProperties()) {
if (property.getID() == 0x11) {
count = (Integer) property.getValue();
break;
}
}
System.out.println("GKPIDDSI_CCHWITHSPACES: " + count);
The moment at which Word's internal algorithm that updates those values kicks in is rather unpredictable to me. So what you see in Word's own statistics may not necessarily be the same as when running the above code.
this is the way i made an XML file to a Java object(s).
i used "xjc" and a valid XML schema, and i got back some "generated" *.java files.
i imported them into a different package in eclipse.
I am reading the XML file in 2 way now.
1) Loading the XML file:
System.out.println("Using FILE approach:");
File f = new File ("C:\\test_XML_files\\complex.apx");
JAXBElement felement = (JAXBElement) u.unmarshal(f);
MyObject fmainMyObject = (MyObject) felement.getValue ();
2) Using a DOM buider:
System.out.println("Using DOM BUILDER Approach:");
JAXBElement element = (JAXBElement) u.unmarshal(test());;
MyObject mainMyObject = (MyObject ) element.getValue ();
now in method "test()" the code below is included:
public static Node test(){
Document document = parseXmlDom();
return document.getFirstChild();
}
private static Document parseXmlDom() {
Document document = null;
try {
// getting the default implementation of DOM builder
DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
DocumentBuilder builder = factory.newDocumentBuilder();
// parsing the XML file
document = builder.parse(new File("C:\\test_XML_files\\MyXML_FILE.apx"));
} catch (Exception e) {
// catching all exceptions
System.out.println();
System.out.println(e.toString());
}
return document;
}
is this the standard way of doing XML to an Java Object?
I tested if I could access the object and everything works fine. (so far)
Do you suggest a different approach?? or is this one sufficient?
I don't know about a "standard way", but either way looks OK to me. The first way looks simpler ( less code ) so that's the way I'd probably do it, unless there were other factors / requirements.
FWIW, I'd expect that the unmarshal(File) method was implemented to do pretty much what you are doing in your second approach.
However, it is really up to you (and your co-workers) to make judgments about what is "sufficient" for your project.