Develop Reference

How can I create an accessible PDF with Java PDFBox 2.0.8 library that is also verifiable with PAC 2 tool?

Background
I have small project on GitHub in which I am trying to create a section 508 compliant (section508.gov) PDF which has form elements within a complex table structure. The tool recommended to verify these PDFs is at http://www.access-for-all.ch/en/pdf-lab/pdf-accessibility-checker-pac.html and my program’s output PDF does pass most of these checks. I will also know what every field is meant for at runtime, so adding tags to structure elements should not be an issue.
The Problem
The PAC 2 tool seems to have an issue with two particular items in the output PDF. In particular, my radio buttons’ widget annotations are not nested inside of a form structure element and my marked content is not tagged (Text and Table Cells).
PAC 2 verifies the P structure element that is within top-left cell but not the marked content…
However, PAC 2 does identify the marked content as an error (i.e. Text/Path object not tagged).
Also, the radio button widgets are detected, but there seems to be no APIs to add them to a form structure element.
What I Have Tried
I have looked at several questions on this website and others on the subject including this one Tagged PDF with PDFBox, but it seems that there are almost no examples for PDF/UA and very little useful documentation (That I have found). The most useful tips that I have found have been at sites that explain specs for tagged PDFs like https://taggedpdf.com/508-pdf-help-center/object-not-tagged/.
The Question
Is it possible to create a PAC 2 verifiable PDF with Apache PDFBox that includes marked content and radio button widget annotations? If it is possible, is it doable using higher level (non-deprecated) PDFBox APIs?
Side Note: This is actually my first StackExchange question (Although I have used the site extensively) and I hope everything is in order! Feel free to add any necessary edits and ask any questions that I may need clarify. Also, I have an example program on GitHub which generates my PDF document at https://github.com/chris271/UAPDFBox.
Edit 1: Direct link to Output PDF Document
*EDIT 2: After using some of the lower-level PDFBox APIs and viewing raw data streams for fully compliant PDFs with PDFDebugger, I was able to generate a PDF with nearly identical content structure compared to the compliant PDF's content structure... However, the same errors appear that the text objects are not tagged and I really can't decide where to go from here... Any guidance would be greatly appreciated!
Edit 3: Side-by-side raw PDF content comparison.
Edit 4: Internal structure of the generated PDF
and the compliant PDF
Edit 5: I have managed to fix the PAC 2 errors for tagged path/text objects thanks in part to suggestions from Tilman Hausherr! I will add an answer if I manage to fix the issues regarding 'annotation widgets not being nested inside form structure elements'.

After going through a large amount of the PDF Spec and many PDFBox examples I was able to fix all issues reported by PAC 2. There were several steps involved to create the verified PDF (with a complex table structure) and the full source code is available here on github. I will attempt to do an overview of the major portions of the code below. (Some method calls will not be explained here!)
Step 1 (Setup metadata)
Various setup info like document title and language
//Setup new document
pdf = new PDDocument();
acroForm = new PDAcroForm(pdf);
pdf.getDocumentInformation().setTitle(title);
//Adjust other document metadata
PDDocumentCatalog documentCatalog = pdf.getDocumentCatalog();
documentCatalog.setLanguage("English");
documentCatalog.setViewerPreferences(new PDViewerPreferences(new COSDictionary()));
documentCatalog.getViewerPreferences().setDisplayDocTitle(true);
documentCatalog.setAcroForm(acroForm);
documentCatalog.setStructureTreeRoot(structureTreeRoot);
PDMarkInfo markInfo = new PDMarkInfo();
markInfo.setMarked(true);
documentCatalog.setMarkInfo(markInfo);
Embed all fonts directly into resources.
//Set AcroForm Appearance Characteristics
PDResources resources = new PDResources();
defaultFont = PDType0Font.load(pdf,
new PDTrueTypeFont(PDType1Font.HELVETICA.getCOSObject()).getTrueTypeFont(), true);
resources.put(COSName.getPDFName("Helv"), defaultFont);
acroForm.setNeedAppearances(true);
acroForm.setXFA(null);
acroForm.setDefaultResources(resources);
acroForm.setDefaultAppearance(DEFAULT_APPEARANCE);
Add XMP Metadata for PDF/UA spec.
//Add UA XMP metadata based on specs at https://taggedpdf.com/508-pdf-help-center/pdfua-identifier-missing/
XMPMetadata xmp = XMPMetadata.createXMPMetadata();
xmp.createAndAddDublinCoreSchema();
xmp.getDublinCoreSchema().setTitle(title);
xmp.getDublinCoreSchema().setDescription(title);
xmp.createAndAddPDFAExtensionSchemaWithDefaultNS();
xmp.getPDFExtensionSchema().addNamespace("http://www.aiim.org/pdfa/ns/schema#", "pdfaSchema");
xmp.getPDFExtensionSchema().addNamespace("http://www.aiim.org/pdfa/ns/property#", "pdfaProperty");
xmp.getPDFExtensionSchema().addNamespace("http://www.aiim.org/pdfua/ns/id/", "pdfuaid");
XMPSchema uaSchema = new XMPSchema(XMPMetadata.createXMPMetadata(),
"pdfaSchema", "pdfaSchema", "pdfaSchema");
uaSchema.setTextPropertyValue("schema", "PDF/UA Universal Accessibility Schema");
uaSchema.setTextPropertyValue("namespaceURI", "http://www.aiim.org/pdfua/ns/id/");
uaSchema.setTextPropertyValue("prefix", "pdfuaid");
XMPSchema uaProp = new XMPSchema(XMPMetadata.createXMPMetadata(),
"pdfaProperty", "pdfaProperty", "pdfaProperty");
uaProp.setTextPropertyValue("name", "part");
uaProp.setTextPropertyValue("valueType", "Integer");
uaProp.setTextPropertyValue("category", "internal");
uaProp.setTextPropertyValue("description", "Indicates, which part of ISO 14289 standard is followed");
uaSchema.addUnqualifiedSequenceValue("property", uaProp);
xmp.getPDFExtensionSchema().addBagValue("schemas", uaSchema);
xmp.getPDFExtensionSchema().setPrefix("pdfuaid");
xmp.getPDFExtensionSchema().setTextPropertyValue("part", "1");
XmpSerializer serializer = new XmpSerializer();
ByteArrayOutputStream baos = new ByteArrayOutputStream();
serializer.serialize(xmp, baos, true);
PDMetadata metadata = new PDMetadata(pdf);
metadata.importXMPMetadata(baos.toByteArray());
pdf.getDocumentCatalog().setMetadata(metadata);
Step 2 (Setup document tag structure)
You will need to add the root structure element and all necessary structure elements as children to the root element.
//Adds a DOCUMENT structure element as the structure tree root.
void addRoot() {
PDStructureElement root = new PDStructureElement(StandardStructureTypes.DOCUMENT, null);
root.setAlternateDescription("The document's root structure element.");
root.setTitle("PDF Document");
pdf.getDocumentCatalog().getStructureTreeRoot().appendKid(root);
currentElem = root;
rootElem = root;
}
Each marked content element (text and background graphics) will need to have an MCID and an associated tag for reference in the parent tree which will be explained in step 3.
//Assign an id for the next marked content element.
private void setNextMarkedContentDictionary(String tag) {
currentMarkedContentDictionary = new COSDictionary();
currentMarkedContentDictionary.setName("Tag", tag);
currentMarkedContentDictionary.setInt(COSName.MCID, currentMCID);
currentMCID++;
}
Artifacts (background graphics) will not be detected by the screen reader. Text needs to be detectable so a P structure element is used here when adding text.
//Set up the next marked content element with an MCID and create the containing TD structure element.
PDPageContentStream contents = new PDPageContentStream(
pdf, pages.get(pageIndex), PDPageContentStream.AppendMode.APPEND, false);
currentElem = addContentToParent(null, StandardStructureTypes.TD, pages.get(pageIndex), currentRow);
//Make the actual cell rectangle and set as artifact to avoid detection.
setNextMarkedContentDictionary(COSName.ARTIFACT.getName());
contents.beginMarkedContent(COSName.ARTIFACT, PDPropertyList.create(currentMarkedContentDictionary));
//Draws the cell itself with the given colors and location.
drawDataCell(table.getCell(i, j).getCellColor(), table.getCell(i, j).getBorderColor(),
x + table.getRows().get(i).getCellPosition(j),
y + table.getRowPosition(i),
table.getCell(i, j).getWidth(), table.getRows().get(i).getHeight(), contents);
contents.endMarkedContent();
currentElem = addContentToParent(COSName.ARTIFACT, StandardStructureTypes.P, pages.get(pageIndex), currentElem);
contents.close();
//Draw the cell's text as a P structure element
contents = new PDPageContentStream(
pdf, pages.get(pageIndex), PDPageContentStream.AppendMode.APPEND, false);
setNextMarkedContentDictionary(COSName.P.getName());
contents.beginMarkedContent(COSName.P, PDPropertyList.create(currentMarkedContentDictionary));
//... Code to draw actual text...//
//End the marked content and append it's P structure element to the containing TD structure element.
contents.endMarkedContent();
addContentToParent(COSName.P, null, pages.get(pageIndex), currentElem);
contents.close();
Annotation Widgets (form objects in this case) will need to be nested within Form structure elements.
//Add a radio button widget.
if (!table.getCell(i, j).getRbVal().isEmpty()) {
PDStructureElement fieldElem = new PDStructureElement(StandardStructureTypes.FORM, currentElem);
radioWidgets.add(addRadioButton(
x + table.getRows().get(i).getCellPosition(j) -
radioWidgets.size() * 10 + table.getCell(i, j).getWidth() / 4,
y + table.getRowPosition(i),
table.getCell(i, j).getWidth() * 1.5f, 20,
radioValues, pageIndex, radioWidgets.size()));
fieldElem.setPage(pages.get(pageIndex));
COSArray kArray = new COSArray();
kArray.add(COSInteger.get(currentMCID));
fieldElem.getCOSObject().setItem(COSName.K, kArray);
addWidgetContent(annotationRefs.get(annotationRefs.size() - 1), fieldElem, StandardStructureTypes.FORM, pageIndex);
}
//Add a text field in the current cell.
if (!table.getCell(i, j).getTextVal().isEmpty()) {
PDStructureElement fieldElem = new PDStructureElement(StandardStructureTypes.FORM, currentElem);
addTextField(x + table.getRows().get(i).getCellPosition(j),
y + table.getRowPosition(i),
table.getCell(i, j).getWidth(), table.getRows().get(i).getHeight(),
table.getCell(i, j).getTextVal(), pageIndex);
fieldElem.setPage(pages.get(pageIndex));
COSArray kArray = new COSArray();
kArray.add(COSInteger.get(currentMCID));
fieldElem.getCOSObject().setItem(COSName.K, kArray);
addWidgetContent(annotationRefs.get(annotationRefs.size() - 1), fieldElem, StandardStructureTypes.FORM, pageIndex);
}
Step 3
After all content elements have been written to the content stream and tag structure has been setup, it is necessary to go back and add the parent tree to the structure tree root. Note: Some method calls (addWidgetContent() and addContentToParent()) in the above code setup the necessary COSDictionary objects.
//Adds the parent tree to root struct element to identify tagged content
void addParentTree() {
COSDictionary dict = new COSDictionary();
nums.add(numDictionaries);
for (int i = 1; i < currentStructParent; i++) {
nums.add(COSInteger.get(i));
nums.add(annotDicts.get(i - 1));
}
dict.setItem(COSName.NUMS, nums);
PDNumberTreeNode numberTreeNode = new PDNumberTreeNode(dict, dict.getClass());
pdf.getDocumentCatalog().getStructureTreeRoot().setParentTreeNextKey(currentStructParent);
pdf.getDocumentCatalog().getStructureTreeRoot().setParentTree(numberTreeNode);
}
If all widget annotations and marked content were added correctly to the structure tree and parent tree then you should get something like this from PAC 2 and PDFDebugger.
Thank you to Tilman Hausherr for pointing me in the right direction to solve this! I will most likely make some edits to this answer for additional clarity as recommended by others.
Edit 1:
If you want to have a table structure like the one I have generated you will also need to add correct table markup to fully comply with the 508 standard... The 'Scope', 'ColSpan', 'RowSpan', or 'Headers' attributes will need to be correctly added to each table cell structure element similar to this or this. The main purpose for this markup is to allow a screen reading software like JAWS to read the table content in an understandable way. These attributes can be added in a similar way as below...
private void addTableCellMarkup(Cell cell, int pageIndex, PDStructureElement currentRow) {
COSDictionary cellAttr = new COSDictionary();
cellAttr.setName(COSName.O, "Table");
if (cell.getCellMarkup().isHeader()) {
currentElem = addContentToParent(null, StandardStructureTypes.TH, pages.get(pageIndex), currentRow);
currentElem.getCOSObject().setString(COSName.ID, cell.getCellMarkup().getId());
if (cell.getCellMarkup().getScope().length() > 0) {
cellAttr.setName(COSName.getPDFName("Scope"), cell.getCellMarkup().getScope());
}
if (cell.getCellMarkup().getColspan() > 1) {
cellAttr.setInt(COSName.getPDFName("ColSpan"), cell.getCellMarkup().getColspan());
}
if (cell.getCellMarkup().getRowSpan() > 1) {
cellAttr.setInt(COSName.getPDFName("RowSpan"), cell.getCellMarkup().getRowSpan());
}
} else {
currentElem = addContentToParent(null, StandardStructureTypes.TD, pages.get(pageIndex), currentRow);
}
if (cell.getCellMarkup().getHeaders().length > 0) {
COSArray headerA = new COSArray();
for (String s : cell.getCellMarkup().getHeaders()) {
headerA.add(new COSString(s));
}
cellAttr.setItem(COSName.getPDFName("Headers"), headerA);
}
currentElem.getCOSObject().setItem(COSName.A, cellAttr);
}
Be sure to do something like currentElem.setAlternateDescription(currentCell.getText()); on each of the structure elements with text marked content for JAWS to read the text.
Note: Each of the fields (radio button and textbox) will need a unique name to avoid setting multiple field values. GitHub has been updated with a more complex example PDF with table markup and improved form fields!

Batik copy element from one document to another

I'm trying to compose an svg with batik from elements from a source document. This works as long as the elements don't reference things defined in the defs section (like gradients or filters). But when a filter is referenced I get an exception. I tried to also copy over the defs section but that didn't help.
String parser = XMLResourceDescriptor.getXMLParserClassName();
SAXSVGDocumentFactory factory = new SAXSVGDocumentFactory(parser);
Document source = factory.createDocument("source", getClass().getResourceAsStream("/artwork/source.svg"));
SVGOMDocument target = (SVGOMDocument) domImpl.createDocument(svgNS, "svg", null);
Node defs = source.getElementsByTagName("defs").item(0).cloneNode(true);
target.adoptNode(defs);
target.getRootElement().appendChild(defs);
BridgeContext ctx = new BridgeContext(new UserAgentAdapter());
GVTBuilder builder = new GVTBuilder();
builder.build(ctx, target);
Element sourceEl = getElementByXPath(source,
"//*[#id='IMAGES']/*[#id='" + selection.getImageSet() + "']/*[#id='"
+ suit.abbreviation() + value + "-" + selection.getImageSet() + "']");
SVGOMElement complete = (SVGOMElement) sourceEl.cloneNode(true);
target.adoptNode(complete);
target.getDocumentElement().appendChild(complete);
Rectangle2D completeBBox = builder.build(ctx, complete).getSensitiveBounds();
The last line gives the exception. Any help would be appreciated.
EDIT:
If I save the target svg after adding the defs section and then reload it again it works. But I would like to avoid this aditional step.

I found a solution to avoid the extra save/load. Use importNode instead of clone/adopt for the defs.
Node defs = target.importNode(source.getElementsByTagName("defs").item(0), true);
target.getRootElement().appendChild(defs);

How to dynamically create a document in the GAE Search API?

I am trying to understand how i can dynamically add fields in my document.
I am following the doc . However i'd like to use the addField method inside a "for" statement and thus programmatically generate fields.
I've tried this :
Builder document = Document.newBuilder();
document.setId(id);
// for statement start
com.google.appengine.api.search.Field.Builder field = Field.newBuilder().setName(key).setAtom(value);
document.addField(field);
// for statement end
document.build();
But it doesn't seems to work. When i'm trying i got this printed out
com.google.appengine.api.search.Document$Builder#2604ed54
Whereas when i use the original code :
String myDocId = "PA6-5000";
Document doc = Document.newBuilder()
// Setting the document identifer is optional.
// If omitted, the search service will create an identifier.
.setId(myDocId)
.addField(Field.newBuilder().setName("content").setText("the rain in spain"))
.addField(Field.newBuilder().setName("email").setText(userEmail))
.addField(Field.newBuilder().setName("domain").setAtom(userDomain))
.addField(Field.newBuilder().setName("published").setDate(new Date()))
.build();
I get the expected result with the code above.
How i can manipulate this in order to put addfield() into for statement ?
thanks for helping.

The docs imply the Document.Builder is mutable, but typically you should reassign the builder after each invocation in case it isn't.
Builder document = Document.newBuilder();
document.setId(id);
com.google.appengine.api.search.Field.Builder field = Field.newBuilder().setName(key).setAtom(value);
document = document.addField(field);
document.build();
In your case what you're seeing is the toString() method on the builder, not the document itself. The problem is the last line.
Try this:
Builder builder = Document.newBuilder();
builder.setId(id);
com.google.appengine.api.search.Field.Builder field = Field.newBuilder().setName(key).setAtom(value);
builder = builder.addField(field);
Document document = builder.build();
Then you can just drop in your for loop and away you go.

Parser JSoup change the tags to lower case letter

I did some research and it seems that is standard Jsoup make this change. I wonder if there is a way to configure this or is there some other Parser I can be converted to a document of Jsoup, or some way to fix this?

Unfortunately not, the constructor of Tag class changes the name to lower case:
private Tag(String tagName) {
this.tagName = tagName.toLowerCase();
}
But there are two ways to change this behavour:
If you want a clean solution, you can clone / download the JSoup Git and change this line.
If you want a dirty solution, you can use reflection.
Example for #2:
Field tagName = Tag.class.getDeclaredField("tagName"); // Get the field which contains the tagname
tagName.setAccessible(true); // Set accessible to allow changes
for( Element element : doc.select("*") ) // Iterate over all tags
{
Tag tag = element.tag(); // Get the tag of the element
String value = tagName.get(tag).toString(); // Get the value (= name) of the tag
if( !value.startsWith("#") ) // You can ignore all tags starting with a '#'
{
tagName.set(tag, value.toUpperCase()); // Set the tagname to the uppercase
}
}
tagName.setAccessible(false); // Revert to false

Here is a code sample (version >= 1.11.x):
Parser parser = Parser.htmlParser();
parser.settings(new ParseSettings(true, true));
Document doc = parser.parseInput(html, baseUrl);

There is ParseSettings class introduced in version 1.9.3.
It comes with options to preserve case for tags and attributes.

You must use xmlParser instead of htmlParser and the tags will remain unchanged. One line does the trick:
String html = "<camelCaseTag>some text</camelCaseTag>";
Document doc = Jsoup.parse(html, "", Parser.xmlParser());

I am using 1.11.1-SNAPSHOT version which does not have this piece of code.
private Tag(String tagName) {
this.tagName = tagName.toLowerCase();
}
So I checked ParseSettings as suggested above and changed this piece of code from:
static {
htmlDefault = new ParseSettings(false, false);
preserveCase = new ParseSettings(true, true);
}
to:
static {
htmlDefault = new ParseSettings(true, true);
preserveCase = new ParseSettings(true, true);
}
and skipped test cases while building JAR.

Loading settings from XML file into a Java class

I have an XML file containing a bunch of simulation settings (partial example below). I would like to load these settings into a Java class, so that the settings are available later without having to write cumbersome DOM/XPath jargon (and import the associated packages) every time I (or another programmer who isn't fluent in DOM/XPath) want to access a specific setting.
Right now I set up a number of sub-classes that represent each level of information in the XML tree, and "manually" parse out the info to all these sub-classes. The result is, for example, that if I want to get Direction number 3, I can write:
settings.setup.directions[3]
I guess this works ok, but it sure feels rigid.
Is there a smarter way of doing this? Should we just stick to the DOM and skip this conversion business? (Please no!)
Note that I am not looking for instructions on how to load an XML file -- I know how to load it into a DOM document and parse it with XPath.
<?xml version="1.0"?>
<Settings>
<Identity>
<JobNumber>1234567</JobNumber>
<SimulationName>MyTest</SimulationName>
</Identity>
<PreProcessing >
<Tolerance>0.01</Tolerance>
</PreProcessing >
<PreprocessedInputData>
<PreChewedThing></PreChewedThing>
<OtherThing></OtherThing>
</PreprocessedInputData>
<Setup>
<DomainExtent>
<XMin>260</XMin>
<XMax>290</XMax>
<YMin>523</YMin>
<YMax>565</YMax>
</DomainExtent>
<Directions>
<Direction Index = "1">0</Direction>
<Direction Index = "2">10</Direction>
<Direction Index = "3">20</Direction>
<Direction Index = "4">30</Direction>
</Directions>
</Setup>
</Settings>

You can use JAXB for this purpose, it is meant to bind XML to Java classes.
There is a useful guide on http://jaxb.java.net/guide/ and a tutorial on http://jaxb.java.net/tutorial/

If you have flexibility over the layout of the XML file, and don't especially need to use a settings class with JAXB, try Commons Configuration:
Creating an XML settings file is as easy as:
XMLConfiguration configCreate = new XMLConfiguration();
configCreate.setFileName("settings.xml");
configCreate.addProperty("somesetting", "somevalue");
configCreate.save();
Reading from the XML settings file:
XMLConfiguration configRead = new XMLConfiguration("settings.xml");
String settingValue = configRead.getString("somesetting");
See http://commons.apache.org/configuration/

In my opinion, the best and simpliest way is using Java and XPath. This is an example:
<settings>
<type>jdbc-mysql</tipus>
<usr>usr</usr>
<pass>pass</pass>
<url>jdbc:mysql://192.168.1.123:3306/notifications_db</url>
<schema>notificacions_db</schema>
<date_format>yyyy-MM-dd HH:mm:ss</date_format>
<prefix_package>false</prefix_package>
<use_ssl>false</use_ssl>
<auto_reconnect>true</auto_reconnect></settings>
Java main class Example:
public static void main(String[] args) {
XPath xpath = XPathFactory.newInstance().newXPath();
String xpathExpression = "/settings";
InputSource inputSource = new InputSource("basedao-settings.xml");
try {
NodeList lstRoot = (NodeList) xpath.compile(xpathExpression).evaluate(inputSource, XPathConstants.NODESET);
NodeList lstChilds = lstRoot.item(0).getChildNodes();
for (int i = 0; i < lstChilds.getLength(); i++) {
System.out.println(lstChilds.item(i).getLocalName());
System.out.println(lstChilds.item(i).getTextContent());
}
} catch (XPathExpressionException e) {
e.printStackTrace();
}
}