Develop Reference

Best parser to parse 1 GB xml data in java [duplicate]

This question already has answers here:
Processing large xml files
(5 answers)
Which is the best library for XML parsing in java [closed]
(7 answers)
Closed 4 years ago.
My Requirement is :
I have 1GB xml file and want to remove few nodes from xml file.Here removing xml nodes can be anything in entire file which is based on the input.What is the best parser in JAVA.
I'm Currently using DOM parser and it is working fine for 100MB files but it is throwing out of memory error :heap space for 1 GB file.
Can anyone suggest best approach for my code below:
public static void main(String[] args) {
DocumentBuilder docBuilder = null;
File inputFile = new File("/scratch/bigfile/final.txt");
// Parse the xml file using DOM parser
try{
DocumentBuilderFactory docBuilderFactory =DocumentBuilderFactory.newInstance();
docBuilderFactory.setExpandEntityReferences(false);
docBuilderFactory.setFeature(XMLConstants.FEATURE_SECURE_PROCESSING, true);
docBuilder = docBuilderFactory.newDocumentBuilder();
Document doc= docBuilder.parse(inputFile);
// Remove unwanted nodes from xml file
Element element1 = (Element) doc.getElementsByTagName("G_SUMMARY_ROWSET").item(0);
element1.getParentNode().removeChild(element1);
Element element2 = (Element) doc.getElementsByTagName("G_JRNLSOURCE_ROWSET").item(0);
element2.getParentNode().removeChild(element2);
Element element3 = (Element) doc.getElementsByTagName("G_JRNLSOURCE_UNMATCHED_ROWSET").item(0);
element3.getParentNode().removeChild(element3);
Element element4 = (Element) doc.getElementsByTagName("G_JRNLDETAILS_UNMATCHED_ROWSET").item(0);
element4.getParentNode().removeChild(element4);
// Convbert Dom Document to Byte array
TransformerFactory transformerFactory = TransformerFactory.newInstance();
Transformer transformer = transformerFactory.newTransformer();
DOMSource source = new DOMSource(doc);
ByteArrayOutputStream bos=new ByteArrayOutputStream();
StreamResult result=new StreamResult(bos);
transformer.transform(source, result);
byte []array=bos.toByteArray();
System.out.println(array.length);
}
catch (Exception e) {
e.printStackTrace();
}
}

Consider using SAXParser. It is generally better to use a SAXParser for larger files because the data is not stored in memory and discards most elements after they have been processed. This would solve your issue of running out of memory.
This is contrasted with a DOM (Document Object Model) parser where the entire document is loaded into memory.

How Can I convert a tilde separated csv file into xml in java

1RW~800~4~22~1~48:2~true
1RW~800~16~39~1~48:2~true
1RW~800~640~330~1~48:2~true
1RW~800~40~124~1~48:2~true
this is my csv file text and i want to read this and put each value in a row separated by ~ into different xml element . I will loop through this and form 4 different xmls

You can use OpenCSV (short tutorial here) to read your CSV file and write it to a XML.
Here is one way this could be done using the mentioned technologies
// create the CSVReader
CSVReader reader = new CSVReader(new FileReader("yourfile.csv"), '~');
// create the DocumentBuilder which will create our xml documents
DocumentBuilderFactory docFactory = DocumentBuilderFactory.newInstance();
DocumentBuilder docBuilder = docFactory.newDocumentBuilder();
// create the transoformer which will make the xml ready for storing
TransformerFactory transformerFactory = TransformerFactory.newInstance();
Transformer transformer = transformerFactory.newTransformer();
// loop through the 4 rows
for (int i = 0; i < 4; i++) {
// read one row from the csv
String [] line = reader.readNext()
// create an xml document
Document doc = docBuilder.newDocument();
Element rootElement = doc.createElement("data-"+i);
doc.appendChild(rootElement);
// iterate over the entries in the row
for (String s : line) {
Element element = doc.createElement("element");
element.appendChild(doc.createTextNode(s));
rootElement.appendChild(element);
}
// finally save it:
DOMSource source = new DOMSource(doc);
StreamResult result = new StreamResult(new File("file_"+i+".xml"));
transformer.transform(source, result);
System.out.println("File " + i + " saved!");
}

Adding element to XML using javax parser without document modification

i am trying to add elements to xml document. Elements are added successfuly but problem is, that parser modifies original xml file in other places e.g it swaps namespace and id attributes or deletes duplicate namespace definitions. I need to get precisely the same document (same syntax, preserved whitespaces) only with specific elements added. I would greatly appreciate any suggestions. Here is my code:
public void appendTimestamp(String timestamp, String signedXMLFile, String timestampedXMLFile){
DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
try{
DocumentBuilder builder = factory.newDocumentBuilder();
Document doc = builder.parse(new File(signedXMLFile));
XPath xPath = XPathFactory.newInstance().newXPath();
NodeList list = (NodeList)xPath.evaluate("//*[local-name()='Signature']/*[local-name()='Object']/*[local-name()='QualifyingProperties']", doc, XPathConstants.NODESET);
if(list.getLength() != 1){
throw new Exception();
}
Node node = list.item(0);
Node unsignedProps = doc.createElement("xades:UnsignedProperties");
Node unsignedSignatureProps = doc.createElement("xzep:UnsignedSignatureProperties");
Node timestampNode = doc.createElement("xzep:SignatureTimeStamp");
timestampNode.appendChild(doc.createTextNode(timestamp));
unsignedSignatureProps.appendChild(timestampNode);
unsignedProps.appendChild(unsignedSignatureProps);
node.appendChild(unsignedProps);
Transformer transformer = TransformerFactory.newInstance().newTransformer();
transformer.setOutputProperty(OutputKeys.INDENT, "no");
transformer.setOutputProperty(OutputKeys.METHOD, "xml");
transformer.setOutputProperty(OutputKeys.OMIT_XML_DECLARATION, "yes");
DOMSource source = new DOMSource(doc);
StringWriter writer = new StringWriter();
StreamResult stringWriter = new StreamResult(writer);
transformer.transform(source, stringWriter);
writer.flush();
System.out.println(writer.toString());
}catch(Exception e){
e.printStackTrace();
}
}
The original xml file:
...
<ds:Object Id="objectIdVerificationObject" xmlns:ds="http://www.w3.org/2000/09/xmldsig#">
...
Modified xml file:
...
<ds:Object xmlns:ds="http://www.w3.org/2000/09/xmldsig#" Id="objectIdVerificationObject">
...

If you use the dom model, then the whole xml file is read, then represented in the memory as the node tree and then saved to xml in a way determined by the writer. So it is almost impossible to preserve the original xml format as you don't have the control over it and for example whitespaces are not represented at all in the node tree.
You need to read partially the original xml and ouptut its content to the new file preserving what was read, then in the "right" place you need to add your new content adn then continue simple coapying of the original.
For example you could use the XMLStreamWriter and XMLStreamReader to achieve that as they offer "the low" level operations.
But probably it would be much easier to just copy the xml as the text line by one till you recognize the insertion point, then create new xml portion and append it as text and continue with copying.

Prevent transformer.transform( source, result ) from escaping special character

I'm updating node and text content of the xml using DOM parser. To save that DOM parser I'm using transformer.transform method.
Below is the sample code.
String xmlText = "<uc>abcd><name>mine</name>efgh\netg<tag>sd</tag></uc>";
DocumentBuilderFactory documentBuilderFactory = DocumentBuilderFactory.newInstance();
DocumentBuilder documentBuilder = documentBuilderFactory.newDocumentBuilder();
InputSource inStream = new InputSource();
inStream.setCharacterStream(new StringReader(xmlText));
Document document = documentBuilder.parse(inStream);
Node node = document.getDocumentElement();
node.normalize();
NodeList childNodes = node.getChildNodes();
for(int i=0; i<childNodes.getLength(); i++) {
if(childNodes.item(i).getNodeType() == Node.TEXT_NODE) {
System.out.println(childNodes.item(i).getTextContent());
childNodes.item(i).setTextContent("123>");
}
}
TransformerFactory tFactory = TransformerFactory.newInstance();
Transformer transformer = tFactory.newTransformer();
transformer.setOutputProperty(OutputKeys.OMIT_XML_DECLARATION, "yes");
transformer.setOutputProperty(OutputKeys.ENCODING, "US-ASCII");
transformer.setOutputProperty(OutputKeys.INDENT, "yes");
DOMSource source = new DOMSource( document );
OutputStream xml = new ByteArrayOutputStream();
StreamResult result = new StreamResult( xml );
transformer.transform( source, result );
String formattedXml = xml.toString();
System.out.println(formattedXml);
Since my updated document is having text content like ">", transformer.transform method is changing it to &g t;
Is there a way to get the output without escaping special characters.
I can't use other parser because of some project constraints.
I can't use StringEscapeUtils.unescapeXml(). The reason is xml can have &g t;. If i use this utility method, &g t; which was originally present in the xml will also get changed.
So i want a mechanism which will not escape any special character.

The transformer you create with
Transformer transformer = tFactory.newTransformer();
is initialized with a default stylesheed that implements the identity transformation. That means it will simply serialize your DOM to a well-formed XML document. Output escaping is automatically applied where necessary.
If you want better control over the output, and possibly generate something that does not adhere to XML document structures, you can use a custom stylesheet that switches the output method to text. This way you control more of the structure but can do more mistakes in the XML area.
More information at
https://docs.oracle.com/en/java/javase/11/docs/api/java.xml/javax/xml/transform/TransformerFactory.html#newTransformer()
https://www.w3.org/TR/xslt20/#element-output

Writing XML in Java FileNotFoundException

i am facing some problem for this following codes
try {
DocumentBuilderFactory docFactory = DocumentBuilderFactory.newInstance();
DocumentBuilder docBuilder = docFactory.newDocumentBuilder();
//root elements
Document doc = docBuilder.newDocument();
Element rootElement = doc.createElement("subcompany");
doc.appendChild(rootElement);
//id elements
Element id = doc.createElement("id");
id.appendChild(doc.createTextNode(subCompanyId != null ? subCompanyId : " "));
rootElement.appendChild(id);
TransformerFactory transformerFactory = TransformerFactory.newInstance();
Transformer transformer = transformerFactory.newTransformer();
DOMSource source = new DOMSource(doc);
transformer.setOutputProperty(OutputKeys.INDENT, "yes");
transformer.setOutputProperty("{http://xml.apache.org/xslt}indent-amount", "2");
String xmlPath = "/project/MallDirectory/mall";
//EDITED for File creation before writing.
boolean isFileCreated = new File(xmlPath, "subcompany.xml").createNewFile();
System.out.println(isFileCreated);
StreamResult result = new StreamResult(new File(xmlPath, "subcompany.xml"));
transformer.transform(source, result);
} catch (Exception ex) {
ex.printStackTrace();
}
After i run, i get this following error:
javax.xml.transform.TransformerException: java.io.FileNotFoundException: file:/project/MallDirectory/mall/subcompany.xml (No such file or directory)
It used to work on my other project, however not this time round. What exactly went wrong here ?
EDITED:
Here is the path that i am trying to write into. The file is created, but it is empty.

I managed to solve the problem.
Here is the Error:
javax.xml.transform.TransformerException: java.io.FileNotFoundException: file:/project/MallDirectory/mall/subcompany.xml (No such file or directory)
I am thinking maybe the transformer is trying to write the xml to this path 'file:/project/MallDirectory/mall/subcompany.xml'. I don't know how it happened, as i have specifically set the file path '/project/MallDirectory/mall/subcompany.xml', and does not prefixed with 'file:/'.
Therefore, i somehow managed to fix it by doing this:
...
//ERROR CODE:
//StreamResult result = new StreamResult(new File(xmlPath, "subcompany.xml"));
//
StreamResult result = new StreamResult(new File(xmlPath, "subcompany.xml").getPath());
transformer.transform(source, result);
...

The number of files that can be in open state at any point of time is specific to the OS (offcourse ,Can be configured) and you have reached the upper limit of that.Look in the code base of your application ,whehter you have some code that is trying to open a file but not closing the stream after its use.Check for such codes.

The directory path you have defined is incorrect. Take a look at the JavaDoc to determine which form of the directory path you need to get to your file location.
JavaDoc java.io.File

If "/project" is your project name then try "./MallDirectory/mall" else try "./project/MallDirectory/mall". Please observe the string carefully it contains dot in it.