Develop Reference

How to parse a big rdf file in rdf4j

I want to parse a huge file in RDF4J using the following code but I get an exception due to parser limit;
public class ConvertOntology {
public static void main(String[] args) throws RDFParseException, RDFHandlerException, IOException {
String file = "swetodblp_april_2008.rdf";
File initialFile = new File(file);
InputStream input = new FileInputStream(initialFile);
RDFParser parser = Rio.createParser(RDFFormat.RDFXML);
parser.setPreserveBNodeIDs(true);
Model model = new LinkedHashModel();
parser.setRDFHandler(new StatementCollector(model));
parser.parse(input, initialFile.getAbsolutePath());
FileOutputStream out = new FileOutputStream("swetodblp_april_2008.nt");
RDFWriter writer = Rio.createWriter(RDFFormat.TURTLE, out);
try {
writer.startRDF();
for (Statement st: model) {
writer.handleStatement(st);
}
writer.endRDF();
}
catch (RDFHandlerException e) {
}
finally {
out.close();
}
}
The parser has encountered more than "100,000" entity expansions in this document; this is the limit imposed by the application.
I execute my code as following as suggested on the RDF4J web site to set up the two parameters (as in the following command)
mvn -Djdk.xml.totalEntitySizeLimit=0 -DentityExpansionLimit=0 exec:java
any help please

The error is due to the Apache Xerces XML parser, rather than the default JDK XML parser.
So Just delete Xerces XML folder from you .m2 repository and the code works fine.

Why can't avro take the schema from the .avro file?

Here is the deserializer from tutorialspoint.
public class Deserialize {
public static void main(String args[]) throws Exception{
//Instantiating the Schema.Parser class.
Schema schema = new Schema.Parser().parse(new File("/home/Hadoop/Avro/schema/emp.avsc"));
DatumReader<GenericRecord> datumReader = new GenericDatumReader<GenericRecord>(schema);
DataFileReader<GenericRecord> dataFileReader = new DataFileReader<GenericRecord>(new File("/home/Hadoop/Avro_Work/without_code_gen/mydata.txt"), datumReader);
GenericRecord emp = null;
while (dataFileReader.hasNext()) {
emp = dataFileReader.next(emp);
System.out.println(emp);
}
System.out.println("hello");
}
}
My question is: If there is already a schema in the .avro file why do I have to pass the schema as well? I find it very inconvenient having to provide the schema in order to parse the file.

Avro requires two schemas for resolution - a reader schema and a writer schema.
The writer schema is included in the file.
And you can parse the schema out of the file
String filepath = ...;
DataFileReader<Void> reader = new DataFileReader<>(Util.openSeekableFromFS(filepath),
new GenericDatumReader<>());
System.out.println(reader.getSchema().toString(true));
This is how java -jar avro-tools.jar getschema works
And you may need the Util.openSeekableFromFS method since it seems to be package private

cannot get cdata result using xstream, stax2, woodstox

sample xml
<root>
<first>
<second><![CDATA[hello, world]]></second>
</first>
<root>
I'm using xstream(1.4.7), stax2-api(3.1.4), woodstox(5.0.3).
#Test
public void xmlInputFactoryTest() throws XMLStreamException, IOException {
ClassPathResource resource = new ClassPathResource("/sample/input.xml");
NameCoder nameCoder = new XmlFriendlyNameCoder();
XMLInputFactory factory = XMLInputFactory.newInstance();
XMLEventReader reader = factory.createXMLEventReader(resource.getInputStream());
XMLStreamReader streamReader = StaxUtils.createEventStreamReader(reader);
HierarchicalStreamReader hsr = new StaxReader(new QNameMap(), streamReader, nameCoder);
write(hsr);
}
public void write(HierarchicalStreamReader hsr) {
System.out.println(hsr.getNodeName() + ", " + hsr.getValue()); // should be print "hello,world" in second tag
if (hsr.hasMoreChildren()) {
hsr.moveDown();
write(hsr);
}
}
here is my sample test code
I can not get cdata string result in path "root/first/second"
I think XMLInputFactory.newInstance() return class affect to result.
when I add
factory.setProperty(XMLInputFactory.IS_COALESCING, true);
this code after get XMLInputFactory.newInstance().
It works.
but it does not seem to be the answer.
.
Is not version compatible?
Is there any other way possible?

java - spring - xsl transformation dtd not found error

I have a code which transform xml file using xsl, my peace of code as following. My problem is when i run the execution point it gives me following error.
StackTrace: javax.xml.transform.TransformerException: javax.xml.transform.TransformerException: com.sun.org.apache.xml.internal.utils.WrappedRuntimeException: /home/app/myapp/bin/xhtml11-flat.dtd (No such file or directory)
at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(TransformerImpl.java:720)
at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(TransformerImpl.java:313)
at com.core.util.XmlUtils.transform(XmlUtils.java:151)
at com.core.util.XmlUtils.transform(XmlUtils.java:147)
Long story short it is trying to find dtd file inside the bin directory from where i executed the application.
/home/app/myapp/bin/xhtml11-flat.dtd
I have the xhtml11-flat.dtd file if i copy this file in bin directory it works fine, instead of the bin directory i want to load it from classpath any idea how can i achieve this with minimum changes ?
I don't know from where it is laoding .dtd code so that i can set my path in it.
//Execution Point
function transform(){
Templates templates = getTemplates();
StringWriter result = new StringWriter();
XmlUtils.transform(templates.newTransformer(), input, new StreamResult(result));
...
}
private Templates getTemplates() throws Exception {
if (templates == null) {
templates = XmlUtils.createTemplates(XslRdcSourceDocTransformer.class.getResourceAsStream("/xsl/" + getXslFileName()));
}
return templates;
}
public static Templates createTemplates(InputStream stream) throws Exception {
TransformerFactory tfactory = TransformerFactory.newInstance();
return tfactory.newTemplates(new StreamSource(stream));
}

Your xml files are probably containing a doctype declaration containing a relative path to the dtd:
<!DOCTYPE html SYSTEM "xhtml11-flat.dtd">
The transformer api tries to resolve this path to the current working directory of the java program. To customize how the path is resolved you need to implement a EntityResolver. This EntityResolver can return an InputSource referring to a copy of the dtd loaded from the classpath.
public InputSource resolveEntity(final String publicId, final String systemId) throws SAXException {
if ("xhtml11-flat.dtd".equals(systemId)) {
ClassLoader cl = Thread.currentThread().getContextClassLoader();
InputSource is = new InputSource();
is.setSystemId(systemId);
is.setByteStream(cl.getResourceAsStream("/com/example/dtd/xhtml11-flat.dtd"));
return is;
} else {
return null;
}
}
How you use this class depends on the type of source for your transformation. For a DOMSource you have to configure the DocumentBuilder:
DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
factory.setValidating(false);
factory.setNamespaceAware(true);
DocumentBuilder builder = factory.newDocumentBuilder();
DocumentBuilder builder = ...
builder.setEntityResolver(entityResolver);
Source source = new DOMSource(builder.parse(inputStream));
For a SAXSource the setting is on the XMLReader instance:
SAXParserFactory factory1 = SAXParserFactory.newInstance();
factory1.setValidating(false);
factory1.setNamespaceAware(true);
SAXParser parser = factory1.newSAXParser();
XMLReader xmlreader = parser.getXMLReader();
xmlreader.setEntityResolver(entityResolver);
Source source = new SAXSource(xmlreader, new InputSource(stream));
The code for the transformation is the same regardless of the source type and should look similar to the code you currently have in your XmlUtils class:
Templates templates = ...
Result result = new StreamResult(...);
Transformer transformer = templates.newTransformer();
transformer.transform(source, result);

SyndFeedInput().build in Java: Cannot access org.jdom.Document class file for org.jdom.Document not found

i'm using Netbeans IDe 7.0.1.
I'm testing a program in Java which is using ROME in order to parse the xml.
public class RSSNew {
public static void main(String[] args) throws Exception {
URL url = new URL("RSS URL");
XmlReader reader = null;
try {
reader = new XmlReader(url);
SyndFeed feed = new SyndFeedInput().build(reader); /* HERE */
}
finally {
if (reader != null)
reader.close();
}
}
}
The error is:
**cannot access org.jdom.Document
class file for org.jdom.Document not found
SyndFeed feed = new SyndFeedInput().build(reader);
Note: C:\Users\User PC\Documents\NetBeansProjects\RSS\src\rss\RSS.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
1 error**
Have you eperienced an error like this?
Thanks, in advance
ps. i have added the following jar files in my project which are:
feed4j.jar
rome-1.0.jar
rome-1.0-javadoc.jar

I just had the same error message - you need to add jdom.jar to your project classpath as well, it's used by rome. You can get it from here: http://www.jdom.org/dist/binary/