I have a OMElement object and from that I want to get an InputStream or reader object. What I want is to stream the xml from the OMElement which I have, without getting loaded into memory. I only could be able to get XMLStreamReader object from that. But I can't find a way to get InputStream/Reader out of that.
OMElement element /*I have this object */
XMLStreamReader xmlreader = element.getXMLStreamReaderWithoutCaching();
Can anyone please help me ? I really appreciate!
Thanks
PS: I have come a way to get an InputStream but it gives an exception which I need help to resolve it.
XMLStreamReader xmlReader = element.getXMLStreamReader(false);
try {
if (xmlReader.getEventType() == XMLStreamReader.START_DOCUMENT) {
xmlReader.next();
}
DataHandler handler = XMLStreamReaderUtils.getDataHandlerFromElement(xmlReader);
handler.getInputStream();
The call to getDataHandlerFromElement generate the exception javax.xml.stream.XMLStreamException: Error during base64 decoding
The purpose of the getDataHandlerFromElement method is to decode base64 encoded binary data contained in an element, which is not what you want.
Axiom currently doesn't have a streaming pull serializer (and I don't know any other XML library that supports this). It only supports serializing to XML in push mode, i.e. by writing to an OutputStream or Writer. You could do the serialization in a separate thread and use a PipedOutputStream/PipedInputStream (similar to what Activation does for DataHandler#getInputStream() for a DataHandler that is not backed by a DataSource). This would satisfy the constant memory requirement, but I guess that the overhead caused by using a separate thread would not be acceptable in your case.
I thought about the possibility to implement such a pull serializer in the past, and I think it's technically feasible to do that (without using a separate thread). Please open a feature request for Axiom so that we can implement that in one of the next releases. I think it would be an interesting addition to Axiom.
Related
I'm trying to parse a web service response message in the following format (message tree):
Message
Properties
Properties..[]
DFDL
ObjectIWantUnmarshalled
AllItsDataIwant[]
And unmarshal the "ObjectIWantUnmarshalled". However, this data is in DFDL format.
In my request, I use the following line in order to format from XML to DFDL:
Document outDocument = outMessage.createDOMDocument(MbDFDL.PARSER_NAME);
But there doesn't seem to be a way to to the opposite, of DFDL to XML.
I have tried:
Document outDocument = inMessage.createDOMDocument(MbXMLNSC.PARSER_NAME);
As well as other attempts to simply unmarshal the data directly from the MbMessage:
jaxbContext_COBOL.createUnmarshaller().unmarshal(inMessage.getDOMDocument())
But I have not been able to get a Document node this way, or any other way, it is always null.
Probably a lot too late, but you were going about this the wrong way.
When using WMB and IIB you should use the built-in XML support - not the javax.XML.* class library. So instead of using the JAXB unmarshaller, you should
create an XMLNSC tree under the output message root
copy the input DFDL message tree to the output XMLNSC message tree ( one line )
...and the message flow will serialize ( unmarshall ) the tree as XML whenever it needs to - when it encounters an output node, or when you call outMessage.toBitstream().
I am reading from an API provided by a company, but the problem is that one of the accounts from which I am getting the data has around 22000 json objects, it reads fine with small amounts of data, i would say up to 8000 records, but then I get issues like the json is not well formatted besides the problem of being able to read the response.
The response comes this way:
<?xml version="1.0" encoding="utf-8"?>
<string xmlns="http://ywers.com">
[{"Name":"Edward", "LastName":"Jones", "Address":"{accepted}"}
,{"Name":"Carlos", "LastName":"Ramirez", "Address":"{Rejected}"}, ....... 22k more records here]</string>
I asked for some help earlier on here for the best way to do this, and i got a response about reading it using the xml parser and then a json parser, i am using GSON.
String XML = "<Your XML Response>";
XPathExpression xpath = XPathFactory.newInstance()
.newXPath().compile("/*[local-name()='string']");
String json = xpath.evaluate(new InputSource(new StringReader(XML)));
and then
JSONArray jsonRoot = new JSONArray(json.trim());
System.out.println(jsonRoot.getJSONObject(0).getString("Address")); // {accepted}
The problem with this is approach i am having is that it throws errors when reading the XML, it starts reading but after a while it stops with errors like:
java.lang.OutOfMemoryError
at java.lang.AbstractStringBuilder.enlargeBuffer(AbstractBuilder.java:94)
at java.lang.StringBuffer.append(StringBuffer.java:219)
at org.apache.harmony.xml.dom.CharacterDataImpl.appendData(CharacterDataImpl.java:43)
......
I would appreciate any advise on how to proceed with this, I am kind of new to android.
I don't know who would wrap 22k objects inside a xml string, but apparently someone is doing that. From my experience, your out of memory is because the you try to convert all the response to string but the response is too big to be handled. I recommend you to stream the JSON data. You can do stream the JSON data from the inputstream response that you get from the your HTTP post, but you need to skip the XML part by creating another input stream from the original response input stream and skip the XML part
Before I use the streaming API from google GSON I also got OOM error because the JSON data I got is very big data (many images and sounds in Base64 encoding) but with GSON streaming I can overcome that error because it reads the data per token not all at once. And for alternative you can also use Jackson JSON library I think it also have streaming API and how to use it almost same with my implementation with google GSON. I hope my answer can help you and if you have another question about my answer feel free to ask in the comment :)
If a mail is send to my inbox, I recieve a message, and I'm inserting the contents into DB.
I have a org.springframework.integration.core.Message something like follows:
public void receive(Message<?> message)
{
//I am inserting message contents into DB
}
Now in the event of failure, I wanted to have fail safe recovery mechanism, what I am thinking is to serialize the Message object into a file and later deserialize and update to DB.
Question
1. In this situation how to serialize the Message object?
2. Other than serialization any other mechanism that can be used?
EDIT
I have not done Serialization before, I heard like the class should implements Serializable in order to use ObjectOutputStream, in this case I don't want to create a subclass of Message, So how to serialize Message to File?
Sure, there are many serialization mechanisms apart from the jvm one.
XML
JSON
BSON
MessagePack
protobufs
...
Some of them are text-based, some are binary. All have drawbacks and pluses. Text-based ones are human-readable, binary ones are faster and take up less space.
There are java libraries that handle all the above formats: JAXB (XML), Jackson (JSON), etc.
In this situation how to serialize the Message object? Other than serialization any other mechanism that can be used?
Extract all the data you need from the Message and save it. You can do this in any manner you choose.
You can deserialize it by populating a new Message with the data you saved.
I don't know if I probably understood it al right.. but assuming Message is not much more than lots of strings and some integers you can just use directly an ObjectOutputStream and write it to a file (binary) and then readin later. Why not?
Message e = new Message();
ObjectOutputStream oos = new ObjectOutputStream(new FileOutputStream("whatever");
oos.writeObject(message);
// read in
ObjectInputStream ois = new ObjectInputStream(new FileInputStream("whatever");
Message e = (Message) ois.readObject();
I need to seek into a huge XML file, start reading XML data from a byte position, basically reading a chunk. I'm creating an XMLStreamReader like this:
final XMLInputFactory factory = XMLInputFactory.newInstance();
reader = factory.createXMLStreamReader(inputStream);
This reader can start reading at any position in the stream, but I want it to skip stuff until it finds the start of a tag. I do not want it to validate anything.
For example, it could start reading this, in the middle of firstAttr:
rstAttr='12345' secondAttr='QWERTY'>
</endElement>
<startReadingAtLeastFromHere>
...
I don't really need to read the </endElement>, but I want the parser to read at least <startReadingAtLeastFromHere> and all content after that.
My problem is that factory.setXMLReporter(reporter) is not invoked, so I cannot use that to tell the reader to continue.
I'm not specifying the javax.xml.stream.XMLInputFactory system property for now, and it defaults to Sun's XMLInputFactoryImpl. It uses StaxErrorReporter, which has a field named fContinueAfterFatalError, but I can't find how to change its value, or how to replace the StaxErrorReporter with my own.
If that's too tricky, is there any pull-based XML parser that can be configured to ignore errors and keep reading?
I have program that needs to parse XML that contains character entities. The program itself doesn't need to have them resolved, and the list of them is large and will change, so I want to avoid explicit support for these entities if I can.
Here's a simple example:
<?xml version="1.0" encoding="UTF-8"?>
<xml>Hello there &something;</xml>
Is there a Java XML API that can parse a document successfully without resolving (non-standard) character entities? Ideally it would translate them into a special event or object that could be handled specially, but I'd settle for an option that would silently suppress them.
Answer & Example:
Skaffman gave me the answer: use a StAX parser with IS_REPLACING_ENTITY_REFERENCES set to false.
Here's the code I whipped up to try it out:
XMLInputFactory inputFactory = XMLInputFactory.newInstance();
inputFactory.setProperty(XMLInputFactory.IS_REPLACING_ENTITY_REFERENCES, false);
XMLEventReader reader = inputFactory.createXMLEventReader(
new FileInputStream("your file here"));
while (reader.hasNext()) {
XMLEvent event = reader.nextEvent();
if (event.isEntityReference()) {
EntityReference ref = (EntityReference) event;
System.out.println("Entity Reference: " + ref.getName());
}
}
For the above XML, it will print "Entity Reference: something".
The STaX API has support for the notion of not replacing character entity references, by way of the IS_REPLACING_ENTITY_REFERENCES property:
Requires the parser to replace
internal entity references with their
replacement text and report them as
characters
This can be set into an XmlInputFactory, which is then in turn used to construct an XmlEventReader or XmlStreamReader. However, the API is careful to say that this property is only intended to force the implementation to perform the replacement, rather than forcing it to not replace them. Still, it's got to be worth a try.
Works for me only when disabling support of external entities:
XMLInputFactory inputFactory = XMLInputFactory.newInstance();
inputFactory.setProperty(XMLInputFactory.IS_REPLACING_ENTITY_REFERENCES, false);
inputFactory.setProperty(XMLInputFactory.IS_SUPPORTING_EXTERNAL_ENTITIES, false);
A SAX parse with an org.xml.sax.EntityResolver might suit your purpose. You could for sure suppress them, and you could probably find a way to leave them unresolved.
This tutorial seems the most relevant: it shows how to resolve entities into strings.
I am not a Java developer, but I "think" Java xml classes support a similar functionality to .net for accomplishing this. IN .net the xmlreadersettings class you set the ProhibitDtd property false and set the XmlResolver property to null. This will cause the parser to ignore externally referenced entities without throwing an exception when they are read. I just did a google search for "Java ignore enity" and got lots of hits, some of which appear to address this topic. I realize this is not a total answer to your question but it should point you in a useful direction.