xml to json conversion using AtlasMap - java

While exploring the efficient conversion tool for xml to json, I find AtlasMap! (https://atlasmap.io) is a very useful.
AtlasMap standalone application helps to formulate source to target mapping file. It supports transforming data to and from XML, JSON and Java objects.
Most important thing, my project is heavily Apache camel depended and it supports camel component too.
Though it is working fine for small payload, it is giving very bad performance for large payload for xml to json conversion.
I believe it is using DOM parser for xml processing, if so how can I configure STAX or Woodstox parser? Or what to do for large xml to json conversion if I want to use AtlasMap? Thanks

Thank you for your interest in AtlasMap. Unfortunately we haven't supported stream processing for XML/JSON yet, but it's definitely on the table. If you could file an issue on our GitHub repo and attracts other people's vote, it helps to be prioritized.
https://github.com/atlasmap/atlasmap
cf. https://github.com/atlasmap/atlasmap/issues/45

Related

ElasticSearch : plain text file instead of JSON

Interested in elasticsearch and working with txt files not json. Can elasticsearch support plain text file? If yes, Is there any java API, I can use ( I tested crud operations with postman on JSON document and it's working fine )
Thanks for the help.
No, the elasticsearch document api supports only JSON.
But there is a workaround for this problem using ingestion pipelines running on ingestion nodes in your cluster https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest.html
. By defaut each elasticsearch server instance is a ingestion node.
Please have a look on this wery well described approach for CSV https://www.elastic.co/de/blog/indexing-csv-elasticsearch-ingest-node which is easily adaptable for flat files.
Another option is to use a second tool like Filebeat or Logstash for file ingestion. Have a look here: https://www.elastic.co/products/beats or here https://www.elastic.co/products/logstash
Having a Filebeat in place will solve many problems with minimal effort. Give it a chance ;)

Apache Solr, SolrJ vs Data Import Handler for parsing XML

I'm hoping to use Solr to run searches from info parsed from XML files.
These XML files are not in Solr's document format, as such I have to parse them and get the fields I need that way.
I am familiar with Java programming and was wondering if SolrJ would be an easier method than using the Data Import Handler. I'm considering running through each XML file I have and parsing the fields that I need from each. Is there any downside to one method over the other? I imagine since I have familiarity with Java it may be easier to parse the XML that way?
I will probably need multiple conditions and regular expressions. If anything, a reliable way to get my fields from relatively unstructured XML.
How would SolrJ work with the interface? That is, if I index using SolrJ, can I do my queries through the interface still?
DIH was designed for prototyping, though some people do use it for production. You can start from it, but be ready to jump to SolrJ or other methods if you hit its limitations. And if you have very complex mappings, you may be better off with SolrJ to start from.
You can also apply XSLT transform on an incoming XML document to map it to the Solr format.
And as said elsewhere, search is a separate issue from indexing.
How you index your content into Solr is orthogonal to how you query it. You can index any way you want, as long as it produces the right docs in the index.
Now, regarding indexing, if DIH will get what you need without much tweaking go for it. But if you need to do a lot of tweaking of the data, in the end you might finish faster if you just write some java with Solr. With Solr you have all the flexibility, with DIH you are more constrained (think of the 80/20 rule).

How to create an XML document programmatically in Java?

Given a pre-described set of tags which I can use(for example, a structure like the AndroidManifest.xml), how can I create an XML document (for a manifest file) from scratch in Java, preferably not using any third party libraries?
Also, what is(if any) the best way to do this?
There are many ways to generate XML without third party framework
SAX and DOM (powerful yet complex)
Jax-B (relatively easy)
The simplest way to generate an XML is to use Jax-B (which comes as a standard from JEE6 onwards).Refer here for more details on Jax-B
But if your trying to implement this in Android(since you mentioned AndroidManifest.xml) then I recommend you to use some third party framework like Simple to avoid unwanted complications, because it is more lightweight than the rest of the above mentioned technologies and easy to use.
If your strict on not using third party library(which I do not recommend), then use SaxParser in Android, refer here for some examples.
Hope this helps
Overview of Java DOM API for XMLType
Oracle XML DB supports the Java Document Object Model (DOM) Application Program Interface (API) for XMLType. This is a generic API for client and server, for both XML schema-based and non-schema-based documents. It is implemented using the Java package oracle.xdb.dom. DOM is an in-memory tree-based object representation of XML documents that enables programmatic access to their elements and attributes. The DOM object and interface are part of a W3C recommendation. DOM views the parsed document as a tree of objects.
http://docs.oracle.com/cd/B19306_01/appdev.102/b14259/xdb11jav.htm
http://www.c-sharpcorner.com/UploadFile/yougerthen/how-to-generate-an-xml-document-programmatically-using-system-xml-xmlwriter-part-iii/

What are the differences between mapping,binding and parsing?

I am starting to learn web-services in java EE6.
I did web development before, but never nothing related to web services.
All is new to me and the books and the tutorials i find in the web are to technical.
I started learning about .xsd schemas and also .xml.
In that topic i feel confident, i understand what are the schemas used for and what validation means.
Now my next step is learning about JAX-B(Java Api for XML Binding). I rode some about it and i did also some practice in my IDE. But i have lots of basic doubts, that make me stuck and cannot feel confident to continue to the next topic.
Ill appreciate a lot if someone could explain me well my doubts:
What does it mean mapping and what is a mapping tool?
What does it mean binding and what is a binding tool?
What does it mean parsing and what is a parsing tool?
How is JAX-B related to mapping,binding and parsing?
I am seeking for a good answer built by you, not just a copy paste from google(Ive already been online a few hours and only got confused).
Based on what I understand..
What does it mean mapping and what is a mapping tool?
In case of Java/XML, mapping is nothing but representing a Java object model
in to an XML document representation ( and vice versa. )
A mapping tool will allow you to convert from one format to another. This is just the definition step.
What does it mean binding and what is a binding tool?
Binding is the process of in-memory(as the application is running) conversion of XML document to object represantation.
Binding is achieved through unmarshalling.
What does it mean parsing and what is a parsing tool?
Parsing is reading an input stream of data and checking whether if the stream of data coforms to certain grammar. Parsing tools consume stream of data and generate errors when the data fails to conforms to grammar that the tool is checking. It would also generate events to indicate that it has received certain "tokens" from the stream. In java/xml scenario, there are multiple types of parsers such as DOM, StAx, SAX...
How is JAX-B related to mapping,binding and parsing?
JAXB mapping:
is when you use xjc to generate java class hierachy based on an XSD
The mapping occurs when classes are generated with JAXB annotations
Mapping tool in the scenario is xjc
JAXB Binding :
Occurs when an application unmarshalls an XML document to Object represanation
(JAXBElement) unmarshaller.unmarshal(
new File("some.xml"));
Parsing :
in order to convert XML document to object represantation, the JAXB engine has to
first "parse" the xml document to ensure correctness and then tokenize to instatiate java objects. This happens internally and you do not control ito ensure correctness and then tokenize to instatiate java objects.
Would strongly suggest that you get a copy of Java Web Services Up and Running and start at chapter 1 and start from there. It is not easy or intuitive, get ready for a long learning curve. Note. All the binding stuff will just confuse you unless you have a basic understanding of how all the pieces work.

Declarative XML -> POJO conversion

I have to write a process (in Java) which periodically hits a URL, reads the returned XML document, and persists that data into the DB. This data is further used by my application, so I have modeled them as Hibernate-mapped POJOs.
I can parse the XML and then create appropriate POJOs, but I was looking for a simpler declarative approach. What libraries are available which can take a input configuration and create the POJOs from the XML document?
Another alternative could be JiBX
Also, although you said you don't want to parse the XML, XPath can be a very concise way of extracting the content you are interested in?
JAXB can automatically create classes based on an XML Schema (assuming you have one for the XML source). At runtime, it can then convert the XML document into POJOs representing the XML. It is declarative in that you can tweak the Schema-to-class mapping, a little.
If I understand your task correctly, this is pretty much the use-case JAXB was designed for (though it can do other things too). It's part of Java 1.6 (maybe 1.5 too?), in packages: javax.xml.bind.*
You can use XStream to deserialize the XML and map it directly to the Hibernate-mapped POJOs.
Cheers.
Using Hibernate you can directly map XML to table. This is experimental feature. Check here
http://www.hibernate.org/hib_docs/v3/reference/en-US/html/xml.html
EclipseLink JAXB (MOXy) has extensions for mapping JPA entities to XML (JPA entities have things like embedded ID classes, lazy loading, and compound key relationships that need special handling), I'm not aware of any other OXM solution that does this.
For more information see:
http://wiki.eclipse.org/EclipseLink/Examples/MOXy/JPA

Categories

Resources