Develop Reference

Index XML files in Apache Solr as plain text

Is there any way to dump all contents of xml file in a single content field??
schema.xml
<field name="id" type="string" indexed="true" stored="true" required="true" multiValued="false" />
<field name="content" type="text_general" indexed="true" stored="true" multiValued="false" termVectors="true" termPositions="true" termOffsets="true"/>
code used for indexing
HttpUrlConnection solrHttpURLConnection = "http://localhost:7892/solr/myCore/update/extract?literal.id=1234&commit=true "
solrHttpURLConnection.setDoOutput(true);
solrHttpURLConnection.setDoInput(true);
solrHttpURLConnection.setUseCaches(false);
solrHttpURLConnection.setAllowUserInteraction(false);
solrHttpURLConnection.setRequestProperty("Content-type", type);
solrHttpURLConnection.connect();
i am taking outputstream from this url and writing contents by taking input stream from dataServer.
NOTE:
the above code works for all file formats except xml,csv and json.
no error message is coming from solr
Sample XML File
<?xml version="1.0" encoding="UTF-8"?>
<content>just a test
</content>

Set the content type to "text/xml"
Add the following lines to your code:
OutputStreamWriter writer = new OutputStreamWriter(solrHttpURLConnection.getOutputStream());
writer.write(your_xml_file);
writer.flush();
Execute the request with this url http://localhost:7892/solr/myCore/update?literal.id=1234&commit=true
For json files use /update/json/docs
Please also check this documentation about uploading data with index handlers https://cwiki.apache.org/confluence/display/solr/Uploading+Data+with+Index+Handlers#UploadingDatawithIndexHandlers-XMLUpdateCommands

Java Castor Unmarshal byte array

Good Day Community,
I have a question regarding castor unmarshaling in java. As mentioned I am using castor to unmarshal a webservice response that contains a byte array (byte[])
.. please refer to below:
public class ViewReportResponse {
private String reportId;
private byte[] report;
//getters and setters here ...
I have used castor before to unmarhsal webservice responses, but admittedly, the responses previously where always strings. The problem lyes with the byte array returned as I believe castor chokes on unmarshaling it.
My castor-mapping file is provided below:
<class name="com.domain.reporting.client.service.ViewReportResponse">
<map-to xml="viewReportResponse"
xsi:schemaLocation="http://domain.com/schemas/reportingService/generateReport"
ns-uri="http://domain.com/schemas/reportingService/generateReport"
ns-prefix="ns2" />
<field name="reportId">
<bind-xml name="reportId"
node="element"
type="string"/>
</field>
<field name="report">
<bind-xml name="report"
node="element"
type="bytes" />
</field>
I'm not sure what I am missing, but the message is received but fails at the point of unmarshaling.
I've attached a clip of the error below.
org.springframework.oxm.UnmarshallingFailureException: Castor unmarshalling exception; nested exception is org.exolab.castor.xml.MarshalException: unable to find FieldDescriptor for 'report' in ClassDescriptor of viewReportResponse.
Please any assistance is much appreciated.
Thank-you kindly

Solved:
The general problem is not the maping to and from the byte[]. The problem is related to the use of namespace in this case.
<field name="reportId">
<bind-xml name="ns:reportId" xmlns:ns="http://domain.com/schemas/reportingService/generateReport"
node="element"
type="string"/>
<field name="report">
<bind-xml name="ns:report" xmlns:ns="http://domain.com/schemas/reportingService/generateReport"
node="element"
type="bytes"/>
This post is now effectively solved and closed.

Why does j8583 Configparser fails with no default value to template fields?

I am using j8583 to build and parse ISO messages.
I have my template and parse config.xml in place, but when there are no default values specified to template fields it fails with NullPointerException.
Below is the template which fails with NullPointerException.
Field 3 doesn't have any default value.
<template type="0200">
<field num="3" type="NUMERIC" length="6"></field>
<field num="32" type="LLVAR">456</field>
<field num="35" type="LLVAR">4591700012340000=</field>
<field num="43" type="ALPHA" length="40">SOLABTEST TEST-3 DF MX</field>
<field num="49" type="ALPHA" length="3">484</field>
<field num="60" type="LLLVAR">B456PRO1+000</field>
<field num="61" type="LLLVAR"> 1234P</field>
<field num="100" type="LLVAR">999</field>
<field num="102" type="LLVAR">ABCD</field>

Well you're specifying field 3 to be a NUMERIC field of 6 digits and you didn't set any value. It's weird that you get NPE, should be NumberFormatException when it tries to parse the empty string.
Please file an issue on github.com/chochos/j8583 and I'll look into it (I'm the author of j8583).
In the meantime, try setting the value to 0. Or, if you don't need the field, just omit it from the template.

Solr could not load MySQL JDBC Driver

I am using Solr 4.6.1 on Mac. I try to figure out how to use data import to load data from MySQL. But I end up with fail to load JDBC Driver class. Here is what I did:
Put mysql-connector-java-5.1.29-bin.jar to ~/Tools/Solr/example/lib/
1.Configure ~/Tools/Solr/example/solr/collection1/conf/data-config.xml:
<dataConfig>
<dataSource type="JdbcDataSource"
driver="com.mysql.jdbc.Driver"
url="jdbc:mysql://localhost:3306/test"
user="root"
password="root"/>
<document>
<entity name="SolrTest"
query="select * from SolrTest">
<field column="ID" name="id"/>
<field column="Name" name="name"/>
<field column="Class" name="class"/>
<field column="Score" name="score"/>
</entity>
</document>
</dataConfig>
2.Configure ~/Tools/Solr/example/solr/collection1/conf/solrconfig.xml:
add the following under < config> < /config>
<requestHandler name="/dataimport"
class="org.apache.solr.handler.dataimport.DataImportHandler">
<lst name="defaults">
<str name="config">data-config.xml</str>
</lst>
</requestHandler>
also
<lib dir="../../../dist/" regex="solr-dataimporthandler-\d.*\.jar" />
<lib dir="../../../contrib/dataimporthandler/lib" regex=".*\.jar" />
3.Add the following to ~/Tools/Solr/example/solr/collection1/conf/schema.xml:
<field name="id" type="int" indexed="true" stored="true" required="true"/>
<field name="name" type="string" indexed="true" stored="true"/>
<field name="class" type="string" indexed="true" stored="true" />
<field name="score" type="int" indexed="true" stored="true"/>
Then run
java -jar start.jar
When I go to
http://localhost:8983/solr/#/collection1/dataimport//dataimport
it shows
No information available (idle)
When I click the execute button, I got the following error message:
31537 [Thread-15] ERROR org.apache.solr.handler.dataimport.DataImporter – Full Import
failed:java.lang.RuntimeException: java.lang.RuntimeException:
org.apache.solr.handler.dataimport.DataImportHandlerException: Could
not load driver: com.mysql.jdbc.Driver Processing Document # 1 at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:270)
at
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:411)
at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:476)
at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:457)
Caused by: java.lang.RuntimeException:
org.apache.solr.handler.dataimport.DataImportHandlerException: Could
not load driver: com.mysql.jdbc.Driver Processing Document # 1 at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:410)
at
org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:323)
at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:231)
... 3 more Caused by:
org.apache.solr.handler.dataimport.DataImportHandlerException: Could
not load driver: com.mysql.jdbc.Driver Processing Document # 1 at
org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:71)
at
org.apache.solr.handler.dataimport.JdbcDataSource.createConnectionFactory(JdbcDataSource.java:116)
at
org.apache.solr.handler.dataimport.JdbcDataSource.init(JdbcDataSource.java:64)
at
org.apache.solr.handler.dataimport.DataImporter.getDataSourceInstance(DataImporter.java:383)
at
org.apache.solr.handler.dataimport.ContextImpl.getDataSource(ContextImpl.java:99)
at
org.apache.solr.handler.dataimport.SqlEntityProcessor.init(SqlEntityProcessor.java:53)
at
org.apache.solr.handler.dataimport.EntityProcessorWrapper.init(EntityProcessorWrapper.java:74)
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:427)
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:408)
... 5 more Caused by: java.lang.ClassNotFoundException: Unable to
load com.mysql.jdbc.Driver or
org.apache.solr.handler.dataimport.com.mysql.jdbc.Driver at
org.apache.solr.handler.dataimport.DocBuilder.loadClass(DocBuilder.java:916)
at
org.apache.solr.handler.dataimport.JdbcDataSource.createConnectionFactory(JdbcDataSource.java:114)
... 12 more Caused by: org.apache.solr.common.SolrException: Error
loading class 'com.mysql.jdbc.Driver' at
org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:470)
at
org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:401)
at
org.apache.solr.handler.dataimport.DocBuilder.loadClass(DocBuilder.java:906)
... 13 more Caused by: java.lang.ClassNotFoundException:
com.mysql.jdbc.Driver at
java.net.URLClassLoader$1.run(URLClassLoader.java:366) at
java.net.URLClassLoader$1.run(URLClassLoader.java:355) at
java.security.AccessController.doPrivileged(Native Method) at
java.net.URLClassLoader.findClass(URLClassLoader.java:354) at
java.lang.ClassLoader.loadClass(ClassLoader.java:423) at
java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:789) at
java.lang.ClassLoader.loadClass(ClassLoader.java:356) at
java.lang.Class.forName0(Native Method) at
java.lang.Class.forName(Class.java:264) at
org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:454)
... 15 more
How to solve this?

Create a folder inside your solr installation directory. (say solr-4.6.1/lib)
Place the mysql-connector-java-5.1.29-bin.jar inside the folder.
Edit solrconfig.xml and put the jar's path
<lib dir="../../../lib/" regex="mysql-connector-java-5.1.29-bin.jar" />
Restart Solr.
Also please check your URL. It's having //
http://localhost:8983/solr/#/collection1/dataimport//dataimport

So for those unfortunate souls who end up here. I'm using the latest Solr Docker image (8.4) which is configured so that the solr Unix-user doesn't have any permissions to create/delete folders and whatnot. For some reason I managed to create a lib file which I took for being a directory. After debugging a while I noticed this error, and promptly just changed the
<lib dir="${solr.install.dir:../../../..}/contrib/dataimporthandler/lib" regex=".*\.jar" />
to
<lib dir="${solr.install.dir:../../../..}/contrib/dataimporthandler/" regex=".*\.jar" />
where my postgresql-42.2.10.jar driver is. Otherwise all the other steps apply (edit solrconfig.xml, add data-config.xml, add schema.xml, copy posgresql-*.jar to the container's /opt/solr-8.4.1/contrib/dataimporthandler). Maybe one thing worth of mentioning is that I used Docker's internal network URL instead of localhost: url="jdbc:postgresql://host.docker.internal:5600/my_postgres_db" but I don't think it matters.
And oh, remember to restart Solr after adding the files / making changes =).

Data Import Handler in Solr

I am trying the Data Import Handler for MySQL Database.
I added the DIhandler in solrconfig.xml, created a data-config.xml according to my database scheme and also added a field in the schema.xml which was different. I am connecting with MySQL database
After i connect and I run the dataimport?command=full-import i get this response
"00C:\solr\conf\data-config.xmlfull-importidle1102011-03-05 15:01:04Indexing completed. Added/Updated: 0 documents. Deleted 0 documents.2011-03-05 15:01:042011-03-05 15:01:040:0:0.400This response format is experimental. It is likely to change in the future."
The xml files are in this http://pastebin.com/iKebKGSZ

<field column="manu" name="manu" />
<field column="id" name="id" />
<field column="weight" name="weight" />
<field column="price" name="price" />
<field column="popularity" name="popularity" />
<field column="instock" name="inStock" />
<field column="includes" name="includes" />
Are these fields also in your schema.xml?
I couldnt see them in the pastebin link. Make sure you have all fields in your schema as well.