OS: Ubuntu 12.04
I just upgraded my system from using Solr 1.4 to Solr 4.3.0 and can't seem to get the MySQL driver to work (or so I suspect). Solr seems to work fine (accessing it through the browser, etc) until I add the following lines to the solrconfig.xml
<requestHandler name="/dataimport" class="org.apache.solr.handler.dataimport.DataImportHandler">
<lst name="defaults">
<str name="config">/opt/solr/core01/conf/data-config.xml</str>
</lst>
</requestHandler>
After adding those lines into the config, restart Tomcat7 and refreshing my localhost:8080/solr, I get the following error:
HTTP Status 500 - {msg=SolrCore 'core01' is not available due to init failure: org/apache/solr/util/plugin/SolrCoreAware,trace=org.apache.solr.common.SolrException:
SolrCore 'core01' is not available due to init failure: org/apache/solr/util/plugin/SolrCoreAware at
And then a bunch of misc. filters, etc. I only see this error via the browser - I can't seem to find it in any of the logs for Tomcat.
Another thing to note is that I have included the required JAR files for the:
solr-dataimporthandler-extras-4.3.0.jar
solr-dataimporthandler-4.3.0.jar
as they seem to not come with the download for Solr 4.3.0
This problem has caused so much time to be wasted so hopefully someone on here can lend a hand and see what is wrong.
Thanks in advance!
EDIT:
This is what my data-config.xml looks like
<?xml version="1.0" encoding="UTF-8"?>
<dataConfig>
<dataSource type="JdbcDataSource" driver="com.mysql.jdbc.Driver" url="jdbc:mysql://localhost/db_name" user="db_user" password="db_password" />
<document>
<entity name="table" query="SELECT * FROM table">
<field column="id" name="id" />
<field column="name" name="name" />
</entity>
</document>
</dataConfig>
With the obvious variables replaced with real data.
After looking at the DIHQuickStart page and looking at the error meessage, you might want to remove the path from your config setting, as Solr will look in the conf folder of the core where you have defined the DIH first by default. So change it to:
<str name="config">data-config.xml</str>
Related
we are getting below exception in liberty running in linux box for jax-rs based web service application. But while running in local windows version Liberty (21.0.0.6) , it works fine and no exception thrown. Also we tried various option available in stack overflow and google, nothing worked. can anyone help us on this
E SRVE0271E: Uncaught init() exception created by servlet [JAX-RS Servlet] in application [ABC_WSv2.1]: java.lang.ClassCastException: com.abc.occws.SearchingApplication incompatible with javax.ws.rs.core.Application
server.xml configuration below
<?xml version="1.0" encoding="UTF-8"?>
<server description="new server">
<!-- Enable features -->
<featureManager>
<feature>webProfile-8.0</feature>
</featureManager>
<!-- To access this server from a remote client add a host attribute to the following element, e.g. host="*" -->
<httpEndpoint httpPort="9080" httpsPort="9443" id="defaultHttpEndpoint" />
<dataSource id="ABCDataSource" jndiName="jdbc/ABCDS" queryTimeout="20m" type="javax.sql.DataSource">
<connectionManager agedTimeout="1000m" minPoolSize="1" numConnectionsPerThreadLocal="10"/>
<jdbcDriver libraryRef="OracleLib"/>
<properties.oracle URL="jdbc:oracle:thin:#//10.135.157.5:1565/DOABC_APP" password="xxx" user="xxxx"/>
</dataSource>
<!-- Define JDBC Drivers -->
<library id="OracleLib">
<file name="C:/dev/Liberty/ojdbc8-19.3.0.0.jar"/>
</library>
<applicationMonitor updateTrigger="mbean" />
<application id="ABC_WSv2.1" location="C:\Dev\Liberty\ABC_WSv2.1.ear" name="ABC_WSv2.1" type="ear" context-root="/abc-ws">
<classloader commonLibraryRef="OracleLib"/>
</application>
<webContainer uppressHtmlRecursiveErrorOutput="true"/>
</server>
On the surface this looks like this problem is classloader related. On Windows I suspect you are running this application is isolation whereas on Linux it is running in concert with other things and likely Parent-Last classloading is involved.
I would suggest collecting Liberty classloader trace to see where the affected classes are being loaded from. Here are some instructions for doing that: Liberty Classloader Mustgather
Note: This is assuming that you are running the same version of your application on both Windows and Linux and in both cases your application class extends javax.ws.rs.core.Application. If that is not the case then that is certainly causing the problem.
I am trying to implement a custom SOLR FieldStreamDataSource. This is the code of my Java class:
package MySOLR;
import org.apache.solr.handler.dataimport.*;
public class MyFieldStreamDataSource extends FieldStreamDataSource {
}
My SOLR is installed in C:\solr\solr-7.4.0. So I added the following jar to my NetBeans project
C:\solr\solr-7.4.0\dist\solr-dataimporthandler-7.4.0.jar
and compiled it to MySOLR.jar
Then I placed MySOLR.jar into C:\solr\solr-7.4.0\server\lib\ext and added the following to my SOLR DIH config file:
...
<dataSource name="fieldStreamDataSource" type="MySOLR.MyFieldStreamDataSource" />
...
<entity name="tika_RTF_NOTES" onError="continue" processor="TikaEntityProcessor" dataField="ts0_tika.RTF_NOTES" dataSource="fieldStreamDataSource">
<field name="text" column="text" />
</entity>
...
I added the following entries to my SOLR core's solrconfig.xml file:
<lib dir="C:/solr/solr-7.4.0/contrib/dataimporthandler/lib" regex=".*\.jar" />
<lib dir="C:/solr/solr-7.4.0/dist/" regex="solr-dataimporthandler-\d.*\.jar" />
<!--lib path="C:/solr/solr-7.4.0/dist/solr-dataimporthandler-7.4.0.jar" /-->
<lib dir="C:/solr/solr-7.4.0/contrib/dataimporthandler-extras/lib" regex=".*\.jar" />
<lib dir="C:/solr/solr-7.4.0/dist/" regex="solr-dataimporthandler-extras-\d.*\.jar" />
After re-starting my SOLR instance (Windows service) I try to run the DIH via the SOLR web interface. I get the following exception in the solr.log file (and in the web interface):
2019-03-18 11:08:12.164 ERROR (qtp1986417638-24) [ x:iET] o.a.s.s.HttpSolrCall null:java.lang.RuntimeException: java.lang.NoClassDefFoundError: org/apache/solr/handler/dataimport/FieldStreamDataSource
at org.apache.solr.servlet.HttpSolrCall.sendError(HttpSolrCall.java:662)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:530)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:377)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:323)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
...
I am pretty sure that my entries in solrconfig.xml are correct as when I remove the entries I see that SOLR is picking up less libraries. I can see that in the solr.log file when looking at
o.a.s.c.SolrResourceLoader [...] Added ... libs to classloader, from paths: ...
What am I missing?
Many thanks in advance,
Michael
By chance I found the solution. There are 2 alternatives (for me):
Place MySOLR.jar in c:\solr\solr-7.4.0\contrib\dataimporthandler\lib
Place MySOLR.jar in c:\solr\solr-7.4.0\dist and add a reference to it in solrconfig.xml
I am trying to index some xml files into Solr 6.2.1 using their DataImportHandler.
For that purpose I have added the needed import and this RequestHandler into the solrconfig.xml:
<lib dir="${solr.install.dir:../../../..}/contrib/dataimporthandler/lib/" regex=".*\.jar" />
<lib dir="${solr.install.dir:../../../..}/dist/" regex="solr-dataimporthandler-.*\.jar" />
<requestHandler name="/dataimport" class="org.apache.solr.handler.dataimport.DataImportHandler" startup="lazy">
<lst name="default">
<str name="config">data-config.xml</str>
</lst>
</requestHandler>
Then I wrote the data-config.xml and put it into the same path as the solrconfig.xml:
<dataConfig>
<dataSource type="FileDataSource" encoding="UTF-8"/>
<document>
<entity name="pickupdir"
processor="FileListEntityProcessor"
dataSource="null"
baseDir="/vagrant/TREC8all/Adhoc/"
recursive="true"
fileName="^[\w\d-]+\.xml$" />
<entity name="trec8_simple"
processor="XPathEntityProcessor"
stream="true"
datasource="pickupdir"
url="${pickupdir.fileAbsolutePath}"
forEach="/DOCS/DOC">
<field column="id" xpath="/DOCS/DOC/DOCNO"/>
<field column="header" xpath="/DOCS/DOC/HEADER"/>
<field column="text" xpath="/DOCS/DOC/TEXT"/>
</entity>
</document>
</dataConfig>
This should make the ImportHandler iterate recursively through all xml files in the directory and index them according to the xpaths.
When I call the requestHandler like this: (I am running solr in a vagrant box instead of locally)
http://192.168.155.156:8983/solr/trec8/dataimport?command=full-import&entity=trec8_simple
I am getting this Exception in the solr.log:
ERROR (Thread-14) [ x:trec8] o.a.s.h.d.DataImporter Full Import failed:java.lang.NullPointerException
at org.apache.solr.handler.dataimport.DataImporter.createPropertyWriter(DataImporter.java:325)
at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:412)
at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:475)
at org.apache.solr.handler.dataimport.DataImporter.lambda$runAsync$0(DataImporter.java:458)
at java.lang.Thread.run(Thread.java:745)
Im assuming this should be the source for the DataImportHandler:
https://github.com/sudarshang/lucene-solr/blob/master/solr/contrib/dataimporthandler/src/java/org/apache/solr/handler/dataimport/DataImporter.java
I have trouble figuring out what is causing this exception and what it is meaning. Would be nice if somebody could help me out. Thanks!
EDIT:
I think this has something to do with the DataImportHandler not beeing able to finde the data-config.xml. When I remove it will throw the exact same exception
Ok I found the issue!
Problem was in the solrconfig,
<lst name="default">
<str name="config">data-config.xml</str>
</lst>
should have been
<lst name="defaults">
<str name="config">data-config.xml</str>
</lst>
Configuring my solr server, am able to start and stop the server and can see the dashboard etc.
Just created a core called "Wish". So in the wish folder of the server I added the data source details in solrconfig.xml
Here is the essential part of it
<requestHandler name="/dataimport" class="org.apache.solr.handler.dataimport.DataImportHandler">
<lst name="defaults">
<str name="config">data-config.xml</str>
<lst name="datasource">
<str name="driver">com.mysql.jdbc.Driver</str>
<str name="url">jdbc:mysql://localhost:3306/wish</str> //db name wish
<str name="user">root</str>
<str name="password"></str>
</lst>
</lst>
</requestHandler>
And the data config xml file content id
<dataConfig>
<dataSource type="JdbcDataSource"
driver="com.mysql.jdbc.Driver"
url="jdbc:mysql://localhost:3306/wish"
user="root"
password=""/>
<document name="retailer">
<entity name="retailer" query="select * from retailer"></entity>
</document>
</dataConfig>
If you look at it I am just trying to add an entitiy retailer and my retailer table consists of 2 rows so far. But when I invoke the solr API request like below, nothing showing :(
Here is the API request
http://localhost:8983/solr/wish/select?q=*:*
but always the result is
<response>
<lst name="responseHeader">
<int name="status">0</int>
<int name="QTime">0</int>
<lst name="params">
<str name="q">*:*</str>
</lst>
</lst>
<result name="response" numFound="0" start="0"/>
</response>
Is something is wrong my config ? or the way I am invoking it wrong ?
Any clues and help will be much appreciated. Thanks in advance.
I am using Solr 4.6.1 on Mac. I try to figure out how to use data import to load data from MySQL. But I end up with fail to load JDBC Driver class. Here is what I did:
Put mysql-connector-java-5.1.29-bin.jar to ~/Tools/Solr/example/lib/
1.Configure ~/Tools/Solr/example/solr/collection1/conf/data-config.xml:
<dataConfig>
<dataSource type="JdbcDataSource"
driver="com.mysql.jdbc.Driver"
url="jdbc:mysql://localhost:3306/test"
user="root"
password="root"/>
<document>
<entity name="SolrTest"
query="select * from SolrTest">
<field column="ID" name="id"/>
<field column="Name" name="name"/>
<field column="Class" name="class"/>
<field column="Score" name="score"/>
</entity>
</document>
</dataConfig>
2.Configure ~/Tools/Solr/example/solr/collection1/conf/solrconfig.xml:
add the following under < config> < /config>
<requestHandler name="/dataimport"
class="org.apache.solr.handler.dataimport.DataImportHandler">
<lst name="defaults">
<str name="config">data-config.xml</str>
</lst>
</requestHandler>
also
<lib dir="../../../dist/" regex="solr-dataimporthandler-\d.*\.jar" />
<lib dir="../../../contrib/dataimporthandler/lib" regex=".*\.jar" />
3.Add the following to ~/Tools/Solr/example/solr/collection1/conf/schema.xml:
<field name="id" type="int" indexed="true" stored="true" required="true"/>
<field name="name" type="string" indexed="true" stored="true"/>
<field name="class" type="string" indexed="true" stored="true" />
<field name="score" type="int" indexed="true" stored="true"/>
Then run
java -jar start.jar
When I go to
http://localhost:8983/solr/#/collection1/dataimport//dataimport
it shows
No information available (idle)
When I click the execute button, I got the following error message:
31537 [Thread-15] ERROR org.apache.solr.handler.dataimport.DataImporter – Full Import
failed:java.lang.RuntimeException: java.lang.RuntimeException:
org.apache.solr.handler.dataimport.DataImportHandlerException: Could
not load driver: com.mysql.jdbc.Driver Processing Document # 1 at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:270)
at
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:411)
at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:476)
at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:457)
Caused by: java.lang.RuntimeException:
org.apache.solr.handler.dataimport.DataImportHandlerException: Could
not load driver: com.mysql.jdbc.Driver Processing Document # 1 at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:410)
at
org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:323)
at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:231)
... 3 more Caused by:
org.apache.solr.handler.dataimport.DataImportHandlerException: Could
not load driver: com.mysql.jdbc.Driver Processing Document # 1 at
org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:71)
at
org.apache.solr.handler.dataimport.JdbcDataSource.createConnectionFactory(JdbcDataSource.java:116)
at
org.apache.solr.handler.dataimport.JdbcDataSource.init(JdbcDataSource.java:64)
at
org.apache.solr.handler.dataimport.DataImporter.getDataSourceInstance(DataImporter.java:383)
at
org.apache.solr.handler.dataimport.ContextImpl.getDataSource(ContextImpl.java:99)
at
org.apache.solr.handler.dataimport.SqlEntityProcessor.init(SqlEntityProcessor.java:53)
at
org.apache.solr.handler.dataimport.EntityProcessorWrapper.init(EntityProcessorWrapper.java:74)
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:427)
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:408)
... 5 more Caused by: java.lang.ClassNotFoundException: Unable to
load com.mysql.jdbc.Driver or
org.apache.solr.handler.dataimport.com.mysql.jdbc.Driver at
org.apache.solr.handler.dataimport.DocBuilder.loadClass(DocBuilder.java:916)
at
org.apache.solr.handler.dataimport.JdbcDataSource.createConnectionFactory(JdbcDataSource.java:114)
... 12 more Caused by: org.apache.solr.common.SolrException: Error
loading class 'com.mysql.jdbc.Driver' at
org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:470)
at
org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:401)
at
org.apache.solr.handler.dataimport.DocBuilder.loadClass(DocBuilder.java:906)
... 13 more Caused by: java.lang.ClassNotFoundException:
com.mysql.jdbc.Driver at
java.net.URLClassLoader$1.run(URLClassLoader.java:366) at
java.net.URLClassLoader$1.run(URLClassLoader.java:355) at
java.security.AccessController.doPrivileged(Native Method) at
java.net.URLClassLoader.findClass(URLClassLoader.java:354) at
java.lang.ClassLoader.loadClass(ClassLoader.java:423) at
java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:789) at
java.lang.ClassLoader.loadClass(ClassLoader.java:356) at
java.lang.Class.forName0(Native Method) at
java.lang.Class.forName(Class.java:264) at
org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:454)
... 15 more
How to solve this?
Create a folder inside your solr installation directory. (say solr-4.6.1/lib)
Place the mysql-connector-java-5.1.29-bin.jar inside the folder.
Edit solrconfig.xml and put the jar's path
<lib dir="../../../lib/" regex="mysql-connector-java-5.1.29-bin.jar" />
Restart Solr.
Also please check your URL. It's having //
http://localhost:8983/solr/#/collection1/dataimport//dataimport
So for those unfortunate souls who end up here. I'm using the latest Solr Docker image (8.4) which is configured so that the solr Unix-user doesn't have any permissions to create/delete folders and whatnot. For some reason I managed to create a lib file which I took for being a directory. After debugging a while I noticed this error, and promptly just changed the
<lib dir="${solr.install.dir:../../../..}/contrib/dataimporthandler/lib" regex=".*\.jar" />
to
<lib dir="${solr.install.dir:../../../..}/contrib/dataimporthandler/" regex=".*\.jar" />
where my postgresql-42.2.10.jar driver is. Otherwise all the other steps apply (edit solrconfig.xml, add data-config.xml, add schema.xml, copy posgresql-*.jar to the container's /opt/solr-8.4.1/contrib/dataimporthandler). Maybe one thing worth of mentioning is that I used Docker's internal network URL instead of localhost: url="jdbc:postgresql://host.docker.internal:5600/my_postgres_db" but I don't think it matters.
And oh, remember to restart Solr after adding the files / making changes =).