Develop Reference

Lucene/Solr - Indexing publications/texts

I want to be able to search publications with facets. These documents will be annotated so I will upload the annotation to the solr instance. The annotation will have fields which are the terms in the document. Here is an example:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<add>
<doc>
<field name="Title">High Glucose Increases the Expression of Inflammatory Cytokine Genes in
Macrophages Through H3K9 Methyltransferase Mechanism.</field>
<field name="Cytokine">INTERFERON </field>
<field name="Cytokine">CYTOKINE </field>
<field name="Cytokine">CYTOKINE</field>
<field name="Cytokine">MEC</field>
<field name="Cytokine">EPA</field>
<field name="Cytokine">DIA</field>
<field name="Cytokine">FIC</field>
<field name="Cytokine">CYTOKINES</field>
<field name="Cytokine">INTERLEUKIN-6 </field>
<field name="Cytokine">INTERLEUKIN</field>
<field name="Cytokine">IL-12P40</field>
<field name="Cytokine">IL-12</field>
<field name="Cytokine">IL-1</field>
<field name="Cytokine">P40</field>
<field name="Cytokine">MACROPHAGE INFLAMMATORY PROTEIN-1</field>
<field name="Cytokine">MACROPHAGE INFLAMMATORY PROTEIN</field>
</doc>
</add>
These terms are all from a Cytokine ontology.
I want be able to set the facet as Cytokine, then select the term and find all of the documents that contain the selected term.
Here is the catch:
I want to be able to store the location of said term found in the
document (it can show up in multiple locations. So I can highlight
later). All of these locations are stored in the annotation.
I want to be able to select one of the terms from the facet and also
bring up documents that contains that terms synonyms but not upload it as a term in the facet (or it being distinguished as a synonym some how (like subcategory)). e.g. automobile
and car
I want to be able to do a cross search e.g. find documents that
contain MEC and EPA.
I have a list of terms I do want to index and want to search the
documents by. These terms have synonyms which I have entered into the
synonyms.txt file.
Also When a term shows up multiple times in the document the annotation has multiple instances of this term with different locations, how should I handle this? Will solr automatically deal with duplication and not give me the documents twice?
One more thing: What about uploading the entire publication to solr, and indexing it on the predefined list of terms?

I understand that, you have synonyms and a search term should be verified directly and also with synonyms and return the results. Let me know if I got it.
If you have all the synonyms while indexing, then you can index them as multi valued field and search on that field.
Faceting is for searching, where the results are grouped.

Mapping an Object[] with dozer

With the help of Dozer I want to map an Object[] to a DTO class
<mapping>
<class-a>com.example.myDtoClass</class-a>
<class-b>java.lang.Object[]</class-b>
<field>
<a>prop</a>
<!-- <b key="1">this</b> -->
<b>this[1]</b>
<b-hint>java.lang.String</b-hint>
</field>
</mapping>
but both settings <b>this[1]</b> and <b key="1">this</b> do not work and the prop field is set with the entire Object[] field so the result in returned JSON looks like
"prop": "[Ljava.lang.Object;#40147864"
what am I missing? I am expecting to have the 1 element of the Object[] in the prop variable

I made a bad mistake. Yes I get a Object[] to map, but the 1 element of that Object[] again is of type Object[]. I got confused and thought it was the main element. I did though expect a String, but got a array of binary code that has to be joined back to a string.

How to give weight to the specific field?

I am using Apache Solr for indexing and searching. I have to give weight to the specific field so that If I make search then search has to perform on that field which is most weighted and then on others.
I am using SolrJ, Java, and GWT for development.

To boost at index time you need to supply a boost statement in your update doc.
<add overwrite="true">
<doc boost="2.0">
<field name="id">1234</field>
<field name="type">type1</type>
</doc>
<doc>
<field name="id">2345</field>
<field name="type" boost="0.5">type2</type>
</doc>
</add>
The above example demonstrates how to boost a complete document (elevation) as well as how to boost a specific field.
For more documentation look here and here

Using the dismax (or edismax) query handler, you can set the qf (Query Fields) parameter to assign boosts to different fields. It uses this format:
field1^boost_val field2^boost_val....etc.
There are other good parameters to help you control your result ranking as well.
http://wiki.apache.org/solr/ExtendedDisMax

file descriptor written in xml

I am looking to parse log files (text files usually delimited by certain characters), and I want to make the job easier by making a parser that can take in an xml file which describes the format of the log file coming in. This way I can feed my parser any type of log file and with my xml description my parser and extract the useful information. I am not sure how to generically describe the log file format in xml. Also I am unsure about how to then match lines in my log file with the descriptions given. I would greatly appreciate any help and guidance on this. Thank you very much to anyone who replies.

<logfile location="/usr/local/logs/logfile.log">
<delimiter>#</delimiter>
<format>
<field index="1" length="10">
<target>Date</target>
</field>
<field index="2">
<target>Message</target>
</field>
<field index="3" length="20">
<target>Component</target>
</field>
<field index="5" length="20">
<target>Details</target>
</field>
</format>
</logfile>
Your code will read this XML and parse each line in the log file dynamically according to the rules defined here. The program will then store the values in appropriate 'target' fields.
log file:
2012/12/12 10.10.10#Critical application state#Error in component XYZ#Useless information#Useful information
Output:
Date: 2012/12/12
Message: Critical application state
Component: Error in component XYZ
Details: Useful information

JAXB multiple mappings for attribute

I'm just changing design errors made in the past, but want to keep backwards compatibility of my software. For this I would need some way to map two flavors of an xml file into one java bean. Can this be done using two JAXB annotations on one attribute/element? I understand the marshalling would be ambiguous, but the unmarshalling could work. Is there some nice way of doing this?
p.s.: I don't care about marshalling.

You can map twice:
the first time using annotations
the second time using XML resources.
Or just two XML mappings instead of annotations.
For XML resource mappings, there's a number of options:
Annox: http://confluence.highsource.org/display/ANX/JAXB+User+Guide (I wrote it so it comes first :))
EclipseLink Moxy: http://eclipse.org/eclipselink/moxy.php
JAXB Intoductions: http://community.jboss.org/wiki/JAXBIntroductions
With Annox you can easily map twice using XML mapping resources with different extensions like MyClass.ann1.xml or MyClass.ann2.xml. (It's MyClass.ann.xml per default, but the adjustment is trivial.)
Here's a sample of what mappings look like:
<class xmlns="http://annox.dev.java.net" xmlns:annox="http://annox.dev.java.net" xmlns:jaxb="http://annox.dev.java.net/javax.xml.bind.annotation">
<jaxb:XmlAccessorType value="FIELD"/>
<jaxb:XmlType name="" propOrder="productName quantity usPrice comment shipDate"/>
<field name="productName">
<jaxb:XmlElement required="true"/>
</field>
<field name="usPrice">
<jaxb:XmlElement name="USPrice" required="true"/>
</field>
<field name="shipDate">
<jaxb:XmlSchemaType name="date"/>
</field>
<field name="partNum">
<jaxb:XmlAttribute required="true"/>
</field>
</class>