I am trying to perform search result aggregation (count and sum) grouping by several fields in a nested fashion.
For example, with the schema shown at the end of this post, I'd like to be able to get the sum of "size" grouped by "category" and sub-grouped further by "subcategory" and get something like this:
<category name="X">
<subcategory name="X_A">
<size sum="..." />
</subcategory>
<subcategory name="X_B">
<size sum="..." />
</subcategory>
</category>
....
I've been looking primarily at Solr's Stats component which, as far as I can see, doesn't allow nested aggregation.
I'd appreciate it if anyone knows of some way to implement this, with or without the Stats component.
Here is a cut-down version of the target schema:
<types>
<fieldType name="string" class="solr.StrField" />
<fieldType name="text" class="solr.TextField">
<analyzer><tokenizer class="solr.StandardTokenizerFactory" /></analyzer>
</fieldType>
<fieldType name="date" class="solr.DateField" />
<fieldType name="int" class="solr.TrieIntField" precisionStep="0" omitNorms="true" positionIncrementGap="0" />
</types>
<fields>
<field name="id" type="string" indexed="true" stored="true" />
<field name="category" type="text" indexed="true" stored="true" />
<field name="subcategory" type="text" indexed="true" stored="true" />
<field name="pdate" type="date" indexed="true" stored="true" />
<field name="size" type="int" indexed="true" stored="true" />
</fields>
The new faceting module in Solr 5.1 can do this, it was added in https://issues.apache.org/jira/browse/SOLR-7214
Here is how you would add sum(size) to every facet bucket, and sort descending by that statistic.
json.facet={
categories:{terms:{
field:category,
sort:"total_size desc", // this will sort the facet buckets by your stat
facet:{
total_size:"sum(size)" // this calculates the stat per bucket
}
}}
}
And this is how you would add in the subfacet on subcategory:
json.facet={
categories:{terms:{
field:category,
sort:"total_size desc",
facet:{
total_size:"sum(size)",
subcat:{terms:{ // this will facet on the subcategory field for each bucket
field:subcategory,
facet:{
sz:"sum(size)" // this calculates the sum per sub-cat bucket
}}
}
}}
}
So the above will give you the sum(size) at both the category and subcategory levels. Documentation for the new facet module is currently at http://yonik.com/json-facet-api/
There is a patch SOLR-3583, which adds percentiles and averages to facets, pivot facets, and distributed pivot facets by making use of range facet internals. It is possible to add sums to pivot facets by improving this patch.
For example, averages can be calculated for categories using this url:
http://localhost:8983/solr/select?q=*%3A*
&facet=true
&facet.pivot=category,subcategory
&facet.stats.percentiles=true
&facet.stats.percentiles.averages=true
&facet.stats.percentiles.field=size
&f.size.stats.percentiles.requested=25,50,75
&f.size.stats.percentiles.lower.fence=0
&f.size.stats.percentiles.upper.fence=1000
&f.size.stats.percentiles.gap=10
See also this video and slides for more details.
1. Counts
To get the counts, you can use Pivot Faceting. It will generate a list very similar to what you asked but with counts only.
You'll need to append this to your query:
&facet=true&facet.pivot=category,subcategory
Note that this works on Solr 4.0 and after.
2. Sums
As for the sums, I think you can achieve them with ordinary facets but using a facet query instead of facet field.. I'm not entirely sure of this one, I'll try it out and re-post if found anything useful.
Related
I have several different XML files. I want to display them in HTML format using JSP file.
The structure of these files can be different. They may have different depth and tag's names.
I've read about XSLT but I found only examples with manually created xsl. (I can't do that because i will have many different xml files).
How can i produce my html?
EDIT:
Example of form:
<form name="Company">
<field name="name" required="true" type="inputText"/>
<field name="registrationDate" required="true" type="date"/>
<field name="isActive" required="false" type="boolean"/>
<field name="unregistrationDate" type="date"/>
<field name="type" type="comboBox">
<values>
<value>type1</value>
<value>type3</value>
<value>type3</value>
</values>
<defaultSelected>type2</defaultSelected>
</field>
</form>
I recently upgraded BeanIO in my project from 1.2.3 to 2.0.6 and as part of that upgrade I had to modify some of my mappings that where using the Bean element to start using Segment element.
But what I noticed is that previously if a line in a stream had to little fields beanio would still initialize the object defined in the bean element in the mapping. Where as now in the same scenario the property that should be set with the object is simply set to null.
Old code that worked and populated the testName property with a empty Test object:
<bean name="testName" class="com.project.Test">
<field name="test1" type="string" default=""/>
<field name="test2" type="string" default=""/>
</bean>
New code that doesn't initialize the Test object and sets the testName property to null:
<segment name="testName" class="com.project.Test" minOccurs="0">
<field name="test1" type="string" default="" minOccurs="0"/>
<field name="test2" type="string" default="" minOccurs="0"/>
</segment>
Is there a way to force the Segment element to always initialize an object?
(Yes I do need to keep minOccurs="0" and no if I remove it, it doesn't solve the issue)
Thx,
goe
Try setting org.beanio.createMissingBeans=true in a beanio.properties file at the root of your classpath.
I have an index named LocationIndex in solr with fields as follows:
<fields>
<field name="solr_id" type="string" stored="true" required="true" indexed="true"/>
<field name="solr_ver" type="string" stored="true" required="true" indexed="true" default="0000"/>
// and some more fields
</fields>
<uniqueKey>solr_id</uniqueKey>
But now I want to change schema so that unique key must be composite of two already present fields solr_id and solr_ver... something as follows:
<fields>
<field name="solr_id" type="string" stored="true" required="true" indexed="true"/>
<field name="solr_ver" type="string" stored="true" required="true" indexed="true" default="0000"/>
<field name="composite-id" type="string" stored="true" required="true" indexed="true"/>
// and some more fields
</fields>
<uniqueKey>solr_ver-solr_id</uniqueKey>
After searching I found that it's possible by adding following to schema: (ref: Solr Composite Unique key from existing fields in schema)
<updateRequestProcessorChain name="composite-id">
<processor class="solr.CloneFieldUpdateProcessorFactory">
<str name="source">docid_s</str>
<str name="source">userid_s</str>
<str name="dest">id</str>
</processor>
<processor class="solr.ConcatFieldUpdateProcessorFactory">
<str name="fieldName">id</str>
<str name="delimiter">--</str>
</processor>
<processor class="solr.LogUpdateProcessorFactory" />
<processor class="solr.RunUpdateProcessorFactory" />
</updateRequestProcessorChain>
So I changed schema and finally it looks like:
<updateRequestProcessorChain name="composite-id">
<processor class="solr.CloneFieldUpdateProcessorFactory">
<str name="source">solr_ver</str>
<str name="source">solr_id</str>
<str name="dest">id</str>
</processor>
<processor class="solr.ConcatFieldUpdateProcessorFactory">
<str name="fieldName">id</str>
<str name="delimiter">-</str>
</processor>
<processor class="solr.LogUpdateProcessorFactory" />
<processor class="solr.RunUpdateProcessorFactory" />
</updateRequestProcessorChain>
<fields>
<field name="solr_id" type="string" stored="true" required="true" indexed="true"/>
<field name="solr_ver" type="string" stored="true" required="true" indexed="true" default="0000"/>
<field name="id" type="string" stored="true" required="true" indexed="true"/>
// and some more fields
</fields>
<uniqueKey>id</uniqueKey>
But while adding a document it's giving me error:
org.apache.solr.client.solrj.SolrServerException: Server at http://localhost:8983/solr/LocationIndex returned non ok status:400, message:Document [null] missing required field: id
I'm not getting what changes in schema are required to work as desired?
In a document I add, it contain fields solr_ver and solr_id. How and where it'll (solr) create id field by combining both these field something like solr_ver-solr_id?
EDIT:
At this link It's given how refer to this chain. Bu I'm unable to understand how would it be used in schema? And where should I make changes?
So it looks like you have your updateRequestProcessorChain defined appropriately and it should work. However, you need to add this to the solrconfig.xml file and not the schema.xml. The additional link you provided shows you how to modify your solrconfig.xml file and add your defined updateRequestProcessorChain to the current /update request handler for your solr instance.
So find do the following:
Move your <updateRequestProcessorChain> to your solrconfig.xml file.
Update the <requestHandler name="/update" class="solr.UpdateRequestHandler"> entry in your solrconfig.xml file and modify it so it looks like the following:
<requestHandler name="/update" class="solr.UpdateRequestHandler">
<lst name="defaults">
<str name="update.chain">composite-id</str>
</lst>
</requestHandler>
This should then execute your defined update chain and populate the id field when new documents are added to the index.
The described above solution may have some limitations, what if "dest" is over maximum length because concatenated fields are too long.
There is also one more solution with MD5Signature (A class capable of generating a signature String from the concatenation of a group of specified document fields, 128 bit hash used for exact duplicate detection)
<!-- An example dedup update processor that creates the "id" field on the fly
based on the hash code of some other fields. This example has
overwriteDupes set to false since we are using the id field as the
signatureField and Solr will maintain uniqueness based on that anyway. -->
<updateRequestProcessorChain name="dedupe">
<processor class="org.apache.solr.update.processor.SignatureUpdateProcessorFactory">
<bool name="enabled">true</bool>
<bool name="overwriteDupes">false</bool>
<str name="signatureField">id</str>
<str name="fields">name,features,cat</str>
<str name="signatureClass">org.apache.solr.update.processor.Lookup3Signature</str>
</processor>
<processor class="solr.LogUpdateProcessorFactory" />
<processor class="solr.RunUpdateProcessorFactory" />
</updateRequestProcessorChain>
From here: http://lucene.472066.n3.nabble.com/Solr-duplicates-detection-td506230.html
I'd like to add this as a comment, but it's impossible to get the creds these days... anyway, here is a better link:
https://wiki.apache.org/solr/Deduplication
My data set is title, description and tags.
I would like to store and index in the SOLR the tag_name and their relative tag_id.
As can be understood, each record has one title, one description bt many tag names + tag ids.
I guess I can store the tags as "some-tag-[id]" but his seems wrong.
You can index tags and tags_id as multivalued fields and add in order.
The order is maintained so you can map them within the fields.
<field name="tags" type="string" indexed="true" stored="true" multiValued="true"/>
<field name="tags_id" type="string" indexed="false" stored="true" multiValued="true"/>
Response -
<arr name="tags">
<str>tag1</str>
<str>tag2</str>
<str>tag3</str>
</arr>
<arr name="tags_id">
<str>id1</str>
<str>id2</str>
<str>id3</str>
</arr>
I am trying the Data Import Handler for MySQL Database.
I added the DIhandler in solrconfig.xml, created a data-config.xml according to my database scheme and also added a field in the schema.xml which was different. I am connecting with MySQL database
After i connect and I run the dataimport?command=full-import i get this response
"00C:\solr\conf\data-config.xmlfull-importidle1102011-03-05 15:01:04Indexing completed. Added/Updated: 0 documents. Deleted 0 documents.2011-03-05 15:01:042011-03-05 15:01:040:0:0.400This response format is experimental. It is likely to change in the future."
The xml files are in this http://pastebin.com/iKebKGSZ
<field column="manu" name="manu" />
<field column="id" name="id" />
<field column="weight" name="weight" />
<field column="price" name="price" />
<field column="popularity" name="popularity" />
<field column="instock" name="inStock" />
<field column="includes" name="includes" />
Are these fields also in your schema.xml?
I couldnt see them in the pastebin link. Make sure you have all fields in your schema as well.