Develop Reference

Lucene is limiting the query terms

I'm trying to use Lucene (5.4.1) MoreLikeThis to tag(classify) texts. It's kind of working, but I'm getting poor results, and I think that the problem is related with the Query object.
The example bellow works, but the highest topdoc isn't the one that I expect. By debuging the query object, it shows only content:erro. From a complete portuguese phrase (see into the example) the query was constructed with just one word.
I'm not using stop words or any other kind of filter.
So why lucene is picking just erro as a query term?
To init main objects
Analyzer analyzer = new PortugueseAnalyzer();
Directory indexDir = new RAMDirectory();
IndexWriterConfig config = new IndexWriterConfig(analyzer);
config.setOpenMode(OpenMode.CREATE_OR_APPEND);
To index
try (IndexWriter indexWriter = new IndexWriter(indexDir, config)) {
FieldType type = new FieldType();
type.setIndexOptions(IndexOptions.DOCS_AND_FREQS_AND_POSITIONS_AND_OFFSETS);
type.setStored(true);
type.setStoreTermVectors(true);
Document doc = new Document();
doc.add(new StringField("id", "880b2bbc", Store.YES));
doc.add(new Field("content", "erro", type));
doc.add(new Field("tag", "atag", type));
indexWriter.addDocument(doc);
indexWriter.commit();
}
To search
try (IndexReader idxReader = DirectoryReader.open(indexDir)) {
IndexSearcher indexSearcher = new IndexSearcher(idxReader);
MoreLikeThis mlt = new MoreLikeThis(idxReader);
mlt.setMinTermFreq(0);
mlt.setMinDocFreq(0);
mlt.setFieldNames(new String[] { "content" });
mlt.setAnalyzer(analyzer);
Reader sReader = new StringReader("Melhorias no controle de sessão no sistema qquercoisa quando expira, ao logar novamente no sistema é exibido o erro "xpto");
Query query = mlt.like("content", sReader);
TopDocs topDocs = indexSearcher.search(query, 3);
}

Well, I decided to take a look inside MoreLokeThis class and I found the answer.
The Query query = mlt.like("content", sReader); call the createQueue(Map<String, Int> words) method in MoreLokeThis class.
Inside it, the tokenized terms/words from sReader (that were converted to a Map) are checked against the index.
Only terms/words that are present into the index are used to create a query.
Using the example that I provided, since my index contains only a document with the word erro, this is the only word that is kept from the phrase that I passed.

retrieve documents after grouping search in lucene

I am doing lucene search on my resources.I have a case where I search for a particular product and I need to do it on a grouping search via 'keywords' field.I can get to know the total number of products grouped by keywords associated with it.How can I get all the documents related to this search, so that I can retrieve other needed fields from it. I tried using AbstractAllGroupHeadsCollector but couldnt find and got confused with its usage.Here is my code.
Thanks in advance.
Integer totalGroupCount = null;
IndexReader ir = DirectoryReader.open(indexLocation);
IndexSearcher is = new IndexSearcher(ir);
GroupingSearch groupingSearch = new GroupingSearch("keywords");
groupingSearch.setGroupSort(Sort.RELEVANCE);
groupingSearch.setFillSortFields(true);
groupingSearch.setCachingInMB(4.0, true);
groupingSearch.setAllGroups(true);
//TermQuery query = new TermQuery(new Term("products", "wfa packages"));
TopGroups<BytesRef> result = groupingSearch.search(is, query, 0, 10);
// Render groupsResult...
totalGroupCount = result.totalGroupCount; // The group count
GroupDocs<BytesRef>[] d=result.groups;
System.out.println("total groups="+result.totalGroupedHitCount);

You have your GroupDocs array, that's most of the way there already. You can then get the scoreDocs from each GroupDocs, and lookup the document with the doc id, from ScoreDoc.doc, like:
for (GroupDocs<BytesRef> group : d) {
for (ScoreDoc scoredoc : group.scoreDocs) {
Document doc = is.doc(scoredoc.doc);
//Do stuff
}
}

Lucene: get newest document for category

I'm pretty new to lucene index, so I apologize in advance if what I am trying to do is trivial.
I have an index where the documents contain (among other) two fields:
documentoId and employeeId.
Each employee can submit various documents. The structure is pretty much the same as in the bookstore example.
What I am trying to achieve, is to get all the newest documents matching a query, meaning with the highest documentoId for each employeeId.
In SQL, this would be something like:
select max(documentoId ), employeeId
from documents
where content like 'mySearchValue'
group by employeeId
I don't know if I should use facet API, or if this can be done with queries, or with the searchAfter method...I'm pretty lost with the documentation.
Any help would be greatly appreciated!
Thanks

Lucene supports grouping search; what you need to do is to define your group and how does it have to be sorted. In the example below, I group by documentoId and sort in descending order.
public static void main(String[] args) throws IOException, ParseException {
StandardAnalyzer standardAnalyzer = new StandardAnalyzer(Version.LUCENE_46);
RAMDirectory ramDirectory = new RAMDirectory();
IndexWriter indexWriter = new IndexWriter(ramDirectory, new IndexWriterConfig(Version.LUCENE_46, standardAnalyzer));
Document d0 = new Document();
d0.add(new TextField("employeeId", "foo", Field.Store.YES));
d0.add(new IntField("documentoId", 1, Field.Store.YES));
indexWriter.addDocument(d0);
Document d1 = new Document();
d1.add(new TextField("employeeId", "bar", Field.Store.YES));
d1.add(new IntField("documentoId", 20, Field.Store.YES));
indexWriter.addDocument(d1);
Document d2 = new Document();
d2.add(new TextField("employeeId", "baz", Field.Store.YES));
d2.add(new IntField("documentoId", 3, Field.Store.YES));
indexWriter.addDocument(d2);
indexWriter.commit();
GroupingSearch groupingSearch = new GroupingSearch("documentoId");
Sort groupSort = new Sort(new SortField("documentoId", SortField.Type.INT, true)); // in descending order
groupingSearch.setGroupSort(groupSort);
groupingSearch.setSortWithinGroup(groupSort);
IndexReader reader = DirectoryReader.open(ramDirectory);
IndexSearcher searcher = new IndexSearcher(reader);
TopGroups<?> groups = groupingSearch.search(searcher, new MatchAllDocsQuery(), 0, 10);
Document highestScoredDocument = reader.document(groups.groups[0].scoreDocs[0].doc);
System.out.println(
"Descending order, first document is " +
"employeeId:" + highestScoredDocument.get("employeeId") + " " +
"documentoId:" + highestScoredDocument.get("documentoId")
);
}
The above code detects that the d1 (middle document) scores at the top and prints the following:
Descending order, first document is employeeId:bar documentoId:20
Above code does not address content like 'mySearchValue' part, you would have to replace MatchAllDocsQuery with a relevant query to do that.

Custom sorting of the hits will do the trick. Google the search.sort parameter in Lucene.

For those in the same situation, I solved my problem using mindas comment and modifying it to use my group field:
GroupingSearch groupingSearch = new GroupingSearch("employeeId");
Sort groupSort = new Sort(new SortField("documentoId", SortField.Type.INT, true)); // in descending order
groupingSearch.setGroupSort(groupSort);
groupingSearch.setSortWithinGroup(groupSort);
int offset = 0;
int limitGroup = 50;
TopGroups<?> groups = groupingSearch.search(is,query, offset, limitGroup);
List<Document> result = new ArrayList();
for (int i=0; i<groups.groups.length; i++) {
ScoreDoc sdoc = groups.groups[i].scoreDocs[0]; // first result of each group
Document d = is.doc(sdoc.doc);
result.add(d);
}

How to use MultiFieldQueryParser from Lucene?

I am using Version.Lucene_29. Using the normal string query method i could do the following:
Directory directory = new FSDirectory(...);
//Start Lucene retrieval.
IndexSearcher iSearch = new IndexSearcher(directory, true);
Analyzer analyzer = new WhitespaceAnalyzer();
QueryParser parser = new QueryParser(Version.LUCENE_29, "content", analyzer);
String str = 'filename:testfile.txt AND filetext:"Singapore food"'
Query query = parser.parse(str);
ScoreDoc[] hits = iSearch.search(query, 1000).scoreDocs;
How do i fire a query using MultiFieldQueryParser in Lucene similar to the string query method?
MultiFieldQueryParser multiParser = new MultiFieldQueryParser(
Version.LUCENE_29, new String[] {"content", "ne"}, analyzer);
str = ???
Query = ????
ScoreDoc[] hits = iSearch.search(query, 1000).scoreDocs;

MultiFieldQueryParser allows you to search for a "WORD" in more then one Fileds with same Analyzer.
e.g.
Query query = MultiFieldQueryParser.parse("development",
new String[]{"title", "subject"},
new SimpleAnalyzer());
it will look for word development in Field : "title" and Field : "subject"

MultiFieldQueryParser is-a QueryParser, MultiFieldQueryParser creates the two Queries into a BooleanClause in this case. So it also supports filename:testfile.txt AND filetext:"Singapore food".

How to specify two Fields in Lucene QueryParser?

I read How to incorporate multiple fields in QueryParser? but i didn't get it.
At the moment i have a very strange construction like:
parser = New QueryParser("bodytext", analyzer)
parser2 = New QueryParser("title", analyzer)
query = parser.Parse(strSuchbegriff)
query2 = parser.Parse(strSuchbegriff)
What can i do for something like:
parser = New QuerParser ("bodytext" , "title",analyzer)
query =parser.Parse(strSuchbegriff)
so the Parser looks for the searching word in the field "bodytext" an in the field "title".

There are 3 ways to do this.
The first way is to construct a query manually, this is what QueryParser is doing internally. This is the most powerful way to do it, and means that you don't have to parse the user input if you want to prevent access to some of the more exotic features of QueryParser:
IndexReader reader = IndexReader.Open("<lucene dir>");
Searcher searcher = new IndexSearcher(reader);
BooleanQuery booleanQuery = new BooleanQuery();
Query query1 = new TermQuery(new Term("bodytext", "<text>"));
Query query2 = new TermQuery(new Term("title", "<text>"));
booleanQuery.add(query1, BooleanClause.Occur.SHOULD);
booleanQuery.add(query2, BooleanClause.Occur.SHOULD);
// Use BooleanClause.Occur.MUST instead of BooleanClause.Occur.SHOULD
// for AND queries
Hits hits = searcher.Search(booleanQuery);
The second way is to use MultiFieldQueryParser, this behaves like QueryParser, allowing access to all the power that it has, except that it will search over multiple fields.
IndexReader reader = IndexReader.Open("<lucene dir>");
Searcher searcher = new IndexSearcher(reader);
Analyzer analyzer = new StandardAnalyzer();
MultiFieldQueryParser queryParser = new MultiFieldQueryParser(
new string[] {"bodytext", "title"},
analyzer);
Hits hits = searcher.Search(queryParser.parse("<text>"));
The final way is to use the special syntax of QueryParser see here.
IndexReader reader = IndexReader.Open("<lucene dir>");
Searcher searcher = new IndexSearcher(reader);
Analyzer analyzer = new StandardAnalyzer();
QueryParser queryParser = new QueryParser("<default field>", analyzer);
// <default field> is the field that QueryParser will search if you don't
// prefix it with a field.
string special = "bodytext:" + text + " OR title:" + text;
Hits hits = searcher.Search(queryParser.parse(special));
Your other option is to create new field when you index your content called bodytextandtitle, into which you can place the contents of both bodytext and title, then you only have to search one field.

We can not use
BooleanQuery booleanQuery = new BooleanQuery();
We have to use builder
BooleanQuery.Builder finalQuery = new BooleanQuery.Builder();
then we can use finalQuery.build(); to get query
more generic way to do this is
private static TopDocs search(Map filters, IndexSearcher searcher) throws Exception {
StandardAnalyzer analyzer = new StandardAnalyzer();
BooleanQuery.Builder finalQuery = new BooleanQuery.Builder();
for(String attribute : filters.keySet()) {
QueryParser queryParser = new QueryParser(attribute, analyzer);
Query query = queryParser.parse(filters.get(attribute));
finalQuery.add(query, Occur.MUST);
}
TopDocs hits = searcher.search(finalQuery.build(),10);
return hits;
}