I have the following code:
Document doc = new Document();
String description = "This is a description text";
Field descField = new StringField("description ", description , Field.Store.YES);
doc.add(descField);
doc.add(new TextField("location", "Berlin", Field.Store.YES));
doc.add(new TextField("location", "Munich", Field.Store.YES));
doc.add(new TextField("location", "Vienna", Field.Store.YES));
writer.addDocument(doc);
How is the field 'location' physically stored in Lucene? Is it mapped to one single field (with offsets kept internally) or are there actually 3 fields with the same name separately stored in the inverted index?
The actually question I have this this: Are there performance issues (runtimes/space) or other issues if I choose to dynamically generate (e.g. from a data source at run-time) these location fields in comparison with adding them to a single field and hence reducing the number of fields to always just two (description and location)?
If somebody knows a pointer or an answer right out of their head it would be appreciated.
It will be mapped to a single field. This:
doc.add(new TextField("location", "Berlin", Field.Store.YES));
doc.add(new TextField("location", "Munich", Field.Store.YES));
doc.add(new TextField("location", "Vienna", Field.Store.YES));
Is very much the same as:
doc.add(new TextField("location", "Berlin Munich Vienna", Field.Store.YES));
(Assuming you are using StandardAnalyzer)
Which you choose should make no discernible difference at index time, and no difference whatsoever in terms of which search result you get back.
The difference between the two is in their stored representation. When adding them separately, you will be able to get the stored result back in an array, rather than a string:
IndexableField[] locations = doc.getFields("location")
for (IndexableField location : location)
//Do stuff
Related
I'm trying to use Lucene to add a search feature but can't seem to get an index to work with significant whitespace. I've got the following test case setup:
RAMDirectory directory = new RAMDirectory();
KeywordAnalyzer analyzer = new KeywordAnalyzer();
IndexWriterConfig config = new IndexWriterConfig(analyzer);
IndexWriter writer = new IndexWriter(directory, config);
Document doc = new Document();
doc.add(new TextField("content", "Bill Evans", Field.Store.NO));
writer.addDocument(doc);
writer.close();
IndexReader reader = DirectoryReader.open(directory);
IndexSearcher searcher = new IndexSearcher(reader);
QueryParser parser = new QueryParser("content", analyzer);
parser.setSplitOnWhitespace(false);
Query query = parser.parse("Bill E");
TopDocs docs = searcher.search(query, 1);
assertTrue(docs.totalHits > 0);
I'm using Lucene 6.6.0 and from what I understand the KeywordAnalyzer is what I'm looking for:
"Tokenizes" the entire stream as a single token. This is useful for data like zip codes, ids, and some product names.
But I can't seem to get any matching documents that contain whitespace.
Any ideas on how to solve this?
When you index, you have a single document with a single field and with a single term with value - Bill Evans
When you are going to search, TermQuery produced by QueryParser tries to search with term value - Bill E and that term obviously doesn't exist in index so you get zero hits.
if you replace your search string with - Bill Evans , you will get results.
Please refer this question too
First , you need to separate your indexing and searching concerns. You can only search what is indexed. If you are indexing full texts without breaking into tokens then at search times - you need to produce WildCardQuery , FuzzyQuery , PhraseQuery etc if your input string at search time is different than what in indexed. TermQuery searches for exact term values.
My suggestion would to be to store full text value ( without tokens - StringField would do that ) as well as generate additional tokens breaking on space using something like - SimpleAnalyzer .
So Something like,
doc.add(new TextField("content", "Bill Evans", Field.Store.NO));
doc.add(new StringField("storedcontent", "Bill Evans", Field.Store.YES));
Above code with SimpleAnalyzer , you will now have terms - bill & evans
( as well as full text as stored field ) and if you now search with same analyzer , your query would be like - content:bill content:e & you will get a result.
All in all - system is working the way you have coded it :)
So understand your requirements first as what you wish to index and what kind of queries you wish to perform on that index.
Story:
I need to search for a list of transactionIds be a given username query e.g "Peter M*".
Question: How is it possible to keep the stored transactionIds unique?
I have populated my index with following documents:
Document doc = new Document();
doc.add(new StoredField(TRANSACTION_ID, data.getTransactionId()));
doc.add(new TextField(MARCHANT_NAME, data.getName(), Store.NO));
I have tried allready two strategies (to avoid duplicate entries) to add a new entry.
IndexWriter.updateDocument with a Term holding the transactionId to store.
Search for the current transactionId, delete it and store it:
You are using a StoredField for the TRANSACTION_ID field. That means it can be retrieved from the index, but is not indexed and can't be searched, and as such, it can't be used as a key to updateDocument. Use a StringField, instead.
I am writing a code in java and use lucene 3.4 to index the text documents. Each document has an id and some other numerical values as well as content and title.
I add each document to the index according to the following code:
Document doc = new Document();
doc.add(new NumericField("id").setIntValue(writer.numDocs()));
doc.add(new NumericField("year").setIntValue(1988));
doc.add(new Field("content", new FileReader(file)));
writer.addDocument(doc);
writer.close();
But when I search and want to get the results, it returns null for these fields. I know that whenever I add a field and set the Field.Store.NO, it returns null, but why it happens right now? What should I do to get the value of these fields?
doc.get("id"); //why it returns null? what should I do?
Numeric fields are by default not stored.
Use the NumericField(String, Field.Store, boolean) constructor to specify that it should be stored if you would like to retrieve it later.
I am creating a text search application for log files using apache lucene. I am using the bellow code to index the files
doc.add(new LongField("modified", file.lastModified(), Field.Store.NO));
doc.add(new TextField("contents", new BufferedReader(new InputStreamReader(fis, "UTF-8"))));
doc.add(new StoredField("filename", file.getCanonicalPath()));
Here i am creating 3 indexes for each file But when searching i can retrieve the value of only one index other two come as null. This is the search side code
Document d = searcher.doc(docId);
System.out.println(i+":File name is"+d.get("filename"));
System.out.println(i+":File name is"+d.get("modified"));
System.out.println(i+":File name is"+d.get("contents"));
The output I am getting is
2 total matching documents
0:File name is/home/maclean/NetBeansProjects/LogSearchEngine/src/SimpleSearcher.java
0:File name isnull
0:File name isnull
1:File name is/home/maclean/NetBeansProjects/LogSearchEngine/src/SimpleFileIndexer.java
1:File name isnull
1:File name isnull
What am i doing wrong
In Lucene, if you want to retrieve the value for a field, you need to store that field. If a field is not stored, on searching its value will be null.
For modified, you've explicitly specified it as a un-stored field by passing the argument Field.Store.NO; as a result it's value is not being stored in the index and hence, null is returned on search. To store and retrieve its value, you need to change the constructor call to:
doc.add(new LongField("modified", file.lastModified(), Field.Store.YES));
For contents, the constructor you've used creates un-stored field. You need to change its constructor to:
doc.add(new TextField("contents", new BufferedReader(new InputStreamReader(fis, "UTF-8")), Field.Store.YES));
After these changes, you should be able to retrieve both the fields.
You are able to retrieve values for filename because you are using a constructor that creates stored fields by default.
I am new to lucene. I have to index date field.
i am using Following IndexWriter constructor in lucene 3.0.0.
IndexWriter writer = new IndexWriter(FSDirectory.open(indexDir), new WhitespaceAnalyzer(), true, IndexWriter.MaxFieldLength.UNLIMITED)
my point is:
Why it needs a analyzer when date fields are not analyzed,while indexing I used Field.Index.NOT_ANALYZED.
You can store date field in this fashion..
Document doc = new Document();
doc.add(new Field("modified",
DateTools.timeToString(f.lastModified(), DateTools.Resolution.MINUTE),
Field.Store.YES, Field.Index.NOT_ANALYZED));
where f is a file object...
Now use the above document for indexwriter...
checkout the sample code comes with lucene... and the following link...
http://lucene.apache.org/java/2_2_0/api/org/apache/lucene/document/DateTools.html
UPDATE
Field.Index NOT_ANALYZED
Index the field's value without using
an Analyzer, so it can be searched. As
no analyzer is used the value will be
stored as a single term. This is
useful for unique Ids like product
numbers.
As per lucene javadoc you don't need analyzer for fields using Field.Index NOT_ANALYZED but i think by design the IndexWriter expects an analyzer as indexing the exact replica of data is not efficient in terms of storage and searching.