I am creating a text search application for log files using apache lucene. I am using the bellow code to index the files
doc.add(new LongField("modified", file.lastModified(), Field.Store.NO));
doc.add(new TextField("contents", new BufferedReader(new InputStreamReader(fis, "UTF-8"))));
doc.add(new StoredField("filename", file.getCanonicalPath()));
Here i am creating 3 indexes for each file But when searching i can retrieve the value of only one index other two come as null. This is the search side code
Document d = searcher.doc(docId);
System.out.println(i+":File name is"+d.get("filename"));
System.out.println(i+":File name is"+d.get("modified"));
System.out.println(i+":File name is"+d.get("contents"));
The output I am getting is
2 total matching documents
0:File name is/home/maclean/NetBeansProjects/LogSearchEngine/src/SimpleSearcher.java
0:File name isnull
0:File name isnull
1:File name is/home/maclean/NetBeansProjects/LogSearchEngine/src/SimpleFileIndexer.java
1:File name isnull
1:File name isnull
What am i doing wrong
In Lucene, if you want to retrieve the value for a field, you need to store that field. If a field is not stored, on searching its value will be null.
For modified, you've explicitly specified it as a un-stored field by passing the argument Field.Store.NO; as a result it's value is not being stored in the index and hence, null is returned on search. To store and retrieve its value, you need to change the constructor call to:
doc.add(new LongField("modified", file.lastModified(), Field.Store.YES));
For contents, the constructor you've used creates un-stored field. You need to change its constructor to:
doc.add(new TextField("contents", new BufferedReader(new InputStreamReader(fis, "UTF-8")), Field.Store.YES));
After these changes, you should be able to retrieve both the fields.
You are able to retrieve values for filename because you are using a constructor that creates stored fields by default.
Related
Story:
I need to search for a list of transactionIds be a given username query e.g "Peter M*".
Question: How is it possible to keep the stored transactionIds unique?
I have populated my index with following documents:
Document doc = new Document();
doc.add(new StoredField(TRANSACTION_ID, data.getTransactionId()));
doc.add(new TextField(MARCHANT_NAME, data.getName(), Store.NO));
I have tried allready two strategies (to avoid duplicate entries) to add a new entry.
IndexWriter.updateDocument with a Term holding the transactionId to store.
Search for the current transactionId, delete it and store it:
You are using a StoredField for the TRANSACTION_ID field. That means it can be retrieved from the index, but is not indexed and can't be searched, and as such, it can't be used as a key to updateDocument. Use a StringField, instead.
I have the following code:
Document doc = new Document();
String description = "This is a description text";
Field descField = new StringField("description ", description , Field.Store.YES);
doc.add(descField);
doc.add(new TextField("location", "Berlin", Field.Store.YES));
doc.add(new TextField("location", "Munich", Field.Store.YES));
doc.add(new TextField("location", "Vienna", Field.Store.YES));
writer.addDocument(doc);
How is the field 'location' physically stored in Lucene? Is it mapped to one single field (with offsets kept internally) or are there actually 3 fields with the same name separately stored in the inverted index?
The actually question I have this this: Are there performance issues (runtimes/space) or other issues if I choose to dynamically generate (e.g. from a data source at run-time) these location fields in comparison with adding them to a single field and hence reducing the number of fields to always just two (description and location)?
If somebody knows a pointer or an answer right out of their head it would be appreciated.
It will be mapped to a single field. This:
doc.add(new TextField("location", "Berlin", Field.Store.YES));
doc.add(new TextField("location", "Munich", Field.Store.YES));
doc.add(new TextField("location", "Vienna", Field.Store.YES));
Is very much the same as:
doc.add(new TextField("location", "Berlin Munich Vienna", Field.Store.YES));
(Assuming you are using StandardAnalyzer)
Which you choose should make no discernible difference at index time, and no difference whatsoever in terms of which search result you get back.
The difference between the two is in their stored representation. When adding them separately, you will be able to get the stored result back in an array, rather than a string:
IndexableField[] locations = doc.getFields("location")
for (IndexableField location : location)
//Do stuff
I am writing a code in java and use lucene 3.4 to index the text documents. Each document has an id and some other numerical values as well as content and title.
I add each document to the index according to the following code:
Document doc = new Document();
doc.add(new NumericField("id").setIntValue(writer.numDocs()));
doc.add(new NumericField("year").setIntValue(1988));
doc.add(new Field("content", new FileReader(file)));
writer.addDocument(doc);
writer.close();
But when I search and want to get the results, it returns null for these fields. I know that whenever I add a field and set the Field.Store.NO, it returns null, but why it happens right now? What should I do to get the value of these fields?
doc.get("id"); //why it returns null? what should I do?
Numeric fields are by default not stored.
Use the NumericField(String, Field.Store, boolean) constructor to specify that it should be stored if you would like to retrieve it later.
I'm 99% sure I had this working in the past, maybe I'm wrong.
Anyway, I'd like to delete a Lucene document by a Field which is stored but not analyzed and contains text.
So the problem, it seems, is that calling luceneWriter.deleteDocuments(query) doesn't delete the document unless the field referenced in query is Field.Index.ANALYZED or a simple number.
Some code:
Integer myId = 1234;
Document doc = new Document();
Field field = new Field("MyIdField", myId, Field.Store.YES, Field.Index.ANALYZED);
doc.add(field);
indexWriter.add(doc);
indexWriter.commit();
...
QueryParser parser = new QueryParser(VERSION, "MyIdField", ANALYZER);
Query query = parser.parse("MyIdField:1234");
indexWriter.deleteDocuments(query);
indexWriter.commit();
Everything works!
Sweet.. what if the field is not analyzed?
Field field = new Field("MyIdField", myId, Field.Store.YES, Field.Index.NOT_ANALYZED);
Still works!
Awesome, what if it's not just a number?
Field field = new Field("MyIdField", "ID" + myId, Field.Store.YES, Field.Index.NOT_ANALYZED);
...
Query query = parser.parse("MyIdField:ID1234");
Doesn't work!.. darn.
The query doesn't match the document and so it isn't deleted.
What if we do index it?
Field field = new Field("MyIdField", "ID" + myId, Field.Store.YES, Field.Index.ANALYZED);
...
Query query = parser.parse("MyIdField:ID1234");
It works again!
Ok, so if the field is not analyzed it can still be queried if it only contains a number? Am I missing something?
Thanks for taking some time.
Note:
Technically, there are two fields, making it an AND query. As such, I'd prefer to delete the documents with a Query rather than a Term. I'm not sure if that makes a difference but wanted to emphasize I would like to stick with a solution using a Query.
According to this question, you have to use a PhraseQuery to search a not analyzed field. Your code
Query query = parser.parse("MyIdField:ID1234");
would yield a TermQuery instead, and thus won't match.
I recommend you to try a KeywordAnalyzer instead (remember that, even if your field isn't analyzed, the query parser could still analyze your query string and therefore your match could fail anyway).
I am new to lucene. I have to index date field.
i am using Following IndexWriter constructor in lucene 3.0.0.
IndexWriter writer = new IndexWriter(FSDirectory.open(indexDir), new WhitespaceAnalyzer(), true, IndexWriter.MaxFieldLength.UNLIMITED)
my point is:
Why it needs a analyzer when date fields are not analyzed,while indexing I used Field.Index.NOT_ANALYZED.
You can store date field in this fashion..
Document doc = new Document();
doc.add(new Field("modified",
DateTools.timeToString(f.lastModified(), DateTools.Resolution.MINUTE),
Field.Store.YES, Field.Index.NOT_ANALYZED));
where f is a file object...
Now use the above document for indexwriter...
checkout the sample code comes with lucene... and the following link...
http://lucene.apache.org/java/2_2_0/api/org/apache/lucene/document/DateTools.html
UPDATE
Field.Index NOT_ANALYZED
Index the field's value without using
an Analyzer, so it can be searched. As
no analyzer is used the value will be
stored as a single term. This is
useful for unique Ids like product
numbers.
As per lucene javadoc you don't need analyzer for fields using Field.Index NOT_ANALYZED but i think by design the IndexWriter expects an analyzer as indexing the exact replica of data is not efficient in terms of storage and searching.