My Spring Boot app is using Couchbase 5.1 community.
My app needs both a primary & several secondary indexes.
Currently, in order to create the needed indexes, I access the UI and the query page and manually create the indexes that the app needs as described here.
I was looking for a way to do it automatically via code, so when the app is starting, it will check if the indexes are missing and will create them if needed.
Is there a way to do it via Spring Data or via the Couchbase client?
You can create them by using the DSL from the index class. There's an example of using it in the documentation under "Indexing the Data: N1QL & GSI"
From that example:
You can also create secondary indexes on specific fields of the JSON,
for better performance:
Index.createIndex("index_name").on(bucket.name(), "field_to_index")
In this case, give a name to your index, specify the target bucket AND
the field(s) in the JSON to index.
If the index already exists, there will be an IndexAlreadyExistsException (see documentation), so you'll need to check for that.
So this is how I solve it:
import com.couchbase.client.java.Bucket;
public class MyCouchBaseRepository{
private Bucket bucket;
public MyCouchBaseRepository(<My Repository that extends CouchbasePagingAndSortingRepository> myRepository){
bucket = myRepository.getCouchbaseOperations().getCouchbaseBucket();
createIndices();
}
private void createIndices(){
bucket.bucketManager().createN1qlPrimaryIndex(true, false)
bucket.query(N1qlQuery.simple("CREATE INDEX xyz ON `myBucket`(userId) WHERE _class = 'com.example.User'"))
...
}
}
Related
We're in the process of converting our java application from Hibernate Search 5 to 6 with an Elasticsearch backend.
For some good background info, see How to do highlighting within HibernateSearch over Elasticsearch for a question we had when upgrading our highlighting code from a Lucene to Elasticsearch backend and how it was resolved.
Hibernate Search 6 seems to support using 2 backends at the same time, Lucene and Elasticsearch, so we'd like to use Elasticsearch for all our queries and Lucene for the highlighting, if that's possible.
Here is basically what we're trying to do:
public boolean matchPhoneNumbers() {
String phoneNumber1 = "603-436-1234";
String phoneNumber2 = "603-436-1234";
LuceneBackend luceneBackend =
Search.mapping(entityManager.getEntityManagerFactory())
.backend().unwrap(LuceneBackend.class);
Analyzer analyzer = luceneBackend.analyzer("phoneNumberKeywordAnalyzer").get();
//... builds a Lucene Query using the analyzer and phoneNumber1 term
Query phoneNumberQuery = buildQuery(analyzer, phoneNumber1, ...);
return isMatch("phoneNumberField", phoneNumber2, phoneNumberQuery, analyzer);
}
private boolean isMatch(String field, String target, Query sourceQ, Analyzer analyzer) {
Highlighter highlighter = new Highlighter(new QueryScorer(sourceQ, field));
highlighter.setTextFragmenter(new NullFragmenter());
try {
String result = highlighter.getBestFragment(analyzer, field, target);
return StringUtils.hasText(result);
} catch (IOException e) {
...
}
}
What I've attempted so far is to configure two separate backends in the configuration properties, per the documentation, like this:
properties.setProperty("hibernate.search.backends.elasticsearch.analysis.configurer", "com.bt.demo.search.AnalysisConfigurer");
properties.setProperty("hibernate.search.backends.lucene.analysis.configurer", "com.bt.demo.search.CustomLuceneAnalysisConfigurer");
properties.setProperty("hibernate.search.backends.elasticsearch.type", "elasticsearch");
properties.setProperty("hibernate.search.backends.lucene.type", "lucene");
properties.setProperty("hibernate.search.backends.elasticsearch.uris", "http://127.0.0.1:9200");
The AnalysisConfigurer class implements ElasticsearchAnalysisConfigurer and
CustomLuceneAnalysisConfigurer implements from LuceneAnalysisConfigurer.
Analyzers are defined twice, once in the Elasticsearch configurer and again in the Lucene configurer.
I don't know why both hibernate.search.backends.elasticsearch.type and hibernate.search.backends.lucene.type are necessary but if I don't include the lucene.type, I get Ambiguous backend type: configuration property 'hibernate.search.backends.lucene.type' is not set.
But if I do have both backend properties types set, I get
HSEARCH000575: No default backend. Check that at least one entity is configured to target the default backend, when attempting to retrieve the Lucene backend, like:
Search.mapping(entityManager.getEntityManagerFactory())
.backend().unwrap(LuceneBackend.class);
And the same error when trying to retrieve the Elasticsearch backend.
I've also added #Indexed(..., backend = "elasticsearch") to my entities since I wish to have them saved into Elasticsearch and don't need them in Lucene. I also tried adding a fake entity with #Indexed(..., backend = "lucene") but it made no difference.
What have I got configured wrong?
I don't know why both hibernate.search.backends.elasticsearch.type and hibernate.search.backends.lucene.type are necessary but if I don't include the lucene.type, I get Ambiguous backend type: configuration property 'hibernate.search.backends.lucene.type' is not set.
That's because the backend name is just that: a name. Hibernate Search doesn't infer particular information from it, even if you name your backend "lucene" or "elasticsearch". You could have multiple Elasticsearch backends for all it knows :)
But if I do have both backend properties types set, I get HSEARCH000575: No default backend. Check that at least one entity is configured to target the default backend, when attempting to retrieve the Lucene backend, like:
Search.mapping(entityManager.getEntityManagerFactory())
.backend().unwrap(LuceneBackend.class);
``
You called .backend(), which retrieves the default backend, i.e. the backend that doesn't have a name and is configured through hibernate.search.backend.* instead of hibernate.search.backends.<somename>.* (see https://docs.jboss.org/hibernate/stable/search/reference/en-US/html_single/#configuration-structure ).
But you are apparently mapping all your entities to a named backends, one named elasticsearch and one named lucene. So the default backend just doesn't exist.
You should call this:
Search.mapping(entityManager.getEntityManagerFactory())
.backend("lucene").unwrap(LuceneBackend.class);
I've also added #Indexed(..., backend = "elasticsearch") to my entities since I wish to have them saved into Elasticsearch
Since you obviously only want to use one backend for indexing, I would recommend reverting that change (keeping #Indexed without setting #Indexed.backend) and simply making using the default backend.
In short, remove the #Indexed.backend and replace this:
properties.setProperty("hibernate.search.backends.elasticsearch.analysis.configurer", "com.bt.demo.search.AnalysisConfigurer");
properties.setProperty("hibernate.search.backends.lucene.analysis.configurer", "com.bt.demo.search.CustomLuceneAnalysisConfigurer");
properties.setProperty("hibernate.search.backends.elasticsearch.type", "elasticsearch");
properties.setProperty("hibernate.search.backends.lucene.type", "lucene");
properties.setProperty("hibernate.search.backends.elasticsearch.uris", "http://127.0.0.1:9200");
With this
properties.setProperty("hibernate.search.backend.analysis.configurer", "com.bt.demo.search.AnalysisConfigurer");
properties.setProperty("hibernate.search.backends.lucene.analysis.configurer", "com.bt.demo.search.CustomLuceneAnalysisConfigurer");
properties.setProperty("hibernate.search.backend.type", "elasticsearch");
properties.setProperty("hibernate.search.backends.lucene.type", "lucene");
properties.setProperty("hibernate.search.backend.uris", "http://127.0.0.1:9200");
You don't technically have to do that, but I think it will be simpler in the long term. It keeps the Lucene backend as a separate hack that doesn't affect your whole application.
I also tried adding a fake entity with #Indexed(..., backend = "lucene")
I confirm you will need that fake entity mapped to the "lucene" backend, otherwise Hibernate Search will not create the "lucene" backend.
Using Hibernate Search 5.11.3 with programmatic API (no annotations), is there a way to facet on dynamic fields added in a class or field bridge? I don't see any 'facet' config available in FieldMetadataBuilder when using MetadataProvidingFieldBridge.
I have tried various combinations of luceneOptions.addSortedDocValuesFieldToDocument() and luceneOptions.addFieldToDocument() in the set() method. This successfully updates the index, but I cannot perform facet queries.
I am trying to do a basic attribute facet/filter where I have a generic table of attributes with id/name and attribute values associated with products. For various reasons I am using the programmatic API and especially for attributes I can't make use of the #Facet annotation. So for a product, I added this class bridge to Product.class:
public class ProductClassTagValuesBridge implements FieldBridge
{
#Override
public void set(String name, Object value, Document document, LuceneOptions luceneOptions)
{
Product product = (Product) value;
for (TagValue v : product.getTagValues())
{
Tag tag = v.getTag();
String tagName = "tag-" + tag.getId();
String tagValue = v.getId().toString();
// not sure if this line is required? Have tried with and without
luceneOptions.addFieldToDocument(tagName, tagValue, document);
luceneOptions.addSortedDocValuesFieldToDocument(tagName, tagValue, document);
}
}
}
Then I build my (test) faceting request to search tag-56 (which I confirmed is in the index using Luke):
FacetParameterContext context = queryBuilder.facet()
.name("tag-56")
.onField("tag-56")
.discrete();
FacetingRequest facetingRequest = context.createFacetingRequest();
Which when used in the search/FacetManager gives me the error:
org.hibernate.search.exception.SearchException: HSEARCH000268: Facet request 'TAG_56' tries to facet on field 'tag-56' which either does not exist or is not configured for faceting (via #Facet). Check your configuration.
I have also tried the custom config solution from the solution in this post: Hibernate Search: configure Facet for custom FieldBridge
For the custom field I added a field bridge to tagValues on my product. The same error occurs.
mapping.entity(Product.class).indexed()
.property("tagValues", ElementType.FIELD).field()
.analyze(Analyze.NO).store(Store.YES)
.bridge(ProductTagValuesFieldBridge.class)
Short answer: Hibernate Search does not allow that... yet.
Long answer:
Hibernate Search 5 allows dynamic fields, but does not allow faceting on fields declared in custom bridges.
That is to say, you can add arbitrary values to your index that don't fit a pre-defined schema, but you cannot use faceting on those fields.
Hibernate search 6 allows faceting (now called "aggregations") on fields declared in custom bridges (just declare them as .aggregable(Aggregable.YES)), but does not allow dynamic fields yet.
EDIT: Starting with 6.0.0.Beta7, dynamic fields are supported thanks to field templates. So the rest of my message is not useful anymore.
See this section of the documentation for more information about field templates. It's totally possible to declare an aggregable, dynamic field in your bridge.
Original message about ways to work without dynamic fields (obsolete):
That is to say, if you know the list of tags upon startup, are able to list them all, and are certain they won't change while your application is up, you could declare the fields upfront and use faceting on them. But if you don't know the list of tags upon startup, none of this is possible (yet).
Until dynamic fields are added to Hibernate Search 6, the only solution is to use Hibernate Search 5 and to re-implement faceting yourself. As you can expect, this will be complex and you will have to get your hands dirty with Lucene. You will have to:
Add fields of type SortedSetDocValuesFacetField to your document in your custom bridge.
Ensure Hibernate Search calls FacetsConfig.build on your documents after they are populated. One way to do that (through a hack) would be to declare a dummy #Facet field on your entity, even if you don't use it.
Completely ignore Hibernate Search's query feature and perform faceting yourself from an IndexReader. You can get an IndexReader from Hibernate Search as explained here. There's an example of how to perform faceting in org.hibernate.search.query.engine.impl.QueryHits#updateStringFacets.
I have two applications which use the same Elasticsearch instance as a search engine. Both applications share the same code base and have only minor differences.
Applications run against different databases, and hence, the different ES indices should be used.
I try to parameterize index name using SpEL like this:
#Indexed(index="${es.index.users}")
public UserEntity {}
However, it doesn't work.
The second option I've tried was setting a different prefix for different applications via hibernate.search.default.indexBase=<app_name>. However, it works only for the Lucene engine but not for ES.
Is there a way to pass the index name into #Indexed annotation on the runtime?
If not, is there another way to pass the index which should be used?
At the moment, the only solution would be to use the programmatic mapping API. This will allow you to execute code to set the index names. If you need to retrieve the index names from configuration files, that will be on you...
First, remove the #Indexed annotations from your indexed entities.
Then, implement a mapping factory:
package com.myCompany;
// ... imports ...
public class MyAppSearchMappingFactory {
#Factory
public SearchMapping getSearchMapping() {
SearchMapping mapping = new SearchMapping();
for ( Map.Entry<Class<?>, String> entry : getIndexNames() ) {
mapping.entity( entry.getKey() ).indexed().indexName( entry.getValue() );
}
return mapping;
}
private Map<Class<?>, String> getIndexNames() {
// Fetch the index names somehow. Maybe just use a different implementation of this class in each application?
}
}
Then reference it in the Hibernate ORM properties (persistence.xml, hibernate.properties, or some framework-specific file, depending on what you use):
hibernate.search.model_mapping com.myCompany.MyAppSearchMappingFactory;
And you should be all set.
I'm trying to use Morphia to interface with MongoDB, and my Morphia entity looks like this:
#Entity(some params about storing the entity)
public class Entity implements Serializable {
<Some other fields here>
#Indexed(options =
#IndexOptions(unique = true, sparse = true)
)
private String field;
<Some other fields here>
}
I would like this field to be unique if present, but not required (and not unique if not present; multiple entries should be able to exclude this field). My understanding of how to do this is with a unique sparse index, as I've tried to set up.
The problem I'm running into is that when I check the index configuration in Studio3T, it appears that my index is being created as unique, but the sparse property is not applied.
What am I doing wrong?
Thanks.
EDIT: Upon further research, this appears like it might be an issue with Microsoft Azure CosmosDB. When I run this code locally, it works fine, but it does not create the sparse index properly on Azure CosmosDB. Updating tags accordingly.
I need to retrieve data from legacy Couchbase bucket without a specific schema. It can be mapped as com.couchbase.client.java.document.StringDocument from the Couchbase Java client. I can do this directly using Java client:
bucket.get(key, StringDocument.class)
But how can I map this StringDocument using org.springframework.data.repository.CrudRepository?
I can't create such interface interface UserRepository extends CrudRepository<StringDocument, String> because Spring Data Couchbase requires #Document and #Id annotations.
All entities should be annotated with the #Document annotation. Also, every field in the entity should be annotated with the #Field annotation. While this is - strictly speaking - optional, it helps to reduce edge cases and clearly shows the intent and design of the entity.
There is also a special #Id annotation which needs to be always in place. Best practice is to also name the property id.
Should I directly use Bucket, create my own similar entity, or there is another solution?
I use following version of spring-boot-starter-data-couchbase: 1.5.9.RELEASE.
If its just a StringDocument (and it wasn't even saved by spring-data-couchbase to begin with.. which injects the _class attribute for deserialization purposes) then I'd just use the bucket to retrieve it.
Keep in mind if your repository (I'm talking about any other repository you have defined that is mapped to the same bucket the StringDocument is in) is defined as CouchbaseRepository you will be able to access the bucket itself from the repository methods like so repository.getCouchbaseOperations().getCouchbaseBucket(). Or maybe you can try the following repository.getCouchbaseOperations().findById("id", StringDocument.class).