I notice strange behavior when querying the GAE datastore. Under certain circumstances Filter does not work for integer fields. The following java code reproduces the problem:
log.info("start experiment");
DatastoreService datastore = DatastoreServiceFactory.getDatastoreService();
int val = 777;
// create and store the first entity.
Entity testEntity1 = new Entity(KeyFactory.createKey("Test", "entity1"));
Object value = new Integer(val);
testEntity1.setProperty("field", value);
datastore.put(testEntity1);
// create the second entity by using BeanUtils.
Test test2 = new Test(); // just a regular bean with an int field
test2.setField(val);
Entity testEntity2 = new Entity(KeyFactory.createKey("Test", "entity2"));
Map<String, Object> description = BeanUtilsBean.getInstance().describe(test2);
for(Entry<String,Object> entry:description.entrySet()){
testEntity2.setProperty(entry.getKey(), entry.getValue());
}
datastore.put(testEntity2);
// now try to retrieve the entities from the database...
Filter equalFilter = new FilterPredicate("field", FilterOperator.EQUAL, val);
Query q = new Query("Test").setFilter(equalFilter);
Iterator<Entity> iter = datastore.prepare(q).asIterator();
while (iter.hasNext()) {
log.info("found entity: " + iter.next().getKey());
}
log.info("experiment finished");
the log looks like this:
INFO: start experiment
INFO: found entity: Test("entity1")
INFO: experiment finished
For some reason it only finds the first entity even though both entities are actually stored in the datastore and both 'field' values are 777 (I see it in the Datastore Viewer)! Why does it matter how the entity is created? I would like to use BeanUtils, because it is convenient.
The same problem occurs on the local devserver and when deployed to GAE.
Ok I found out what is going on. The "problem" is that for some reason BeanUtils transforms integers into strings. A string looks exactly the same in the datastore viewer but it is of course not the same. This pretty much fooled me. I should have studied the apache BeanUtils manual or something.
Have you given the datastore 1 second after writing before you query the data? Sometimes you don't have to (ancestor queries, perhaps) but other times you do. The GAE/J documentation will give full details.
The fact that the entities are created with BeanUtils is completely irrelevant. If the entities are in the datastore (you can see them in the viewer) and the field value is indexed (it does not show "unindexed" next to value in datastore viewer) then you can query for them using a filter. This works... its is the basic functionality of the datastore.
Given the entities are created and indexed, I suggest that Ian Marshalls suggestion is probably correct. To test this, go to the preferences for App Engine and un-tick "Enable local HRD support". This will ensure that when you write an Entity you can query for it immediately.
It is not important if you store an Integer or int or any other numeric value - they are all stored as a long value internally and when you read your value back you will get a Long (despite storing an Integer)
Related
I have written a code to fetch data from Google Datastore in my Google Cloud Dataflow program. I am able to fetch all fields of the entity except Id field which is autogenerated field. I have tried to use entity.getKey() but I am getting null.
Below is my code snippet,
Datastore datastore = DataflowDatastoreService.getDatastoreObject(null, null, null);
Query.Builder queryBuilder = Query.newBuilder();
Filter filter1 = Filter.newBuilder()
.setPropertyFilter(PropertyFilter.newBuilder() .setProperty(PropertyReference.newBuilder().setName("cId"))
.setOp(PropertyFilter.Operator.EQUAL)
.setValue(Value.newBuilder().setIntegerValue(1059438885900008L).build()).build()).build();
Filter filter2 = Filter.newBuilder()
.setPropertyFilter(PropertyFilter.newBuilder()
.setProperty(PropertyReference.newBuilder().setName("active"))
.setOp(PropertyFilter.Operator.EQUAL)
.setValue(Value.newBuilder().setBooleanValue(Boolean.TRUE).build()).build()).build();
Filter composeFilter = Filter.newBuilder().setCompositeFilter(CompositeFilter.newBuilder()
.addFilters(filter1).setOp(Operator.AND).addFilters(filter2).build()).build();
queryBuilder.addKind(KindExpression.newBuilder().setName("MyMaster").build());
queryBuilder.setFilter(composeFilter).build();
RunQueryRequest request = DataflowDatastoreService.makeRequest(queryBuilder.build(), null);
RunQueryResponse response = datastore.runQuery(request);
QueryResultBatch batch = response.getBatch();
List<EntityResult> entityResutls = batch.getEntityResultsList();
List<Entity> myEntities = new ArrayList<>();
Map<String, Value> entityMap = myEntities(0).getPropertiesMap();
In my code I am able to get all fields in entityMap key but I am not getting key, is there any other way through which I can fetch all the fields with Id.
Note: I'm not a java user, answer based on python experience
Indeed, entities returned in a regular query result do not contain the entity key/ID. Attempting to obtain that from the entity is rather inefficient - you need to reach to the datastore for each individual entity (not even looking at why that doesn't appear to be working for you).
If I need the entity keys/IDs I'd instead use keys-only queries - obtaining the keys, from which I can easily get:
the key IDs, locally, without making actual datastore calls (in python via key.id(), I don't know the java equivalent)
the entities via direct key lookup, which can be batched for efficiency.
entity.getKey().getPathList().get(0).getId()
This help me to achieve the result. Getting entity Id through getKey method.
I'm trying to do upsert using mongodb driver, here is a code:
BulkWriteOperation builder = coll.initializeUnorderedBulkOperation();
DBObject toDBObject;
for (T entity : entities) {
toDBObject = morphia.toDBObject(entity);
builder.find(toDBObject).upsert().replaceOne(toDBObject);
}
BulkWriteResult result = builder.execute();
where "entity" is morphia object. When I'm running the code first time (there are no entities in the DB, so all of the queries should be insert) it works fine and I see the entities in the database with generated _id field. Second run I'm changing some fields and trying to save changed entities and then I receive the folowing error from mongo:
E11000 duplicate key error collection: statistics.counters index: _id_ dup key: { : ObjectId('56adfbf43d801b870e63be29') }
what I forgot to configure in my example?
I don't know the structure of dbObject, but that bulk Upsert needs a valid query in order to work.
Let's say, for example, that you have a unique (_id) property called "id". A valid query would look like:
builder.find({id: toDBObject.id}).upsert().replaceOne(toDBObject);
This way, the engine can (a) find an object to update and then (b) update it (or, insert if the object wasn't found). Of course, you need the Java syntax for find, but same rule applies: make sure your .find will find something, then do an update.
I believe (just a guess) that the way it's written now will find "all" docs and try to update the first one ... but the behavior you are describing suggests it's finding "no doc" and attempting an insert.
I have an entity kind called Account. One of the fields is a String named selfie, which is basically the url to a selfie uploaded by a user. I want to fetch for users who have a selfie (so if a user does not have a selfie they should not be included in the result set). I have the following query. but it won't work because I have "NULL" as a string. What is the correct way for doing this? Again, I only want users who have selfies.
Filter selfie = new FilterPredicate("selfie", FilterOperator.GREATER_THAN, "NULL");
Query query = new Query(Account.class.getSimpleName()).setFilter(selfie);
FetchOptions options = FetchOptions.Builder.withLimit(30);
DatastoreService datastore = DatastoreServiceFactory.getDatastoreService();
QueryResultList<Entity> entities = datastore.prepare(query).asQueryResultList(options);
Also I am open to a JDO/JPA way of doing this on App-Engine (but it must work on App-Engine).
After some trial and error on https://console.cloud.google.com/datastore/entities/query/gql
I found this work around expression for IS NOT NULL:
> NULL
Example:
Select * from State where customer_id > NULL
I haven't tested this on a field with negative values.
This should be as simple as
Filter selfie = new FilterPredicate("selfie", FilterOperator.NOT_EQUAL, null);
where you actually pass the Java null value (not a String) and filter with not equals (see https://stackoverflow.com/a/22200137).
Personally, I've been using JDO for my datastore management, which has its pros and cons. Let me know if you'd like to see a solution using JDO as well. Cheers!
I have an application in which I want to overwrite an individual entity. This is how I originally create entity logs:
Entity log = new Entity("Log", "Logkey");
String property1 = req.getParameter("property1");
String property2 = req.getParameter("property2");
log.setProperty("property1", property1);
log.setProperty("property2", property2);
datastore.put(log);
Here is how entity logs are retrieved to be overwritten:
Query query = new Query("Log", "Logkey")
.setFilter(timeStampFilter);
List<Entity> logs = datastore.prepare(query).asList(FetchOptions.Builder.withLimit(1));
request.setAttribute("logs", logs);
and sent to a jsp form page as value="${log.properties.property1}" where they should be overwritten. This entry is then sent to a second servlet with the POST method and requested as parameters as in the earlier code but saved as a new entity with the same kind:
Entity edit_log = new Entity("Log", "Logkey");
String property1 = req.getParameter("property1");
String property2 = req.getParameter("property2");
edit_log.setProperty("property1", property1);
For rewriting and existing entity, after retreiving a specific log by timestamp, you can get the key of this log using getKey() method and then create an entity with this key and the new details. Now when you put this new entity to the datastore it will replace the earlier one with the same key
With the code you've written, you only have a single Log entity in your datastore with the key "Logkey" that you are constantly overwriting.
If you're using some other code to retrieve entities and rewrite them, then you'll need to show that other code. Otherwise, this question is poorly written, because the code given is already describing what you want to do (always overwrite the same entity).
If you have code elsewhere creating/saving entities, it's best to show that too.
Edit: It looks like you end up creating a nested entity with the data from the old entity in a new entity with the same key. It's far easier just to reuse the entity you received from the query.
log = logs.get(0)
log.setProperty("property1", req.getParameter("property1");
log.setProperty("property2", req.getParameter("property2");
datastore.put(log);
Furthermore, since you actually know the key ("Logkey"), you don't need to issue a datastore query, you can just fetch the entity by key - which is good because you get around eventual-consistency behavior.
If your new entity has the same key as your original one, then when you store it it will override the old entity.
I have created entities in the datastore. I want to use the index assigned to them by the datstore for queries.
i.e get an iterable back where the id is greater than a given number
e.g.
Query q = new Query("MyEntity");
q.addFilter("id",Query.FilterOperator.GREATER_THAN_OR_EQUAL, startId);
PreparedQuery pq = datastore.prepare(q);
I know I can get back an individual entity back via id - but how to get a list ?
Cheers,
Iterable<Entity> myEntities = pq.asIterable();
List<Entity> myEntitiesInAList = pq.asList(FetchOptions.Builder.withChunkSize(500));
Side note:
"id" is not the name of the key assigned automatically by App Engine. I think you must use __key__ as the property name to do queries on it, and construct a full key as the search parameter, not just the long id.
If your intent is to remember where you left off and continue there on a subsequent query, consider using a query cursor. The example on that page might give you some further options.