Develop Reference

Azure blob storage - findBlobsByTags always fails with error Error parsing query at or near character position 1: unexpected '1'

I am using Azure Blob storage. While persisting blobs, I have persisted index tags. I am able to see index tags when I download the blobs through Microsoft Azure storage explorer.
Programatically, I am trying to filter the tags(using azure-storage-blob version 12.15.0)
I am using this api to get details from DB - findBlobsByTags
This is the code I am trying
FindBlobsOptions findByTags = new FindBlobsOptions(searchExpression));
findByTags.setMaxResultsPerPage(100);
PagedIterable<TaggedBlobItem> blobsByTags = blobContainerClient.findBlobsByTags(findByTags, Duration.ofSeconds(30), Context.NONE);
I have tried following queries
1. String searchExpression = "where=#container = 'container-name' AND \"field\" = \'value\'";
2. String searchExpression = "where=field=value";
3. String searchExpression = "where=\"field\" =\'value\'";
String escapeJava = StringEscapeUtils.escapeJava(searchExpression);
4. String searchExpression = "&where=\"trackingId\"=\'TA00965650\'";
and I am getting this exception.
[Request processing failed; nested exception is
com.azure.storage.blob.models.BlobStorageException: Status code 400, "
InvalidQueryParameterValueError parsing
query at or near character position 1: unexpected '1'
RequestId:78043a6d-901e-00de-0797-510acf000000
Time:2022-04-16T13:42:00.7475672Zwherewhere="field"='value'This query parameter value is invalid."] with root cause
com.azure.storage.blob.models.BlobStorageException: Status code 400, "
InvalidQueryParameterValueError parsing
query at or near character position 1: unexpected '1'
RequestId:78043a6d-901e-00de-0797-510acf000000
Time:2022-04-16T13:42:00.7475672Zwherewhere="trackingId"='TA00965650'This
query parameter value is invalid." at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at
com.azure.core.http.rest.RestProxy.instantiateUnexpectedException(RestProxy.java:390)
at
From the exception log it looks like the query I am passing to the findBlobsByTags method has issues in parsing. Please let me know what I am doing wrong. If some escaping/encoding needs to be done

As mentioned in comments, please remove where= from your search expression. Documentation seems to be incorrect. You may want to create an issue here: https://github.com/Azure/azure-sdk-for-java/issues so that the documentation can be corrected.
You can see working code here: https://github.com/Azure/azure-sdk-for-java/blob/azure-storage-blob_12.16.0/sdk/storage/azure-storage-blob/src/test/java/com/azure/storage/blob/ContainerAPITest.groovy.

I also had a similar issue. I updated the query as below and it works fine.
BlobServiceClient blobServiceClient = getBlobServiceClient();
PagedIterable<TaggedBlobItem> findBlobsByTags = blobServiceClient.findBlobsByTags("\"Company\"='ABC'");
for(TaggedBlobItem taggedBlobItem : findBlobsByTags) {
System.out.println("Find Blob, ContainerName - " + taggedBlobItem.getContainerName()+", Name - "+taggedBlobItem.getName());
}

Bulk Import in CosmosDb with Stored Procedure and Partitioning

I've been using Azure's cosmosDb for a while. Recently i had done bulk import using stored procedure in a collection of my database, and that used to work fine. Now i've to do the same in another collection which uses partitioning; I searched azure code samples and modified my previous bulk insert function like this:
public void createMany(JSONArray aDocumentList, PartitionKey aPartitionKey) throws DocumentClientException {
List<String> aList = new ArrayList<String>();
for(int aIndex = 0; aIndex < aDocumentList.length(); aIndex++) {
JSONObject aJsonObj = aDocumentList.getJSONObject(aIndex);
aList.add(aJsonObj.toString());
}
String aSproc = getCollectionLink() + BULK_INSERTION_PROCEDURE;
RequestOptions requestOptions = new RequestOptions();
requestOptions.setPartitionKey(aPartitionKey);
String result = documentClient.executeStoredProcedure(aSproc,
requestOptions
, new Object[] { aList}).getResponseAsString();
}
but this code gives me error:
com.microsoft.azure.documentdb.DocumentClientException: Message: {"Errors":["Encountered exception while executing function. Exception = Error: {\"Errors\":[\"Requests originating from scripts cannot reference partition keys other than the one for which client request was submitted.\"]}\r\nStack trace: Error: {\"Errors\":[\"Requests originating from scripts cannot reference partition keys other than the one for which client request was submitted.\"]}\n at callback (bulkInsertionStoredProcedure.js:1:1749)\n at Anonymous function (bulkInsertionStoredProcedure.js:689:29)"]}
I'm not quite certain what that error actually means. Since partitionKey is just a JSON key in the document, why would it need it in other Documents also. Do i need to append this in my document also(with partitionKey key) .Could anyone please tell me what i'm missing here? I've searched over the internet and haven't found anything useful that could make it work.

I've already answered this question here. The gist of it is that the documents you're inserting with your SPROC must have a partitionKey that matches the one you pass with the request options
// ALL documents inserted must have a parititionKey value that matches
// "aPartitionKey" value
requestOptions.setPartitionKey(aPartitionKey);
So if aPartitionKey == 123456 then all the documents you are inserting with the SPROC are required to belong to that partition. If you have documents spanning multiple partitions that you want to bulk insert you will have to group them by partition key and run the SPROC separately for each grouping.

No response with a query by ID on Azure DocumentDB

I'm currently facing very slow/ no response on a collection looking by ID. I have ~ 2 milion of documents in a partitioned collection. If lookup the document using the partitionKey and id the response is immediate
SELECT * FROM c WHERE c.partitionKey=123 AND c.id="20566-2"
if I try using only the id
SELECT * FROM c WHERE c.id="20566-2"
the response never returns, java client seems freezed and I have the same situation using the Data Explorer from Azure Portal. I tried also looking up by another field that isn't the id or the partitionKey and the response always returns. When I try the select from Java client I always set the flag to enable cross partition query.
The next thing to try is to avoid the character "-" in the ID to test if this character blocks the query (anyway I didn't find anything on the documentation)

The issue is related to your Java code. Due to Azure DocumentDB Java SDK wrapped the DocumentDB REST APIs, according to the reference of REST API Query Documents, as #DanCiborowski-MSFT said, the header x-ms-documentdb-query-enablecrosspartition explains your issue reason as below.
Header: x-ms-documentdb-query-enablecrosspartition
Required/Type: Optional/Boolean
Description: If the collection is partitioned, this must be set to True to allow execution across multiple partitions. Queries that filter against a single partition key, or against single-partitioned collections do not need to set the header.
So you need to set True to enable cross partition for querying across multiple partitions without a partitionKey in where clause via pass a instance of class FeedOption to the method queryDocuments, as below.
FeedOptions queryOptions = new FeedOptions();
queryOptions.setEnableCrossPartitionQuery(true); // Enable query across multiple partitions
String collectionLink = collection.getSelfLink();
FeedResponse<Document> queryResults = documentClient.queryDocuments(
collectionLink,
"SELECT * FROM c WHERE c.id='20566-2'", queryOptions);

save and find json string in mongodb

I'm building a logging application that does the following:
gets JSON strings from many loggers continuously and saves them to a db
serves the collected data as a per logger bulk
my intention is to use a document based NoSQL storage to have the bulk structure right away. After some research I decided to go for MongoDB because of the following features:
- comprehensive functions to insert data into existing structures ($push, (capped) collection)
- automatic sharding with a key I choose (so I can shard on a per logger basis and therefore serve bulk data in no time - all data already being on the same db server)
The JSON I get from the loggers looks like this:
[
{"bdy":{
"cat":{"id":"36494h89h","toc":55,"boc":99},
"dataT":"2013-08-12T13:44:03Z","had":0,
"rng":23,"Iss":[{"id":10,"par":"dim, 10, dak"}]
},"hdr":{
"v":"0.2.7","N":2,"Id":"KBZD348940"}
}
]
The logger can send more than one element in the same array. I this example it is just one.
I started coding in Java with the mongo driver and the first problem I discovered was: I have to parse my with no doubt valid JSON to be able to save it in mongoDB. I learned that this is due to BSON being the native format of MongoDB. I would have liked to forward the JSON string to the db directly to save that extra execution time.
so what I do in a first Java test to save just this JSON string is:
String loggerMessage = "...the above JSON string...";
DBCollection coll = db.getCollection("logData");
DBObject message = (DBObject) JSON.parse(loggerMessage);
coll.insert(message);
the last line of this code causes the following exception:
Exception in thread "main" java.lang.IllegalArgumentException: BasicBSONList can only work with numeric keys, not: [_id]
at org.bson.types.BasicBSONList._getInt(BasicBSONList.java:161)
at org.bson.types.BasicBSONList._getInt(BasicBSONList.java:152)
at org.bson.types.BasicBSONList.get(BasicBSONList.java:104)
at com.mongodb.DBCollection.apply(DBCollection.java:767)
at com.mongodb.DBCollection.apply(DBCollection.java:756)
at com.mongodb.DBApiLayer$MyCollection.insert(DBApiLayer.java:220)
at com.mongodb.DBApiLayer$MyCollection.insert(DBApiLayer.java:204)
at com.mongodb.DBCollection.insert(DBCollection.java:76)
at com.mongodb.DBCollection.insert(DBCollection.java:60)
at com.mongodb.DBCollection.insert(DBCollection.java:105)
at mongomockup.MongoMockup.main(MongoMockup.java:65)
I tried to save this JSON via the mongo shell and it works perfectly.
How can I get this done in Java?
How could I maybe save the extra parsing?
What structure would you choose to save the data? Array of messages in the same document, collection of messages in single documents, ....

It didn't work because of the array. You need a BasicDBList to be able to save multiple messages. Here is my new solution that works perfectly:
BasicDBList data = (BasicDBList) JSON.parse(loggerMessage);
for(int i=0; i < data.size(); i++){
coll.insert((DBObject) data.get(i));
}

OData4j 0.7 Exception while retrieving workitems

I am trying to retrieve workitems from TFS Server using OData4j 0.7 in Java.
Here is my code:
public List<TFSWorkItem> getWorkItems(final String projectName)
{
final List<TFSWorkItem> tfsWorkItems = new ArrayList<TFSWorkItem>();
String filter = String.format("Project eq '%s'", projectName);
Enumerable<OEntity> workItems = consumer.getEntities("WorkItems").filter(filter).execute();
for (OEntity workitem : workItems)
{
System.out.println(workitem.getProperty("Title", String.class));
}
}
When I run this code I get a
Exception in thread "main" java.lang.IllegalArgumentException: Illegal datetime format 2013-03-15T14:22:08.077+05:30
at org.odata4j.internal.InternalUtil.parseDateTimeFromXml(InternalUtil.java:96)
On further debugging the code I found that OData4j when trying to map the retrieved date from TFS server finds it incompatible .
Date retrieved from TFS :
2013-03-15T14:22:08.077+05:30
Date expected by OData4j :
2013-03-15T14:22:08.077
Is there a way where I can avoid this?
Updated
For anybody who is facing the same issue.
I have modified my code to :
final String fields = "Id,Project,Title";
Enumerable<OEntity> workItems = consumer.getEntities("WorkItems").filter(filter).select(fields.toString()).execute();
for (OEntity workitem : workItems)
{
System.out.println("Id : " + workitem.getProperty("Id").getValue());
System.out.println("Project : "+workitem.getProperty("Project").getValue());
System.out.println("Title : "+workitem.getProperty("Title").getValue());
}
Since I need only these fields to process I have given a select query to select "Id","Project" and "Title" instead of fetching all fields.
This is a temporary fix unless I find a better solution.

I think I figured this out. Check out this post: OData4J exception
Basically, it boils down to setting the value of an Edm.DateTime column in C# code using DateTime.Now. The OData Producer should be using DateTime.UtcNow.
DateTime.Now includes local time zone information. Microsoft's code sees the local time zone info in the DateTime structure and sends back the Edm.DataTime field formatted as an Edm.DateTimeOffset string. The DateTime.UtcNow property does not include any time zone information, so it is correctly formatted as an Edm.DateTime string when it is sent back to the OData Consumer.
I suspect this might be a bug in Microsoft's WCF Data Services stack or their Entity Framework.