I'd like to do something like
SELECT e1.sender
FROM email as e1, email as e2
WHERE e1.sender = e2.receiver;
but in MongoDB. I found many forums about JOIN, which can be implemented via MapReduce in MongoDB, but I don't understand how to do it in this example with self-join.
I was thinking about something like this:
var map1 = function(){
var output = {
sender:db.collectionSender.email,
receiver: db.collectionReceiver.findOne({email:db.collectionSender.email}).email
}
emit(this.email, output);
};
var reduce1 = function(key, values){
var outs = {sender:null, receiver:null
values.forEach(function(v) {
if(outs.sender == null){
outs.sender = v.sender
}
if(outs.receivers == null){
outs.receiver = v.receiver
}
});
return outs; }};
db.email.mapReduce(map2,reduce2,{out:'rec_send_email'})
to create 2 new collections - collectionReceiver containing only receiver email and collectionSender containing only sender email
OR
var map2 = function(){
var output = {sender:this.sender,
receiver: db.email.findOne({receiver:this.sender})}
emit(this.sender, output);
};
var reduce2 = function(key, values){
var outs = {sender:null, receiver:null
values.forEach(function(v){
if(outs.sender == null){
outs.sender = v.sender
}
if(outs.receiver == null){
outs.receiver = v.receiver
}
});
return outs; };};
db.email.mapReduce(map2,reduce2,{out:'rec_send_email'})
but none of them is working and I don't understand this MapReduce-thing well. Could somebody explain it to me please? I was inspired by this article http://tebros.com/2011/07/using-mongodb-mapreduce-to-join-2-collections/ .
Additionally, I need to write it in Java. Is there any way how to solve it?
If you need to implement a "self-join" when using MongoDB then you may have structured your schema incorrectly (or sub-optimally).
In MongoDB (and noSQL in general) the schema structure should reflect the queries you will need to run against them.
It looks like you are assuming a collection of emails where each document has one sender and one receiver and now you want to find all senders who also happen to be receivers of email? The only way to do this would be via two simple queries, and not via map/reduce (which would be far more complex, unnecessary and the way you've written them wouldn't work as you can't query from within map function).
You are writing in Java - why not make two queries - the first to get all unique senders and the second to find all unique receivers who are also in the list of senders?
In the shell it would be:
var senderList = db.email.distinct("sender");
var receiverList = db.email.distinct("receiver", {"receiver":{$in:senderList}})
Related
I'm using azure-documentdb java SDK in order to create and use "User Defined Functions (UDFs)"
So from the official documentation I finally find the way (with a Java client) on how to create an UDF:
String regexUdfJson = "{"
+ "id:\"REGEX_MATCH\","
+ "body:\"function (input, pattern) { return input.match(pattern) !== null; }\","
+ "}";
UserDefinedFunction udfREGEX = new UserDefinedFunction(regexUdfJson);
getDC().createUserDefinedFunction(
myCollection.getSelfLink(),
udfREGEX,
new RequestOptions());
And here is a sample query :
SELECT * FROM root r WHERE udf.REGEX_MATCH(r.name, "mytest_.*")
I had to create the UDF one time only because I got an exception if I try to recreate an existing UDF:
DocumentClientException: Message: {"Errors":["The input name presented is already taken. Ensure to provide a unique name property for this resource type."]}
How should I do to know if the UDF already exists ?
I try to use "readUserDefinedFunctions" function without success. Any example / other ideas ?
Maybe for the long term, should we suggest a "createOrReplaceUserDefinedFunction(...)" on azure feedback
You can check for existing UDFs by running query using queryUserDefinedFunctions.
Example:
List<UserDefinedFunction> udfs = client.queryUserDefinedFunctions(
myCollection.getSelfLink(),
new SqlQuerySpec("SELECT * FROM root r WHERE r.id=#id",
new SqlParameterCollection(new SqlParameter("#id", myUdfId))),
null).getQueryIterable().toList();
if (udfs.size() > 0) {
// Found UDF.
}
An answer for .NET users.
`var collectionAltLink = documentCollections["myCollection"].AltLink; // Target collection's AltLink
var udfLink = $"{collectionAltLink}/udfs/{sampleUdfId}"; // sampleUdfId is your UDF Id
var result = await _client.ReadUserDefinedFunctionAsync(udfLink);
var resource = result.Resource;
if (resource != null)
{
// The UDF with udfId exists
}`
Here _client is Azure's DocumentClient and documentCollections is a dictionary of your documentDb collections.
If there's no such UDF in the mentioned collection, the _client throws a NotFound exception.
I would like to update a specific collection in MongoDb via Spark in Java.
I am using the MongoDB Connector for Hadoop to retrieve and save information from Apache Spark to MongoDb in Java.
After following Sampo Niskanen's excellent post regarding retrieving and saving collections to MongoDb via Spark, I got stuck with updating collections.
MongoOutputFormat.java includes a constructor taking String[] updateKeys, which I am guessing is referring to a possible list of keys to compare on existing collections and perform an update. However, using Spark's saveAsNewApiHadoopFile() method with parameter MongoOutputFormat.class, I am wondering how to use that update constructor.
save.saveAsNewAPIHadoopFile("file:///bogus", Object.class, Object.class, MongoOutputFormat.class, config);
Prior to this, MongoUpdateWritable.java was being used to perform collection updates. From examples I've seen on Hadoop, this is normally set on mongo.job.output.value, maybe like this in Spark:
save.saveAsNewAPIHadoopFile("file:///bogus", Object.class, MongoUpdateWritable.class, MongoOutputFormat.class, config);
However, I'm still wondering how to specify the update keys in MongoUpdateWritable.java.
Admittedly, as a hacky way, I've set the "_id" of the object as my document's KeyValue so that when a save is performed, the collection will overwrite the documents having the same KeyValue as _id.
JavaPairRDD<BSONObject,?> analyticsResult; //JavaPairRdd of (mongoObject,result)
JavaPairRDD<Object, BSONObject> save = analyticsResult.mapToPair(s -> {
BSONObject o = (BSONObject) s._1;
//for all keys, set _id to key:value_
String id = "";
for (String key : o.keySet()){
id += key + ":" + (String) o.get(key) + "_";
}
o.put("_id", id);
o.put("result", s._2);
return new Tuple2<>(null, o);
});
save.saveAsNewAPIHadoopFile("file:///bogus", Object.class, Object.class, MongoOutputFormat.class, config);
I would like to perform the mongodb collection update via Spark using MongoOutputFormat or MongoUpdateWritable or Configuration, ideally using the saveAsNewAPIHadoopFile() method. Is it possible? If not, is there any other way that does not involve specifically setting the _id to the key values I want to update on?
I tried several combination of config.set("mongo.job.output.value","....") and several combination of
.saveAsNewAPIHadoopFile(
"file:///bogus",
classOf[Any],
classOf[Any],
classOf[com.mongodb.hadoop.MongoOutputFormat[Any, Any]],
mongo_config
)
and none of them worked.
I made it to work by using MongoUpdateWritable class as output of my map method:
items.map(row => {
val mongo_id = new ObjectId(row("id").toString)
val query = new BasicBSONObject()
query.append("_id", mongo_id)
val update = new BasicBSONObject()
update.append("$set", new BasicBSONObject().append("field_name", row("new_value")))
val muw = new MongoUpdateWritable(query,update,false,true)
(null, muw)
})
.saveAsNewAPIHadoopFile(
"file:///bogus",
classOf[Any],
classOf[Any],
classOf[com.mongodb.hadoop.MongoOutputFormat[Any, Any]],
mongo_config
)
The raw query executed in mongo is then something like this:
2014-11-09T13:32:11.609-0800 [conn438] update db.users query: { _id: ObjectId('5436edd3e4b051de6a505af9') } update: { $set: { value: 10 } } nMatched:1 nModified:0 keyUpdates:0 numYields:0 locks(micros) w:24 3ms
I'm using the below code to get all the available volumes under EC2. But I can't find any Ec2 api to get already attached volumes with an instance. Please let me know how to get all attached volumes using instanceId.
EC2Api ec2Api = computeServiceContext.unwrapApi(EC2Api.class);
List<String> volumeLists = new ArrayList<String>();
if (null != volumeId) {
volumeLists.add(volumeId);
}
String[] volumeIds = volumeLists.toArray(new String[0]);
LOG.info("the volume IDs got from user is ::"+ Arrays.toString(volumeIds));
Set<Volume> ec2Volumes = ec2Api.getElasticBlockStoreApi().get()
.describeVolumesInRegion(region, volumeIds);
Set<Volume> availableVolumes = Sets.newHashSet();
for (Volume volume : ec2Volumes) {
if (volume.getSnapshotId() == null
&& volume.getStatus() == Volume.Status.AVAILABLE) {
LOG.debug("available volume with no snapshots ::" + volume.getId());
availableVolumes.add(volume);
}
}
The AWS Java SDK now provides a method to get all the block device mappings for an instance. You can use that to get a list of all the attached volumes:
// First get the EC2 instance from the id
DescribeInstancesRequest describeInstancesRequest = new DescribeInstancesRequest().withInstanceIds(instanceId);
DescribeInstancesResult describeInstancesResult = ec2.describeInstances(describeInstancesRequest);
Instance instance = describeInstancesResult.getReservations().get(0).getInstances().get(0);
// Then get the mappings
List<InstanceBlockDeviceMapping> mappingList = instance.getBlockDeviceMappings();
for(InstanceBlockDeviceMapping mapping: mappingList) {
System.out.println(mapping.getEbs().getVolumeId());
}
You can filter the output of the EC2 DescribeVolumes API call. There are various attachment.* filters available, the one you want is filtering by attached instance ID. Try the following code:
Multimap<String, String> filter = ArrayListMultimap.create();
filter.put("attachment.instance-id", instanceId);
filter.put("attachment.status", "attached");
Set<Volume> volumes = ec2Api.getElasticBlockStoreApi().get()
.describeVolumesInRegionWithFilter(region, volumeIds, filter);
The filter is a Multimap with the keys and values you want to filter on. You can actually specify the same filter multiple times, for example to get all volumes attached to a number of different instances.
You can use volumeAttachmentApi.listAttachmentsOnServer() to do this.
NovaApi novaApi = context.unwrapApi(NovaApi.class);VolumeApi volumeApi = novaApi.getVolumeExtensionForZone(region).get();
VolumeAttachmentApi volumeAttachmentApi = novaApi.getVolumeAttachmentExtensionForZone(region).get();
volumeAttachmentApi.listAttachmentsOnServer(serverId)
iam trying to orderlookup droplet API by passing some parameters.I assume that the parameters which are mandatory is userId and organisationIds which i have passed and additionally i have also passed "state" parameter.All these params are passed thru request and then the service method of droplet is invoked.But the service method returns nothing.My goal is to check whether this droplet this retrieving the expected set of orders or not.We can use droplet invoker but i tried that way but it didnt work may be i missed something.Please help me out!!
this is my code when i tried to use OrderLookUp API
DynamoHttpServletRequest request = ServletUtil.getCurrentRequest();
mTestService.setCurrentRequest(request);
if (request == null) {
mTestService.vlogError("Request is null.");
Assert.fail("Request is null ");
}
else
{
Object droplet = mTestService
.getRequestScopedComponent("OrderLookupDroplet");
OrderLookupDroplet=(OrderLookup) droplet;
request.setParameter("state", "submitted");
request.setParameter("organisationIds", organizationIds);
request.setParameter("userId", userId);
ByteBuffer buffer = ByteBuffer.allocate(1024);
DynamoHttpServletRequest dynRequest = (DynamoHttpServletRequest) request;
TestingDynamoHttpServletRequest wrappedRequest = new TestingDynamoHttpServletRequest(
dynRequest, buffer);
TestingDynamoHttpServletResponse wrappedResponce = new TestingDynamoHttpServletResponse(
dynRequest.getResponse());
OrderLookupDroplet.service(wrappedRequest, wrappedResponce);
}
the above sample is only part of the code..
this is the code when i tried using droplet invoker
DropletInvoker invoker = new DropletInvoker(mNucleus);
invoker.getRequest().setParameter("state", "submitted");
// String [] siteIds = {"siteA", "siteB"};
// invoker.getRequest().setParameter("siteIds", Arrays.asList(siteIds));
String [] organizationIds = {"OrgA", "OrgB"};
invoker.getRequest().setParameter("organizationIds", organizationIds);
String [] orderIds = {"orderautouser001OrgA" , "orderautouser001OrgB"};
invokeDroplet(invoker, "autouser001", orderIds);
......
protected void invokeDroplet(DropletInvoker pInvoker, String pUserId, String[] pOrderIds) throws Exception
{
Map<String, Object> localParams = new HashMap();
localParams.put("userId", pUserId);
DropletResult result = pInvoker.invokeDroplet("/atg/commerce/order/OrderLookup", localParams);
RenderedOutputParameter oparam = result.getRenderedOutputParameter("output", 0);
assertNotNull("'output' oparam was not rendered", oparam);
assertEquals("Check totalCount.", pOrderIds.length, oparam.getFrameParameter("totalCount"));
List<Order> orders = (List<Order>)oparam.getFrameParameter("result");
assertEquals("Check order array length.", pOrderIds.length, orders.size());
for (int index = 0; index < pOrderIds.length; index++) {
boolean found = false;
for (Order order: orders) {
if (pOrderIds[index].equals(order.getId())) {
found = true;
break;
}
}
assertTrue("Expected orderId " + pOrderIds[index] + " not found in result array", found);
}
in first case i donno how to retrieve the orders by directly using orderlookup api....and in second case though i know how to use it ,iam still failing!! please help me out..thanks in advance
You should't use droplets in java classes they should be used only inside jsp pages. Documentation of OrderLookup with example hot to use it on jsp page is here.
If you want to get orders or any other data stored in a repository you should use repository API with RQL (Repository Query Language). Example how to get data from repository you can find here and RQL grammar here.
Thanks for giving your opinions.Good news is we can invoke droplets from any other API
OrderLookup droplet = (OrderLookup) sNucleus.resolveName("/atg/commerce/order/OrderLookup");
ServletTestUtils utils = new ServletTestUtils();
mRequest = utils.createDynamoHttpServletRequestForSession(sNucleus, null, null);
ServletUtil.setCurrentRequest(mRequest);
mResponse = new DynamoHttpServletResponse();
mRequest.setResponse(mResponse);
mResponse.setRequest(mRequest);
mResponse.setResponse(new GenericHttpServletResponse());
mRequest.setParameter("userId", "publishing");
droplet.setSearchByUserId(true);
droplet.service(mRequest, mResponse);
ArrayList<Order> orders = (ArrayList<Order>) mRequest.getObjectParameter("result");
here the "result" param is output param which this droplet sets.and the userId i have hardcoded as "publishing" which i have created.Ignore servletTestUtils class that is created by me which has not much to do with droplet theory here :)
I assume from your code example, and the fact that you mention DropletInvoker that you are writing a unit test, and that this is not functional code.
If it is functional code, you really, really, should not invoke a droplet from another Nucleus component. A droplet exists solely to be used in a JSP page. If you need the functionality of the droplet in Java code, you should refactor the droplet into a service that holds the main logic, and a droplet that simply acts as a façade to the service to allow it to be invoked from a page.
In the case of the OrderLookup look droplet, you don't need to refactor anything. The service to use should be OrderManager or OrderTools depending on what you need. Note, there is a difference between Order objects and Order repository items, and you should prefer to use order objects - so only use the Order Repository directly if you really need to.
How to get the details of Multiple products in a Single Call in Android using XMLRPC from Magento.I am able to get the list of products using the function catalog_product.list using XMLRPC.
Now, i have the SKU id's of all the products.I am able to get the media details of each product using the function product_media.list.
If suppose I have 10 products,i have to call 10 times product_media.list method for each product which takes long time.
So,how can I call the multiCall function of Magento in Android. Many tutorials in php for calling the multiCall function are posted but I am not able to imitate the same in Android.
So please help me if you have similar code snippet that can make me understand multiCall function(for Android) so that I can Advance further using that.
Thanks.
PHP code Example from Josua Marcel C 's Answer:
$session = $client->call('login', array('apiUser', 'apiKey'));
$client->call('call', array($session,'somestuff.method', array('arg1', 'arg2', 'arg3')));
$client->call('call', array($session, 'somestuff.method', 'arg1'));
$client->call('call', array($session, 'somestuff.method'));
$client->call('multiCall',
array($session,
array(
array('somestuff.method', 'arg1'),
array('somestuff.method', array('arg1', 'arg2')),
array('somestuff.method')
)
)
);
I would like to imitate the above php code in Android that calls the multiCall() function of the Magento.
After making long long Research, I got half-way Solution that calls the multiCall() method without any Error,but Still I don't know how to get the response of the Server in a variable and use it.
AnyOne who has knowledge of it can Edit my Answer, I will be thankful to him.
The Code that I have Used is :
Object[] skuid=new Object[product_list.size()];
Object calling[]=new Object[product_list.size()];
for(int m=0;m<product_list.size();m++)
{
skuid[m]=new Object[]{product_list.get(m).getp_Sku()};
calling[m]=new Object[]{"catalog_product_attribute_media.list",skuid[m]};
}
try
{
client.callEx("multiCall",new Object[]{Utils.sessionId,calling});
}
catch (XMLRPCException e)
{
e.printStackTrace();
}
AcknowledgeMents :
I have worked on the Answer posted by Iain.
The Answer
since android is based java application, You can use this.
package org.apache.xmlrpc;
import java.util.Hashtable;
import java.util.Vector;
public class MultiCall
implements ContextXmlRpcHandler
{
public Object execute(String method, Vector params, XmlRpcContext context)
throws Exception
{
if ("multicall".equals(method))
{
return multicall(params, context);
}
throw new NoSuchMethodException("No method '" + method + "' in " + this.getClass().getName());
}
public Vector multicall(Vector requests, XmlRpcContext context)
{
// The array of calls is passed as a single parameter of type array.
requests=(Vector)requests.elementAt(0);
Vector response = new Vector();
XmlRpcServerRequest request;
for (int i = 0; i < requests.size(); i++)
{
try
{
Hashtable call = (Hashtable) requests.elementAt(i);
request = new XmlRpcRequest((String) call.get("methodName"),
(Vector) call.get("params"));
Object handler = context.getHandlerMapping().getHandler(request.getMethodName());
Vector v = new Vector();
v.addElement(XmlRpcWorker.invokeHandler(handler, request, context));
response.addElement(v);
}
catch (Exception x)
{
String message = x.toString();
int code = (x instanceof XmlRpcException ?
((XmlRpcException) x).code : 0);
Hashtable h = new Hashtable();
h.put("faultString", message);
h.put("faultCode", new Integer(code));
response.addElement(h);
}
}
return response;
}
}
Source
Since Magento support SOAP API why didn't you use SOAP API v1? because SOAP is powerful. try to go here What's the difference between XML-RPC and SOAP?
Parsing of Soap messages is not included in Android runtime, so it isn't really straightforward. You should use an external library. I'm using ksoap2.
If you search here on StackOverflow you'll see many examples on how to use it. For instance here
more references: link 1 link 2
MultiCall with PHP
$client = new Zend_XmlRpc_Client('http://magentohost/api/xmlrpc/');
// If somestuff requires api authentification,
// we should get session token
$session = $client->call('login', array('apiUser', 'apiKey'));
$client->call('call', array($session, 'somestuff.method', array('arg1', 'arg2', 'arg3')));
$client->call('call', array($session, 'somestuff.method', 'arg1'));
$client->call('call', array($session, 'somestuff.method'));
$client->call('multiCall', array($session,
array(
array('somestuff.method', 'arg1'),
array('somestuff.method', array('arg1', 'arg2')),
array('somestuff.method')
)
));
// If you don't need the session anymore
$client->call('endSession', array($session));
First login in whatever way works for calling catalog_product.list. Make sure session, client and product_ids have the right values. If you don't need to log in for these operations, set session = null (and if that doesn't work, try not passing session at all :) ). Then:
Object[][] calls = new Object[product_ids.length];
for (int i = 0; i < product_ids.length; i++) {
calls[i] = new Object[] { "product_media.list", product_ids[i] };
}
product_media_ids = client.call("multiCall", new Object[] { session, calls });
product_media_ids should then be an array of arrays of product images - that is, each element of product_media_ids will be a return value from product_media.list.
The code is untested, I'm afraid.