Develop Reference

Unit test using HBaseTestingUtility

I am trying to debug the java code using HBaseTestingUtility library. I already have table created. I need to:
- Insert a value with a key in "myTable"
- Get the value from "myTable" with the key
- Verify the returned value is equal to the value I created
Here is the code that I filled out:
package HbaseUniteTest;
import jdk.nashorn.api.scripting.ScriptUtils;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.*;
import org.apache.hadoop.hbase.client.*;
import org.apache.hadoop.hbase.io.compress.Compression;
import org.apache.hadoop.hbase.util.Bytes;
import org.apache.hadoop.hbase.HTableDescriptor;
import org.junit.Assert;
import static org.junit.Assert.assertEquals;
public class TestCreateTableClass
{
private final static String tableName = "myTable";
private static ScriptUtils HTableUtil;
public static void main( String[] args ) throws Exception {
//Start the "mini cluster"
HBaseTestingUtility testingUtility = new HBaseTestingUtility();
testingUtility.startMiniCluster();
//Get the configuration
//Configuration conf = ...
Configuration conf = testingUtility.getConfiguration();
//Instantiate a connection
Connection connection = ConnectionFactory.createConnection(conf);
//Define table "myTable"
HTableDescriptor table = new HTableDescriptor(TableName.valueOf(tableName));
table.addFamily(new HColumnDescriptor("cf1").setCompressionType(Compression.Algorithm.NONE));
//Create table "myTable"
connection.getAdmin().createTable(table);
//Get the first (and only) table name
String first_table = connection.getAdmin().listTableNames()[0].getNameAsString();
//Verify the returned Table name is equal to the table name we provided
assertEquals(tableName,first_table);
//Insert a value with a key in "myTable"
byte[] key = Bytes.toBytes("some-key");
Put put = new Put(key);
put.add(Bytes.toBytes("colfam1"), Bytes.toBytes("qual1.1"), System.currentTimeMillis(), Bytes.toBytes("val1.1"));
put.add(Bytes.toBytes("colfam1"), Bytes.toBytes("qual1.2"), System.currentTimeMillis(), Bytes.toBytes("val1.2"));
put.add(Bytes.toBytes("colfam2"), Bytes.toBytes("qual2.1"), System.currentTimeMillis(), Bytes.toBytes("val2.1"));
Result converted = HTableUtil.convert(put);
table.put(put);
Result readFromTable = table.get(new Get(key));
Assert.assertArrayEquals(readFromTable.raw(), converted.raw());
//Get the value from "myTable" with the key
//Verify the returned value is equal to the value you created
//Stop the mini cluster
testingUtility.shutdownMiniCluster();
System.out.println("END OF TEST");
}
public static void setHTableUtil(ScriptUtils HTableUtil) {
TestCreateTableClass.HTableUtil = HTableUtil;
}
}
However, I got the following error:
1. The error at this line of code with the function put.add()
put.add(Bytes.toBytes("colfam1"), Bytes.toBytes("qual1.1"), System.currentTimeMillis(), Bytes.toBytes("val1.1"));
The 2nd error on this line of code:
Result converted = HTableUtil.convert(put);
Java cannot find symbol for these 3 methods put(), get(), raw()
table.put(put);
Result readFromTable = table.get(new Get(key));
Assert.assertArrayEquals(readFromTable.raw(), converted.raw());
I also notice some warnings regarding the class HTableDescriptor, HColumnDescriptor have been deprecated. I checked on internet and they advice to use for example "TableDescriptorBuilder" instead but I am not sure how to use it. (Ref: https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java)

1. The error at this line of code with the function put.add().
I think you can use addColumn() like this for adding column.
put.addColumn(Bytes.toBytes("colfam1"), Bytes.toBytes("qual1.1"), System.currentTimeMillis(), Bytes.toBytes("val1.1"));
put.addColumn(Bytes.toBytes("colfam1"), Bytes.toBytes("qual1.2"), System.currentTimeMillis(), Bytes.toBytes("val1.2"));
put.addColumn(Bytes.toBytes("colfam2"), Bytes.toBytes("qual2.1"), System.currentTimeMillis(), Bytes.toBytes("val2.1"));
2. The 2nd error on this line of code:
I'm not familiar with 'ScriptUtils', But I think It works.
Result converted = (Result) HTableUtil.convert(put, Result.class);
3. Java cannot find symbol for these 3 methods put(), get(), raw()
It because you keep using 'HTableDescriptor' to put(), get(), or raw(). 'HTableDescriptor' is used to create table like DDL. You need to use Table class to manipulate using put(), get(), or raw().
Table createdTable = connection.getTable(TableName.valueOf(tableName));
createdTable.put(put);
Result readFromTable = createdTable.get(new Get(key));
Also, I believe class 'Result' doesn't provide raw(). So, you can compare both Results using Result.compareResults() like this.
Result.compareResults(readFromTable, converted);
4. How to use 'TableDescriptorBuilder'
Like I said above, 'Descriptor' is the class for defining your table, column family, column, and so on. So, you need to use it when you make/create them.
//Define table "myTable"
TableDescriptorBuilder table = TableDescriptorBuilder.newBuilder(TableName.valueOf(tableName));
table.setColumnFamily(ColumnFamilyDescriptorBuilder.newBuilder(Bytes.toBytes("cf1")).setCompressionType(Compression.Algorithm.NONE).build());
//Create table "myTable"
connection.getAdmin().createTable(table.build());

Java insert value to Array in MongoDB

im new to MongoDB and completely confused by the queries. I simply need to update a document in a mongodb database by adding a string value (example: Temperature) to a list of strings. From research I know that I have to use the $push method for that. I think the code has to look somehow like this:
BasicDBObject newDocument = new BasicDBObject().append("$set",
new BasicDBObject().append("Subscribed Topics", topic));
collection.update(new BasicDBObject().append("Sensor Type", sensorType), newDocument);
new BasicDBObject("$push",
new BasicDBObject("Subscribed Topics", topic));
The field with the array is called "Subscribed Topics", "topic" is a String (Temperature). Then I want to update the document in the collection with the corresponding "Sensor Type". However, I do not really know how to call the $push part correctly. I hope someone can help me sort this part of the code.
Best regards.
Update, I tried to implemented as suggested in the duplicate question but still got error. Very unsure if thats the right way anyway.
DBObject listItem = new BasicDBObject("Subscribed Topics", "Light");
DBObject updateQuery = new BasicDBObject("$push", listItem);
collection.update(query, updateQuery);`
I create a new Object with the value Light in for Key Subscribed Topics (the array). Why do I push it to a new Object then?

My goodness! This question got me descending into the long forgotten world of Java again - after all these years... ;) Anyhoo, here's a complete working example that might give you a clue of what's going on. You can run the code several times and see how the number of elements in the "Subscribed Topics" array increases.
I used the following driver: https://oss.sonatype.org/content/repositories/releases/org/mongodb/mongo-java-driver/3.3.0/mongo-java-driver-3.3.0.jar
import com.mongodb.MongoClient;
import com.mongodb.client.MongoDatabase;
import com.mongodb.client.MongoCollection;
import org.bson.Document;
import org.bson.conversions.Bson;
import static com.mongodb.client.model.Filters.*;
import static com.mongodb.client.model.Updates.*;
public class MongoDbPush {
public static void main(String[] args)
{
MongoClient mongoClient = new MongoClient();
MongoDatabase database = mongoClient.getDatabase("pushExampleDb");
MongoCollection<Document> collection = database.getCollection("pushExampleCollection");
String sensorType = "Temperature";
// try to load existing document from MongoDB
Document document = collection.find(eq("Sensor Type", sensorType)).first();
if(document == null)
{
// no test document, let's create one!
document = new Document("Sensor Type", sensorType);
// insert it into MongoDB
collection.insertOne(document);
// read it back from MongoDB
document = collection.find(eq("Sensor Type", sensorType)).first();
}
// see what it looks like in JSON (on the first run you will notice that it has got an "_id" but no "Subscribed Topics" array yet)
System.out.println(document.toJson());
// update the document by adding an entry to the "Subscribed Topics" array
Bson filter = eq("Sensor Type", sensorType);
Bson change = push("Subscribed Topics", "Some Topic");
collection.updateOne(filter, change);
// read one more time from MongoDB
document = collection.find(eq("Sensor Type", sensorType)).first();
// see what the document looks like in JSON (on the first run you will notice that the "Subscribed Topics" array has been created and has got one element in it)
System.out.println(document.toJson());
mongoClient.close();
}
}

The above method still works, however, with updated Mongo Driver the below is also a viable mechanism.
The below works for Mongo Driver 3.6 onward (in this case using 3.12.4)
MongoClient mongoClient = new MongoClient();
MongoDatabase database = mongoClient.getDatabase("pushExampleDb");
MongoCollection<Document> collection = database.getCollection("pushExampleCollection");
collection.findOneAndUpdate(Filters.eq("Sensor Type",<theSensorTypeNameComesHere>),
Updates.pushEach("Subscribed Topics",<listContainingTheValuesComeHere>));
Refer: $push and $each from MongoDB Manual

JDO Query with cursors on GAE returns inconsistent results

I am running GWT RPC calls to a GAE server, querying for article objects that are stored in the datastore with JDO, and I am paginating the results by using cursors.
I send an initial RPC call to start the pagination with a "range" of 10 results. I store the query cursor in the memcache, and retrieve it when the user requests for the next page of 10 results. The code that implements this is shown below.
The range is always the same, 10 results. However, some subsequent RPC calls return 2 results, or 12 results. It is very inconsistent. The calls also sometimes return duplicate results.
I have read this Google developers documentation: https://developers.google.com/appengine/docs/java/datastore/queries#Java_Limitations_of_cursors. It mentions that: "Cursors don't always work as expected with a query that uses an inequality filter or a sort order on a property with multiple values. The de-duplication logic for such multiple-valued properties does not persist between retrievals, possibly causing the same result to be returned more than once."
As you can see in the code, I am sorting on a "date" property. This property only has one value.
Can you let me see what I am doing wrong here. Thanks.
This is the code that executes the RPC call on the GAE server:
import java.util.ArrayList;
import java.util.HashMap;
import java.util.Iterator;
import java.util.Map;
import java.util.List;
import java.util.logging.Logger;
import javax.jdo.PersistenceManager;
import javax.jdo.Query;
import javax.servlet.http.HttpSession;
import com.google.appengine.api.datastore.Cursor;
import com.google.appengine.datanucleus.query.JDOCursorHelper;
import com.google.gwt.user.server.rpc.RemoteServiceServlet;
//...
private void getTagArticles(String tag, int range, boolean start) {
PersistenceManager pm = PMF.getNonTxnPm();
ArticleStreamItemSummaryDTO aDTO = null;
ArticleStreamItem aDetached = null;
summaryList = new ArrayList<ArticleStreamItemSummaryDTO>();
String cursorString = null;
session = getThreadLocalRequest().getSession();
UserAccount currentUser = LoginHelper.getLoggedInUser(session, pm);
String cursorID = currentUser.getId().toString() + tag;
if (start) { // The start or restart of the query
CacheSupport.cacheDelete(String.class.getName(), cursorID);
}
Object o = CacheSupport.cacheGet(String.class.getName(), cursorID);
if (o != null && o instanceof String) {
cursorString = (String) o;
}
Query q = null;
try {
q = pm.newQuery(ArticleStreamItem.class);
if (cursorString != null) {
Cursor cursor = Cursor.fromWebSafeString(cursorString);
Map<String, Object> extensionMap = new HashMap<String, Object>();
extensionMap.put(JDOCursorHelper.CURSOR_EXTENSION, cursor);
q.setExtensions(extensionMap);
}
q.setFilter("tag == tagParam");
q.declareParameters("String tagParam");
q.setOrdering("date desc");
q.setRange(0, range);
#SuppressWarnings("unchecked")
List<ArticleStreamItem> articleStreamList = (List<ArticleStreamItem>) q.execute(tag);
if (articleStreamList.iterator().hasNext()) {
Cursor cursor = JDOCursorHelper.getCursor(articleStreamList);
cursorString = cursor.toWebSafeString();
CacheSupport.cacheDelete(String.class.getName(), cursorID);
CacheSupport.cachePutExp(String.class.getName(), cursorID, cursorString, CACHE_EXPIR);
for (ArticleStreamItem a : articleStreamList) {
aDetached = pm.detachCopy(a);
aDTO = aDetached.buildSummaryItem();
summaryList.add(aDTO);
}
}
}
catch (Exception e) {
// e.printStackTrace();
logger.warning(e.getMessage());
}
finally {
q.closeAll();
pm.close();
}
}

The code snippet that I provided in the question above actually works well. The problem arose from the client side. The RPC calls were sometimes being made a few milliseconds from each other, and that was creating the inconsistent behavior that I was seeing in the results returned.
I changed the client side code to make a single RPC call every 5 seconds, and that fixed it.

insert text data values eg(date time PH value) in Mongodb every 3 hours automatically using java

I have a mongodb database, I need to insert text data values eg(date time PH value) every 3 hours automatically using java.
Need help
I have made a mongodb database called project and collection called Water Monetering system
also here is the basic layout of java- mongodb integration
import com.mongodb.MongoClient;
import com.mongodb.MongoException;
import com.mongodb.WriteConcern;
import com.mongodb.DB;
import com.mongodb.DBCollection;
import com.mongodb.BasicDBObject;
import com.mongodb.DBObject;
import com.mongodb.DBCursor;
import com.mongodb.ServerAddress;
import java.util.Arrays;
public class MongoDBJDBC{
public static void main( String args[] ){
try{
// To connect to mongodb server
MongoClient mongoClient = new MongoClient( "localhost" , 27017 );
// Now connect to your databases
DB db = mongoClient.getDB( "project" );
System.out.println("Connect to database successfully");
DBCollection coll = db.getCollection("WaterMoneteringSystem");
System.out.println("Collection WaterMoneteringSystem selected successfully");
BasicDBObject doc = new BasicDBObject("title", "Watermoneteringsystem").
append("Date", "date").
append("time", "time").
append("value", "ph").
coll.insert(doc);
System.out.println("Document inserted successfully");
}catch(Exception e){
System.err.println( e.getClass().getName() + ": " + e.getMessage() );
}
}
}
I am not getting how data(date time PH) in text box can be implemented
Thank you

Instead of:
BasicDBObject doc = new BasicDBObject("title", "Watermoneteringsystem").
append("Date", "date").
append("time", "time").
append("value", "ph");
You need something like:
BasicDBObject doc = new BasicDBObject("title", "Watermoneteringsystem").
append("Date", new Date()).
append("value", "ph");
The new Date() call will set today's date and time in there (you don't need separate fields for date and time).
But the "ph" bit needs to come from some other place - where are you getting the ph values from? Is it going to be an argument (i.e. in args[])? Is it coming from a file, or external system?
Assuming it's something you can pass in to the method, you can do something like:
String phValue = args[0];
BasicDBObject doc = new BasicDBObject("title", "Watermoneteringsystem").
append("Date", new Date()).
append("value", phValue);
I'd suggest renaming the field to something like "ph" or "phValue" as well, since "value" is not a helpful field name.
Take a look at the MongoDB and Java documentation, there are more examples there in how to use MongoDB from Java.

not able to insert documents into collection using mongodb java driver

I'm using mongo mongo java driver version 2.11.2. I want to insert some few documents into my dbin mongodb and when I try to do it from command line it all works fine. But when I use mongo java driver, it is not working. I'm using BasicDBObject to populate the document. But collection.insert(BasicDBObject).getN() gives me 0 always. Nothing is getting inserted into the collection. Am I missing something here?
Adding the code:
mongo = new MongoClient("localhost", 27017);
DB db = mongo.getDB("db");
DBCollection collection = db.getCollection("collection");
BasicDBObject o = new BasicDBObject();
o.put("key1", "value1");
o.put("key2", "value2");
collection.insert(o);
No update is made in DB after this.

The 'n' value from the getlasterror of an insert is always zero. (The 'n' value is what the WriteResult.getN() returns.)
See this MongoDB Jira ticket: https://jira.mongodb.org/browse/SERVER-4381. Which has been closed in preference to a new insert, update, remove mechanism: https://jira.mongodb.org/browse/SERVER-9038
Long story short. You are not mad or missing anything. It is a "feature" of MongoDB that will hopefully finally be fixed with the 2.6 release.
Rob.
Edit:
I modified your example slightly to print the saved document. Can you try running this version in your environment?
import java.net.UnknownHostException;
import com.mongodb.BasicDBObject;
import com.mongodb.DB;
import com.mongodb.DBCollection;
import com.mongodb.DBObject;
import com.mongodb.MongoClient;
import com.mongodb.WriteConcern;
public class StackOverFlow {
public static void main(String[] args) throws UnknownHostException {
MongoClient mongo = new MongoClient("localhost:27017");
DB db = mongo.getDB("db");
DBCollection collection = db.getCollection("collection");
BasicDBObject o = new BasicDBObject();
o.put("key1", "value1");
o.put("key2", "value2");
collection.insert(WriteConcern.SAFE, o);
for (DBObject doc : collection.find()) {
System.out.println(doc);
}
}
}
On my machine it outputs:
{ "_id" : { "$oid" : "5235f98495302901eb70e7a4"} , "key1" : "value1" , "key2" : "value2"}