Develop Reference

Oracle STRUCT use OBJECT TYPE with CLOB attribute

I am trying to use STRUCT for the insertion of massive data in my table DATA_TABLE, but generate error (java.sql.SQLException: Fail to convert to internal representation) with data type field CLOB and I can't find a solution to my problem, my code:
My table
CREATE TABLE DATA_TABLE (DAT_ID NUMBER,
DAT_CODE VARCHAR2(10),
DAT_TEXT CLOB);
Create type object
CREATE OR REPLACE TYPE TY_OBJ_DATA AS OBJECT (DAT_ID NUMBER,
DAT_CODE VARCHAR2(10),
DAT_TEXT CLOB);
create type table from type object
CREATE OR REPLACE TYPE TY_TABLE_DATA AS TABLE OF SCHEMA.TY_OBJ_DATA;
My simplified java method
public static void bulkData(List<DataTable> listDataInfo) throws Exception {
DataSource ds = (DataSource) getEntityManager().getEntityManagerFactory().getProperties().get("javax.persistence.jtaDataSource");
OracleConnection connection = ds.getConnection().unwrap(OracleConnection.class);
try{
StructDescriptor typeTableObject = StructDescriptor.createDescriptor("SCHEMA.TY_OBJ_DATA", conect);
STRUCT[] structData = new STRUCT[DataTable.size()];
int counter= 0;
for (DataTable d : listDataInfo) {
Clob clob = connection.createClob();
STRUCT m = new STRUCT(typeTableObject, connection,
new Object[]{d.getDatId(),
d.getDatCode,
clob.setString(1, d.getDatText)});
structData [counter++] = m;
}
ArrayDescriptor tyTable = ArrayDescriptor.createDescriptor("SCHEMA.TY_TABLE_DATA", connection);
ARRAY array = new ARRAY(tyTable, connection, structData);
String sqlQuery = "{ CALL PACKAGE_BULK.PL_BULK_DATA }";
CallableStatement cst = conect.prepareCall(sqlQuery);
cst.setArray(1, array );
cst.execute();
} catch (Exception e) {
throw new Exception(e);
} finally {
try {
connection.close();
} catch (SQLException e) {
throw new Exception(e);
}
}
}
I omit the package code, because it is working correctly and is not my main problem. i Use ojdbc6 version 11.2.0, java 8 and Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit.
Is it possible to use STRUCT with fields of type CLOB? I am doing something wrong? my DatText field when it is remapped is of the String type and that was the best conversion that I managed to do from String to a clob but I still have problems, any idea how I can solve this? Thank you.

I know this is very old, but as I just came by:
I solved this issue by simply skipping the descriptors and using:
cst.setArray(i, ((OracleConnection) connection).createOracleArray("TY_TABLE_DATA", shiftArrayOneUp(new Object[]{d.getDatId(),
d.getDatCode(),
d.getDatText()}));
shiftArrayOneUp is just adding one empty value at the beginning (oracle array starts at 1, not at 0)
private Object[] shiftArrayOneUp(Object[] values) {
Object[] result = new Object[values.length + 1];
System.arraycopy(values, 0, result, 1, values.length);
return result;
}
As I had to adjust my code accordingly, I did not test what I was writing here.
Maybe some small adjustments need to be done

MongoDB - find() calls getting stuck sometimes - Timeout

We are using MongoDb for saving and fetching data.
All calls that are putting data into collections are working fine and are through common method.
All calls that are fetching data from collections are working fine sometimes and are through common method.
But Sometimes, only for one of the collection, i get my calls being stuck for forever, consuming CPU usage. I have to manually kill the threads otherwise it consumes my whole CPU.
Mongo Connection
MongoClient mongo = new MongoClient(hostName , Integer.valueOf(port));
DB mongoDb = mongo.getDB(dbName);
Code To fetch
DBCollection collection = mongoDb.getCollection(collectionName);
DBObject dbObject = new BasicDBObject("_id" , key);
DBCursor cursor = collection.find(dbObject);
Though i have figured out the collection for which it is causing issues, but how can i improve upon this, since it is occurring for this particular collection and sometimes.
EDIT
Code to save
DBCollection collection = mongoDb.getCollection(collectionName);
DBObject query = new BasicDBObject("_id" , key);
DBObject update = new BasicDBObject();
update.put("$set" , JSON.parse(value));
collection.update(query , update , true , false);
Bulk Write / collection
DB mongoDb = controllerFactory.getMongoDB();
DBCollection collection = mongoDb.getCollection(collectionName);
BulkWriteOperation bulkWriteOperation = collection.initializeUnorderedBulkOperation();
Map<String, Object> dataMap = (Map<String, Object>) JSON.parse(value);
for (Entry<String, Object> entrySet : dataMap.entrySet()) {
BulkWriteRequestBuilder bulkWriteRequestBuilder = bulkWriteOperation.find(new BasicDBObject("_id" ,
entrySet.getKey()));
DBObject update = new BasicDBObject();
update.put("$set" , entrySet.getValue());
bulkWriteRequestBuilder.upsert().update(update);
}
How can i set timeout for fetch calls..??

A different approach is to use the proposed method for MongoDB 3.2 Driver. Keep in mind that you have to update your .jar libraries (if you haven't) to the latest version.
public final MongoClient connectToClient(String hostName, String port) {
try {
MongoClient client = new MongoClient(hostName, Integer.valueOf(port));
return client;
} catch(MongoClientException e) {
System.err.println("Cannot connect to Client.");
return null;
}
}
public final MongoDatabase connectToDB(String databaseName) {
try {
MongoDatabase db = client.getDatabase(databaseName);
return db;
} catch(Exception e) {
System.err.println("Error in connecting to database " + databaseName);
return null;
}
public final void closeConnection(MongoClient client) {
client.close();
}
public final void findDoc(MongoDatabase db, String collectionName) {
MongoCollection<Document> collection = db.getCollection(collectionName);
try {
FindIterable<Document> iterable = collection
.find(new Document("_id", key));
Document doc = iterable.first();
//For an Int64 field named 'special_id'
long specialId = doc.getLong("special_id");
} catch(MongoException e) {
System.err.println("Error in retrieving document.");
} catch(NullPointerException e) {
System.err.println("Document with _id " + key + " does not exist.");
}
}
public final void insertToDB(MongoDatabase db, String collectioName) {
try {
db.getCollection(collectionName).insertOne(new Document()
.append("special_id", 5)
//Append anything
);
catch(MongoException e) {
System.err.println("Error in inserting new document.");
}
}
public final void updateDoc(MongoDatabase db, String collectionName, long id) {
MongoCollection<Document> collection = db.getCollection(collectionName);
try {
collection.updateOne(new Document("_id", id),
new Document("$set",
new Document("special_id",
7)));
catch(MongoException e) {
System.err.println("Error in updating new document.");
}
}
public static void main(String[] args) {
String hostName = "myHost";
String port = "myPort";
String databaseName = "myDB";
String collectionName = "myCollection";
MongoClient client = connectToClient(hostName, port);
if(client != null) {
MongoDatabase db = connectToDB(databaseName);
if(db != null) {
findDoc(db, collectionName);
}
client.closeConnection();
}
}
EDIT: As the others suggested, check from the command line if the procedure of finding the document by its ID is slow too. Then maybe this is a problem with your hard drive. The _id is supposed to be indexed but for better or for worse, re-create the index on the _id field.

The answers posted by others are great, but did not solve my purpose.
Actually issue was in my existing code itself , my cursor was waiting in while loop infinite time.
I was missing few checks which has been resolved now.

Just some possible explanations/thoughts.
In general "query by id" has to be fast since _id is supposed to be indexed, always. The code snippet looks correct, so probably the reason is in mongo itself. This leads me to a couple of suggestions:
Try to connect to mongo directly from the command line and run the "find" from there. The chances are that you'll still be able to observe occasional slowness.
In this case:
Maybe its about the disks (maybe this particular server is deployed on the slow disk or at least there is a correlation with some slowness of accessing the disk).
Maybe your have a sharded configuration and one shard is slower than others
Maybe its a network issue that occurs sporadically. If you run mongo locally/on staging env. with the same collection does this reproduce?
Maybe (Although I hardly believe that) the query runs in sub un-optimal way. In this case you can use "explain()" as someone has already suggested here.
If you happen to have replica set, please figure out what is the [Read Preference]. Who knows, maybe you prefer to get this id from the sub-optimal server

How to mass delete multiple rows in hbase?

I have the following rows with these keys in hbase table "mytable"
user_1
user_2
user_3
...
user_9999999
I want to use the Hbase shell to delete rows from:
user_500 to user_900
I know there is no way to delete, but is there a way I could use the "BulkDeleteProcessor" to do this?
I see here:
https://github.com/apache/hbase/blob/master/hbase-examples/src/test/java/org/apache/hadoop/hbase/coprocessor/example/TestBulkDeleteProtocol.java
I want to just paste in imports and then paste this into the shell, but have no idea how to go about this. Does anyone know how I can use this endpoint from the jruby hbase shell?
Table ht = TEST_UTIL.getConnection().getTable("my_table");
long noOfDeletedRows = 0L;
Batch.Call<BulkDeleteService, BulkDeleteResponse> callable =
new Batch.Call<BulkDeleteService, BulkDeleteResponse>() {
ServerRpcController controller = new ServerRpcController();
BlockingRpcCallback<BulkDeleteResponse> rpcCallback =
new BlockingRpcCallback<BulkDeleteResponse>();
public BulkDeleteResponse call(BulkDeleteService service) throws IOException {
Builder builder = BulkDeleteRequest.newBuilder();
builder.setScan(ProtobufUtil.toScan(scan));
builder.setDeleteType(deleteType);
builder.setRowBatchSize(rowBatchSize);
if (timeStamp != null) {
builder.setTimestamp(timeStamp);
}
service.delete(controller, builder.build(), rpcCallback);
return rpcCallback.get();
}
};
Map<byte[], BulkDeleteResponse> result = ht.coprocessorService(BulkDeleteService.class, scan
.getStartRow(), scan.getStopRow(), callable);
for (BulkDeleteResponse response : result.values()) {
noOfDeletedRows += response.getRowsDeleted();
}
ht.close();
If there exists no way to do this through JRuby, Java or alternate way to quickly delete multiple rows is fine.

Do you really want to do it in shell because there are various other better ways. One way is using the native java API
Construct an array list of deletes
pass this array list to Table.delete method
Method 1: if you already know the range of keys.
public void massDelete(byte[] tableName) throws IOException {
HTable table=(HTable)hbasePool.getTable(tableName);
String tablePrefix = "user_";
int startRange = 500;
int endRange = 999;
List<Delete> listOfBatchDelete = new ArrayList<Delete>();
for(int i=startRange;i<=endRange;i++){
String key = tablePrefix+i;
Delete d=new Delete(Bytes.toBytes(key));
listOfBatchDelete.add(d);
}
try {
table.delete(listOfBatchDelete);
} finally {
if (hbasePool != null && table != null) {
hbasePool.putTable(table);
}
}
}
Method 2: If you want to do a batch delete on the basis of a scan result.
public bulkDelete(final HTable table) throws IOException {
Scan s=new Scan();
List<Delete> listOfBatchDelete = new ArrayList<Delete>();
//add your filters to the scanner
s.addFilter();
ResultScanner scanner=table.getScanner(s);
for (Result rr : scanner) {
Delete d=new Delete(rr.getRow());
listOfBatchDelete.add(d);
}
try {
table.delete(listOfBatchDelete);
} catch (Exception e) {
LOGGER.log(e);
}
}
Now coming down to using a CoProcessor. only one advice, 'DON'T USE CoProcessor' unless you are an expert in HBase.
CoProcessors have many inbuilt issues if you need I can provide a detailed description to you.
Secondly when you delete anything from HBase it's never directly deleted from Hbase there is tombstone marker get attached to that record and later during a major compaction it gets deleted, so no need to use a coprocessor which is highly resource exhaustive.
Modified code to support batch operation.
int batchSize = 50;
int batchCounter=0;
for(int i=startRange;i<=endRange;i++){
String key = tablePrefix+i;
Delete d=new Delete(Bytes.toBytes(key));
listOfBatchDelete.add(d);
batchCounter++;
if(batchCounter==batchSize){
try {
table.delete(listOfBatchDelete);
listOfBatchDelete.clear();
batchCounter=0;
}
}}
Creating HBase conf and getting table instance.
Configuration hConf = HBaseConfiguration.create(conf);
hConf.set("hbase.zookeeper.quorum", "Zookeeper IP");
hConf.set("hbase.zookeeper.property.clientPort", ZookeeperPort);
HTable hTable = new HTable(hConf, tableName);

If you already aware of the rowkeys of the records that you want to delete from HBase table then you can use the following approach
1.First create a List objects with these rowkeys
for (int rowKey = 1; rowKey <= 10; rowKey++) {
deleteList.add(new Delete(Bytes.toBytes(rowKey + "")));
}
2.Then get the Table object by using HBase Connection
Table table = connection.getTable(TableName.valueOf(tableName));
3.Once you have table object call delete() by passing the list
table.delete(deleteList);
The complete code will look like below
Configuration config = HBaseConfiguration.create();
config.addResource(new Path("/etc/hbase/conf/hbase-site.xml"));
config.addResource(new Path("/etc/hadoop/conf/core-site.xml"));
String tableName = "users";
Connection connection = ConnectionFactory.createConnection(config);
Table table = connection.getTable(TableName.valueOf(tableName));
List<Delete> deleteList = new ArrayList<Delete>();
for (int rowKey = 500; rowKey <= 900; rowKey++) {
deleteList.add(new Delete(Bytes.toBytes("user_" + rowKey)));
}
table.delete(deleteList);

How to Update an Existing Record in a Dataset

I can't seem to update an existing record in my table using a strongly-typed dataset. I can add a new record, but if I make changes to an existing record it doesn't work.
Here is my code:
private void AddEmplMaster()
{
dsEmplMast dsEmpMst = new dsEmplMast();
SqlConnection cn = new SqlConnection();
cn.ConnectionString = System.Configuration.ConfigurationSettings.AppSettings["cn.ConnectionString"];
SqlDataAdapter da1 = new SqlDataAdapter("SELECT * FROM UPR00100", cn);
SqlCommandBuilder cb1 = new SqlCommandBuilder(da1);
da1.Fill(dsEmpMst.UPR00100);
DataTable dtMst = UpdateEmpMst(dsEmpMst);
da1.Update(dsEmpMst.UPR00100);
}
This procedure is called from above to assign the changed fields to a record:
private DataTable UpdateEmpMst(dsEmplMast dsEmpMst)
{
DataTable dtMst = new DataTable();
try
{
dsEmplMast.UPR00100Row empRow = dsEmpMst.UPR00100.NewUPR00100Row();
empRow.EMPLOYID = txtEmplId.Text.Trim();
empRow.LASTNAME = txtLastName.Text.Trim();
empRow.FRSTNAME = txtFirstName.Text.Trim();
empRow.MIDLNAME = txtMidName.Text.Trim();
empRow.ADRSCODE = "PRIMARY";
empRow.SOCSCNUM = txtSSN.Text.Trim();
empRow.DEPRTMNT = ddlDept.SelectedValue.Trim();
empRow.JOBTITLE = txtJobTitle.Text.Trim();
empRow.STRTDATE = DateTime.Today;
empRow.EMPLOYMENTTYPE = "1";
dsEmpMst.UPR00100.Rows.Add(empRow);
}
catch { }
return dtMst;
}
Thank you
UPDATE:
Ok I figured it out. In my UpdateEmpMst() procedure I had to check if the record exists then to retrieve it first. If not then create a new record to add. Here is what I added:
try
{
dsEmplMast.UPR00100Row empRow;
empRow = dsEmpMst.UPR00100.FindByEMPLOYID(txtEmplId.Text.Trim());
if (empRow == null)
{
empRow = dsEmpMst.UPR00100.NewUPR00100Row();
dsEmpMst.UPR00100.Rows.Add(empRow);
}
then I assign my data to the new empRow I created and updates fine.

In order to edit an existing record in a dataset, you need to access a particular column of data in a particular row. The data in both typed and untyped datasets can be accessed via the following:
With the indices of the tables, rows, and columns collections.
By passing the table and column names as strings to their respective collections.
Although typed datasets can use the same syntax as untyped datasets, there are additional advantages to using typed datasets. For more information, see the "To update existing records using typed datasets" section below.
To update existing records in either typed or untyped datasets
Assign a value to a specific column within a DataRow object.
The table and column names of untyped datasets are not available at design time and must be accessed through their respective indices.

HBase setting timestamp

I'm having problems setting row timestamp using java api.
When I'm trying to add a timestamp value to put constructor (or into put.add()) nothing happens and after reading rows from table I get system provided timestamps.
public static boolean addRecord(String tableName, String rowKey,
String family, String qualifier, Object value)
{
try {
HTable table = new HTable(conf, tableName);
Put put = new Put(Bytes.toBytes(rowKey), 12345678l);
put.add(Bytes.toBytes(family), Bytes.toBytes(qualifier), Bytes.toBytes(value.toString()));
table.put(put);
return true;
} catch (Exception e) {
e.printStackTrace();
return false;
}
}
HBase 0.92.1 running in standalone mode.
Thanks in advance for any help!

Most likely, you already have rows in the table that have timestamp > 12345678l. To confirm that this is not the case, try it with a very large value for timestamp, say Long.MAX_VALUE.
If it is indeed the case, you can simply delete the older versions. Then this entry will show up.