Develop Reference

Spark Dataset - How to create a new column by modifying an existing column value

I have a Dataset like below
Dataset<Row> dataset = ...
dataset.show()
| NAME | DOB |
+------+----------+
| John | 19801012 |
| Mark | 19760502 |
| Mick | 19911208 |
I want to convert it to below (formatted DOB)
| NAME | DOB |
+------+------------+
| John | 1980-10-12 |
| Mark | 1976-05-02 |
| Mick | 1991-12-08 |
How can I do this? Basically, I am trying to figure out how to manipulate existing column string values in a generic way.
I tried using dataset.withColumn but couldn't quite figure out how to achieve this.
Appreciate any help.

With "substring" and "concat" functions:
df.withColumn("DOB_FORMATED",
concat(substring($"DOB", 0, 4), lit("-"), substring($"DOB", 5, 2), lit("-"), substring($"DOB", 7, 2)))

Load the data into a dataframe(deltaData) and just use the following line
deltaData.withColumn("DOB", date_format(to_date($"DOB", "yyyyMMdd"), "yyyy-MM-dd")).show()

Assuming DOB is a String you could write a UDF
def formatDate(s: String): String {
// date formatting code
}
val formatDateUdf = udf(formatDate(_: String))
ds.select($"NAME", formatDateUdf($"DOB").as("DOB"))

JAVA - Storing result set in hash table by grouping data efficiently

I'd like to store in a hash table a result set coming from a query execution.
The hash table is something like this
Map<List<String>,List<Object>>
where
List<String>, the hash table key, is a subset of the extracted fields
Object is a Java object corresponding to a database tuple (all fields)
So, first, data have to be grouped in order to create each key and group all the items sharing this key.
The pseudo-code related to my current approach is:
while(iterate){
while(rs.next){
if(key is empty)
// build REFERENCE KEY and delete rs entry
else
// build key for i-th rs entry and compare it with the REFERENCE key.
Eventually, get data and delete rs entry
}
rs.beforeFirst()
}
In other words, the result set is iterated many times and each time a new key is created, in order to compare the ramaining result set entries with it. Each time the processed entry is deleted to exit the outer loop.
Since the result set is very large (and also each List(Object) ), performance are poor (a very high loading time per key).
Appending an order by clause to the query (in order to preliminarily group data) doesn't alleviate the problem.
Is there a more efficient approach?
Thanks everyone.
EDIT
Input ResultSet
---------------------------------------------------------------
| Field1 | Field2 | Field3 | Field4 | Field5 | Field6 | Field7 |
---------------------------------------------------------------
| X | A | val1_3 | val1_4 | val1_5 | val1_6 | val1_7 |
| X | A | val2_3 | val2_4 | val2_5 | val2_6 | val2_7 |
| Y | B | val3_3 | val3_4 | val3_5 | val3_6 | val3_7 |
| Z | C | val4_3 | val4_4 | val4_5 | val4_6 | val4_7 |
| Y | D | val5_3 | val5_4 | val5_5 | val5_6 | val5_7 |
----------------------------------------------------------------
Key_Fields : [Field1, Field2]
Output Map
-----------------------------------
| KEY | VALUE |
-----------------------------------
| [X,A] | [Object1, Object2] |
| [Y,B] | [Object3] |
| [Z,C] | [Object4] |
| [Y,D] | [Object5] |
-----------------------------------
I'm using List<String> for key because another ResultSet can have a Key_Fields of different lenght.
Here, my current time-consuming Java code
while(itera){
key = new ArrayList<String>();
values = new ArrayList<AbstractClass>();
while(rs.next()){
if(key.isEmpty()){
// build REFERENCE KEY
// add first OBJECT to List<AbstractClass>
// delete this data from ResultSet
}
else{
// Build KEY_TO_BE_COMPARED
List<String> row_to_be_compared = new ArrayList<String>();
// If this key equals to REFERENCE KEY
if(row_to_be_compared.equals(key)){
AbstractClass value_object = new AbstractClass();
...
rs.deleteRow();
}
// ORDERBY clause in query ensures that, if keys don't match, then all objects related to REFERENCE KEY have been collected
else{
break;
}
}
}
rs.beforeFirst();
map.put(key, values);
if(!rs.next() || items_loaded==max_hash_size)
itera = false;
else
rs.beforeFirst();
}
}

Instead of using List as key. Use a class having List as its instance variable. Override equals very carefully.

Why don't you simplify your key and make it a String that contains all the concatenated fields concatenated by a special character (say .)?

Using cassandra triggers

I'm have a cassandra table like this:
keyspace_name| columnfamily_name | column_name | component_index |
-------------+-------------------+-------------+-----------------+
aw | test | as_of_date | 0 |
aw | test | data | 1 |
aw | test | record_id | null |
aw | test | upload_time | 1 |
And I'm won't to create trigger that will print(slf4j for example) rows that whould be inserted in next format:
key = key1
column_name1=value1
column_name2=value2
...
column_namen=valuen
Is it possible to get column name in trigger?
I try example from the internet, but it prints incorrect data.
public Collection<RowMutation> augment(ByteBuffer key, ColumnFamily update) {
String localKey = new String(key.array(), Charset.forName("UTF-8"));
logger.info("key={}.", localKey);
for (Column cell : update) {
try {
String name = ByteBufferUtil.string(cell.name());
logger.info("name={}.", name);
String value = ByteBufferUtil.string(cell.value());
logger.info("value={}.", value);
} catch (Exception e) {
logger.info("Exception={}.", e.getMessage());
}
}
As I understand, i am must convert cell.value() to specific data type like this:
Date date = TimestampType.instance.compose(cell.value());
But I don't know, how to detect field type and i am don't understand why i can't get column name using ByteBufferUtil.string(cell.name()).

To properly format cellname and values you must use the CFMetaData. The correct version of the code should be:
public Collection<Mutation> augment(ByteBuffer key, ColumnFamily update)
{
CFMetaData cfm = update.metadata();
String localKey = cfm.getKeyValidator().getString(key);
logger.info("key={}.", localKey);
for (Cell cell : update)
{
try
{
String name = cfm.comparator.getString(cell.name());
logger.info("name={}.", name);
String value = cfm.getValueValidator(cell.name()).getString(cell.value());
logger.info("value={}.", value);
} catch (Exception e) {
logger.info("Exception={}.", e.getMessage());
}
}
return Collections.emptyList();
}

JTable getting values of specific column

How could I get the values under specific column in JTable
example :
_________________________
| Column 1 | Column 2 |
________________________
| 1 | a |
________________________
| 2 | b |
________________________
| 3 | c |
_________________________
How could I get the values under Column 1 that is [1, 2, 3]
In the form of some data structure ( preferable array)?

you can do something like this
ArrayList list = new ArrayList();
for(int i = 0;i<table.getModel().getRowCount();i++)
{
list.add(table.getModel().getValueAt(i,0)); //get the all row values at column index 0
}

List the child nodes in tree when parent expanded

I have a problem in implementing the tree structure of OID. when I click the parent , i need to display only child details, not the sub child of a child.
i.e., i need not display an OID which contains a "." (dot).
For example, if my OID structure is private.MIB.sample.first
private.MIB.sample.second and so on.
when I click on MIB, it should display only "sample" not first and second.
first and second is to be displayed when I click sample.
How can I implement this in java.
My datyabase is MySQL. The code which I tried is given below
FilteredRowSet rs = new FilteredRowSetImpl();
// for Other Types Like OBJECT-TYPE, Object_IDENTIFIER
rs = new FilteredRowSetImpl();
rs.setCommand("Select * from MIBNODEDETAILS where " + "mn_OID like '" + OID
+ ".%' order by mn_NodeType, mn_OID");
rs.setUrl(Constants.DB_CONNECTION_URL);
rs.setFilter(new MibRowFilter(1, expString));
rs.execute();
rs.absolute(1);
rs.beforeFirst();
I guess the change is to be made in the setCommand argument.
How can I do this?
Structure of mobnodedetails table
+--------------------+-------------------+-------------+
| mn_OID | mn_name | mn_nodetype |
+--------------------+-------------------+-------------+
| 1 | iso | 0 |
| 1.3 | org | 1 |
| 1.3.6 | dod | 1 |
| 1.3.6.1 | internet | 1 |
| 1.3.6.1.1 | directory | 1 |
| 1.3.6.1.2 | mgmt | 1 |
| 1.3.6.1.2.1 | mib-2 | 0 |
| 1.3.6.1.2.1.1 | system | 1 |
| 1.3.6.1.2.1.10 | transmission | 1 |

You can use something like
SELECT *
FROM mibnodedetails
WHERE mn_oid LIKE "+mn_OID+"%
AND LENGTH ("+mn_OID+") + 2 = LENGTH (mn_oid)
ORDER BY mn_nodetype, mn_oid
So if you pass mm_OID as 1.3.6.1 (|1.3.6.1 |internet |1 |)
You will get following result:
| 1.3.6.1.1 | directory | 1 |
| 1.3.6.1.2 | mgmt | 1 |
Working Demo
PS: This will not work for child more than 9 as we are using length + 2

The function given below dispalys the tree as required.
public void populateMibValues()
{
final DefaultTreeModel model = (DefaultTreeModel) this.m_mibTree.getModel();
model.setRoot(null);
this.rootNode.removeAllChildren();
final String query_MibNodeDetailsSelect = "Select * from MIBNODEDETAILS where LENGTH(mn_oid)<=9 "
+ " and mn_OID<='1.3.6.1.4.1' order by mn_OID"; // only
this.innerNodeNames.clear();
this.innerNodes.clear();
this.innerNodesOid = null;
try {
final ResultSet deviceRS = Application.getDBHandler().executeQuery(query_MibNodeDetailsSelect, null);// inner
// nodes
while (deviceRS.next()) {
final mibNode mb = new mibNode(deviceRS.getString("mn_OID").toString(), deviceRS.getString("mn_name")
.toString());
mb.m_Type = Integer.parseInt(deviceRS.getString("mn_nodetype").toString());
createMibTree(mb);
}
}
catch (final Exception e) {
Application.showErrorInConsole(e);
NmsLogger.writeErrorLog("ERROR creating MIB tree failed", e.toString());
}