How to cast all columns of Spark dataset to string using Java - java

I have a dataset with so many columns and I want to cast all columns to the string using Java.
I tried below steps, I want to know if there is any better way to achieve this?
Dataset<Row> ds = ...;
JavaRDD<String[]> stringArrRDD = ds.javaRDD().map(row->{
int length = row.length();
String[] columns = new String[length];
for(int i=0; i<length;i++){
columns[i] = row.get(i) !=null? row.get(i).toString():"";
}
return columns;});

You can iterate over columns:
for (String c: ds.columns()) {
ds = ds.withColumn(c, ds.col(c).cast("string"));
}

If you want to use objects only:
import org.apache.spark.sql.types.*;
...
for (String c: ds.columns()) {
ds = ds.withColumn(c, ds.col(c).cast(DataTypes.StringType));
}

Related

Creating n Object of array in java

I am having issues understanding how to create an array of n objects in Java.
This is the constructor of class ServicePath as follows:
public ServicePath(String id) {
this.id = id;
}
This is the elements of the array that I would like to create the objects.
String ServicePathArrays[] = {"SH11","SH13","SH17","SH110","SH111","SH112","SH115", ...}
I tried the following, but it creates it manually.
ServicePath[] servicePathArray = new ServicePath[ServicePathArrays.length];
For example, manually it creates the following
ServicePath[0] = new ServicePath("SH11");
ServicePath[1] = new ServicePath("SH13");
..
..
I would like to create it automatically using
String ServicePathArrays in such way:
ServicePath[0].id = "SH11";
ServicePath[1].id = "SH12";
ServicePath[2].id = "SH13";
..
..
This could be done using the functional behavior of jdk8+ :
String servicePathArray[] = {"SH11", "SH13", "SH17",
"SH110", "SH111", "SH112", "SH115"};
List<ServicePath> collection = Stream.of(servicePathArray)
.map(ServicePath::new)
.collect(Collectors.toList());
System.out.println(collection);
String ServicePathArrays[] = {"SH11","SH13","SH17","SH110","SH111","SH112","SH115", ...};
ServicePath[] servicePathArray = new ServicePath[ServicePathArrays.length];
for(int i = 0; i < ServicePathArrays.length; i++) {
servicePathArray [i] = new ServicePath(ServicePathArrays[i]);
}

How to dynamically create 2d array with for loop in java?

i have 2d array in this form:
GrupaArtikala [] grupaArtikala = parser.getObjektiArtikli();
Object[][] data = {
{grupaArtikala[1].getId(), grupaArtikala[1].getSifra(), grupaArtikala[1].getNaziv(), grupaArtikala[1].getIkonaID()},
{grupaArtikala[2].getId(), grupaArtikala[2].getSifra(), grupaArtikala[2].getNaziv(), grupaArtikala[2].getIkonaID()},
{grupaArtikala[3].getId(), grupaArtikala[3].getSifra(), grupaArtikala[3].getNaziv(), grupaArtikala[3].getIkonaID()},
{grupaArtikala[4].getId(), grupaArtikala[4].getSifra(), grupaArtikala[4].getNaziv(), grupaArtikala[4].getIkonaID()},
{grupaArtikala[5].getId(), grupaArtikala[5].getSifra(), grupaArtikala[5].getNaziv(), grupaArtikala[5].getIkonaID(),}
};
But i wat to create it with nested for loop, Any help please?
Here is one way you could incorporate a loop into this:
GrupaArtikala[]grupaArtikala=parser.getObjektiArtikli();
int length = grupaArtikala.length;
Object[][] data = new Object[length][4];
for(int i=0;i<length;i++){
GrupaArtikala temp = grupaAtrikala[i];
data[i][0] = temp.getId();
data[i][1] = temp.getSifra();
//add the rest of your attributes
}

Merge data by date

I have below data which I fetched from database by using Hibernate NamedQuery
TXN_ID END_DATE
---------- ------------
121 15-JUN-16
122 15-JUN-16
123 16-MAY-16
Each row data can be store in Java class Object.
Now I want to combined data depending on the END_DATE. If END_DATE are same then merge TXN_ID data.
From the above data output would be :
TXN_ID END_DATE
---------- ------------
121|122 15-JUN-16
123 16-MAY-16
I want to do this program in java. What is the easy program for that?
Using the accepted function printMap, to iterate through the hashmap in order to see if output is correct.
With the code below:
public static void main(String[] args) {
String[][] b = {{"1","15-JUN-16"},{"2","16-JUN-16"},{"3","13-JUN-16"},{"4","16-JUN-16"},{"5","17-JUN-16"}};
Map<String, String> mapb = new HashMap<String,String>();
for(int j=0; j<b.length; j++){
String c = mapb.get(b[j][1]);
if(c == null)
mapb.put(b[j][1], b[j][0]);
else
mapb.put(b[j][1], c+" "+b[j][0]);
}
printMap(mapb);
}
You get the following output:
13-JUN-16 = 3
16-JUN-16 = 2 4
17-JUN-16 = 5
15-JUN-16 = 1
I think this will solve your problem.
With hibernate you can put query result in a list of object
Query q = session.createSQLQuery( sql ).addEntity(ObjDataQuery.class);
List<ObjDataQuery> res = q.list();
Now you can create an hashmap to storage final result, to populate this object you can iterate over res
Map<String, String> finalResult= new HashMap<>();
for (int i=0; i<res.size(); i++){
if (finalResult.get(res.get(i).date!=null){
//new element
finalResult.put(res.get(i).date,res.get(i).txn)
} else {
//update element
finalResult.put(res.get(i).date,
finalResult.get(res.get(i).date) + res.get(i).txn)
}
}
I've not tested it by logic should be correct.
Another way is to change the query to obtain direct the final result (in oracle see LISTAGG)

How to convert rows of sqlite data to a ArrayList<double[]>()?

I get the data type of data from sqlite database is string. How to put these data into ArrayList()? Thank you!
try below code:-
ArrayList<String> array = new ArrayList<String>();
Levels l = new Levels(getBaseContext()); // level ur table name
l.open();
Cursor c_l = l.selectAll();
for (int i = 0; i < c_l.getCount()+1; i++)
{
array.add(c_l.getString(0)); // getstring 0 means your column
}
c_l.close();
l.close();

incompatible type of double array and properties string.split()

public static void main(String[] args)
{
String input="jack=susan,kathy,bryan;david=stephen,jack;murphy=bruce,simon,mary";
String[][] family = new String[50][50];
//assign family and children to data by ;
StringTokenizer p = new StringTokenizer (input,";");
int no_of_family = input.replaceAll("[^;]","").length();
no_of_family++;
System.out.println("family= "+no_of_family);
String[] data = new String[no_of_family];
int i=0;
while(p.hasMoreTokens())
{
data[i] = p.nextToken();
i++;
}
for (int j=0;j<no_of_family;j++)
{
family[j][0] = data[j].split("=")[0];
//assign child to data by commas
StringTokenizer v = new StringTokenizer (data[j],",");
int no_of_child = data[j].replaceAll("[^,]","").length();
no_of_child++;
System.out.println("data from input = "+data[j]);
for (int k=1;k<=no_of_child;k++)
{
family[j][k]= data[j].split("=")[1].split(",");
System.out.println(family[j][k]);
}
}
}
i have a list of family in input string and i seperate into a family and i wanna do it in double array family[i][j].
my goal is:
family[0][0]=1st father's name
family[0][1]=1st child name
family[0][2]=2nd child name and so on...
family[0][0]=jack
family[0][1]=susan
family[0][2]=kathy
family[0][3]=bryan
family[1][0]=david
family[1][1]=stephen
family[1][2]=jack
family[2][0]=murphy
family[2][1]=bruce
family[2][2]=simon
family[2][3]=mary
but i got the error as title: in compatible types
found:java.lang.String[]
required:java.lang.String
family[j][k]= data[j].split("=")[1].split(",");
what can i do?i need help
nyone know how to use StringTokenizer for this input?
Trying to understand why you can't just use split for your nested operation as well.
For example, something like this should work just fine
for (int j=0;j<no_of_family;j++)
{
String[] familySplit = data[j].split("=");
family[j][0] = familySplit[0];
String[] childrenSplit = familySplit[1].split(",");
for (int k=0;k<childrenSplit.length;k++)
{
family[j][k+1]= childrenSplit[k];
}
}
You are trying to assign an array of strings to a string. Maybe this will make it more clear?
String[] array = data.split("=")[1].split(",");
Now, if you want the first element of that array you can then do:
family[j][k] = array[0];
I always avoid to use arrays directly. They are hard to manipulate versus dynamic list. I implemented the solution using a Map of parent to a list of childrens Map<String, List<String>> (read Map<Parent, List<Children>>).
public static void main(String[] args) {
String input = "jack=susan,kathy,bryan;david=stephen,jack;murphy=bruce,simon,mary";
Map<String, List<String>> parents = new Hashtable<String, List<String>>();
for ( String family : input.split(";")) {
final String parent = family.split("=")[0];
final String allChildrens = family.split("=")[1];
List<String> childrens = new Vector<String>();
for (String children : allChildrens.split(",")) {
childrens.add(children);
}
parents.put(parent, childrens);
}
System.out.println(parents);
}
The output is this:
{jack=[susan, kathy, bryan], murphy=[bruce, simon, mary], david=[stephen, jack]}
With this method you can directory access to a parent using the map:
System.out.println(parents.get("jack"));
and this output:
[susan, kathy, bryan]

Categories

Resources