Develop Reference

How can I match double rows with a threshold in Spark?

I have a very simple dataframe:
+--+------+
|Id|Amount|
+--+------+
|0 |3.47 |
|1 |-3.47 |
|2 |3.47 |
|3 |3.47 |
|4 |2.02 |
|5 |-2.01 |
|6 |-2.01 |
|7 |7.65 |
|8 |7.65 |
+--+------+
I'd like to match lines that cancel each other given a threshold value (let's say 0.5).
So in this case, match line 0 and 1, 4 and 5, and return line 2 and 3. There are several solutions to this, returning lines 0 and 2 is also fine.
The general idea is that they should be matched 2 by 2 and the leftovers returned. It should return nothing if every line has a match and should return all lines that couldn't be paired that way.
Any idea how to do that?
Expected result:
+--+------+
|Id|Amount|
+--+------+
|0 |3.47 |
|2 |3.47 |
|6 |-2.01 |
|7 |7.65 |
|8 |7.65 |
+--+------+
I've been thinking about using an UserDefinedAggregateFunction, but I'm not sure whether or not it's enough. Especially because I think it can only return one value per group of lines.

I went with an UDF. Writing UDFs in Java is seriously overcomplicated...
If anybody can see a way to simplify that mess, please post or comment.
private UDF1<WrappedArray<Row>, Row[]> matchData() {
return (data) -> {
List<Data> dataList = JavaConversions.seqAsJavaList(data).stream().map(Data::fromRow).collect(Collectors.toList());
Set<Data> matched = new HashSet<>();
for (Data element : dataList) {
if (matched.contains(element)) continue;
dataList.stream().filter(e -> !matched.contains(e) && e != element)
.filter(e -> Math.abs(e.getAmount() + element.getAmount()) < THRESHOLD
&& Math.signum(e.getAmount()) != Math.signum(element.getAmount()))
.min(Comparator.comparingDouble(e -> Math.abs(e.getAmount() + element.getAmount())))
.ifPresent(e -> {
matched.add(e);
matched.add(element);
});
}
if (matched.size() != dataList.size()) {
return dataList.stream().map(Data::toRow).toArray(Row[]::new);
} else {
return new Row[0];
}
};
}
With the Data class (using Lombok):
#AllArgsConstructor
#EqualsAndHashCode
#Data
public final class Data {
private String name;
private Double amount;
public static Data fromRow(Row r) {
return new Data(
r.getString(r.fieldIndex("name")),
r.getDouble(r.fieldIndex("amount")));
}
public Row toRow() {
return RowFactory.create(name, amount);
}
}
I'm returning the whole set in case it didn't work, this is actually what I need in my case.

How to get parameters of a step with multiple pipelines in cucumber for the java code?

I have below steps in my feature file for a scenario.
Given my_first_step
And my_second_step
| Themes | one | three |
| Service Windows | two | four |
And my_third_step
| Create Apps |
| Config |
we can get parameters of 'my_third_step' as below in the java code as a list
public void my_third_step(List listOfItems) {}
but how can get parameters in 'my_second_step' ?
I need to get a rows as array of elements in the java code. How can I do that ?

You have to pass a list of objects, your object will look like
public class MyObject {
private Integer themes;
private Integer service;
public Integer getThemes() {
return this.themes;
}
public void setThemes(Integer themes) {
this.themes = themes;
}
public Integer getService() {
return this.service;
}
public void setService(Integer service) {
this.service = service;
}
}
Then you can pass a List<MyObject> to the method.
public void my_second_step(List<MyObject>) {
...
}
In the feature file change the definition as follows:
And my_second_step
| Themes | Service |
| one | two |
| three | four |
I hope this helps.

Using Header we can implement Data Table in much clean & precise way and considering Data Table looks like below one -
And my_second_step
| Heading_1 | Heading_2 | Heading_3 |
| Themes | one | three |
| Service Windows | two | four |
public void my_second_step(DataTable table) throws Throwable {
List<Map<String, String>> list = table.asMaps(String.class,String.class);
System.out.println(list.get(0).get("Heading_1") + " : " + list.get(1).get("Heading_1"));
System.out.println(list.get(0).get("Heading_2") + " : " + list.get(1).get("Heading_2"));
System.out.println(list.get(0).get("Heading_3") + " : " + list.get(1).get("Heading_3"));
}

Looping through 2d array with String.split()

I've got a simple problem, but I'm new to Java coming from PHP. I need to split a delimited text file into an array. I've broken it down into an array of lines, each one would look something like this:
{
{Bob | Smithers | Likes Cats | Doesnt Like Dogs},
{Jane | Haversham | Likes Bats | Doesnt Like People}
}
I need to turn this into a 2 dimensional array.
In PHP, it's a cinch. You just use explode(); I tried using String.split on a 1d array and it wasn't that bad either. The things is, I haven't yet learned how to be nice to Java. So I don't know how to loop through the array and turn it into a 2d. This is what I have:
for (i = 0; i < array.length; i++) {
String[i][] 2dArray = array[i].split("|", 4);
}
PHP would be
for ($i = 0; $i < count($array); $i++) {
$array[i][] = explode(",", $array[i]);
}

You can loop the array like this:
// Initialize array
String[] array = {
"Bob | Smithers | Likes Cats | Doesnt Like Dogs",
"Jane | Haversham | Likes Bats | Doesnt Like People"
};
// Convert 1d to 2d array
String[][] array2d = new String[2][4];
for(int i=0;i<array.length;i++) {
String[] temp = array[i].split(" \\| ");
for(int j=0;j<temp.length;j++) {
array2d[i][j] = temp[j];
}
}
// Print the array
for(int i=0;i<array2d.length;i++) {
System.out.println(Arrays.toString(array2d[i]));
}
Notes: I used \\|to split the pipe character.

Problem
If I got you right you have an input like this:
{{Bob | Smithers | Likes Cats | Doesnt Like Dogs},{Jane | Haversham | Likes Bats | Doesnt Like People}}
Readable version:
{
{Bob | Smithers | Likes Cats | Doesnt Like Dogs},
{Jane | Haversham | Likes Bats | Doesnt Like People}
}
And you want to represent that structure in a 2-dimensional String aray, String[][].
Solution
The key is the method String#split which splits a given String into substrings delimited by a given symbol. This is , and | in your example.
First of all we remove all {, } as we don't need them (as long as the text itself does not contain delimiter):
String input = ...
String inputWithoutCurly = input.replaceAll("[{}]", "");
The text is now:
Bob | Smithers | Likes Cats | Doesnt Like Dogs,Jane | Haversham | Likes Bats | Doesnt Like People
Next we want to create the outer dimension of the array, that is split by ,:
String[] entries = inputWithoutCurly.split(",");
Structure now is:
[
"Bob | Smithers | Likes Cats | Doesnt Like Dogs",
"Jane | Haversham | Likes Bats | Doesnt Like People"
]
We now want to split each of the inner texts into their components. We therefore iterate all entries, split them by | and collect them to the result:
// Declaring a new 2-dim array with unknown inner dimension
String[][] result = new String[entries.length][];
// Iterating all entries
for (int i = 0; i < entries.length; i++) {
String[] data = entries[i].split(" | ");
// Collect data to result
result[i] = data;
}
Finally we have the desired structure of:
[
[ "Bob", "Smithers", "Likes Cats", "Doesnt Like Dogs" ],
[ "Jane", "Haversham", "Likes Bats", "Doesnt Like People"]
]
Everything compact:
String[] entries = input.replaceAll("[{}]", "").split(",");
String[][] result = new String[entries.length][];
for (int i = 0; i < entries.length; i++) {
result[i] = entries[i].split(" | ");
}
Stream
If you have Java 8 or newer you can use the Stream API for a compact functional style:
String[][] result = Arrays.stream(input.replaceAll("[{}]", "").split(","))
.map(entry -> entry.split(" | "))
.toArray(String[][]::new);

what is the best way to find the topper by subject?

I have a string array of student info:
StudentNumber Integer, Subject String, mmarks integer
What would be the best way to use the java & collection to find out the topper in each subject.
ArrayList<String> strings = new ArrayList<String>();
strings.add("1 | Computers | 48");
strings.add("2 | Data Structures | 89");
strings.add("33 | English | 35");
strings.add("24 | Maths | 70");
strings.add("15 | Computers | 58");
strings.add("6 | Data Structures | 55");
strings.add("7 | English | 40");
strings.add("18 | Maths | 73");
for (String str : strings) {
String [] strArray = str.split("\\|");
// in completed code
for (int i = 0; i < str.length(); i++) {
sam.put(strArray[0], strArray[2]);
s.add(strArray[1]);
}
}
Expected output
15 Computers
2 Data structures
7 English
18 Maths

Create a Result class to store the information:
class Result {
private int studentNumber;
private String subject;
private int mark;
// constructor, getters and setters go here ...
}
Now convert your List<String> to a List<Result>:
List<Result> results = new ArrayList<>();
for (String s : strings){
String[] sa = s.split(" \\| ");
results.add(new Result(Integer.parseInt(sa[0]), sa[1], Integer.parseInt(sa[2])));
}
Create a stream from the results list, group by subject, and find the student with the highest mark:
Map<String, Integer> map = results.stream()
.collect(Collectors.groupingBy(Result::getSubject,
Collectors.collectingAndThen(Collectors.maxBy(Comparator.comparing(Result::getMark)), r -> r.get().getStudentNumber())));
Print the result:
map.forEach((k,v) -> System.out.println(v + " " + k));
15 Computers
18 Maths
7 English
2 Data Structures

Split Java Spring

I have a db with 2 columns, key and value. record:
------------------------------------
| key | value |
------------------------------------
| A | 1,desc 1;2,desc 2;3,desc 3 |
------------------------------------
I want to split value column become json format:
[{"key":"1","value":"desc 1"},{"key":"2","value":"desc 2"},{"key":"3", "value":"desc 3"}]
Where I am put split function? in service? because too dificult for 2 split. How to solve this problem?
Thanks,
Bobby

That depends on how your application is usually working with this value. If the usual case is using some specific data from this column, I would parse this at repository level already:
public static void main(String[] args) {
// You actually get this from DB
String value = "1,desc 1;2,desc 2;3,desc 3";
JSONArray j = new JSONArray();
Stream.of(value.split(";")).forEach((pair -> {
String[] keyValue = pair.split(",");
JSONObject o = new JSONObject();
o.put("key", keyValue[0]);
o.put("value", keyValue[1]);
j.put(o);
}));
System.out.println(j);
}