Develop Reference

Count distinct while aggregating others?

This is how my dataset looks like:
+---------+------------+-----------------+
| name |request_type| request_group_id|
+---------+------------+-----------------+
|Michael | X | 1020 |
|Michael | X | 1018 |
|Joe | Y | 1018 |
|Sam | X | 1018 |
|Michael | Y | 1021 |
|Sam | X | 1030 |
|Elizabeth| Y | 1035 |
+---------+------------+-----------------+
I want to calculate the amount of request_type's per person and count unique request_group_id's
Result should be following:
+---------+--------------------+---------------------+--------------------------------+
| name |cnt(request_type(X))| cnt(request_type(Y))| cnt(distinct(request_group_id))|
+---------+--------------------+---------------------+--------------------------------+
|Michael | 2 | 1 | 3 |
|Joe | 0 | 1 | 1 |
|Sam | 2 | 0 | 2 |
|John | 1 | 0 | 1 |
|Elizabeth| 0 | 1 | 1 |
+---------+--------------------+---------------------+--------------------------------+
What I've done so far: (helps to derive first two columns)
msgDataFrame.select(NAME, REQUEST_TYPE)
.groupBy(NAME)
.pivot(REQUEST_TYPE, Lists.newArrayList(X, Y))
.agg(functions.count(REQUEST_TYPE))
.show();
How to count distinct request_group_id's in this select? Is it possible to do within it?
I think it's possible only via two datasets join (my current result + separate aggregation by distinct request_group_id)

Example with "countDistinct" ("countDistinct" is not worked over window, replaced with "size","collect_set"):
val groupIdWindow = Window.partitionBy("name")
df.select($"name", $"request_type",
size(collect_set("request_group_id").over(groupIdWindow)).alias("countDistinct"))
.groupBy("name", "countDistinct")
.pivot($"request_type", Seq("X", "Y"))
.agg(count("request_type"))
.show(false)

How to perform a query using a field that is a merge of 2 columns?

I'm building up a series of distribution analysis using Java Spark library. This is the actual code I'm using to fetch the data from a JSON file and save the output.
Dataset<Row> dataset = spark.read().json("local/foods.json");
dataset.createOrReplaceTempView("cs_food");
List<GenericAnalyticsEntry> menu_distribution= spark
.sql(" ****REQUESTED QUERY ****")
.toJavaRDD()
.map(row -> Triple.of( row.getString(0), BigDecimal.valueOf(row.getLong(1)), BigDecimal.valueOf(row.getLong(2))))
.map(GenericAnalyticsEntry::of)
.collect();
writeObjectAsJsonToHDFS(fs, "/local/output/menu_distribution_new.json", menu_distribution);
The query I'm looking for is based on this structure:
+------------+-------------+------------+------------+
| FIRST_FOOD | SECOND_FOOD | DATE | IS_SPECIAL |
+------------+-------------+------------+------------+
| Pizza | Spaghetti | 11/02/2017 | TRUE |
+------------+-------------+------------+------------+
| Lasagna | Pizza | 12/02/2017 | TRUE |
+------------+-------------+------------+------------+
| Spaghetti | Spaghetti | 13/02/2017 | FALSE |
+------------+-------------+------------+------------+
| Pizza | Spaghetti | 14/02/2017 | TRUE |
+------------+-------------+------------+------------+
| Spaghetti | Lasagna | 15/02/2017 | FALSE |
+------------+-------------+------------+------------+
| Pork | Mozzarella | 16/02/2017 | FALSE |
+------------+-------------+------------+------------+
| Lasagna | Mozzarella | 17/02/2017 | FALSE |
+------------+-------------+------------+------------+
How can I achieve this (written below) output from the code written above?
+------------+--------------------+----------------------+
| FOODS | occurrences(First) | occurrences (Second) |
+------------+--------------------+----------------------+
| Pizza | 2 | 1 |
+------------+--------------------+----------------------+
| Lasagna | 2 | 1 |
+------------+--------------------+----------------------+
| Spaghetti | 2 | 3 |
+------------+--------------------+----------------------+
| Mozzarella | 0 | 2 |
+------------+--------------------+----------------------+
| Pork | 1 | 0 |
+------------+--------------------+----------------------+
I've of course tried to figure out a solution by myself but had no luck with the my tries, I may be wrong, but I need something like this:
"SELECT (first_food + second_food) as menu, COUNT(first_food), COUNT(second_food) from cs_food GROUP BY menu"

From the example data, this looks like it will produce the output you want:
select
foods,
first_count,
second_count
from
(select first_food as food from menus
union select second_food from menus) as f
left join (
select first_food, count(*) as first_count from menus
group by first_food
) as ff on ff.first_food=f.food
left join (
select second_food, count(*) as second_count from menus
group by second_food
) as sf on sf.second_food=f.food
;

Simple combination of flatMap and groupBy should do the job like this (sorry, can't check if it 100% correct right now):
import spark.sqlContext.implicits._
val df = Seq(("Pizza", "Pasta"), ("Pizza", "Soup")).toDF("first", "second")
df.flatMap({case Row(first: String, second: String) => Seq((first, 1, 0), (second, 0, 1))})
.groupBy("_1")

Spark - sample() function duplicating data?

I want to randomly select a subset of my data and then limit it to 200 entries. But after using the sample() function, I'm getting duplicate rows, and I don't know why. Let me show you:
DataFrame df= sqlContext.sql("SELECT * " +
" FROM temptable" +
" WHERE conditions");
DataFrame df1 = df.select(df.col("col1"))
.where(df.col("col1").isNotNull())
.distinct()
.orderBy(df.col("col1"));
df.show();
System.out.println(df.count());
Up until now, everything is OK. I get the output:
+-----------+
|col1 |
+-----------+
| 10016|
| 10022|
| 100281|
| 10032|
| 100427|
| 100445|
| 10049|
| 10070|
| 10076|
| 10079|
| 10081|
| 10082|
| 100884|
| 10092|
| 10099|
| 10102|
| 10103|
| 101039|
| 101134|
| 101187|
+-----------+
only showing top 20 rows
10512
with 10512 records without duplicates. AND THEN!
df = df.sample(true, 0.5).limit(200);
df.show();
System.out.println(users.count());
This returns 200 rows full of duplicates:
+-----------+
|col1 |
+-----------+
| 10022|
| 100445|
| 100445|
| 10049|
| 10079|
| 10079|
| 10081|
| 10081|
| 10082|
| 10092|
| 10102|
| 10102|
| 101039|
| 101134|
| 101134|
| 101134|
| 101345|
| 101345|
| 10140|
| 10141|
+-----------+
only showing top 20 rows
200
Can anyone tell me why? This is driving me crazy. Thank you!

You explicitly ask for a sample with replacement so there is nothing unexpected about getting duplicates:
public Dataset<T> sample(boolean withReplacement, double fraction)

Java Splitting With Math Expression

I am trying to split a Math Expression.
String number = "100+500";
String[] split = new String[3];
I want to make
split[0] = "100"
split[1] = "+"
split[2] = "500"
I tried this but I don't know what to write for splitting.
split = number.split(????);

You want to split between digits and non-digits without consuming any input... you need look arounds:
String[] split = number.split("(?<=\\d)(?=\\D)|(?<=\\D)(?=\\d)");
What the heck is that train wreck of a regex?
It's expressing the initial sentence of this answer:
(?<=\d) means the previous character is a digit
(?=\D) means the next character is a non-digit
(?<=\d)(?=\D) together will match between a digit and a non-digit
regexA|regexB means either regexA or regexB is matched, which is used as above points, but non-digit then digit for the visa-versa logic
An important point is that look arounds are non-consuming, so the split doesn't gobble up any of the input during the split.
Here's some test code:
String number = "100+500-123/456*789";
String[] split = number.split("(?<=\\d)(?=\\D)|(?<=\\D)(?=\\d)");
System.out.println(Arrays.toString(split));
Output:
[100, +, 500, -, 123, /, 456, *, 789]
To work with numbers that may have a decimal point, use this regex:
"(?<=[\\d.])(?=[^\\d.])|(?<=[^\\d.])(?=[\\d.])"
which effectively just add . to the characters that are a "number".

Off the bat, I don't know any library routine for the split. A custom splitting routine could be like this:
/**
* Splits the given {#link String} at the operators +, -, * and /
*
* #param string
* the {#link String} to be split.
* #throws NullPointerException
* when the given {#link String} is null.
* #return a {#link List} containing the split string and the operators.
*/
public List<String> split(String string) throws NullPointerException {
if (string == null)
throw new NullPointerException("the given string is null!");
List<String> result = new ArrayList<String>();
// operators to split upon
String[] operators = new String[] { "+", "-", "*", "/" };
int index = 0;
while (index < string.length()) {
// find the index of the nearest operator
int minimum = string.length();
for (String operator : operators) {
int i = string.indexOf(operator, index);
if (i > -1)
minimum = Math.min(minimum, i);
}
// if an operator is found, split the string
if (minimum < string.length()) {
result.add(string.substring(index, minimum));
result.add("" + string.charAt(minimum));
index = minimum + 1;
} else {
result.add(string.substring(index));
break;
}
}
return result;
}
Some test code:
System.out.println(split("100+10*6+3"));
System.out.println(split("100+"));
Output:
[100, +, 10, *, 6, +, 3]
[100, +]

You can also use the Pattern/Matcher classes in Java:
String expression = "100+34";
Pattern p = Pattern.compile("(\\d+)|(\\+)");
Matcher m = p.matcher(expression);
String[] elems = new String[m.groupCount() +1];
int i=0;
while(m.find())
{
elems[i++] = m.group();
}

You can do something simple instead of insane regex; just pad + with white space:
String number = "100+500";
number = number.replace("+", " + ");
Now you can split it at the white space:
String[] split = number.split(" ");
Now your indices will be set:
split[0] = "100";
split[1] = "+";
split[2] = "500";
To check for all arithmetic symbols, you can use the following method if you wish to avoid regex:
public static String replacing(String s) {
String[] chars = {"+", "-", "/", "="};
for (String character : chars) {
if (s.contains(character)) {
s = s.replace(character, " " + character + " ");//not exactly elegant, but it works
}
}
return s;
}
//in main method
number = replacing(number);
String[] split = number.split(" ");

You can split your expression string, then in result having pure tokens and categorized tokens. The mXparser library supports this as well as the calculation process. Please follow the below example:
Your very simple example "100+500":
import org.mariuszgromada.math.mxparser.*;
...
...
Expression e = new Expression("100+500");
mXparser.consolePrintTokens( e.getCopyOfInitialTokens() );
Result:
[mXparser-v.4.0.0] --------------------
[mXparser-v.4.0.0] | Expression tokens: |
[mXparser-v.4.0.0] ---------------------------------------------------------------------------------------------------------------
[mXparser-v.4.0.0] | TokenIdx | Token | KeyW | TokenId | TokenTypeId | TokenLevel | TokenValue | LooksLike |
[mXparser-v.4.0.0] ---------------------------------------------------------------------------------------------------------------
[mXparser-v.4.0.0] | 0 | 100 | _num_ | 1 | 0 | 0 | 100.0 | |
[mXparser-v.4.0.0] | 1 | + | + | 1 | 1 | 0 | NaN | |
[mXparser-v.4.0.0] | 2 | 500 | _num_ | 1 | 0 | 0 | 500.0 | |
[mXparser-v.4.0.0] ---------------------------------------------------------------------------------------------------------------
More sophisticated example "2*sin(x)+(3/cos(y)-e^(sin(x)+y))+10":
import org.mariuszgromada.math.mxparser.*;
...
...
Argument x = new Argument("x");
Argument y = new Argument("y");
Expression e = new Expression("2*sin(x)+(3/cos(y)-e^(sin(x)+y))+10", x, y);
mXparser.consolePrintTokens( e.getCopyOfInitialTokens() );
Result:
[mXparser-v.4.0.0] --------------------
[mXparser-v.4.0.0] | Expression tokens: |
[mXparser-v.4.0.0] ---------------------------------------------------------------------------------------------------------------
[mXparser-v.4.0.0] | TokenIdx | Token | KeyW | TokenId | TokenTypeId | TokenLevel | TokenValue | LooksLike |
[mXparser-v.4.0.0] ---------------------------------------------------------------------------------------------------------------
[mXparser-v.4.0.0] | 0 | 2 | _num_ | 1 | 0 | 0 | 2.0 | |
[mXparser-v.4.0.0] | 1 | * | * | 3 | 1 | 0 | NaN | |
[mXparser-v.4.0.0] | 2 | sin | sin | 1 | 4 | 1 | NaN | |
[mXparser-v.4.0.0] | 3 | ( | ( | 1 | 20 | 2 | NaN | |
[mXparser-v.4.0.0] | 4 | x | x | 0 | 101 | 2 | NaN | |
[mXparser-v.4.0.0] | 5 | ) | ) | 2 | 20 | 2 | NaN | |
[mXparser-v.4.0.0] | 6 | + | + | 1 | 1 | 0 | NaN | |
[mXparser-v.4.0.0] | 7 | ( | ( | 1 | 20 | 1 | NaN | |
[mXparser-v.4.0.0] | 8 | 3 | _num_ | 1 | 0 | 1 | 3.0 | |
[mXparser-v.4.0.0] | 9 | / | / | 4 | 1 | 1 | NaN | |
[mXparser-v.4.0.0] | 10 | cos | cos | 2 | 4 | 2 | NaN | |
[mXparser-v.4.0.0] | 11 | ( | ( | 1 | 20 | 3 | NaN | |
[mXparser-v.4.0.0] | 12 | y | y | 1 | 101 | 3 | NaN | |
[mXparser-v.4.0.0] | 13 | ) | ) | 2 | 20 | 3 | NaN | |
[mXparser-v.4.0.0] | 14 | - | - | 2 | 1 | 1 | NaN | |
[mXparser-v.4.0.0] | 15 | e | e | 2 | 9 | 1 | NaN | |
[mXparser-v.4.0.0] | 16 | ^ | ^ | 5 | 1 | 1 | NaN | |
[mXparser-v.4.0.0] | 17 | ( | ( | 1 | 20 | 2 | NaN | |
[mXparser-v.4.0.0] | 18 | sin | sin | 1 | 4 | 3 | NaN | |
[mXparser-v.4.0.0] | 19 | ( | ( | 1 | 20 | 4 | NaN | |
[mXparser-v.4.0.0] | 20 | x | x | 0 | 101 | 4 | NaN | |
[mXparser-v.4.0.0] | 21 | ) | ) | 2 | 20 | 4 | NaN | |
[mXparser-v.4.0.0] | 22 | + | + | 1 | 1 | 2 | NaN | |
[mXparser-v.4.0.0] | 23 | y | y | 1 | 101 | 2 | NaN | |
[mXparser-v.4.0.0] | 24 | ) | ) | 2 | 20 | 2 | NaN | |
[mXparser-v.4.0.0] | 25 | ) | ) | 2 | 20 | 1 | NaN | |
[mXparser-v.4.0.0] | 26 | + | + | 1 | 1 | 0 | NaN | |
[mXparser-v.4.0.0] | 27 | 10 | _num_ | 1 | 0 | 0 | 10.0 | |
[mXparser-v.4.0.0] ---------------------------------------------------------------------------------------------------------------
To understand what Token.tokenId and Token.tokenTypeId means you need to refer to the API documentation and parsertokens section. For instance in Operator class you have
Operator.TYPE_ID - this corresponds to Token.tokenTypeId if Token is recognized as Operator
Operator.OPERATOR_NAME_ID - this corresponds to Token.tokenId if Token is recognized as particular OPERATOR_NAME.
Please follow mXparser tutorial for better understanding.
Best regards

Since +,-,* basically all mathematically symbols are special characters so you put a "\\" before them inside the split function like this
String number = "100+500";
String[] numbers = number.split("\\+");
for (String n:numbers) {
System.out.println(n);
}

How to resolve Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 2?

Can somebody help me solving this type of error
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 2
I am searching for data in linked list but when I want to insert the data into an array, it turn up to be like this:
matric | nama | sem | cc | ch | fm
32255 | izzat | 1 | ccs2 | 3 | 45.0
| | 2 | ccs3 | 3 | 56.0
32345 | khai] | 3 | ccs4 | 3 | 45.0
| | 2 | ccs5 | 3 | 2.0
32246 | fifi | 1 | cc1 | 3 | 60.0
| | 1 | ccs3 | 4 | 34.0
34567 | dudu | 2 | ccs2 | 2 | 24.0
| | 2 | ccs4 | 6 | 79.0
first-->34567-->32246-->32345-->32255-->null
first-->6-->2-->4-->3-->3-->3-->3-->3-->null
first-->2-->2-->1-->1-->2-->3-->2-->1-->null
first-->dudu-->fifi-->khai]-->izzat-->null
first-->ccs4-->ccs2-->ccs3-->cc1-->ccs5-->ccs4-->ccs3-->ccs2-->null
first-->79.0-->24.0-->34.0-->60.0-->2.0-->45.0-->56.0-->45.0-->null
42insert matric= 032345
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 2
2
khai]
2
3
at inputoutput.LinkedList.getcc(LinkedList.java:141)
at inputoutput.baca.getcc(baca.java:84)
at inputoutput.Inputoutput.main(Inputoutput.java:75)
Java Result: 1
BUILD SUCCESSFUL (total time: 7 seconds)
the code:
String[] getcc(int mat,int sub) {
ListObject2 current = first2;
int count=0;
String b[]=new String[2] ;//2 is the subject number==sub
int x=0;
while (current!=null ) {
if(count==((mat*sub)+x) && ((mat*sub)+0)<((mat*sub)+x)<<((mat*sub)+sub)){
b[x]=current.data2;
x++;
}
current=current.next;
count++;
}
return b;
}
but I will get the input if search for last data in the linked list which is 032255
this is the output:
matric | nama | sem | cc | ch | fm
32255 | izzat | 1 | ccs2 | 3 | 45.0
| | 2 | ccs3 | 3 | 56.0
32345 | khai] | 3 | ccs4 | 3 | 45.0
| | 2 | ccs5 | 3 | 2.0
32246 | fifi | 1 | cc1 | 3 | 60.0
| | 1 | ccs3 | 4 | 34.0
34567 | dudu | 2 | ccs2 | 2 | 24.0
| | 2 | ccs4 | 6 | 79.0
first-->34567-->32246-->32345-->32255-->null
first-->6-->2-->4-->3-->3-->3-->3-->3-->null
first-->2-->2-->1-->1-->2-->3-->2-->1-->null
first-->dudu-->fifi-->khai]-->izzat-->null
first-->ccs4-->ccs2-->ccs3-->cc1-->ccs5-->ccs4-->ccs3-->ccs2-->null
first-->79.0-->24.0-->34.0-->60.0-->2.0-->45.0-->56.0-->45.0-->null
42insert matric= 032255
3
izzat
2
1
ccs3//the data i want to search
ccs2//

You're going into the if statement more than twice while walking the list. If you do that, you'll go past the bounds of the b array (which can only hold two values). You should use an ArrayList instead so you can add as many items as you need.