Spark Java: Passing variable number of arguments to a function - java

Please see section "Programmatically Specifying the Schema". Java section.
The example works. But I have a question about this particular code snippet.
JavaRDD<Row> rowRDD = people.map(
new Function<String, Row>() {
public Row call(String record) throws Exception {
String[] fields = record.split(",");
return Row.create(fields[0], fields[1].trim());
}
The Row create method is being called with a static number of objects determined at compile time.
However, in my code, I need to call the Row.create method for a dynamic number of arguments.
I will only know the number of fields at run time
For example, it may be one of:
return Row.create(fields[0], fields[1].trim(), fields[2]);
or
return Row.create(fields[0]);
or
return Row.create(fields[0],fields[1].trim(), fields[2], fields[3],fields[4]);
How do I do it?

Here is how you can do it. Worked for me.
JavaRDD<Row> rowRDD = people.map(
new Function<String, Row>() {
public Row call(String record) throws Exception {
String[] fields = record.split(",");
//return Row.create(fields[0], fields[1].trim());
Object[] fields_converted = fields;
return Row.create(fields_converted);
}
});

Try using elipsis in your implemented method as below.
public static void create(String ...arg) { ... }
Elipsis accept n number of arguments.

You can specify a method to take multiple arguments by using three dots after the argument, for example:
public static <return_type> create(String...args){
// Yoo can now use the String[] args
}
Replace with your desired return type.
Please change the signature of your call method as you have not specified a return type for it!

Here is what I did in the same situation
new Function<String, Row>(String s) {
public Row call(String s){
int n = /* width of actual schema */
Object rec[] = new Object[n];
for( int i = 0; i < n; ++i )
rec[i] = /* Something that aligns with the type of #i field */
return Row.create( rec );
}
}
There might be dragons here. My version compiles, looks good, not tested yet.

Related

Add method for multiple arrays and ToString?

I'm having trouble understanding what exactly I would put in one of my classes to create the add method for 3 Arrays of the same Type. Here are the generic arrays in the main class
ArrayContainer<Integer> numberContainer = new ArrayContainer<>();
ArrayContainer<String> wordContainer = new ArrayContainer<>();
ArrayContainer<Pokemon> pokedex = new ArrayContainer<>();
My constructor for ArrayContainer is
public ArrayContainer(){
container = (T[]) new Object[defaultSize];
numItems = 0;
}
In my separate class, I'm confused what to put for my
public void add (T item){}
and I'm confused as what to return within my toString. I know you add to an array by putting
arrayName[index] = whatever;
But what would I put in that add method that would add to whatever array I call the method on? Would it be container[index] = item;?
What should I return that would return the element in the array?
Since the number of items in your ArrayContainer is not known beforehand, you should use a dynamic array, also known as List.
The numItems then becomes redundant since you can get it by calling list.size()
Your add function will only need to call list.add. As noted in the comments, it seems you're re-writing/wrapping List
In your toString method, you can return a string that concatenates all results of toString of the items included. StringBuilder can help you create a "format" that suits you. Of course this means that the objects you're putting in the container need to implement toString
Combining all the things will give you something like this:
ArrayContainer
public class ArrayContainer<T> {
private List<T> items;
public ArrayContainer() {
items = new ArrayList<>();
}
public void add(T item) {
items.add(item);
}
public String toString() {
StringBuilder sb = new StringBuilder();
sb.append("[ ");
for (T it: items)
sb.append(it.toString()).append(' ');
sb.append(']');
return sb.toString();
}
}
Main
public class Main {
public static void main(String[] args) {
ArrayContainer<String> stringArrayContainer = new ArrayContainer<>();
stringArrayContainer.add("hello");
stringArrayContainer.add("world");
System.out.println(stringArrayContainer);
// Outputs: [hello world]
}
}

Initialize Runnable Array of Lambdas(with string arg and returns a boolean) and Call Index

I was able to get this working with lambda's returning void and taking in 0 args using a hash table, see here -> Create lambda two dimensional array
Now, I'm trying to create a Runnable[] array, with lambda's in the index, and each lambda takes a String argument and returns a boolean.
Here is the code...
public class testLambdaWithPrimitiveType {
private final String[] numArray = {"One", "Two", "Three"};
private boolean numFound = false;
testLambdaWithPrimitiveType(String num){
setNumFound(num);
}
private void setNumFound(String num){
Runnable[] runnableNumArray = {
() -> isStringOne(num),
() -> isStringTwo(num),
() -> isStringThree(num)
};
for (int numChecked = 0; numChecked < runnableNumArray.length; numChecked++){
if (runnableNumArray[numChecked].run(num)){
this.numFound = true;
}
}
}
private boolean isNumFound(){return this.numFound;}
private boolean isStringOne(String num){
return num.equals(numArray[0]);
}
private boolean isStringTwo(String num){
return num.equals(numArray[1]);
}
private boolean isStringThree(String num){
return num.equals(numArray[2]);
}
public static void main(String[] args) {
testLambdaWithPrimitiveType objectOne = new testLambdaWithPrimitiveType("One");
testLambdaWithPrimitiveType objectTwo = new testLambdaWithPrimitiveType("Two");
testLambdaWithPrimitiveType objectThree = new testLambdaWithPrimitiveType("Three");
testLambdaWithPrimitiveType objectFour = new testLambdaWithPrimitiveType("Four");
System.out.println(objectFour.isNumFound()); // false
System.out.println(objectThree.isNumFound()); // true
System.out.println(objectTwo.isNumFound()); // true
System.out.println(objectOne.isNumFound()); // true
}
}
It looks like the array gets initialized correctly, but when I try to call on the index if (runnableNumArray[numChecked].run(num)){, I get a compile error. Any idea why this is happening?
That is because Runnable has method void run(), with no parameters, and you're trying to call run(num). Since num has already been applied from the setNumFound() parameter, just call using run().
Of course, that leads to second error, i.e. method returns void, so the if (run()) doesn't work.
Seems you might want a method boolean xxx(String), so replace Runnable with Predicate<String>, and you can call it using test(num) instead of run().
That then leads to compilation error Cannot create a generic array of Predicate<String>, so you have to replace the array with a List.
You can then use method references instead.
private void setNumFound(String num){
List<Predicate<String>> runnableNumList = Arrays.asList(
this::isStringOne,
this::isStringTwo,
this::isStringThree
);
for (Predicate<String> runnableNum : runnableNumList){
if (runnableNum.test(num)){
this.numFound = true;
}
}
}
In the Java Language, Runnable instances cannot have parameters, lambdas which do have parameters are Callable instances instead. In other words, your question is inaccurate... you cannot create Runnable array that takes parameters, even though the compiler (wrongly) allows you to.
The error is that the Runnable interface has a run method with the signature,
public abstract void run()
Yet you are trying pass a parameter to that run method.
runnableNumArray[numChecked].run( num )
Removing the num parameter will still give you an error. This is because the run method returns void which is nothing (look again at the signature) but if statements require a boolean value to evaluate.
I am not sure what you are trying to achieve with this Array of lambdas. If you give me more info, I might be able to correct your code. As it stands though, it is unclear what you are expecting the Runnables to achieve.
Here is an example of using Callable instances to achieve something of what you wanted.
private void setNumFound(String num) throws Exception {
Callable[] runnableNumArray = {
() -> isStringOne( num ),
() -> isStringTwo( num ),
() -> isStringThree( num )
};
for ( int numChecked = 0; numChecked < runnableNumArray.length; numChecked++ ){
if ( ( Boolean ) runnableNumArray[numChecked].call() ){
this.numFound = true;
}
}
}
private void setNumFound(String num){
boolean[] a = new boolean[1];
Runnable[] runnableNumArray = {
() -> a[0] = isStringOne(num),
() -> a[0] = isStringTwo(num),
() -> a[0] = isStringThree(num)
};
for (Runnable r : runnableNumArray ) {
r.run();
if ( a[0] ) {
this.numFound = true;
break; }
}
}
I rewrite your method and added boolean variable as array[1]. I think, so wrong write, in general situation you will get error: "local variables referenced from a lambda expression must be final or effectively final" - but its work in Java SE 8 (build 31).

Passign attribute name as parameter automatically in Java-8

I would like to have a method to validate fields kind of
protected void validate(String field, String fieldName){
if (field==null || field.isEmpty){
throw new IllegalArgumentException("Parameter " + fieldName + " cannot be empty");
}
}
and use in my class for example
class Foo {
private String x;
private String y;
...
public void validateAll(){
validate(x, "x");
validate(y, "y");
}
}
It would be great to use in this way
public void validateAll(){
validate(x);
validate(y);
}
and let the compiler pass the name of the variable automatically to validate(field, fieldName) method
How can I achive this in Java-8 ?
You can achieve this in Java by abandoning the idea of having java classes with fields, and instead having a Map which maps Column objects to values. From a usage standpoint, it would look roughly like this:
public static final Column<String> X_COLUMN = new Column<>( "x", String.class );
public static final Column<String> Y_COLUMN = new Column<>( "y", String.class );
public static final Table FOO_TABLE = new Table( "Foo", X_COLUMN, Y_COLUMN, ... );
...
Row fooRow = new Row( FOO_TABLE );
fooRow.setFieldValue( X_COLUMN, "x" );
String x = fooRow.getFieldValue( X_COLUMN );
for( Column<?> column : fooRow.getTable().getColumns() )
doSomethingWithField( fooRow, column );
private static <T> void doSomethingWithField( Row row, Column<T> column )
{
T value = row.getFieldValue( column );
...do something with the field value...
}
Since a value passed as argument to a method bears no information about the field it originated from, if it was read from a field at all, you can’t reconstruct this information. However, since your intent to verify fields, the desired operation is possible when processing the fields in the first place, rather than their contained values:
class Foo {
private String x;
private String y;
//...
public void validateAll() {
for(Field f: Foo.class.getDeclaredFields()) {
if(!Modifier.isStatic(f.getModifiers()) && !f.getType().isPrimitive()) try {
Object o=f.get(this);
if(o==null || o.equals(""))
throw new IllegalArgumentException(f.getName()+" cannot be empty");
} catch(ReflectiveOperationException ex) { throw new AssertionError(); }
}
}
}
The general problem of this approach is that by the time validateAll() reports a problem, the Foo instance already contains the illegal state. It’s preferable to reject invalid values right when they are attempted to set for a property. In that case, the parameter name of the method might not be available reflectively, however, when a method named setX throws an IllegalArgumentException (as would be indicated by the stack trace), there is no need for an additional meta information in the message…

What is best technique to pass big count of arguments to Java class?

I have non-standard question.
Lets say I have class MyClass that I execute from command line with list of arguments
[<key> <value>]
like:
MyClass -key1 arg1 -key2 arg2 -key3 arg3 ... -keyN argN.
therefore my class has followed method that passes input data to fileds:
public class MyClass{
private static String arg1;
private static String arg2;
private static String arg3;
...
private static String argN;
private static void getArgs(String[] args) {
for(int k=0; k<args.length; k=k+2){
if(args[k].equals("-key1")){
arg1 = args[k+1];
}
else if(args[k].equals("-key2")){
arg2 = args[k+1];
}
else if(args[k].equals("-key3")){
arg3 = args[k+1];
}
...
else if(args[k].equals("-keyN")){
argN = args[k+1];
}
}
}
I write automation and have about 30-40 MyClass's where each one has 10-30 parameters.
All classes have the same fields and different ones as well (50/50).
I don't like private static void getArgs(String[] args) method implementation, seems waste of time.
I though to write in Notepad++ some script based on Reg-ex and generates java code but before I want to ask if someone knows other technologies/techniques to take care about input parameters.
[EDIT]
I thought to use reflection, something like:
for(int k=0; k<args.length; k=k+2){
Field field = classInstance.getClass().getDeclaredField(args[k].substring(1));
field.setAccessible(true);
field.set(classInstance, args[k]);
}
Thank you,
Commons CLI already provides a nice solution to this problem. I would suggest using this as it is already well tested and used. No sense reinventing the library.
It is convenient to pass run time arguments to the java program when there are few arguments to be provided. But not in scenarios when there is a huge list of params to be passed to the program. I would suggest instead of using command line arguments to provide your info, you should rather use properties file. And to your advantage, Properties file also holds the data in key value pair and it makes more sense for your requirement. Also using the java Properties class to read values from the file is very convenient.
And if you need you can pass the properties file name(with full path) as an argument to your java class.
In all honesty, I would just parse them and put them in a map.
And use a constant key to get the value from the map whenever you need it: private static final String KEY_FIRSTKEY = "firstkey";.
So I would simply write a function parseArgs to put everything in a map.
(Perhaps this belongs in an utility class). Here is a quick example (without exception handling)
private static Map<String, String> parseArgs(String... pArgs)
{
if (pArgs == null) return null;
Map<String, String> map = new HashMap<String, String>();
String arg;
String key;
String value;
for (int i = 0, iLength = pArgs.length; i < iLength; i += 2)
{
arg = pArgs[i];
// GET KEY
if (!arg.startsWith("-")) continue;
key = arg.substring(1);
// GET VALUE
if ((i + 1) >= pArgs.length) break;
value = pArgs[i + 1];
// PUT IT IN MAP
map.put(key, value);
}
return (map.isEmpty()) ? null : map;
}
But if you really prefer to put them in private fields.
Then I propose to use some Reflection to convert the map to fields afterwards.
Here is an example:
private static void mapStaticFields(Map<String, String> pMap, Class pStaticFieldHolder)
throws NoSuchFieldException
{
if (pMap == null) return;
for (Map.Entry<String, String> entry : pMap.entrySet())
{
try
{
Field field = pStaticFieldHolder.getDeclaredField(entry.getKey());
field.setAccessible(true);
field.set(field, entry.getValue());
}
catch (IllegalAccessException iae)
{
// impossible
}
}
}
Putting it all together in an example:
import java.lang.reflect.Field;
import java.util.HashMap;
import java.util.Map;
public class MainTest
{
private static String key1;
private static String key2;
private static String key3;
public static void main(String... pArgs)
{
try
{
Map<String, String> argMap = parseArgs(pArgs);
mapStaticFields(argMap, MainTest.class);
System.out.println(key1);
System.out.println(key2);
System.out.println(key3);
}
catch (Exception e)
{
e.printStackTrace();
}
}
}
Invoking this from the commandline as java MainTest -key1 val1 -key2 val2 -key3 val3
will result in the following output:
val1
val2
val3
Enjoy !
It should be trivial to build a Map from the argument list:
Map<String, String> map = new HashMap<>();
for (i=0; i < args.length-1; i+=2) {
map.put(args[i], args[i+1];
}
You can then initialize your arguments like:
String arg1 = map.get("-key1");
...

Using Mockito with multiple calls to the same method with the same arguments

Is there a way to have a stubbed method return different objects on subsequent invocations? I'd like to do this to test nondeterminate responses from an ExecutorCompletionService. i.e. to test that irrespective of the return order of the methods, the outcome remains constant.
The code I'm looking to test looks something like this.
// Create an completion service so we can group these tasks together
ExecutorCompletionService<T> completionService =
new ExecutorCompletionService<T>(service);
// Add all these tasks to the completion service
for (Callable<T> t : ts)
completionService.submit(request);
// As an when each call finished, add it to the response set.
for (int i = 0; i < calls.size(); i ++) {
try {
T t = completionService.take().get();
// do some stuff that I want to test
} catch (...) { }
}
How about
when( method-call ).thenReturn( value1, value2, value3 );
You can put as many arguments as you like in the brackets of thenReturn, provided they're all the correct type. The first value will be returned the first time the method is called, then the second answer, and so on. The last value will be returned repeatedly once all the other values are used up.
You can do that using the thenAnswer method (when chaining with when):
when(someMock.someMethod()).thenAnswer(new Answer() {
private int count = 0;
public Object answer(InvocationOnMock invocation) {
if (count++ == 1)
return 1;
return 2;
}
});
Or using the equivalent, static doAnswer method:
doAnswer(new Answer() {
private int count = 0;
public Object answer(InvocationOnMock invocation) {
if (count++ == 1)
return 1;
return 2;
}
}).when(someMock).someMethod();
As previously pointed out almost all of the calls are chainable.
So you could call
when(mock.method()).thenReturn(foo).thenReturn(bar).thenThrow(new Exception("test"));
//OR if you're mocking a void method and/or using spy instead of mock
doReturn(foo).doReturn(bar).doThrow(new Exception("Test").when(mock).method();
More info in Mockito's Documenation.
Almost all of the calls are chainable:
doReturn(null).doReturn(anotherInstance).when(mock).method();
BDD style:
import static org.mockito.BDDMockito.given;
...
given(yourMock.yourMethod()).willReturn(1, 2, 3);
Classic style:
import static org.mockito.Mockito.when;
...
when(yourMock.yourMethod()).thenReturn(1, 2, 3);
Explicit style:
...
when(yourMock.yourMethod())
.thenReturn(1)
.thenReturn(2)
.thenReturn(3);
Depending on an arg
Suppose we have 2 args, and check the size of the 2nd (list) arg:
...
when(yourMock.yourMethod(any(), anyList()))
.thenAnswer(args -> ((List) args.getArgument(1)).size() < 2
? 1
: 3);
args are Objects, so we have to cast an arg to our type. I cast ^^^ to (List) in my case.
BDD
...
given(yourMock.yourMethod(any(), anyList()))
.willAnswer(args -> ((List) args.getArgument(1)).size() < 2
? 1
: 3);
I've implemented a MultipleAnswer class that helps me to stub different answers in every call. Here the piece of code:
private final class MultipleAnswer<T> implements Answer<T> {
private final ArrayList<Answer<T>> mAnswers;
MultipleAnswer(Answer<T>... answer) {
mAnswers = new ArrayList<>();
mAnswers.addAll(Arrays.asList(answer));
}
#Override
public T answer(InvocationOnMock invocation) throws Throwable {
return mAnswers.remove(0).answer(invocation);
}
}
doReturn( value1, value2, value3 ).when( method-call )
Related to #[Igor Nikolaev]'s answer from 8 years ago, using an Answer can be simplified somewhat using a lambda expression available in Java 8.
when(someMock.someMethod()).thenAnswer(invocation -> {
doStuff();
return;
});
or more simply:
when(someMock.someMethod()).thenAnswer(invocation -> doStuff());
If you have a dynamic list of values you can use AdditionalAnswers.returnsElementsOf:
import org.mockito.AdditionalAnswers;
when(mock.method()).thenAnswer(AdditionalAnswers.returnsElementsOf(myListOfValues));
Following can be used as a common method to return different arguments on different method calls. Only thing we need to do is we need to pass an array with order in which objects should be retrieved in each call.
#SafeVarargs
public static <Mock> Answer<Mock> getAnswerForSubsequentCalls(final Mock... mockArr) {
return new Answer<Mock>() {
private int count=0, size=mockArr.length;
public Mock answer(InvocationOnMock invocation) throws throwable {
Mock mock = null;
for(; count<size && mock==null; count++){
mock = mockArr[count];
}
return mock;
}
}
}
Ex. getAnswerForSubsequentCalls(mock1, mock3, mock2); will return mock1 object on first call, mock3 object on second call and mock2 object on third call.
Should be used like when(something()).doAnswer(getAnswerForSubsequentCalls(mock1, mock3, mock2));
This is almost similar to when(something()).thenReturn(mock1, mock3, mock2);
You can use a LinkedList and an Answer. Eg
MyService mock = mock(MyService.class);
LinkedList<String> results = new LinkedList<>(List.of("A", "B", "C"));
when(mock.doSomething(any())).thenAnswer(invocation -> results.removeFirst());
This is not directly related to the question. But wanted to put this in the same chain.
If trying to verify the same method call with multiple arguments, you can use the below times feature by Mockito. You don't need it if you are not verifying.
Mockito.verify(method, times(n)).methoscall();
Here is 'n' is the number of times the mock is invoked.
This might be basic/obvious, but if like me you are trying to mock multiple calls for a method that is called unknown number of times per call to method to be tested, for example:
public String method(String testArg) {
//...
while(condition) {
someValue = someBean.nestedMethod(); // This is called unknown number of times
//...
}
//...
}
You can do something like:
#Test
public void testMethod() {
mockNestedMethodForValue("value1");
assertEquals(method("arg"), "expected1");
mockNestedMethodForValue("value2");
assertEquals(method("arg"), "expected2");
mockNestedMethodForValue("value3");
assertEquals(method("arg"), "expected3");
}
private void mockNestedMethodForValue(String value) {
doReturn(value).when(someBeanMock).nestedMethod();
}
Here is working example in BDD style which is pretty simple and clear
given(carRepository.findByName(any(String.class))).willReturn(Optional.empty()).willReturn(Optional.of(MockData.createCarEntity()));

Categories

Resources