Develop Reference

How do I test Function's code when it's passed as method parameter?

Is it possible to test code that is written in lambda function that is passed inside the method process?
#AllArgsConstructor
public class JsonController {
private final JsonElementProcessingService jsonElementProcessingService;
private final JsonObjectProcessingService jsonObjectProcessingService;
private final JsonArrayProcessingService jsonArrayProcessingService;
public void process(String rawJson) {
jsonElementProcessingService.process(json -> {
JsonElement element = new JsonParser().parse(json);
if (element.isJsonArray()) {
return jsonArrayProcessingService.process(element.getAsJsonArray());
} else {
return jsonObjectProcessingService.process(element.getAsJsonObject());
}
}, rawJson);
}
}
Since the lambda is lazy the function is not invoked (Function::apply) when I call JsonController::process so is there any way to check that jsonArrayProcessingService::process is called?
#RunWith(JMockit.class)
public class JsonControllerTest {
#Injectable
private JsonElementProcessingService jsonElementProcessingService;
#Injectable
private JsonObjectProcessingService jsonObjectProcessingService;
#Injectable
private JsonArrayProcessingService jsonArrayProcessingService;
#Tested
private JsonController jsonController;
#Test
public void test() {
jsonController.process("[{\"key\":1}]");
// how check here that jsonArrayProcessingService was invoked?
}
}

Just make it testable (and readable) by converting it to a method:
public void process(String rawJson) {
jsonElementProcessingService.process(this::parse, rawJson);
}
Object parse(String json) {
JsonElement element = new JsonParser().parse(json);
if (element.isJsonArray()) {
return jsonArrayProcessingService.process(element.getAsJsonArray());
} else {
return jsonObjectProcessingService.process(element.getAsJsonObject());
}
}
The relevant guiding principles I personally follow are:
anytime my lambdas require curly brackets, convert them to a method
organise code so that it can be unit tested
You may need to change the return type of the parse method to match whatever your processing services (which you didn’t show) return.

Given its relatively-basic redirection logic, don't you just want to confirm which of the #Injectables got called:
#Test
public void test() {
jsonController.process("[{\"key\":1}]");
new Verifications() {{
jsonArrayProcessingService.process(withInstanceOf(JsonArray.class));
}};
}

How to read AvroFile into Tuple Class with Java in Flink

I'm Trying to read an Avro file and perform some operations on it, everything works fine but the aggregation functions, when I use them it get the below exception :
aggregating on field positions is only possible on tuple data types
then I change my class to implement Tuple4 (as I have 4 fields) but then when I want to collect the results get AvroTypeException Unknown Type : T0
Here are my data and job classes :
public class Nation{
public Integer N_NATIONKEY;
public String N_NAME;
public Integer N_REGIONKEY;
public String N_COMMENT;
public Integer getN_NATIONKEY() {
return N_NATIONKEY;
}
public void setN_NATIONKEY(Integer n_NATIONKEY) {
N_NATIONKEY = n_NATIONKEY;
}
public String getN_NAME() {
return N_NAME;
}
public void setN_NAME(String n_NAME) {
N_NAME = n_NAME;
}
public Integer getN_REGIONKEY() {
return N_REGIONKEY;
}
public void setN_REGIONKEY(Integer n_REGIONKEY) {
N_REGIONKEY = n_REGIONKEY;
}
public String getN_COMMENT() {
return N_COMMENT;
}
public void setN_COMMENT(String n_COMMENT) {
N_COMMENT = n_COMMENT;
}
public Nation() {
}
public static void main(String[] args) throws Exception {
Configuration parameters = new Configuration();
final ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
Path path2 = new Path("/Users/violet/Desktop/nation.avro");
AvroInputFormat<Nation> format = new AvroInputFormat<Nation>(path2,Nation.class);
format.configure(parameters);
DataSet<Nation> nation = env.createInput(format);
nation.aggregate(Aggregations.SUM,0);
JobExecutionResult res = env.execute();
}
and here's the tuple class and the same code for the job as above:
public class NationTuple extends Tuple4<Integer,String,Integer,String> {
Integer N_NATIONKEY(){ return this.f0;}
String N_NAME(){return this.f1;}
Integer N_REGIONKEY(){ return this.f2;}
String N_COMMENT(){ return this.f3;}
}
I tried with this class and got the TypeException (Used NationTuple everywhere instead of Nation)

I don't think having your class implementing Tuple4 is right way to go. Instead you should add to your topology a MapFunction that converts your NationTuple to Tuple4.
static Tuple4<Integer, String, Integer, String> toTuple(Nation nation) {
return Tuple4.of(nation.N_NATIONKEY, ...);
}
And then in your topology call:
inputData.map(p -> toTuple(p)).returns(new TypeHint<Tuple4<Integer, String, Integer, String>(){});
The only subtle part is that you need to provide a type hint so flink can figure out what kind of tuple your function returns.
Another solution is to use field names instead of tuple field indices when doing your aggregation. For example:
groupBy("N_NATIONKEY", "N_REGIONKEY")
This is all explained here: https://ci.apache.org/projects/flink/flink-docs-stable/dev/api_concepts.html#specifying-keys

Performing IO operations within java.util.function.Predicate

I have a question on the use of IO operations within java.util.function.Predicate. Please consider the following example:
public class ClientGroupFilter implements Predicate<Client> {
private GroupMapper mapper;
private List<String> validGroupNames = new ArrayList<>();
public ClientGroupFilter(GroupMapper mapper) {
this.mapper = mapper;
}
#Override
public boolean test(Client client) {
// this is a database call
Set<Integer> validsIds = mapper.getValidIdsForGroupNames(validGroupNames);
return client.getGroupIds().stream().anyMatch(validIds::contains);
}
public void permit(String name) {
validGroupNames.add(name);
}
}
As you can see this filter accepts any number of server group names, which are resolved by the mapper when a specific client is tested. If the client owns one of the valid server groups, true is returned.
Now, of course it is obivous that this is totally iniffecient if the filter is applied to multiple clients. So, refactoring lead me to this:
public class ClientGroupFilter implements Predicate<Client> {
private GroupMapper mapper;
private List<String> validGroupNames = new ArrayList<>();
private boolean updateRequired = true;
private Set<Integer> validIds = new HashSet<>();
public ClientGroupFilter(GroupMapper mapper) {
this.mapper = mapper;
}
#Override
public boolean test(Client client) {
if(updateRequired) {
// this is a database call
validIds = mapper.getValidIdsForGroupNames(validGroupNames);
updateRequired = false;
}
return client.getGroupIds().stream().anyMatch(validIds::contains);
}
public void permit(String name) {
validGroupNames.add(name);
updateRequired = true;
}
}
The performance is a lot better, of course, but im still not happy with the solution, since i feel like java.util.function.Predicate should not be used like this. However, i still want to be able to provide a fast solution to filter a list of clients, without the need to require the consumer to map the server group name to its ids.
Does anyone have a better idea to refactor this?

If your usage pattern is such that you call permit several times, and then use Predicate<Client> without calling permit again, you can separate the code that collects validGroupNames from the code of your predicate by using a builder:
class ClientGroupFilterBuilder {
private final GroupMapper mapper;
private List<String> validGroupNames = new ArrayList<>();
public ClientGroupFilter(GroupMapper mapper) {
this.mapper = mapper;
}
public void permit(String name) {
validGroupNames.add(name);
}
public Predicate<Client> build() {
final Set<Integer> validIds = mapper.getValidIdsForGroupNames(validGroupNames);
return new Predicate<Client>() {
#Override
public boolean test(Client client) {
return client.getGroupIds().stream().anyMatch(validIds::contains);
}
}
}
}
This restricts building of validIds to the point where we construct the Predicate<Client>. Once the predicate is constructed, no further input is necessary.

Best way to iterate two lists and extract few things?

I have two classes as shown below. I need to use these two classes to extract few things.
public final class ProcessMetadata {
private final String clientId;
private final String deviceId;
// .. lot of other fields here
// getters here
}
public final class ProcMetadata {
private final String deviceId;
private final Schema schema;
// .. lot of other fields here
}
Now I have below code where I am iterating above two classes and extracting schema given a clientId.
public Optional<Schema> getSchema(final String clientId) {
for (ProcessMetadata metadata1 : processMetadataList) {
if (metadata1.getClientId().equalsIgnoreCase(clientId)) {
String deviceId = metadata1.getDeviceId();
for (ProcMetadata metadata2 : procMetadataList) {
if (metadata2.getDeviceId().equalsIgnoreCase(deviceId)) {
return Optional.of(metadata2.getSchema());
}
}
}
}
return Optional.absent();
}
Is there any better way of getting what I need by iterating those two above classes in couple of lines instead of what I have? I am using Java 7.

You're doing a quadratic* search operation, which is inneficient. You can do this operation in constant time by first creating (in linear time) a mapping from id->object for each list. This would look something like this:
// do this once, in the constructor or wherever you create these lists
// even better discard the lists and use the mappings everywhere
Map<String, ProcessMetadata> processMetadataByClientId = new HashMap<>();
for (ProcessMetadata process : processMetadataList) {
processMetadataByClientId.put(process.getClientId(), process);
}
Map<String, ProcMetadata> procMetadataByDeviceId = new HashMap<>();
for (ProcMetadata metadata2 : procMetadataList) {
procMetadataByDeviceId.put(proc.getDeviceId(), proc);
}
Then your lookup simply becomes:
public Optional<Schema> getSchema(String clientId) {
ProcessMetadata process = processMetadataByClientId.get(clientId);
if (process != null) {
ProcMetadata proc = procMetadataByDeviceId.get(process.getDeviceId());
if (proc != null) {
return Optional.of(proc.getSchema());
}
}
return Optional.absent();
}
In Java 8 you could write it like this:
public Optional<Schema> getSchema(String clientId) {
return Optional.fromNullable(processMetadataByClientId.get(clientId))
.map(p -> procMetadataByDeviceId.get(p.getDeviceId()))
.map(p -> p.getSchema());
}
* In practice your algorithm is linear assuming client IDs are unique, but it's still technically O(n^2) because you potentially touch every element of the proc list for every element of the process list. A slight tweak to your algorithm can guarentee linear time (again assuming unique IDs):
public Optional<Schema> getSchema(final String clientId) {
for (ProcessMetadata metadata1 : processMetadataList) {
if (metadata1.getClientId().equalsIgnoreCase(clientId)) {
String deviceId = metadata1.getDeviceId();
for (ProcMetadata metadata2 : procMetadataList) {
if (metadata2.getDeviceId().equalsIgnoreCase(deviceId)) {
return Optional.of(metadata2.getSchema());
}
}
// adding a break here ensures the search doesn't become quadratic
break;
}
}
return Optional.absent();
}
Though of course using maps ensures constant-time, which is far better.

I wondered what could be done with Guava, and accidentally wrote this hot mess.
import static com.google.common.collect.Iterables.tryFind
public Optional<Schema> getSchema(final String clientId) {
Optional<String> deviceId = findDeviceIdByClientId(clientId);
return deviceId.isPresent() ? findSchemaByDeviceId(deviceId.get()) : Optional.absent();
}
public Optional<String> findDeviceIdByClientId(String clientId) {
return tryFind(processMetadataList, new ClientIdPredicate(clientId))
.transform(new Function<ProcessMetadata, String>() {
String apply(ProcessMetadata processMetadata) {
return processMetadata.getDeviceId();
}
});
}
public Optional<Schema> findSchemaByDeviceId(String deviceId) {
return tryFind(procMetadataList, new DeviceIdPredicate(deviceId.get())
.transform(new Function<ProcMetadata, Schema>() {
Schema apply(ProcMetadata procMetadata) {
return processMetadata.getSchema();
}
});
}
class DeviceIdPredicate implements Predicate<ProcMetadata> {
private String deviceId;
public DeviceIdPredicate(String deviceId) {
this.deviceId = deviceId;
}
#Override
public boolean apply(ProcMetadata metadata2) {
return metadata2.getDeviceId().equalsIgnoreCase(deviceId)
}
}
class ClientIdPredicate implements Predicate<ProcessMetadata> {
private String clientId;
public ClientIdPredicate(String clientId) {
this.clientId = clientId;
}
#Override
public boolean apply(ProcessMetadata metadata1) {
return metadata1.getClientId().equalsIgnoreCase(clientId);
}
}
Sorry.

RX Java 2, Observable that accepts new values to be added

I'm looking to create a LocationHandler class that returns an observable<Location> whose I can send a new Location and subscribers get the last one added and any subsequent values.
I've written this class, it works but I don't know if it's the correct way to do it because I've added a callback and I smell it bad.
Thanks for any help.
public class LocationHandler {
private MessageHandler<Location> onNewItem;
private Observable<Location> locationObservable;
public LocationHandler(LocationInitializationBuilder locationInitBuilder) {
locationObservable = getHookedObservable()
.mergeWith(locationInitBuilder.build())
.replay(1).autoConnect();
}
private Observable<Location> getHookedObservable() {
return Observable.create(new ObservableOnSubscribe<Location>() {
#Override
public void subscribe(ObservableEmitter<Location> e) throws Exception {
onNewItem = location -> e.onNext(location);
}
});
}
public Observable<Location> getLocation(){
return locationObservable;
}
public void setLocation(Location address){ // <---------- add new values
if (onNewItem != null){
onNewItem.handleMessage(address);
} else {
throw new IllegalStateException("Cannot add an item to a never subscribed stream");
}
}
}
Following #Blackbelt advice I've modified it with a ReplaySubject.
public class LocationHandler {
private ReplaySubject<Location> inputStream = ReplaySubject.create(1);
private Observable<Location> locationObservable;
public LocationHandler(LocationInitializationBuilder locationInitBuilder) {
locationObservable = locationInitBuilder.build()
.mergeWith(inputStream)
.replay(1).autoConnect();
}
public Observable<Location> getLocation(){
return locationObservable;
}
public void setLocation(Location address){
inputStream.onNext(address);
}
}

you could use a Subject instead of MessageHandler. Subject can act as observable and subscriber at the same time. You could have a method in your LocationHandler that returns Subject#asObservable to which you will subscribe. Internally, when setLocation, you will have to invoke Subject#onNext providing the location. There are different types of Subjects available. Please refer the documentation to choose the one that suits better your needs. E.g.
public class LocationHandler {
BehaviorSubject<GeevLocation> mLocationSubject = BehaviorSubject.create();
public Observable<GeevLocation> getLocation() {
return mLocationSubject.asObservable();
}
public void setLocation(GeevLocation address){
mLocationSubject.onNext(address);
}
}
from the outside call getLocation and subscribe to the returned Observable. When a setLocation is called you will get the object onNext

as Blackbelt already told you, you would use a Subject. In particular I would use a BehaviorSubject. Subjects are hot by default, but they can replay events by subscription. BehaviorSubject will give you the last emitted value or the init-value, if you subscribe. Every subscriber will get the values as the come in. The stream will never finish because it is hot. Please remeber to handle errores, because the second onError will be swallowed.
Example-Code
class Location {
}
class LocationInitializationBuilder {
static Location build() {
return new Location();
}
}
class LocationHandler {
private Subject<Location> locationObservable;
public LocationHandler(LocationInitializationBuilder locationInitBuilder) {
Location initialValue = LocationInitializationBuilder.build();
locationObservable = BehaviorSubject.<Location>createDefault(initialValue).toSerialized();
}
public Observable<Location> getLocation() {
return locationObservable.hide();
}
public void setLocation(Location address) { // <---------- add new values
locationObservable.onNext(address);
}
}
public class LocationTest {
#Test
public void name() throws Exception {
LocationHandler locationHandler = new LocationHandler(new LocationInitializationBuilder());
TestObserver<Location> test = locationHandler.getLocation().test();
locationHandler.setLocation(new Location());
test.assertValueCount(2);
}
}