Develop Reference

Flink State Across multiple Transformers

How can I access a state using the same-id across multiple transformers, for example the following stores an Order object via ValueState in OrderMapper class:
env.addSource(source1()).keyBy(Order::getId).flatMap(new OrderMapper()).addSink(sink1());
Now I would like to access the same Order object via a SubOrderMapper class:
env.addSource(source2()).keyBy(SubOrder::getOrderId).flatMap(new SubOrderMapper()).addSink(sink2());
Edit: Looks like it's not possible to have state maintained across multiple operators, is there a way to have one operator accept multiple inputs, lets say 5 sources?

Take a look at CoProcessFunction
To realize low-level operations on two inputs, applications can use
CoProcessFunction or KeyedCoProcessFunction. This function is bound to
two different inputs and gets individual calls to processElement1(...)
and processElement2(...) for records from the two different inputs.
Also side outputs might be useful for you. side output
Edit:
Union operator my be an option.
Union

You can create a custom EitherOfFive class that contains one of your five different stream values (I'm assuming they are all different). See Flink's Either class for the one of two case.
Each input stream would use a Map function that converts the input class type to an EitherOfFive type.
There would be a getKey() method that would figure out (based on which of the five values is actually set) what key to return. And then you can have a single KeyedProcessFunction that takes as input this EitherOfFive type.
If the output is always the same, then you're all set. Otherwise you'll want side outputs, one per type, that feed the five different sinks.

Using Spring expression language to assert if a property (with comma separated values) loaded from a properties file, contains "value"

I have the following problem, I have a certain configuration class in spring boot which contains beans, which I would like to be created only if a certain property which has a list of values, contains a certain value.
now the configuration class looks something like this:
#ConditionalOnExpression("#{'${conditions.values.options}'.contains('ScenarioOne')}")
ConfigurationForScenarioOne{
#Bean
public StudentBean getStudentBean(){
...
}
}
In the properties file I would have something like so:
conditions.values.options=${CONDITIONS_VALUES_OPTIONS}
Then provide the value of CONDITIONS_VALUES_OPTIONS at runtime like so:
-DCONDITIONS_VALUES_OPTIONS=ScenarioOne,ScenarioTwo,ScenarioFive
The reason why I want to do this is to be able to deploy the app and have different beans be in use depending on the value given at runtime.
I will have several of these Configuration classes that will be created based on which ones of these properties are passed.
TO achieve this I was trying to rely on Spring Expression language to do the following:
1-Read the property conditions.values.options, then convert it to a list, then verify if the list contained the desired configuration string.
So far I have tried several expressions including:
#ConditionalOnExpression("#{'${conditions.values.options}'.contains('ScenarioOne')}")
and other similar expressions, with no luck. Can someone help me see what I am missing here?

I was able to get the desired behavior using this:
#ConditionalOnExpression("#{T(java.util.Arrays).asList('${conditions.values.options}').contains('ScenarioOne')}")
So dissecting this a little bit in the hopes to help others who may come across similar problems:
It seems that spring reads comma separated properties as an Array, this is bogus because the following can be done using spring boot:
The following two statements are valid:
#Value("${property.with.comma.separated.values}")
private List<String> prop;
OR
#Value("${property.with.comma.separated.values}")
private String [] prop;
now the expression:
"#{'${conditions.values.options}'.contains('ScenarioOne')}"
will throw an error saying that the Array class does not contain a method called '.contains()'
hence this is why I had to resort to use some method execution:
T(java.util.Arrays).asList('${conditions.values.options}')
to read the comma separated values, convert the String array into a List and then perform the '.contains(..)' method invocation.
I wish there was a simpler way to do it, to have to do a method invocation in there seems like overkill.
I am open to sugestions 😀

Can you try writing your annotation values like this and check :
#ConditionalOnExpression("#{${conditions.values.options}}.contains('ScenarioOne')")
Write the contains method after the curly brackets.

Wiremock- Verifying unique JSON matches expected?

I'm trying to test my REST application which generates some JSON and uses it to contact another API. I want to ensure the correct JSON is being generated, but the problem is that two of the fields are unique for each run, specifically an ID and a timestamp.
How can I verify this JSON is correct using Wiremock given that the fields are unique each time? Is there a way I can leave those fields to "any" or something in Wiremock?
obj.verfyObjContaining("{\"id\": 123 ,\"timestamp\": 11:11:11}");
Timestamp and ID are unique so this doesn't match.

Setting them to any isn't your only option. You can also mock the timestamp and id. Setting them to a static value during the test run. The advantage of that would be that you at least test that the value is being pushed out with the api call.
Unfortunately without any code posted I can not give more details than that.

Sorted it! Found a "matching()" verification function in Wiremock which takes a regex.

Hibernate Search with Elasticsearch creating fields with .keyword suffix

I just implemented the integration of Hibernate Search with Elasticsearch using hibernate search 5.8 and ES 5.5.
I have several fields created specifically for sorting, and they are all called [field]Sort.
When I was testing it locally, the first time I let Hibernate create the indexes, it created the String sort fields like this:
nameSort -> text
nameSort.keyword -> keyword
I realized that I should use the suffixed field for sorting.
But then, when I destroyed my Elasticsearch cluster, to start over, it didn't create the suffixed fields, it just created the sort fields as keyword directly.
I recreated the cluster 5 or more times again and it never created the suffixed fields again.
When I finally sent my changes to our staging environment, it created the suffixed fields again, causing my queries to fail, because they are trying to sort by a text field, instead of a keyword field.
Now, I'm really not sure of why it sometimes creates the suffix and sometimes doesn't.
Is there any rule?
Is there a way to avoid it creating 2 fields and making it always create only one keyword field with exactly the name I gave it?
Here's an example of a sort field:
#Field(name = "nameSort", analyze = Analyze.NO, store = Store.YES, index = Index.NO)
#SortableField(forField = "nameSort")
public String getNameSort() {
return name != null ? name.toLowerCase(Locale.ENGLISH) : null;
}
Thanks in advance for any help.

Hibernate Search does no such thing as creating a separate keyword field for text fields. It creates either a text field or a keyword field, depending on whether the field should be analyzed. In your case, the field is not analyzed, so it should create a keyword field.
Now, Hibernate Search is not alone here, and this behavior could stem from the Elasticsearch cluster itself. Did you check whether you have particular index templates on your Elasticsearch cluster? It could lead to Elasticsearch creating a keyword field whenever Hibernate Search creates a text property.
On a side note, you may be interested by the fact Hibernate Search 5.8 allows defining normalizers (same thing as Elasticsearch normalizers), which would allow you to annotate the getName() getter directly and avoid doing the lowercase conversion yourself. See this blog post for more information.

Handling null in Freemarker

Situation:
Old java project using freemarker has many finished templates working great.
Every template is using data form Transaction object.
This transaction object is very large, because wraps all data about transaction.
In templates is a lot of expression like this:
get("object1").getNestedObject2().getNestedObject3().getValue();
Problem:
New requirements appear: All templates have to be process for preview with no real data. All numbers should be Zero and all string should be ---.
Unsatisfactory solutions:
Remake all templates to check null values. (Lot of work and not safe)
Create Transaction object that contains all default value. (Lot of work)
Well my question is: Can I say to Freemarker, that if he finds null or finds null along the way, that he should use 0 instead if he was expecting number or --- if he was expecting String.
Or do you see any better solution?

If you need to show a dummy data model to the templates, your best bet is probably a custom ObjectWrapper (see Configuration.setObjectWrapper). Everything that reads the data model runs through the TemplateModel-s, and the root TemplateModel is made by the ObjectWrapper, thus it can control what values the templates get for what names. But the question is, when you have to return a dummy value for a name, how can you tell what its type will be? It's not just about finding out if it will be a string or a number, but also if it will be a method (like getNestedObject2) or a hash (something that can be followed by .). What can help there is that FreeMarker allows a value to have multiple types, so you can return a value that can be used as a method and as a hash and as a string, for example. Depending on the application that hack is might be good enough, except, you still have to decide if the value is a string or a number, because ${} will print the numerical value if the value both a string and a number.