It's complicated for me to articulate a proper title for this. But an example should make it far simpler. Suppose I have this:
final class Cache {
private static final ConcurrentHashMap<String, List<String>> CACHE = ...
static List<String> byName(String name) {
return CACHE.computeIfAbsent(name, x -> // some expensive operation)
}
}
The idea is probably trivial, this acts as a LoadingCache, much like guava or caffeine (in reality it is more complicated, but that is irrelevant to the question).
I would like to be able to tell if this was the first load into the CACHE, or it was a read of an existing mapping. Currently, I do this:
final class Cache {
private static final ConcurrentHashMap<String, List<String>> CACHE = ...
static List<String> byName(String name) {
boolean b[] = new boolean[1];
List<String> result = CACHE.computeIfAbsent(name, x -> {
b[0] = true;
// some expensive operation)
});
if(b[0]) {
// first load into the cache, do X
} else {
// do Y
}
return result;
}
}
This works, but I am afraid I am missing something that ConcurrentHashMap can offer for me that would allow me to do the same. Thank you.
If you want to avoid your single-element array to pass data out of the lambda (which I would rather do with an AtomicReference or AtomicBoolean), you could use a stateful callback object. It doesn't change the behavior or design of your code, but could be considered a little bit cleaner and more OOP-y.
class LoadingAction<K, V> {
private boolean called = false;
public V load(final K key) {
called = true;
// load data
return ...;
}
public void executePostLoad() {
if (called) {
// loaded into cache, do X
} else {
// do Y
}
}
}
final class Cache {
private static final ConcurrentHashMap<String, List<String>> CACHE = new ConcurrentHashMap<>();
static List<String> byName(String name) {
final LoadingAction<String, List<String>> loader = new LoadingAction<>();
final List<String> result = CACHE.computeIfAbsent(name, loader::load);
loader.executePostLoad();
return result;
}
}
Or turn it inside-out:
class Loader<K, V> {
private boolean called = false;
public V load(final Map<K, V> map, final K key) {
final V result = map.computeIfAbsent(key, this::load);
this.executePostLoad();
return result;
}
private V load(final K key) {
called = true;
// load data
return ...;
}
private void executePostLoad() {
if (called) {
// loaded into cache, do X
} else {
// do Y
}
}
}
final class Cache {
private static final ConcurrentHashMap<String, List<String>> CACHE = new ConcurrentHashMap<>();
static List<String> byName(String name) {
final Loader<String, List<String>> loader = new Loader<>();
return loader.load(CACHE, name);
}
}
Construction and loading could be encapsulated in a static method:
class Loader<K, V> {
private boolean called = false;
public static <K, V> V load(final Map<K, V> map, final K key) {
final Loader<K, V> loader = new Loader<>();
return loader.doLoad(map, key);
}
private V doLoad(final Map<K, V> map, final K key) {
final V result = map.computeIfAbsent(key, this::load);
this.executePostLoad();
return result;
}
private V load(final K key) {
called = true;
// load data
return ...;
}
private void executePostLoad() {
if (called) {
// loaded into cache, do X
} else {
// do Y
}
}
}
final class Cache {
private static final ConcurrentHashMap<String, List<String>> CACHE = new ConcurrentHashMap<>();
static List<String> byName(String name) {
return Loader.load(CACHE, name);
}
}
I can think of a couple of ways to do this using the ConcurrentHashMap API. Both reply on using a mapping function that has side-effects. The idea is that the function makes a record of whether it was called, or what arguments it was called with.
The spec for computeIfAbsent says that the mapping function is only called if the key is absent. Alternatively, the spec for compute says that the mapping function is called with a null argument if the key is argument. In either case, if you record what happened in the mapper function via a side-effect on (say) a field of the mapper function/object, you can determine if the cache entry was already present or not.
To make this thread-safe, you need to create a fresh (thread-confined) instance of the mapper function.
Related
This is a beginner's question to kafka-streaming.
How would you collect pairs of messages using the java kafka-streaming library and write them to a new output topic?
I was thinking about something like this:
private void accumulateTwo(KStream<String, String> messages) {
Optional<String> accumulator = Optional.empty();
messages.mapValues(value -> {
if (accumulator.isPresent()) {
String tmp = accumulator.get();
accumulator = Optional.empty();
return Optional.of(new Tuple<>(tmp, value));
}
else {
accumulator = Optional.of(value);
return Optional.empty();
}
}).filter((key, value) -> value.isPresent()).to("pairs");
Yet this will not work, since variables in Java Lambda expressions must be final.
Any ideas?
EDIT:
As suggested in the comments, three additional steps are necessary:
The Transformer must explicitly store its state within a state store. It will get a reference to the state store from the ProcessorContext, which it is getting passed in the init method.
The state store must be registered with the StreamsBuilder
The name of the state store must be passed within the transform method.
In this example it is sufficient to store the last message we have seen. We are using a KeyValueStore for this which will have exactly zero or one entry at each point in time.
public class PairTransformerSupplier<K,V> implements TransformerSupplier<K, V, KeyValue<K, Pair<V,V>>> {
private String storeName;
public PairTransformerSupplier(String storeName) {
this.storeName = storeName;
}
#Override
public Transformer<K, V, KeyValue<K, Pair<V, V>>> get() {
return new PairTransformer<>(storeName);
}
}
public class PairTransformer<K,V> implements Transformer<K, V, KeyValue<K, Pair<V, V>>> {
private ProcessorContext context;
private String storeName;
private KeyValueStore<Integer, V> stateStore;
public PairTransformer(String storeName) {
this.storeName = storeName;
}
#Override
public void init(ProcessorContext context) {
this.context = context;
stateStore = (KeyValueStore<Integer, V>) context.getStateStore(storeName);
}
#Override
public KeyValue<K, Pair<V, V>> transform(K key, V value) {
// 1. Update the store to remember the last message seen.
if (stateStore.get(1) == null) {
stateStore.put(1, value); return null;
}
KeyValue<K, Pair<V,V>> result = KeyValue.pair(key, new Pair<>(stateStore.get(1), value));
stateStore.put(1, null);
return result;
}
#Override
public void close() { }
}
public KStream<String, String> sampleStream(StreamsBuilder builder) {
KStream<String, String> messages = builder.stream(inputTopic, Consumed.with(Serdes.String(), Serdes.String()));
// 2. Create the state store and register it with the streams builder.
KeyValueBytesStoreSupplier store = Stores.persistentKeyValueStore(stateStoreName);
StoreBuilder storeBuilder = new KeyValueStoreBuilder<>(
store,
new Serdes.IntegerSerde(),
new Serdes.StringSerde(),
Time.SYSTEM
);
builder.addStateStore(storeBuilder);
transformToPairs(messages);
return messages;
}
private void transformToPairs(KStream<String, String> messages) {
// 3. reference the name of the state store when calling transform(...)
KStream<String, Pair<String, String>> pairs = messages.transform(
new PairTransformerSupplier<>(),
stateStoreName
);
KStream<String, Pair<String, String>> filtered = pairs.filter((key, value) -> value != null);
KStream<String, String> serialized = filtered.mapValues(Pair::toString);
serialized.to(outputTopic);
}
Changes to the state store can be watched using the console consumer:
./bin/kafka-console-consumer --topic <changelog-topic-name> --bootstrap-server localhost:9092
Full source code here: https://github.com/1123/spring-kafka-stream-with-state-store
Original Answer:
The JavaDoc of the org.apache.kafka.streams.kstream.ValueMapper interface states that it is for stateless record-by-record transformations, and that the org.apache.kafka.streams.kstream.Transformer interface, on the other hand, is
for stateful mapping of an input record to zero, one, or multiple new output records.
Therefore I guess the Transformer interface is the appropriate choice for collecting pairs of messages. This may only be of relevance in case of failure and restart of streaming applications, such that they can recover the state from Kafka.
Hence, here is another solution based upon the org.apache.kafka.streams.kstream.Transformer interface:
class PairTransformerSupplier<K,V> implements TransformerSupplier<K, V, KeyValue<K, Pair<V,V>>> {
#Override
public Transformer<K, V, KeyValue<K, Pair<V, V>>> get() {
return new PairTransformer<>();
}
}
public class PairTransformer<K,V> implements Transformer<K, V, KeyValue<K, Pair<V, V>>> {
private V left;
#Override
public void init(ProcessorContext context) {
left = null;
}
#Override
public KeyValue<K, Pair<V, V>> transform(K key, V value) {
if (left == null) { left = value; return null; }
KeyValue<K, Pair<V,V>> result = KeyValue.pair(key, new Pair<>(left, value));
left = null;
return result;
}
#Override
public KeyValue<K, Pair<V, V>> punctuate(long timestamp) {
return null;
}
public void close() { }
}
The PairTransformerSupplier is then used as follows:
private void accumulateTwo(KStream<String, String> messages) {
messages.transform(new PairTransformerSupplier<>())
.filter((key, value) -> value != null)
.mapValues(Pair::toString)
.to("pairs");
}
Trying out both solutions within a single process on a topic with a single partition yields, however, the exact same results. I have not tried with a topic with multiple partitions and multiple stream consumers.
You should be able to write an accumulator class
class Accumulator implements ValueMapper<String, Optional<Tuple<String>>> {
private String key;
public Optional<Tuple<String>> get(String item) {
if (key == null) {
key = item;
return Optional.empty();
}
Optional<Tuple<String>> result = Optional.of(new Tuple<>(key, item));
key = null;
return result;
}
}
and then process with
messages.mapValues(new Accumulator())
.filter(Optional::isPresent) // I don't think your filter is correct
.to("pairs");
I want to make a HashMap which contains HashSets as values and returns an empty HashSet when the key is not found.
public class IsbnHashMap<K,V> extends HashMap<K,V> {
protected V defaultValue;
public IsbnHashMap(V defaultValue) {
this.defaultValue = defaultValue;
}
#Override
public V get(Object k) {
return containsKey(k) ? super.get(k) : defaultValue;
}
}
However my implementation does not work.
private static IsbnHashMap<String, HashSet<String>> isbnToId = new IsbnHashMap<String, HashSet<String>>();
This returns "HashSet cannot be applied". If I try to change K,V in IsbnHashMap to <String, HashSet<String>> I get some funky errors as well. How can I implement this?
First it should be noted that in Java-8 you can use instead:
isbnToId.computeIfAbsent(isbn, k -> new HashSet<>()).add(_id);
Second, if you really want to do something like this in previous Java versions, you'd better to create separate method for this purpose (for example, getOrDefault()) in order not to violate the contract. Third, you need to create new HashSet<>() for every new key. If you return the same instance, it will be shared between given keys. If you don't expect users to modify it, it's better to use unmodifiable Collections.emptySet() as default value. This way users may safely do isbnToId.getOrDefault(isbn).contains(_id), but trying isbnToId.getOrDefault(isbn).add(_id) will result in exception. If you want to support the modification (prior to Java-8), you can, for example, pass the element class to the constructor instead:
public static class MyMap<K, V> extends HashMap<K, V> {
private Class<?> clazz;
public MyMap(Class<?> clazz) {
this.clazz = clazz;
}
public V getOrCompute(K key) {
V v = get(key);
if(v == null) {
try {
v = (V) clazz.newInstance();
} catch (InstantiationException | IllegalAccessException e) {
throw new RuntimeException(e);
}
put(key, v);
}
return v;
}
}
Usage example:
MyMap<String, Set<String>> map = new MyMap<>(HashSet.class);
map.getOrCompute("a").add("b");
map.getOrCompute("a").add("c");
map.getOrCompute("d").add("e");
System.out.println(map); // {a=[b, c], d=[e]}
Here we assume that instantiating the passed class with default constructor is ok. An alternative would be to pass the factory interface which is capable to produce the default values.
As Jon Skeet said ...
private static IsbnHashMap<String, HashSet<String>> isbnToId = new IsbnHashMap<String, HashSet<String>>(new HashSet<String>());
... however, that would return the same default object as Dunni pointed out.
So this will do:
private static HashMap<String, HashSet<String>> isbnToId = new HashMap<String, HashSet<String>>();
public static void coupleIsbnToId(String isbn, String _id) {
if (!isbnToId.containsKey(isbn)) {
isbnToId.put(isbn, new HashSet<String>());
}
isbnToId.get(isbn).add(_id);
}
Without using getter and setter method how to prevent modification access from child class if super class has protected Hashmap variable?
This Map is mutable (So i should be able to add the values from super class)So can't use UnmodifiableMap(its only applicable immutable collection object)
Class A
{
protected Map<Integer,Integer> m = new HashMap<Integer,Integer>();
A()
{
m.put(10,11)
m.put(11.12)
}
}
Class B extends A
{
B()
{
super.m.put(34,90) —— I don’t want to give access to child class to add
the value and child class and its only should able to get the values.
}
}
Make the map unmodifiable, and populate it in the construction of A.
class A {
protected final Map<Integer,Integer> m;
A() {
Map<Integer, Integer> tempMap = = new HashMap<>();
tempMap.put(10,11);
tempMap.put(11.12);
this.m = java.util.Collections.unmodifiableMap(tempMap);
}
}
If and when B attempts to modify the map, a ´UnsupportedOperationException´ will be thrown.
If you want A to be able to modify the map, then you'll need a different approach in which the map is private, and a protected getter returns an unmodifiable map.
class A {
private final Map<Integer,Integer> m = new HashMap<>();
A() {
m.put(10,11);
m.put(11.12);
// m remains modifiable within the context of A
}
protected Map<Integer, Integer> getMap() {
return java.util.Collections.unmodifiableMap(m);
}
}
EDIT
If you really don't want to use a getter but still have read-only access, you can use this approach.
class A {
private final Map<Integer,Integer> writableMap = new HashMap<>();
protected final Map<Integer,Integer> m = Collections.unmodifiableMap(writableMap);
A() {
writableMap.put(10,11);
writableMap.put(11.12);
}
}
Using this approach, only m is visible outside A, and is read-only. Within A, you can update writableMap and these changes will be visible in m
Here is a variant of what Steve Chaloner presented in his answer:
public class A {
private final Map<Integer, Integer> map = new HashMap<>();
protected final Map<Integer,Integer> m = Collections.unmodifiableMap(map);
public A() {
map.put(10, 11);
map.put(11, 12);
}
}
The private map is modifiable in the A class and changes will be reflected in the protected m whenever changes are made in map.
It is being used this way in Concurrency In Practice for example.
This should be the best solution Composition:
Implement a new Map and keep an internal private modifiable map like this:
class A {
private Map<Integer,Integer> m = new HashMap<>();
protected Map<Integer, Integer> map = new Map<>() {
//implement interface
public Integer put(Integer key, Integer value) {
throw new UnsupportedOperationException();
}
public Integer get(Object key) {
return m.get(key);
}
public void clear() {
throw new UnsupportedOperationException();
}
public boolean containsKey(Object key) {
return m.containsKey(key);
}
public boolean containsValue(Object value) {
return m.containsValue(value);
}
//
// ... And so on
//
// ... with all other methods
}
A() {
m.put(10,11)
m.put(11.12)
}
}
class B extends A {
B() {
super.map.put(34,90) // thorws exception
super.m.put(34,90) // inaccesible
}
}
All modifications are allowed in A via m but subclasses may only acces them by map that was succesfully blocked modifications.
I was wondering if it was possible to have a Java dictionary of objects where one of the fields of the object is defined to be the key of the dictionary.
To be more specific, here's what I would like: I have defined a class with three fields. One of these fields is an Integer and is unique to each object. I would like this field to be the key of the dictionary.
Yes, of course it's possible.
Example :
Map<Integer,MyClass> map = new HashMap<Integer,MyClass>();
MyClass myObject = new MyClass(...);
map.put (myObject.getIntegerKey(), myObject);
If you want to hide the details:
public interface HasOwnKey<K> {
public K getKey();
}
public class MyMap<K, V extends HasOwnKey<K>> {
{
private Map<K,V> map = new HashMap<>();
public V put(V value) {
{
return this.map.put(value.getKey(),value);
}
public V get(K key) {
return this.map.get(key)
}
... etc
}
public class MyClass extends HasOwnKey<String> {
...
#Override String getKey() { return this.key; }
}
MyMap<String, MyClass> myMap = new MyMap<>();
MyClass obj = new MyClass();
obj.setKey("abc");
myMap.put(obj);
Unfortunately Java 7 doesn't seem to be smart enough to infer K from a declaration like
public class MyMap<V extends HasOwnKey<K>> {
so you have to provide the Key type in two places and cannot do
MyMap<MyClass> myMap = new MyMap<>();
You can do that easily as follows :
public class CustomClass
{
private int primaryKey;
private int secondaryField;
private int tertiaryField;
public CustomClass(int primaryKey, int secondaryField, int tertiaryField)
{
this.primaryKey = primaryKey;
this.secondaryField = secondaryField;
this.tertiaryField = tertiaryField;
}
public int getPrimaryKey(CustomClass object)
{
return object.primaryKey;
}
}
public class Test
{
public static void main(String[] args)
{
CustomClass object = new CustomClass(10, 20, 30);
Map map = new HashMap<Integer,CustomClass>();
map.put(object.getPrimaryKey(object), object);
}
}
You may also want to consider using Enums for doing the same, if the number of such records is fairly less, as they provide more readability.
If you already have created a List of those objects you can use an aggregate operation in java 8 like this:
Map<Integer, List<MyClass>> theMap = list
.stream()
.collect( Collectors.groupingBy(MyClass::myIntegerKey) );
I need a mapping from a list of keys to a value. I know I could write my own code like this:
Map<Person, Map<Daytime, Map<Food, Integer>>> eaten = ...;
Now I want to have some get and put methods like these:
Integer numberOfEggsIAteInTheMorning = eaten.get(me, morning, scrambledEggs);
eaten.put(me, evening, scrambledEggs, 1);
Do you know of an existing class that has this kind of API? I'm too lazy of writing it myself. ;)
If you look for a more generic approach, and you might have more than 2 or 3 'chain steps', I would suggest in applying some different structural approach, rather than sticking to using only basic collection classes. I have feeling that Composite Pattern could be the right choice if it's correctly applied.
EDIT: due to example requested
The full example would be somewhat time consuming, so let me just explain my idea with dirty Java/pseudocode mix (I'm not even sure if I've missed something!!!). Let's consider we have class BaseMap:
abstract class BaseMap {
public abstract Object getValue(Object.. keys);
public abstract void putValue(Object value, Object.. keys);
}
Then we could have ObjectMap that would be the 'leaf' of our composite structure:
class ObjectsMap extends BaseMap {
private Map<Object, Object> map = new [...]
public Object getValue(Object.. keys) {
// assert that keys.length == 1
return map.get(keys[0]);
}
public void putValue(Object value, Object.. keys) {
// assert that keys.length = 1
map.put(keys[0], value);
}
}
And the actual composite would be as such:
class CompositeMap extends BaseMap {
private Map<Object, BaseMap> compositeMaps = new [...]
public Object getValue(Object.. keys) {
// assert that keys.length > 1
return compositeMap.get(keys[0]).getValue(/* System.arrayCopy => subset of elements {keys_1, .. ,keys_max} */);
}
public void putValue(Object value, Object.. keys) {
// assert keys.length > 1
BaseMap newMap = null;
if (keys.length = 2) -> newMap = new ObjectsMap()
else newMap = new CompositeMap();
newMap.putValue(value, /*subset of keys {keys_1, .. , keys_max}*/);
}
}
You can use org.apache.commons.collections.keyvalue.MultiKey for that: Map<Multikey, Object>
It would be hard to implement a general chained map.
How would the declaration of the class look like? (You can't have a variable number of type parameters.
class ChainedMap<K1..., V>
Another option would be to have a ChainedMapUtil class that performs put / get recursively.
Here is an example of a recursive get. (Quite ugly solution though I must say.)
import java.util.*;
public class Test {
public static Object chainedGet(Map<?, ?> map, Object... keys) {
Object k = keys[0];
if (!map.containsKey(k)) return null;
if (keys.length == 1) return map.get(k);
Object[] tailKeys = Arrays.copyOfRange(keys, 1, keys.length);
return chainedGet((Map<?,?>) map.get(k), tailKeys);
}
public static void main(String[] arg) {
Map<String, String> m1 = new HashMap<String, String>();
m1.put("ipsum", "dolor");
Map<Integer, Map<String, String>> m2 =
new HashMap<Integer, Map<String, String>>();
m2.put(17, m1);
Map<String, Map<Integer, Map<String, String>>> chained =
new HashMap<String, Map<Integer, Map<String, String>>>();
chained.put("lorem", m2);
System.out.println(chainedGet(chained, "lorem", 17, "ipsum")); // dolor
System.out.println(chainedGet(chained, "lorem", 19, "ipsum")); // null
}
}
If you are going to write your own, I would suggest
eaten.increment(me, evening, scrambledEggs);
You could use a composite key
eaten.increment(Key.of(me, evening, scrambledEggs));
(TObjectIntHashMap supports increment and adjust)
You may not even need a custom key.
eaten.increment(me + "," + evening + "," + scrambledEggs);
It is fairly easy to decompose the key with split()
I once made a map using 3 keys just for fun.May be you can use it instead of using chained maps:
public class ThreeKeyMap<K1,K2,K3,V>{
class wrap{
K1 k1;
K2 k2;
K3 k3;
public wrap(K1 k1,K2 k2,K3 k3) {
this.k1=k1;this.k2=k2;this.k3=k3;
}
#Override
public boolean equals(Object arg0) {
// TODO Auto-generated method stub
wrap o=(wrap)arg0;
if(!this.k1.equals(o.k1))
return false;
if(!this.k2.equals(o.k2))
return false;
if(!this.k2.equals(o.k2))
return false;
return true;
}
#Override
public int hashCode() {
int result=17;
result=37*result+k1.hashCode();
result=37*result+k2.hashCode();
result=37*result+k3.hashCode();
return result;
}
}
HashMap<wrap,V> map=new HashMap<wrap, V>();
public V put(K1 k1,K2 k2,K3 k3,V arg1) {
return map.put(new wrap(k1,k2,k3), arg1);
}
public V get(Object k1,Object k2,Object k3) {
return map.get(new wrap((K1)k1,(K2)k2,(K3)k3));
}
public static void main(String[] args) {
ThreeKeyMap<Integer,Integer,Integer,String> birthDay=new ThreeKeyMap<Integer, Integer, Integer, String>();
birthDay.put(1, 1,1986,"Emil");
birthDay.put(2,4,2009, "Ansih");
birthDay.put(1, 1,1986,"Praveen");
System.out.println(birthDay.get(1,1,1986));
}
}
UPDATE:
As #Arturs Licis suggested.I looked up in net for composite pattern and I wrote a sample using it.I guess this is composite..Please comment if it is not so.
Person class:
public class Person {
private final String name;
private Map<Time, Food> map = new HashMap<Time, Food>();
public Person(String name) {
this.name = name;
}
void addTimeFood(Time time, Food food) {
map.put(time, food);
}
public String getName() {
return name;
}
Food getFood(Time time) {
Food tmp = null;
return (tmp = map.get(time)) == null ? Food.NoFood : tmp;
}
// main to test the person class
public static void main(String[] args) {
Person p1 = new Person("Jack");
p1.addTimeFood(Time.morning, Food.Bread);
p1.addTimeFood(Time.evening, Food.Chicken);
Person p2 = new Person("Jill");
p2.addTimeFood(Time.morning, Food.Egg);
p2.addTimeFood(Time.evening, Food.Rice);
Map<String, Person> map = new HashMap<String, Person>();
map.put(p1.getName(), p1);
map.put(p2.getName(), p2);
System.out.println(map.get("Jack").getFood(Time.evening));
}
#Override
public String toString() {
StringBuilder b = new StringBuilder();
b.append(name).append("\n");
b.append(map);
return b.toString();
}
}
Food class:
public enum Food {
Rice,
Egg,
Chicken,
Bread,
NoFood;
}
Time class:
public enum Time {
morning,
evening,
night
}