Develop Reference

Kafka-Streaming: How to collect pairs of messages and write to a new topic

This is a beginner's question to kafka-streaming.
How would you collect pairs of messages using the java kafka-streaming library and write them to a new output topic?
I was thinking about something like this:
private void accumulateTwo(KStream<String, String> messages) {
Optional<String> accumulator = Optional.empty();
messages.mapValues(value -> {
if (accumulator.isPresent()) {
String tmp = accumulator.get();
accumulator = Optional.empty();
return Optional.of(new Tuple<>(tmp, value));
}
else {
accumulator = Optional.of(value);
return Optional.empty();
}
}).filter((key, value) -> value.isPresent()).to("pairs");
Yet this will not work, since variables in Java Lambda expressions must be final.
Any ideas?

EDIT:
As suggested in the comments, three additional steps are necessary:
The Transformer must explicitly store its state within a state store. It will get a reference to the state store from the ProcessorContext, which it is getting passed in the init method.
The state store must be registered with the StreamsBuilder
The name of the state store must be passed within the transform method.
In this example it is sufficient to store the last message we have seen. We are using a KeyValueStore for this which will have exactly zero or one entry at each point in time.
public class PairTransformerSupplier<K,V> implements TransformerSupplier<K, V, KeyValue<K, Pair<V,V>>> {
private String storeName;
public PairTransformerSupplier(String storeName) {
this.storeName = storeName;
}
#Override
public Transformer<K, V, KeyValue<K, Pair<V, V>>> get() {
return new PairTransformer<>(storeName);
}
}
public class PairTransformer<K,V> implements Transformer<K, V, KeyValue<K, Pair<V, V>>> {
private ProcessorContext context;
private String storeName;
private KeyValueStore<Integer, V> stateStore;
public PairTransformer(String storeName) {
this.storeName = storeName;
}
#Override
public void init(ProcessorContext context) {
this.context = context;
stateStore = (KeyValueStore<Integer, V>) context.getStateStore(storeName);
}
#Override
public KeyValue<K, Pair<V, V>> transform(K key, V value) {
// 1. Update the store to remember the last message seen.
if (stateStore.get(1) == null) {
stateStore.put(1, value); return null;
}
KeyValue<K, Pair<V,V>> result = KeyValue.pair(key, new Pair<>(stateStore.get(1), value));
stateStore.put(1, null);
return result;
}
#Override
public void close() { }
}
public KStream<String, String> sampleStream(StreamsBuilder builder) {
KStream<String, String> messages = builder.stream(inputTopic, Consumed.with(Serdes.String(), Serdes.String()));
// 2. Create the state store and register it with the streams builder.
KeyValueBytesStoreSupplier store = Stores.persistentKeyValueStore(stateStoreName);
StoreBuilder storeBuilder = new KeyValueStoreBuilder<>(
store,
new Serdes.IntegerSerde(),
new Serdes.StringSerde(),
Time.SYSTEM
);
builder.addStateStore(storeBuilder);
transformToPairs(messages);
return messages;
}
private void transformToPairs(KStream<String, String> messages) {
// 3. reference the name of the state store when calling transform(...)
KStream<String, Pair<String, String>> pairs = messages.transform(
new PairTransformerSupplier<>(),
stateStoreName
);
KStream<String, Pair<String, String>> filtered = pairs.filter((key, value) -> value != null);
KStream<String, String> serialized = filtered.mapValues(Pair::toString);
serialized.to(outputTopic);
}
Changes to the state store can be watched using the console consumer:
./bin/kafka-console-consumer --topic <changelog-topic-name> --bootstrap-server localhost:9092
Full source code here: https://github.com/1123/spring-kafka-stream-with-state-store
Original Answer:
The JavaDoc of the org.apache.kafka.streams.kstream.ValueMapper interface states that it is for stateless record-by-record transformations, and that the org.apache.kafka.streams.kstream.Transformer interface, on the other hand, is
for stateful mapping of an input record to zero, one, or multiple new output records.
Therefore I guess the Transformer interface is the appropriate choice for collecting pairs of messages. This may only be of relevance in case of failure and restart of streaming applications, such that they can recover the state from Kafka.
Hence, here is another solution based upon the org.apache.kafka.streams.kstream.Transformer interface:
class PairTransformerSupplier<K,V> implements TransformerSupplier<K, V, KeyValue<K, Pair<V,V>>> {
#Override
public Transformer<K, V, KeyValue<K, Pair<V, V>>> get() {
return new PairTransformer<>();
}
}
public class PairTransformer<K,V> implements Transformer<K, V, KeyValue<K, Pair<V, V>>> {
private V left;
#Override
public void init(ProcessorContext context) {
left = null;
}
#Override
public KeyValue<K, Pair<V, V>> transform(K key, V value) {
if (left == null) { left = value; return null; }
KeyValue<K, Pair<V,V>> result = KeyValue.pair(key, new Pair<>(left, value));
left = null;
return result;
}
#Override
public KeyValue<K, Pair<V, V>> punctuate(long timestamp) {
return null;
}
public void close() { }
}
The PairTransformerSupplier is then used as follows:
private void accumulateTwo(KStream<String, String> messages) {
messages.transform(new PairTransformerSupplier<>())
.filter((key, value) -> value != null)
.mapValues(Pair::toString)
.to("pairs");
}
Trying out both solutions within a single process on a topic with a single partition yields, however, the exact same results. I have not tried with a topic with multiple partitions and multiple stream consumers.

You should be able to write an accumulator class
class Accumulator implements ValueMapper<String, Optional<Tuple<String>>> {
private String key;
public Optional<Tuple<String>> get(String item) {
if (key == null) {
key = item;
return Optional.empty();
}
Optional<Tuple<String>> result = Optional.of(new Tuple<>(key, item));
key = null;
return result;
}
}
and then process with
messages.mapValues(new Accumulator())
.filter(Optional::isPresent) // I don't think your filter is correct
.to("pairs");

What is the best way to search prefixes in a Map implementation?

LinkedHashMap.put("a.a1","11");
LinkedHashMap.put("a.a12","12");
LinkedHashMap.put("b.b1","13");
LinkedHashMap.put("c.c1","14");
LinkedHashMap.put("c.c1","15");
A search on "a." key should return two values.
Which data structure in java should we be using as Trie DS implementation is not available. The next best which i could think of was only LinkedHashMap

You're looking for the Apache Patricia Trie. It is the exact data-structure for your use case.
From their docs:
A PATRICIA Trie is a compressed Trie. Instead of storing all data at the edges of the Trie (and having empty internal nodes), PATRICIA stores data in every node. This allows for very efficient traversal, insert, delete, predecessor, successor, prefix, range, and select(Object) operations. All operations are performed at worst in O(K) time, where K is the number of bits in the largest item in the tree. In practice, operations actually take O(A(K)) time, where A(K) is the average number of bits of all items in the tree.
Most importantly, PATRICIA requires very few comparisons to keys while doing any operation. While performing a lookup, each comparison (at most K of them, described above) will perform a single bit comparison against the given key, instead of comparing the entire key to another key.
In particular, the prefixMap(prefix) operation returns a SortedMap view with all the entries that match the given prefix.
Again, from the docs:
For example, if the Trie contains 'Anna', 'Anael', 'Analu', 'Andreas', 'Andrea', 'Andres', and 'Anatole', then a lookup of 'And' would return 'Andreas', 'Andrea', and 'Andres'.

Have another map that indexes by the prefix. In particular use Guava's Multimap which allows a key to map to a collection values.

I wrote my own MapFilter. I use it mostly for properties files. Essentially you pick a common prefix - say "com." and filter your map, selecting all entries with that prefix.
The elegance of this solution derives from the fact that the filtering process is points back to the underlying map for its values so it is truly a filter. Also, filtering filtered maps has efficiency benefits.
/**
* Allows the filtering of maps by key prefix.
*
* Note that all access through the filter reference the underlying Map so
* adding to a MapFilder results in additions to the Map.
*
* #author OldCurmudgeon
* #param <T>
*/
public class MapFilter<T> implements Map<String, T> {
// The enclosed map -- could also be a MapFilter.
final private Map<String, T> map;
// Use a TreeMap for predictable iteration order.
// Store Map.Entry to reflect changes down into the underlying map.
// The Key is the shortened string. The entry.key is the full string.
final private Map<String, Map.Entry<String, T>> entries = new TreeMap<>();
// The prefix they are looking for in this map.
final private String prefix;
public MapFilter(Map<String, T> map, String prefix) {
// Store my backing map.
this.map = map;
// Record my prefix.
this.prefix = prefix;
// Build my entries.
rebuildEntries();
}
public MapFilter(Map<String, T> map) {
this(map, "");
}
private synchronized void rebuildEntries() {
// Start empty.
entries.clear();
// Build my entry set.
for (Map.Entry<String, T> e : map.entrySet()) {
String key = e.getKey();
// Retain each one that starts with the specified prefix.
if (key.startsWith(prefix)) {
// Key it on the remainder.
String k = key.substring(prefix.length());
// Entries k always contains the LAST occurrence if there are multiples.
entries.put(k, e);
}
}
}
#Override
public String toString() {
return "MapFilter(" + prefix + ") of " + map + " containing " + entrySet();
}
// Constructor from a properties file.
public MapFilter(Properties p, String prefix) {
// Properties extends HashTable<Object,Object> so it implements Map.
// I need Map<String,T> so I wrap it in a HashMap for simplicity.
// Java-8 breaks if we use diamond inference.
this(new HashMap<>((Map) p), prefix);
}
// Helper to fast filter the map.
public MapFilter<T> filter(String prefix) {
// Wrap me in a new filter.
return new MapFilter<>(this, prefix);
}
// Count my entries.
#Override
public int size() {
return entries.size();
}
// Are we empty.
#Override
public boolean isEmpty() {
return entries.isEmpty();
}
// Is this key in me?
#Override
public boolean containsKey(Object key) {
return entries.containsKey(key);
}
// Is this value in me.
#Override
public boolean containsValue(Object value) {
// Walk the values.
for (Map.Entry<String, T> e : entries.values()) {
if (value.equals(e.getValue())) {
// Its there!
return true;
}
}
return false;
}
// Get the referenced value - if present.
#Override
public T get(Object key) {
return get(key, null);
}
// Get the referenced value - if present.
public T get(Object key, T dflt) {
Map.Entry<String, T> e = entries.get((String) key);
return e != null ? e.getValue() : dflt;
}
// Add to the underlying map.
#Override
public T put(String key, T value) {
T old = null;
// Do I have an entry for it already?
Map.Entry<String, T> entry = entries.get(key);
// Was it already there?
if (entry != null) {
// Yes. Just update it.
old = entry.setValue(value);
} else {
// Add it to the map.
map.put(prefix + key, value);
// Rebuild.
rebuildEntries();
}
return old;
}
// Get rid of that one.
#Override
public T remove(Object key) {
// Do I have an entry for it?
Map.Entry<String, T> entry = entries.get((String) key);
if (entry != null) {
entries.remove(key);
// Change the underlying map.
return map.remove(prefix + key);
}
return null;
}
// Add all of them.
#Override
public void putAll(Map<? extends String, ? extends T> m) {
for (Map.Entry<? extends String, ? extends T> e : m.entrySet()) {
put(e.getKey(), e.getValue());
}
}
// Clear everything out.
#Override
public void clear() {
// Just remove mine.
// This does not clear the underlying map - perhaps it should remove the filtered entries.
for (String key : entries.keySet()) {
map.remove(prefix + key);
}
entries.clear();
}
#Override
public Set<String> keySet() {
return entries.keySet();
}
#Override
public Collection<T> values() {
// Roll them all out into a new ArrayList.
List<T> values = new ArrayList<>();
for (Map.Entry<String, T> v : entries.values()) {
values.add(v.getValue());
}
return values;
}
#Override
public Set<Map.Entry<String, T>> entrySet() {
// Roll them all out into a new TreeSet.
Set<Map.Entry<String, T>> entrySet = new TreeSet<>();
for (Map.Entry<String, Map.Entry<String, T>> v : entries.entrySet()) {
entrySet.add(new Entry<>(v));
}
return entrySet;
}
/**
* An entry.
*
* #param <T>
*
* The type of the value.
*/
private static class Entry<T> implements Map.Entry<String, T>, Comparable<Entry<T>> {
// Note that entry in the entry is an entry in the underlying map.
private final Map.Entry<String, Map.Entry<String, T>> entry;
Entry(Map.Entry<String, Map.Entry<String, T>> entry) {
this.entry = entry;
}
#Override
public String getKey() {
return entry.getKey();
}
#Override
public T getValue() {
// Remember that the value is the entry in the underlying map.
return entry.getValue().getValue();
}
#Override
public T setValue(T newValue) {
// Remember that the value is the entry in the underlying map.
return entry.getValue().setValue(newValue);
}
#Override
public boolean equals(Object o) {
if (!(o instanceof Entry)) {
return false;
}
Entry e = (Entry) o;
return getKey().equals(e.getKey()) && getValue().equals(e.getValue());
}
#Override
public int hashCode() {
return getKey().hashCode() ^ getValue().hashCode();
}
#Override
public String toString() {
return getKey() + "=" + getValue();
}
#Override
public int compareTo(Entry<T> o) {
return getKey().compareTo(o.getKey());
}
}
// Simple tests.
public static void main(String[] args) {
String[] samples = {
"Some.For.Me",
"Some.For.You",
"Some.More",
"Yet.More"};
Map map = new HashMap();
for (String s : samples) {
map.put(s, s);
}
Map all = new MapFilter(map);
Map some = new MapFilter(map, "Some.");
Map someFor = new MapFilter(some, "For.");
System.out.println("All: " + all);
System.out.println("Some: " + some);
System.out.println("Some.For: " + someFor);
Properties props = new Properties();
props.setProperty("namespace.prop1", "value1");
props.setProperty("namespace.prop2", "value2");
props.setProperty("namespace.iDontKnowThisNameAtCompileTime", "anothervalue");
props.setProperty("someStuff.morestuff", "stuff");
Map<String, String> filtered = new MapFilter(props, "namespace.");
System.out.println("namespace props " + filtered);
}
}

How do I get a key from a HashMap by providing the value?

private static HashMap<Script, String> scripts = new HashMap<>();
public Script getScriptByName(String name) {
for (String s : scripts.values()) {
if (s.equals(name)) {
...
}
}
return null;
}
Given this code, how can I get the key of a specific value?

Navigate through the entries of the map instead:
for (Map.Entry<String, String> entry : scripts.entrySet()) {
if (entry.getValue().equals(name)) {
return entry.getKey();
}
}
return null;

How to efficiently get the values from a HashMap with a static class?

I have a class with a HashMap<k,v>.
The type of the values of this HashMap is a static class which has two different objects as attributes. i.e.,
public class Example {
private HashMap<String, StaticClassExample> map;
private static class StaticClassExample {
private Object1 o1;
private Object2 o2;
//...
}
//...
}
And my question is how can I do this operation efficiently:
public List<Object1> getAllObject1() {}
I know that I can do: map.values() and then iterate the values collection and get Object1 from each StaticClassExample, but this wouldn't be efficient.
It's possible what I ask or I must create another hashmap for my purpose?

If you don't mind some memory overhead, you could keep a separate list with the o1-values:
public class HashMapList
{
private HashMap<String, StaticClassExample> map = new HashMap<String, HashMapList.StaticClassExample>();
private List<Object> o1List = new LinkedList<Object>();
public static class StaticClassExample
{
private Object o1;
private Object o2;
}
public void addStaticClassExample(String key, StaticClassExample example)
{
StaticClassExample oldVal = map.put(key, example);
if(oldVal != null)
{
o1List.remove(oldVal.o1);
}
o1List.add(example.o1);
}
public StaticClassExample getStaticClassExampleByKey(String key)
{
return map.get(key);
}
public void removeStaticClassExampleByKey(String key)
{
StaticClassExample removed = map.remove(key);
if(removed != null)
{
o1List.remove(removed.o1);
}
}
public List<Object> getAllObject1()
{
return Collections.unmodifiableList(o1List);
}
}
Of course, this requires you to encapsule the HashMap inside the class and never give a straight access to it, because then someone using the class could modify the HashMap directly, and the List would no longer be in sync with the Map. Note that getAllObject1 returns an unmodifiable view of the internal list, so it can't be modified from outside of the class.

MultiKeyMap get method

I want to use MultiKeyMap from Apache Collection, because I need a HashMap with two keys and a value.
To put elements I do this:
private MultiKeyMap multiKey = new MultiKeyMap();
multiKey.put("key1.1", "key2.1", "value1");
And for get element I do this:
String s = multiKey.get("key1.1");
But the String s cames null... If I pass the two keys, like that:
String s = multiKey.get("key1.1", "key2.1");
The String s cames with values value1...
How can I extend the MultiKeyMap to get the right value when I pass only one of the two keys?

If you need only one key to get a value you have a plain old HashMap.
private Map<String, String> map = new HashMap<>();
map.put("key1.1", "value1");
map.put("key2.1", "value1");
And for get element you can do this:
String s = map.get("key1.1"); // s == "value1"
MultiKeyMap is required when both keys must be provided.

If you specify a value with two keys, you are going to need both keys to get it back. The hash function is not designed to return all the possible values that are associated with only one of the two keys. You may need to find a different data structure to do this.

MultiKeyMap is about using tuples as keys, not about matching one value to more than one key. Use a normal map and just put your value twice, with different keys.
Some more caution is needed when removing values. When you remove a value for the first key, do you want to automatically remove other keys with the same value? If so, you need either to loop over all keys and remove those with same value by hand, which could be inefficient, or keep some kind of reverse map to quickly find keys for specific value.

I don't know exact solution to your problem. But I suggest you to implement it like:
Map<K2, K1> m2;
Map<K1, V> m1;
And see: How to implement a Map with multiple keys?

It seems that you just do not need MultiKeyMap. You need regular map. Using it you can associate the same value with as many keys as you want.
Map<String, String> map = new HashMap<String, String>();
Object value = .....
map.put("key1", value);
map.put("key2", value);
..................
if(map.get("key1") == map.get("key2")) {
System.out.println("the same value stored under 2 different keys!");
}

You just can't since it's not the way a MultiKeyMap works. Put the value with separate keys and than try getting it with each key at a time.

Instead of that you can use table data stature from guava.

I would suggest to create a separate class for multiple keys:
public class Test {
Map<Shape, Book> test1 = new HashMap<>();
Book book = new Book("A");
test1.put(Shape, book);
private class Shape {
String id1;
String id2;
public Shape(String id1, String id2) {
this.id1 = id1;
this.id2 = id2;
}
#Override
public boolean equals(Object o) {//}
#Override
public int hashCode() {//}
}
}

Here is a simple MultiKeyMap implementation that worked for me.
import java.util.Collection;
import java.util.HashMap;
import java.util.HashSet;
import java.util.Map;
import java.util.Set;
import java.util.UUID;
public class MultiMap<K, V> implements Map<K, V>
{
private class MultiMapEntery implements java.util.Map.Entry<K, V>
{
private final K key;
private V value;
public MultiMapEntery(K key, V value)
{
this.key = key;
this.value = value;
}
#Override
public K getKey()
{
return key;
}
#Override
public V getValue()
{
return value;
}
#Override
public V setValue(V value)
{
V oldValue = this.value;
this.value = value;
return oldValue;
}
};
private final Map<K, String> keyMap = new HashMap<K, String>();
private final Map<String, Set<K>> inverseKeyMap = new HashMap<String, Set<K>>();
private final Map<String, V> valueMap = new HashMap<String, V>();
#Override
public void clear()
{
keyMap.clear();
inverseKeyMap.clear();
valueMap.clear();
}
#Override
public boolean containsKey(Object key)
{
return keyMap.containsKey(key);
}
#Override
public boolean containsValue(Object value)
{
return valueMap.containsValue(value);
}
#Override
public Set<java.util.Map.Entry<K, V>> entrySet()
{
Set<java.util.Map.Entry<K, V>> entries = new HashSet<>();
for(K key : keyMap.keySet())
{
V value = valueMap.get(key);
entries.add(new MultiMapEntery(key, value));
}
return entries;
}
#Override
public V get(Object key)
{
return valueMap.get(keyMap.get(key));
}
#Override
public boolean isEmpty()
{
return valueMap.isEmpty();
}
#Override
public Set<K> keySet()
{
return keyMap.keySet();
}
#Override
public V put(K key, V value)
{
String id = keyMap.get(key);
if(id == null)
{
id = UUID.randomUUID().toString();
}
keyMap.put(key, id);
Set<K> keys = inverseKeyMap.get(id);
if(keys == null)
{
keys = new HashSet<>();
}
keys.add(key);
inverseKeyMap.put(id, keys);
valueMap.put(id, value);
return value;
}
public V put(Set<K> keys, V value)
{
String id = null;
for(K key : keys)
{
id = keyMap.get(key);
if(id != null) // one of the keys already exists
{
break;
}
}
if(id == null)
{
id = UUID.randomUUID().toString();
}
for(K key : keys)
{
keyMap.put(key, id);
}
inverseKeyMap.put(id, keys);
valueMap.put(id, value);
return value;
}
#Override
public void putAll(Map<? extends K, ? extends V> map)
{
for(java.util.Map.Entry<? extends K, ? extends V> entry : map.entrySet())
{
put(entry.getKey(), entry.getValue());
}
}
#Override
public V remove(Object key)
{
String id = keyMap.get(key);
keyMap.remove(key);
Set<K> keys = inverseKeyMap.get(id);
keys.remove(key);
V value = valueMap.get(id);
if(keys.size() == 0) // it was the last key, now remove the value
{
valueMap.remove(id);
}
return value;
}
#Override
public int size()
{
return valueMap.size();
}
#Override
public Collection<V> values()
{
return valueMap.values();
}
public static void main(String[] args)
{
MultiMap<String, String> m = new MultiMap<>();
m.put("a", "v1");
Set<String> s = new HashSet<>();
s.add("b");
s.add("c");
s.add("d");
m.put(s, "v2");
System.out.println("size:" + m.size());
System.out.println("keys:" + m.keySet());
System.out.println("values:" + m.values().toString());
System.out.println("a:" + m.get("a"));
System.out.println("b:" + m.get("b"));
System.out.println("c:" + m.get("c"));
System.out.println("d:" + m.get("d"));
m.remove("a");
System.out.println("size:" + m.size());
System.out.println("keys:" + m.keySet());
System.out.println("values:" + m.values().toString());
System.out.println("a:" + m.get("a"));
System.out.println("b:" + m.get("b"));
System.out.println("c:" + m.get("c"));
System.out.println("d:" + m.get("d"));
s.add("a");
m.put(s, "v3");
System.out.println("size:" + m.size());
System.out.println("keys:" + m.keySet());
System.out.println("values:" + m.values().toString());
System.out.println("a:" + m.get("a"));
System.out.println("b:" + m.get("b"));
System.out.println("c:" + m.get("c"));
System.out.println("d:" + m.get("d"));
}
}

A little late, but you probably mean to get every result from the map, that matches the first element only, even though it contains multiple results, ignoring the second key (wildcard effect). Apache's MultiKeyMap is not suitable for this.
You could solve this by creating your own filter functionality using the MultiKey of MultiKeyMap. First, filter out only the relevant MultiKeys (which you get from yourMultiKeyMap.keySet() ) . The following method takes those multiKeys, and the first keys you want to filter on:
private Set<MultiKey<? extends String>> filterMultiKeys(Set<MultiKey<? extends String>> multiKeys, final String... keys) {
final List<String> givenKeys = Arrays.asList(keys);
return multiKeys.stream().filter(multiKey -> {
final Object[] actualKeys = multiKey.getKeys();
if (actualKeys.length < givenKeys.size()) {
// Lesser keys, so never a match
return false;
}
final List<Object> trimmedKeys = Arrays.asList(actualKeys).subList(0, givenKeys.size());
return trimmedKeys.equals(givenKeys);
}).collect(Collectors.toSet());
}
Then, use the resulting MultiKeys to get the results:
final Set<String> results = filteredKeys.stream().map(multiKey -> yourMultiKeyMap.get(multiKey)).collect(Collectors.toSet());
For bonus points, one could extend or decorate MultiKeyMap and create MyMultiKeyMap , having a method like match(keys...) using the filter functionality.