I am trying to find a way to create a hot stream where i could insert data in one method and a subscriber could get the data in another method.
I have succeeded using a WorkQueueProcessor, but I am not sure if this is the right way of doing that. Is it possible to do the same thing using Flux.create ?
Here's my working snippet:
Call connect();
Send byte data to server, the client will receive a response from tcp server and workQueueProcessor will emit the data.
#Component
#RequiredArgsConstructor
public class TcpCli {
#Setter
private Connection connection;
private NettyOutbound out;
//Creation of Work Queue Processor, can a Flux.create here can do the same job ?
private WorkQueueProcessor<String> workQueueProcessor = WorkQueueProcessor.<String>builder().build();
public Mono<? extends Connection> connect() {
return TcpClient.create()
.host(tcpConfig.getHost())
.port(tcpConfig.getPort())
.handle(this::handleConnection)
.connect();
}
public Mono<String> sendData(ByteArray data) {
out.sendByteArray(Mono.just(data)).then().subscribe();
//Get emitted data from workQueueProcessor
return workQueueProcessor.next();
}
private Publisher<Void> handleConnection(NettyInbound in, NettyOutbound out) {
this.out = out;
in.receive().asString()
.log("In received")
.subscribe(str -> {
LOGGER.info(String.format("Inbound: %s", str));
//Emit data to workQueueProcessor
workQueueProcessor.onNext(str);
});
return out
.neverComplete() //keep connection alive
.log("Never close");
}
}
If you have multiple input sources and more than one subscribers, I think you are on right way.
If the NettyInbound is the only input source of your steam, then you don't need to use any Processors. Just subscribe to it.
If you have multiple input sources for your stream, and you have only one subscriber, in your case NettyOutbound, you may try 'UnicastProcessor' which is much light weighted.
Can you explain what the #Setter does here. I try the same code and it always gives the out
#Setter
private Connection connection;
private NettyOutbound out;
Related
I'm doing it first time. Where am going to read stream of data using websocket.
Here is my code snippet
RsvpApplication
#SpringBootApplication
public class RsvpApplication {
private static final String MEETUP_RSVPS_ENDPOINT = "ws://stream.myapi.com/2/rsvps";
public static void main(String[] args) {
SpringApplication.run(RsvpApplication.class, args);
}
#Bean
public ApplicationRunner initializeConnection(
RsvpsWebSocketHandler rsvpsWebSocketHandler) {
return args -> {
System.out.println("initializeConnection");
WebSocketClient rsvpsSocketClient = new StandardWebSocketClient();
rsvpsSocketClient.doHandshake(
rsvpsWebSocketHandler, MEETUP_RSVPS_ENDPOINT);
};
}
}
RsvpsWebSocketHandler
#Component
class RsvpsWebSocketHandler extends AbstractWebSocketHandler {
private static final Logger logger =
Logger.getLogger(RsvpsWebSocketHandler.class.getName());
private final RsvpsKafkaProducer rsvpsKafkaProducer;
public RsvpsWebSocketHandler(RsvpsKafkaProducer rsvpsKafkaProducer) {
this.rsvpsKafkaProducer = rsvpsKafkaProducer;
}
#Override
public void handleMessage(WebSocketSession session,
WebSocketMessage<?> message) {
logger.log(Level.INFO, "New RSVP:\n {0}", message.getPayload());
System.out.println("handleMessage");
rsvpsKafkaProducer.sendRsvpMessage(message);
}
}
RsvpsKafkaProducer
#Component
#EnableBinding(Source.class)
public class RsvpsKafkaProducer {
private static final int SENDING_MESSAGE_TIMEOUT_MS = 10000;
private final Source source;
public RsvpsKafkaProducer(Source source) {
this.source = source;
}
public void sendRsvpMessage(WebSocketMessage<?> message) {
System.out.println("sendRsvpMessage");
source.output()
.send(MessageBuilder.withPayload(message.getPayload())
.build(),
SENDING_MESSAGE_TIMEOUT_MS);
}
}
As far I know and read about websocket is that, It needs one time connection and stream of data will be flowing continuously until either party (client or server) stops.
I'm building it first time, so trying to cover major scenarios which can come acroos while dealing with 10000+ messages per minute. Total kafka brokers are two with enough space.
What can be done, if connection gets lost and again start consuming messages from webscoket once connected back where it was left in last failure and push messages into further Kafka broker ?
What can be done to put on hold websocket to keep pushing messages in broker if it has reached to threshold limit of not processed messages (in broker) ?
What can be done, When broker reached to its threshold, run a separate process to check available space in broker to push more messages and give indication to resume pushing messages in kafka broker ?
Please share other issues, which needs to be considered while setting up this thing ?
I would like to ask a simple question,
I have implemented a processor, which process one payload and return an array of entities, such like:
#EnableBinding(Processor.class)
public class SimpleProcessor {
...
public SimpleProcessor () {
...
}
#Transformer(inputChannel = Processor.INPUT, outputChannel = Processor.OUTPUT)
public OutgoingEntity[] processData(IncomingEntity payload) {
// business logic here
return outgoingEntity;
}
I have my stream in SCDF and middleware as kafka like this:
some source | SimpleProcessor | JDBC sink
to validate the messages, I have replaced log sink to relpace JDBC sink and it logs arrays of json. When I use JDBC sink, it throws Exception and says JDBC sink can not access the properties in the object, which makes sense, that it is array of objects instead object...
My question is:
Can I use modify my processor, so it can process a payload once and give out message multiple time, like this,
#Transformer(inputChannel = Processor.INPUT)
public void processData(IncomingEntity payload) {
...
for(OutGoingEntity o: OutgoingEntity[]){
outputMethode();
}
}
#Transformer(outputChannel = Processor.OUTPUT)
private OutGoingEntity outputMethode() {
.....
return outGoingEntity;
}
So it can pass multiple json object to jdbc sink and write in the datatable.
Can I use JDBC sink to deal with arrays? How?
Can I use some other processors or sink to finish this task?
Like Matthias J. Sax suggested in comment, I have used flatMapValue method of KStream to deal with array from input. I put this scdf processor after the one forwards array. That works fine.
#EnableBinding(KafkaStreamsProcessor.class)
public class ArrayProcessor {
#StreamListener("input") #SendTo("output")
public KStream<?, String> process(KStream<?, String> payload) {
return payload.flatMapValues( //impl )
...} }
I've been working on updating a Flink processor (Flink version 1.9) that reads from Kafka and then writes to Kafka. We have written this processor to run towards a Kafka 0.10.2 cluster and now we have deployed a new Kafka cluster running version 2.2. Therefore I set out to update the processor to use the latest FlinkKafkaConsumer and FlinkKafkaProducer (as suggested by the Flink docs). However I've run into some problems with the Kafka producer. I'm unable to get it to Serialize data using deprecated constructors (not surprising) and I've been unable to find any implementations or examples online about how to implement a Serializer (all the examples are using older Kafka Connectors)
The current implementation (for Kafka 0.10.2) is as follows
FlinkKafkaProducer010<String> eventBatchFlinkKafkaProducer = new FlinkKafkaProducer010<String>(
"playerSessions",
new SimpleStringSchema(),
producerProps,
(FlinkKafkaPartitioner) null
);
When trying to implement the following FlinkKafkaProducer
FlinkKafkaProducer<String> eventBatchFlinkKafkaProducer = new FlinkKafkaProducer<String>(
"playerSessions",
new SimpleStringSchema(),
producerProps,
null
);
I get the following error:
Exception in thread "main" java.lang.NullPointerException
at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.<init>(FlinkKafkaProducer.java:525)
at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.<init>(FlinkKafkaProducer.java:483)
at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.<init>(FlinkKafkaProducer.java:357)
at com.ebs.flink.sessionprocessor.SessionProcessor.main(SessionProcessor.java:122)
and I haven't been able to figure out why.
The constructor for FlinkKafkaProducer is also deprecated and when I try implementing the non-deprecated constructor I can't figure out how to serialize the data.
The following is how it would look:
FlinkKafkaProducer<String> eventBatchFlinkKafkaProducer = new FlinkKafkaProducer<String>(
"playerSessions",
new KafkaSerializationSchema<String>() {
#Override
public ProducerRecord<byte[], byte[]> serialize(String s, #Nullable Long aLong) {
return null;
}
},
producerProps,
FlinkKafkaProducer.Semantic.EXACTLY_ONCE
);
But I don't understand how to implement the KafkaSerializationSchema and I find no examples of this online or in the Flink docs.
Does anyone have any experience implementing this or any tips on why the FlinkProducer gets NullPointerException in the step?
If you are just sending String to Kafka:
public class ProducerStringSerializationSchema implements KafkaSerializationSchema<String>{
private String topic;
public ProducerStringSerializationSchema(String topic) {
super();
this.topic = topic;
}
#Override
public ProducerRecord<byte[], byte[]> serialize(String element, Long timestamp) {
return new ProducerRecord<byte[], byte[]>(topic, element.getBytes(StandardCharsets.UTF_8));
}
}
For sending a Java Object:
import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonProcessingException;
import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper;
import org.apache.flink.streaming.connectors.kafka.KafkaSerializationSchema;
import org.apache.kafka.clients.producer.ProducerRecord;
public class ObjSerializationSchema implements KafkaSerializationSchema<MyPojo>{
private String topic;
private ObjectMapper mapper;
public ObjSerializationSchema(String topic) {
super();
this.topic = topic;
}
#Override
public ProducerRecord<byte[], byte[]> serialize(MyPojo obj, Long timestamp) {
byte[] b = null;
if (mapper == null) {
mapper = new ObjectMapper();
}
try {
b= mapper.writeValueAsBytes(obj);
} catch (JsonProcessingException e) {
// TODO
}
return new ProducerRecord<byte[], byte[]>(topic, b);
}
}
In your code
.addSink(new FlinkKafkaProducer<>(producerTopic, new ObjSerializationSchema(producerTopic),
params.getProperties(), FlinkKafkaProducer.Semantic.EXACTLY_ONCE));
To the deal with the timeout in the case of FlinkKafkaProducer.Semantic.EXACTLY_ONCE you should read https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/kafka.html#kafka-011-and-newer, particularly this part:
Semantic.EXACTLY_ONCE mode relies on the ability to commit transactions that were started before taking a checkpoint, after recovering from the said checkpoint. If the time between Flink application crash and completed restart is larger than Kafka’s transaction timeout there will be data loss (Kafka will automatically abort transactions that exceeded timeout time). Having this in mind, please configure your transaction timeout appropriately to your expected down times.
Kafka brokers by default have transaction.max.timeout.ms set to 15 minutes. This property will not allow to set transaction timeouts for the producers larger than it’s value. FlinkKafkaProducer011 by default sets the transaction.timeout.ms property in producer config to 1 hour, thus transaction.max.timeout.ms should be increased before using the Semantic.EXACTLY_ONCE mode.
I'm using JAVA/Spring MVC and I need to make a Connection Pool for a Third Party Application integration in my application becouse when i try to connect it multiple time my application and server System utilize 100% RAM.
here i have to problem, when users start to hit a specific method (callGenerationService()) multiple time, my Heap memory(RAM space) increases and becomes 100% and application going to slow becouse of it connect third party application multiple times ? here i need to create a connection only once and get it multiple times. where my connection like,
public class ClickToCallServiceImpl implements ClickToCallServiceInterface {
Client client = null;
#Override
public ClickToCall callGenerationService(ClickToCall clickToCall) {
client = new Client();
client.connect("127.0.0.1", 8021 , "password", 10); //Every time Connection Connect.
client.setEventSubscriptions("plain", "all");
// client.sendSyncApiCommand("",""); //here i run command on every hit like.
client.sendSyncApiCommand(clickToCall.command1, clickToCall.command2);
client.close();
}
}
and here 'ClickToCall' is a #Component Bean/POJO Class with variables setters and getters.
Is there, how to we create a connection (either pool or only once connect) for above connection where i connect only once and hit clickToCall.Command1 and clickToCall.Command2 multiple times and utilize less RAM? Thanks in advance.
Please note that I'm not an expert of freeswitch esl so you must check the code properly. Anyway this is what I would do.
First I create a Factory for Client
public class FreeSwitchEslClientFactory extends BasePooledObjectFactory<Client> {
#Override
public Client create() throws Exception {
//Create and connect: NOTE I'M NOT AN EXPERT OF ESL FREESWITCH SO YOU MUST CHECK IT PROPERLY
Client client = new Client();
client.connect("127.0.0.1", 8021 , "password", 10);
client.setEventSubscriptions("plain", "all");
return client;
}
#Override
public PooledObject<Client> wrap(Client obj) {
return new DefaultPooledObject<Client>(obj);
}
}
Then I create a shareable GenericObjectPool:
#Configuration
#ComponentScan(basePackages= {"it.olgna.spring.pool"})
public class CommonPoolConfig {
#Bean("clientPool")
public GenericObjectPool<Client> clientPool(){
GenericObjectPool<Client> result = new GenericObjectPool<Client>(new FreeSwitchEslClientFactory());
//Pool config e.g. max pool dimension
result.setMaxTotal(20);
return result;
}
}
Finally I use the created pool in order to get the Client obj:
#Component
public class FreeSwitchEslCommandSender {
#Autowired
#Qualifier("clientPool")
private GenericObjectPool<Client> pool;
public void sendCommand(String command, String param) throws Exception{
Client client = null;
try {
client = pool.borrowObject();
client.sendSyncApiCommand(command, param);
} finally {
if( client != null ) {
client.close();
}
pool.returnObject(client);
}
}
}
I didn't test (also because I can't) it but it should work. In any case I pray you to properly check the configuration. I don't know if it's OK to always create a Client object and connect or if it's better to connect when you want to send command
I hope it can be useful
EDIT INFORMATION
Sorry I made an error early. You must return the client to the pool
I updated my FreeSwitchEslCommandSender class
Angelo
I'd like to do something similar to the example posted in restlet's site (first applicaiton) - with one difference:
I want to stream data - not use primitive types - using an interface.
I want to define some kind of an interface between the client and the server, stream data between them and let restlet handle transmit the data seamlessly.
Example of what I have in mind:
interface Streaming {
InputStream startStream(String streamId);
}
When the client invokes a call, it starts reading from the inputstream. The server receives the call and starts providing the stream by creating an inputstream (for example, a video file, or just some raw data). Restlet should be reading from the inputstream on the server side and provide the data as an inputstream on the client side.
Any idea how can I achieve this? A code sample or link to one would be great. Thanks.
Below's an example code of what I learned so far - an interface with streaming capavilities and a client-server streaming example.
I haven't yet added parameters to the interface and it's only download - no upload yet.
Interface:
public interface DownloadResource {
public ReadableRepresentation download();
}
Interface with protocol: (separation between logic and technology):
public interface DownloadResourceProtocol extends DownloadResource {
#Get
#Override
public ReadableRepresentation download();
}
Client:
ClientResource cr = new ClientResource("http://10.0.2.2:8888/download/");
cr.setRequestEntityBuffering(true);
DownloadResource downloadResource = cr.wrap(DownloadResourceProtocol.class);
// Remote invocation - seamless:
Representation representation = downloadResource.download();
// Using data:
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
IOUtils.copy(representation.getStream(), byteArrayOutputStream);
byte[] byteArray = byteArrayOutputStream.toByteArray();
Log.i("Byte array: " + Arrays.toString(byteArray));
Server:
public class DownloadResourceImpl extends ServerResource implements DownloadResourceProtocol {
#Override
public ReadableRepresentation download() {
InputStreamChannel inputStreamChannel;
try {
inputStreamChannel = new InputStreamChannel(new ByteArrayInputStream(new byte[]{1,2,3,4,5,6,7,8,9,10}));
return new ReadableRepresentation(inputStreamChannel, MediaType.ALL);
} catch (IOException e) {
e.printStackTrace();
return null;
}
}
}
Configuration:
public class SampleApplication extends Application {
#Override
public Restlet createInboundRoot() {
Router router = new Router(getContext());
router.attach("/download/", DownloadResourceImpl.class);
return router;
}
}
Not sure this addresses your problem completely, but one approach is to create a thread that streams data back to the client using ReadableRepresentation and a Pipe.
Create a pipe:
Pipe pipe = Pipe.open();
Create a representation like this:
ReadableRepresentation r = new ReadableRepresentation(pipe.source(), mediatype);
Start a separate thread that writes batches of bytes to the pipe like this:
pipe.sink().write(ByteBuffer.wrap(someBytes));
return the representation to the client.