I am trying to understand the internal nuances of Kafka connect, like how the design has been implemented and which patterns are used.
specifically I want to understand how to develop similar app which can take input configuration and start acting according to the configuration, so when we have to implement some new feature we can just write connects. So others need not to spend more time in reinventing the wheel.
Apache Kafka is open source, and you can find the Connect source code here.
But if you just want to create a source connector, the internals will not really help you do that.
Confluent has a blog post on developing custom source connectors.
Alternatively, you could not "reinvent the wheel" yourself, and use projects like Apache Camel to see if they support a source system you're using.
Related
I am implementing OpenTok/Vonage into our android app in order to enable video conferencing. I'm looking for advice on getting started. For example recommendations on the kind of server I should deploy for the implementation. I notice that a limited functionality server can be deployed at Heroki, but what about something full functionality? Any examples of setup of one of those? Where do I deploy it? Can it operate on AWS or Azure? What spec should I use for the server? How do I implement and install the server? Again all things that aren't touched upon in the Vonage documentation. It's almost as if you should someone intuit it like arcana.
OpenTok Developer Advocate here.
There's a lot to unpack in that question(s). May I suggest asking in the Vonage Community Slack?
Quick answer on the Server SDKs: you can run all of the OpenTok Server SDK code with full functionality in any cloud, including Heroku/AWS/Azure/Google/etc.
As for specs of servers, that is something that is specific to your use cases, bandwidth, expected load, etc. There's not a good way to provide details there with the information you provided.
I am new to Apache Kafka and also to data analytics.
I am able to consume messages from Apache kafka consumer in Java.
Now I want to take these real time data and display it like a dashboard.
I want to visualize all these data using any open source tool.
One tool which I found is Druid but the documentation provided is not enough to learn and proceed with that.
Also I read that Druid is very difficult to install and deploy in production.
Any other tools available to do this? Any help is appreciated.
you can use Apache Zeppelin https://zeppelin.apache.org/ to visualise your kafka topics. It has a web interface for notebook and it supports java. You can write your code on it and visualise its output
I recently started using Metatron Discovery. https://metatron.app/
It is a free and open source software for data visualization.
It supports kafka, so you can visualize your real-time data with a wide variety of charts.
If you are interested in open source, this will be helpful.
https://github.com/metatron-app/metatron-discovery/
You could also use the Elastic stack. If you get your data with Kafka, then store it in Elasticsearch, you could quickly have a dashboard with Kibana. When I had to install and deploy it, I found it very easy to use.
I am using a embedded ActiveMQ in my application, the queue works excellent,
Now we want a way to be able to monitor this ActiveMQ, due to its embedded nature we cannot use the default web console provided by ActiveMQ.
I have had a look here http://activemq.apache.org/how-can-i-monitor-activemq.html , the provided options haven't helped much in my cause due to following reasons
Using JConsole is not a nice option because it uses up much of server resources and causes JVM to be slow.
StatisticsPlugin is a nice approach but doesn't provide a UI view, and gets reset on every server restart (This is what we will use if nothing else is found).
Have also had a look at similar question on SO ActiveMQ: how to programmatically monitor embedded broker , this is not what I want
Recently have heard about a tool called Hawtio , but this also seems to be useful when ActiveMq is running as standalone instance, (Please Correct If I am wrong on this, Any pointers will be definitely helpful)
So the help that I need is
Is Hawtio really useful for embedded instance ?
Are there any other tools availablle to achieve this goal?
Any help really appreciated.
I have used hawtio, its very useful tools for monitoring ActiveMQ. It used JMX internally which takes you java process running and monitor it.
In my web-app, i need to push specific messages to my clients in real time. Web-sockets would be a good idea, but they don't work in IE, which should be supported as well.
Until recently we have been using StreamHub, but it's license has expired and the project itself seems to be dead. We also considered jWebSockets, but they seem a bit of overkill.
Again, all we need to do is to send messages from the server to specific clients as events happen on the server.
Could you recomend a lightweight, free and opensource solution for that?
Here's an opensource Java implementation of Socket.IO. It allows you to use Ajax mulitpart streaming to simulate WebSockets when your client doesn't support them. I've used it on projects in the past. It's horribly out of date with the most recent implementation of the WebSocket protocol, so if you plan on using WebSocket capable browsers in addition to IE, you'll have to update the source code to comply with the latest protocol. At one point I was researching a maintained replacement for Java Socket.IO and jWebSockets sounded like the way to go.
Take a look at the Atmosphere WebSocket/Comet library. I believe it can use long polling when WebSockets aren't supported.
Any help would be great please..!
How to build an intranet chat application using java which includes transform of files like gtalk?
where to start the process?
what all the sw requirements?
The Eclipse Communication Framework includes several use cases based upon real-time shared editing over XMPP/Google Talk and Skype
And ECF is the base for a twitter client (as an example of what you can do with that framework)
you must have learned networking in java (java.net.*)
be sure of what king d of architecture you want to use
client/server
p2p
xmpp
If you need to deploy one, you can use Jabber with one of many Java Clients out there.
If you need to write your own server and/or client, you can start with this question and use an existing XMPP (IM protocol) libraries.
Why build one when good open source solutions based on open standards are available.
Take a look at running an Openfire XMPP (Jabber) server.
Theres a variety of clients available the best being:
Spark
Pidgin
I beleive that a flash client is also available.