I have two questions for which I couldn't find any popular/widely accepted solutions:
What is the easiest way to start zookeeper server using Java Program?
And, is it possible to add servers to zookeeper cluster without having to manually go to each machine and update their config files with new node's id and ip:port entry?
Can someone please help? Thanks!
If you want to start a new ZooKeeper server process from your Java code, you would do it the same way you would start any other external process from Java, e.g. using a ProcessBuilder. There is nothing special here in case of ZooKeeper. You can check the official docs on how the actual command should look like. It gets complicated if you want to supervise the process for production use, so in that case it would be better to use something provided on your OS (e.g. upstart, runit, etc...), or take a look at Exhibitor for code examples: https://github.com/Netflix/exhibitor.
If you are asking about starting a ZooKeeper cluster from your Java program, then you complicate things further, since you would basically need to supervise multiple ZooKeeper JVM processes on different hosts. Also take a look at Exhibitor.
If your question is about starting a ZooKeeper server instance inside the same JVM process as your Java code (embedded), then it is also possible. There are a few important details to keep in mind, take a look at this answer:
Is it possible to start a zookeeper server instance in process, say for unit tests?
Regarding your second question, real support for dynamic cluster reconfiguration was added just recently, in 3.5.0: http://zookeeper.apache.org/doc/trunk/zookeeperReconfig.html.
Prior to this, you can still "add servers to zookeeper cluster without having to manually go to each machine and update their config", but you have to use a configuration management tool like Chef, Puppet or similar, and in this case you would also need to do a restart of the cluster to uptake new config.
Related
We maintain our server once a week.
Sometimes, the customer wishes that we change some settings which is already cached in server.
My colleague always write some JSP code to change these settings which are stored in the memory.
Is it a good method to use this kind of methodology?
If our project is not a Web container, which tools can help me?
Usually, in my experience, the server configuration is not stored only in memory of server:
What happens that after a configuration change, the server has been restarted / just went down for some system reason?
What happens if you have more than one instance of the same server to work on (a cluster of servers in other words)?
So, usually, people opt for various "externalized configuration" options that can range from "file-based" configuration + redeploy the whole cluster upon each configuration change, to configuration management servers (like Consul, etc.d, etc). There are also some solutions that came from (and used in) a java world: Apache Zookeeper, Spring cloud config server to name a few, there are others. In addition, sometimes, it's convenient to store the configurations in a database.
Now to your question: If your project is not a web container and you don't care that configuration will "disappear" after a server restart and you're not running a distributed cluster of servers, then, using JSP indeed doesn't seem appropriate in this case.
Maybe you should take a look at JMX - Java management extensions, that have a built-in solution so that you probably will be able to get rid of a web container (which seems to be not used by your team anyway other than for JSP modifications that you've described).
You basically need in memory cache, there are multiple solutions found in answers which include creating your own implementation or using existing java library. You can also get data from database and add cache over the database layer.
I am creating a storm based project where messages will be filtered by storm. My aim is to allow a user to adapt the filtering performed at runtime by sending configuration information to a zookeeper Znode.
I believe this is possible by setting a zookeeper watcher up within storm but I am struggling to achieve this. I would be gratefull for some guidance or a simple example on how to perfrom this.
I have looked at the Java docs and afraid the way to perfrom this does not seem obvious
I'm working on an application which is comprised of many web containers with WAR-s installed on these containers.
Currently I have a farm of ~ 10 servers like this.
I'm about to start the integration of jolokia/hawtio to track the JMX mbeans exposed on each of these servers.
For this purpose (I think) I'll install a jolokia agent on each of these servers (just put the war into the deployment library.
Now I think to put hawtio on a dedicated node and connect to the remote agents deployed on the rest of my 10 servers.
My question is whether its possible to somehow give a list of predefined agents (host/port/credentials)?
I have a lot of farms to manage like this, I would prefer to use a predefined list generated per farm, rather than dealing with auto-discovery
Thanks in advance
No currently this is not supported, but a good idea. You are welcome to log a ticket about this.
https://github.com/hawtio/hawtio/issues
And also how would you like that list to be configured? Should you need to edit the web.xml and repackage the WAR, or how should that configuration be made easy? That is for something to think about.
We have a Spring Integration application which is polling a mongodb:inbound-channel-adapter like so:
<int-mongodb:inbound-channel-adapter channel="n2s.mongoResults"
collection-name="entities"
query="{_id: {$regex: 'mpl/objectives'}})">
<!-- Run every 15 minutes -->
<int:poller fixed-rate="900000"/>
</int-mongodb:inbound-channel-adapter>
Everything works fine. However, this application is deployed to a cluster and so multiple servers are running the same poller. We'd like to coordinate these servers such that only one runs the pipeline.
Of course, the servers don't know about each other, so we probably need to coordinate them through a locking mechanism in a database. Any suggestions on how to achieve this?
Notes:
We have access to both a MongoDB database and an Oracle database in this workflow. From the perspective of the workflow, it makes more sense to lock on the Oracle database.
It's fine if all server execute the polling step and then one server locks to actually process the records, if that's easier to achieve.
Any suggestions on how to achieve this?
You could use distributed locking tool like like Zookeeper. Another alternative would be to change from a simple fixed trigger to a scheduling framework like Quartz which will ensure that the job only executes on a single node.
It's fine if all server execute the polling step and then one server
locks to actually process the records, if that's easier to achieve.
Yea that's what I would do. I think it's by far the easiest approach. See this post for details on how to do locking with Oracle.
There are several options, including:
Set auto-startup="false" and use some management tool to monitor the servers and ensure that exactly one adapter is running (you can use a control-bus or JMX to start/stop the adapter.
Run the application in SpringXD containers; set the module count for the source (containing the mongo adapter) and the XD admin will make sure an instance is running. It uses Zookeeper to manage state.
Use a distributed lock to ensure only one instance processes messages. Spring Integration itself comes with a RedisLockRegistry for such things or you can use any distributed lock mechanism.
I did some research but i wasnt able to find an answer.
What i like to do is, changing ActiveMQ configuration while it is running.
Lets say I want to increase the memory usage, flow control etc. while activemq is running. But i dont want to restart it. Is this possible? I tried it but the changes didnt kick in.
is there a way to do this?
I do have C++ and java clients (producers and consumers respectively)
The ActiveMQ configuration can be changed at runtime via JMX. However, any changes that are made at runtime will not be persisted to the configuration file.
As mentioned by bsnyder it you can change broker config at runtime with JMX.
An easy way of doing this is to use JConsole to get to the broker's JMX Beans (instructions here), there is also a JMX API if you need to this through code.