I have been following the ehCache documentation regarding replication on https://www.ehcache.org/documentation/2.8/replication/rmi-replicated-caching.html
EhCache is already implemented in our project, but going through the code base, I am seeing at multiple places regarding remoteObjectPort being added in properties while configuring the listener.
Here is our code to add peer information to the peerProviderConfig
FactoryConfiguration peerProviderConfig = new FactoryConfiguration();
peerProviderConfig.setClass(RMICacheManagerPeerProviderFactory.class
.getName());
for (CacheHost remoteHost : remoteHosts) {
// create a RMI URL
rmiUrls += separator
createRmiUrl(remoteHost, cacheName);
separator = "|";
}
// and set it in configuration
peerProviderConfig.setProperties("peerDiscovery=manual, rmiUrls=" + rmiUrls);
I understand this piece of code. This will build and append the url for all the peers.
The createRmiUrl() method returns url as follows: //<hostname>:<hostport>/cacheName
For all the hosts, the hostport is same that is 40001. This means that on all the different hosts, ehCache will be running on port 40001. According to my understanding following things happen:
Whenever a peer comes alive, it will start the ehCache on port 40001
It will also configure all the peers in the cluster as we are using manual discovery
It will listen for some events and then update/ replicate its own data in the cache.
Is my understanding right?
Our PeerListenerProperties are as follows:
String peerListenerProps = "port=" + rmiPort;
peerListenerProps += ", remoteObjectPort=40002";
peerListenerProps += ", socketTimeoutMillis=" + socketTimeoutMillis;
FactoryConfiguration peerListenerConfig = new FactoryConfiguration();
peerListenerConfig.setClass(RMICacheManagerPeerListenerFactory.class.getName());
peerListenerConfig.setProperties(peerListenerProps);
So in the listener properties, we are adding port = rmiPort (which is 40001 and the same will be used by all the peers/hosts). What I don't understand is what is the usage of remoteObjectPort in the listener properties?
Related
I just started with a simple example to read the jmx metrics and used the simple values of HeapMemoryUsage, CPUTime etc to get the feel for it. I need to try and access the kafka server / consumer metrics specifically lag which I can see is visible as a key in the jconsole app under FetcherLagMetrics-ConsumerLag. But programmaticaly I get the following error:
javax.management.InstanceNotFoundException: kafka.consumer:type=consumer-
fetch-manager-metrics
This tells me that the consumer-fetch-manager metrics is the issue as it is not present even in the jconsole. I changed it to the following & still the same issue:
consumerBean = jmxCon.getMBeanServerConnection().getAttribute(new
ObjectName("kafka.server:type=FetcherLagMetrics"),"ConsumerLag");
cd = (CompositeData) consumerBean;
The code trying to access these values is as follows:
jmxCon.getMBeanServerConnection().invoke(new
ObjectName("java.lang:type=Memory"), "gc", null, null);
for (int i = 0; i < 100; i++) {
//get an instance of the HeapMemoryUsage Mbean
memoryMbean = jmxCon.getMBeanServerConnection().getAttribute(new ObjectName("java.lang:type=Memory"), "HeapMemoryUsage");
cd = (CompositeData) memoryMbean;
//get an instance of the OperatingSystem Mbean
osMbean = jmxCon.getMBeanServerConnection().getAttribute(new ObjectName("java.lang:type=OperatingSystem"),"ProcessCpuTime");
//get an instance of the kafka metrics Mbean
consumerBean = jmxCon.getMBeanServerConnection().getAttribute(new ObjectName("kafka.consumer:type=consumer-fetch-manager-metrics"),"MaxLag");
cd = (CompositeData) consumerBean;
consumerBeanII = jmxCon.getMBeanServerConnection().getAttribute(new ObjectName("kafka.server:type=FetcherLagMetrics,name=ConsumerLag"),"Lag");
System.out.println("Used memory: " + " " + cd.get("MaxLag") + " Used cpu: " + consumerBean); //print memory usage
tempMemory = tempMemory + Long.parseLong(cd.get("used").toString());
Thread.sleep(1000); //delay for one second
}
It fails at line consumerBean = ......Could someone please explain or provide the correct way to access the kafka metrics using jmx /JMI implementation?
I'm not sure your version of Kafka, but when I look at mine (v1.1.0) using JConsole there is no such bean for consumer lag that you describe. Therefore I think it is expected that your JMX query will fail.
Instead, I believe this kind of information was moved to the Kafka Admin API interface. So you must use this now to get lag settings, if you don't want to use kafka-consumer-groups.sh command-line utility.
For us we only need these stats for monitoring, so we are using https://github.com/danielqsj/kafka_exporter to obtain the info for Prometheus.
I have problem with vertx HttpClient.
Here's code which shows that tests GET using vertx and plain java.
Vertx vertx = Vertx.vertx();
HttpClientOptions options = new HttpClientOptions()
.setTrustAll(true)
.setSsl(false)
.setDefaultPort(80)
.setProtocolVersion(HttpVersion.HTTP_1_1)
.setLogActivity(true);
HttpClient client = vertx.createHttpClient(options);
client.getNow("google.com", "/", response -> {
System.out.println("Received response with status code " + response.statusCode());
});
System.out.println(getHTML("http://google.com"));
Where getHTML() is from here: How do I do a HTTP GET in Java?
This is my output:
<!doctype html><html... etc <- correct output from plain java
Feb 08, 2017 11:31:21 AM io.vertx.core.http.impl.HttpClientRequestImpl
SEVERE: java.net.UnknownHostException: failed to resolve 'google.com'. Exceeded max queries per resolve 3
But vertx can't connect. What's wrong here? I'm not using any proxy.
For reference: a solution, as described in this question and in tsegismont's comment here, is to set the flag vertx.disableDnsResolver to true:
-Dvertx.disableDnsResolver=true
in order to fall back to the JVM DNS resolver as explained here:
sometimes it can be desirable to use the JVM built-in resolver, the JVM system property -Dvertx.disableDnsResolver=true activates this behavior
I observed this DNS resolution issue with a redis client in a kubernetes environment.
I had this issue, what caused it for me was stale DNS servers being picked up by the Java runtime, i.e. servers registered for a network the machine was no longer connected to. The issue is first in the Sun JNDI implementation, it also exists in Netty which uses JNDI to bootstrap its list of name servers on most platforms, then finally shows up in VertX.
I think a good place to fix this would be in the Netty layer where the set of default DNS servers is bootstrapped. I have raised a ticket with the Netty project so we'll see if they agree with me! Here is the Netty ticket
In the mean time a fairly basic workaround is to filter the default DNS servers detected by Netty, based on whether they are reachable or not. Here is a code Sample in Kotlin to apply before constructing the main VertX instance.
// The default set of name servers provided by JNDI can contain stale entries
// This default set is picked up by Netty and in turn by VertX
// To work around this, we filter for only reachable name servers on startup
val nameServers = DefaultDnsServerAddressStreamProvider.defaultAddressList()
val reachableNameServers = nameServers.stream()
.filter {ns -> ns.address.isReachable(NS_REACHABLE_TIMEOUT)}
.map {ns -> ns.address.hostAddress}
.collect(Collectors.toList())
if (reachableNameServers.size == 0)
throw StartupException("There are no reachable name servers available")
val opts = VertxOptions()
opts.addressResolverOptions.servers = reachableNameServers
// The primary Vertx instance
val vertx = Vertx.vertx(opts)
A little more detail in case it is helpful. I have a company machine, which at some point was connected to the company network by a physical cable. Details of the company's internal name servers were set up by DHCP on the physical interface. Using the wireless interface at home, DNS for the wireless interface gets set to my home DNS while the config for the physical interface is not updated. This is fine since that device is not active, ipconfig /all does not show the internal company DNS servers. However, looking in the registry they are still there:
Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\Interfaces
They get picked up by the JNDI mechanism, which feeds Netty and in turn VertX. Since they are not reachable from my home location, DNS resolution fails. I can imagine this home/office situation is not unique to me! I don't know whether something similar could occur with multiple virtual interfaces on containers or VMs, it could be worth looking at if you are having problems.
Here is the sample code which works for me.
public class TemplVerticle extends HttpVerticle {
public static void main(String[] args) {
Vertx vertx = Vertx.vertx();
// Create the web client and enable SSL/TLS with a trust store
WebClient client = WebClient.create(vertx,
new WebClientOptions()
.setSsl(true)
.setTrustAll(true)
.setDefaultPort(443)
.setKeepAlive(true)
.setDefaultHost("www.w3schools.com")
);
client.get("www.w3schools.com")
.as(BodyCodec.string())
.send(ar -> {
if (ar.succeeded()) {
HttpResponse<String> response = ar.result();
System.out.println("Got HTTP response body");
System.out.println(response.body().toString());
} else {
ar.cause().printStackTrace();
}
});
}
}
Try using web client instead of httpclient, here you have an example (with rx):
private val client: WebClient = WebClient.create(vertx, WebClientOptions()
.setSsl(true)
.setTrustAll(true)
.setDefaultPort(443)
.setKeepAlive(true)
)
open fun <T> get(uri: String, marshaller: Class<T>): Single<T> {
return client.getAbs(host + uri).rxSend()
.map { extractJson(it, uri, marshaller) }
}
Another option is to use getAbs.
Properties props = System.getProperties();
props.put("mail.imap.connectiontimeout",5000);
Session session = Session.getInstance(props);
Store store = session.getStore("imap");
for(50K users){
//login,password changed in loop
String[] folders = {"inbox", "f1", "f2", "f3", "spam"};
store.connect(serverAddress, login + emailSuffix, password);
for (int i = 0; i < folders.length; i++) {
Folder x = store.getFolder(folders[i]);
x.open(Folder.READ_ONLY);
System.out.println("folder " + folders[i] + " of " + login);
x.getUnreadMessageCount();
x.close(false);
}
store.close();
}
I'm using same store for all connections, changed service_count in dovecot according to this answer in order to improve imap-dovecot performance but I see only first iteration and after that code hangs or does next system.out after long time.
Actually, I need to grab all old messages of all users + count all unread messages as I want to migrate from pure Java Mail to some custom format. I didn;t manage even to just iterate over all users and folders for each user because even simple store.connect hangs after 1-st iteration!
I personally think that bottleneck is my dovecot config, but it uses default limits (1000 connections) which looks good.
My I somehow improve my dovecot or connect my store only once for all users or somehow fetch all messages of all users and unreadMessagesCount of all users in other way?
PS. The only alternative to programmatic way is some bash script in maildir which whill read each message from file system and pass it to some rest which converts to my custom format) but it much more harder than Java it's too difficult to parse smptp, parse seen flags from file name and so on.
UPDATE
I found apache commons net imapclient which works very fast.
<dependency>
<groupId>commons-net</groupId>
<artifactId>commons-net</artifactId>
<version>3.3</version>
</dependency>
My code is the following
IMAPClient client = new IMAPClient();
client.connect("localhost");
for(50K users){
client.login(login + emailSuffix, password);
for (int i = 0; i < folders.length; i++) {
System.out.println(client.select("INBOX"); //prints true, it's ok
}
}
Looks like it connects faster than java mailapi because it may
connect once to host and after that login for each user. May I somehow repeat such behavior in JavaMail API?
How may I grab messages with apache commons client? All methods return boolean or void, so it's looks like just server checking library am I right? Is it possible to somehow get useful info from imapclient?
Finally solved my problem by simple iterating over file system (I have maildir format).
I guess Java Mail API makes new dovecot auth for each user in store.connect while it should just connect once (consume dovecot auth) and after that login for each user (consume dovecot imap-login). That's why I waited 1 minute for each iteration - it's std idle for auth process in dovecot config. I'm not sure but looks so.
Apache lib works fast but it's just testing library for pinging server, checking connection and other imap operations. It returns boolean result about operations but not useful information(
Background:
We have 4 physical servers (4 IPS), each one running in JBOSS 6 EAP running on port 80.All requests are redirected to any one of these servers via Load balancer.
Now I tried to implement Java cache system for such distributed env so that our properties gets updated in each servers cache.
POC:
For that we did a small POC on our local systems implementing JCS v1.3 lateral caching.
Enabled it in our maven project. The following config is used in .ccf file :
jcs.default=
jcs.default.cacheattributes=org.apache.jcs.engine.CompositeCacheAttributes
jcs.default.cacheattributes.MaxObjects=1000
jcs.default.cacheattributes.MemoryCacheName=org.apache.jcs.engine.memory.lru.LRUMemoryCache
# PRE-DEFINED CACHE REGION
##############################################################
##### AUXILIARY CACHES
# LTCP AUX CACHE
jcs.auxiliary.LTCP=org.apache.commons.jcs.auxiliary.lateral.socket.tcp.LateralTCPCacheFactory
jcs.auxiliary.LTCP.attributes=org.apache.commons.jcs.auxiliary.lateral.socket.tcp.TCPLateralCacheAttributes
#jcs.auxiliary.LTCP.attributes.TcpServers=152.144.219.209:8080
jcs.auxiliary.LTCP.attributes.TcpListenerPort=1118
jcs.auxiliary.LTCP.attributes.UdpDiscoveryAddr=228.5.6.8
jcs.auxiliary.LTCP.attributes.UdpDiscoveryPort=6780
jcs.auxiliary.LTCP.attributes.UdpDiscoveryEnabled=true
jcs.auxiliary.LTCP.attributes.Receive=true
jcs.auxiliary.LTCP.attributes.AllowGet=true
jcs.auxiliary.LTCP.attributes.IssueRemoveOnPut=false
jcs.auxiliary.LTCP.attributes.FilterRemoveByHashCode=false
jcs.auxiliary.LTCP.attributes.SocketTimeoOt=1001
jcs.auxiliary.LTCP.attributes.OpenTimeOut=2002
jcs.auxiliary.LTCP.attributes.ZombieQueueMaxSize=2000
And implementing the getter and setter methods for saving a string attribute in cache and getting it from cache
public void addProp(String propId)
throws PimsAppException {
try {
configMSCache.put(propId, propId);
} catch (CacheException e) {
e.printStackTrace();
}
}
#Override
public String testProp(String propId) throws PimsAppException {
if(configMSCache!=null){
return (String) configMSCache.get(propId);
}else{
return "It dint work";
}
}
The application is deployed fine no error in getting it up.
TEST METHOD:
deployed the project.war in my local server and in a remote server with different IP. Both machines are in same network, so no firewall issue in accessing each others IP.
Have saved a property in my server and get it. (Worked fine)
Tried to get the saved property via my local by the remote machine. (It returns blank response).
Means the distributed cache feature is NOT achieved.
Doubts :
1. Does the auxiliary caches set up properly? I mean the configurations
2. Am I testing it properly or how can I test it in dev environment.
3. As JCS UDP Discovery,lets us support the same config on multiple machines, then why it dint work on remote machine?
4. Or is there any caching mechanism, with good examples and documentation can suffice my application needs(as mentioned in background section).
Thanks in advance.
This reply might be too late. But I will suggest in case, to log the stats on both servers and see. As could be possible that it is propagating the cache but just in the processing time, there is an issue reading it.
For example:
JCSAdminBean admin = new JCSAdminBean();
LinkedList linkedList = admin.buildCacheInfo();
ListIterator iterator = linkedList.listIterator();
while (iterator.hasNext()) {
CacheRegionInfo info = (CacheRegionInfo)iterator.next();
CompositeCache compCache = info.getCache();
System.out.println("Cache Name: " + compCache.getCacheName());
System.out.println("Cache Type: " + compCache.getCacheType());
System.out.println("Cache Misses (not found): " + compCache.getMissCountNotFound());
System.out.println("Cache Misses (expired): " + compCache.getMissCountExpired());
System.out.println("Cache Hits (memory): " + compCache.getHitCountRam());
System.out.println("Cache value: " + compCache.get(propId));
}
I am trying to write a client utility that to connect to Tomcat via JMX and look at the status of the connection datasource.
I set the following VM arguments in $CATALINA_HOME/bin/setenv.bat and restarted Tomcat
set JAVA_OPTS=-Xms512M -Xmx1024M -XX:MaxPermSize=256M %JAVA_OPTS%
set CATALINA_OPTS=-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9004 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false %CATALINA_OPTS%
I am not very familiar with JMX so i am just having a play with it to get the feel of it.
The utility i am writing will be running outside of Tomcat. I wrote the following test to try and access datasource Mbean object in Tomcat
but for some reason it is not finding it.
public class GuiMonitor {
public static void main(String[] args){
try{
JMXServiceURL url = new JMXServiceURL(
"service:jmx:rmi:///jndi/rmi://localhost:9004/jmxrmi");
JMXConnector jmxc = JMXConnectorFactory.connect(url, null);
final List<MBeanServer> servers = new LinkedList<MBeanServer>();
servers.add(ManagementFactory.getPlatformMBeanServer());
servers.addAll(MBeanServerFactory.findMBeanServer(null));
System.out.println("MbeanServers " + servers.size());
for(final MBeanServer server : servers){
System.out.println("Server : " + server.getClass().getName());
}
MBeanServer mbsc = ManagementFactory.getPlatformMBeanServer();
System.out.println(mbsc.queryMBeans(null, null));
ObjectName on = new ObjectName("Catalina:type=DataSource,path=/appdb,host=localhost,class=javax.sql.DataSource,name=\"jdbc/appdb\"");
System.out.println("ObjectName : " + on.toString());
System.out.println(mbsc.getAttribute(on, "Catalina:type=DataSource,path=/appdb,host=localhost,class=javax.sql.DataSource,name=\"jdbc/appdb\""));
} catch (Exception e) {
e.printStackTrace();
}
}
}
I have a JSP page that i found on the internet which when i upload onto the webapps folder and run it, it displays all of the available
MBeans in Tomcat. The object string/name i used above came from the name that was reported on both the jsp page i used and Jconsole so it does exist.
The output to the above program is shown below
MbeanServers 2
Server : com.sun.jmx.mbeanserver.JmxMBeanServer
Server : com.sun.jmx.mbeanserver.JmxMBeanServer
[com.sun.management.OperatingSystem[java.lang:type=OperatingSystem], sun.management.MemoryPoolImpl[java.lang:type=MemoryPool,name=Tenured Gen], sun.management.MemoryPoolImpl[java.lang:type=MemoryPool,name=Perm Gen], java.util.logging.Logging[java.util.logging:type=Logging], sun.management.CompilationImpl[java.lang:type=Compilation], javax.management.MBeanServerDelegate[JMImplementation:type=MBeanServerDelegate], sun.management.MemoryImpl[java.lang:type=Memory], sun.management.MemoryPoolImpl[java.lang:type=MemoryPool,name=Survivor Space], sun.management.RuntimeImpl[java.lang:type=Runtime], sun.management.GarbageCollectorImpl[java.lang:type=GarbageCollector,name=Copy], sun.management.MemoryPoolImpl[java.lang:type=MemoryPool,name=Eden Space], sun.management.GarbageCollectorImpl[java.lang:type=GarbageCollector,name=MarkSweepCompact], sun.management.ThreadImpl[java.lang:type=Threading], sun.management.MemoryPoolImpl[java.lang:type=MemoryPool,name=Perm Gen [shared-ro]], sun.management.MemoryPoolImpl[java.lang:type=MemoryPool,name=Perm Gen [shared-rw]], sun.management.HotSpotDiagnostic[com.sun.management:type=HotSpotDiagnostic], sun.management.ClassLoadingImpl[java.lang:type=ClassLoading], sun.management.MemoryManagerImpl[java.lang:type=MemoryManager,name=CodeCacheManager], sun.management.MemoryPoolImpl[java.lang:type=MemoryPool,name=Code Cache]]
ObjectName : Catalina:type=DataSource,path=/appdb,host=localhost,class=javax.sql.DataSource,name="jdbc/appdb"
javax.management.InstanceNotFoundException: Catalina:type=DataSource,path=/appdb,host=localhost,class=javax.sql.DataSource,name="jdbc/appdb"
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1094)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:662)
at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638)
at com.bt.c21.c21mon.C21GuiMonitor.main(C21GuiMonitor.java:39)
A couple of questions
Is the URL correct? I know the port number is correct but i am not sure of the service name. The service name "jmxrmi" i am using on the URL is just one that i saw in one of the examples i have been looking at.
I have a feeling that this is connecting to a different MBeanServer. I suspect this because if you look at the output of mbsc.queryMBeans(null, null), there is nothing tomcat specific. What service name do i use for the Tomcat instance?
If the URL is correct then is the service name always jmxrmi? And why does it not find the "Catalina:type=DataSource,path=/appdb,host=localhost,class=javax.sql.DataSource,name=\"jdbc/appdb\"" entry?
I have seen a lot of examples of how to do this and most use a different method to get teh MbeanServer. A couple of examples i have seen are
ManagementFactory.getPlatformMBeanServer()
MBeanServerFactory.findMBeanServer(null)
getMBeanServerConnection()
As mentioned earlier, the utility i am writing is a normal java application that will be running outside of tomcat. Is there any other configuration that i have missed out? I have been looking at several examples and the majority talk about creating MBeans and there is usually references to Listeners. As i am not creating any new Mbeans but only reading the values of existing ones, do i need to configure a listener?
Edit
It seems that getPlatformMbeanServer() is not returning the correct JVM Instance. I tried the following
MBeanServerConnection conn = jmxc.getMBeanServerConnection();
System.out.println("Query2 : " + conn.queryMBeans(null, null));
And this does return some Tomcat specific values. But i am still unable to get the jdbc/appdb datasource.
krtek - I wont be able to use JMX Console as i plan to do it all manually with the intention of automating it.
Edit 2
Ok, i figured out what i was doing wrong. Initially i was trying to retrieve the values as
MBeanServerConnection conn = jmxc.getMBeanServerConnection();
ObjectName on = new ObjectName("Catalina:type=DataSource,path=/appdb,host=localhost,class=javax.sql.DataSource,name=\"jdbc/appdb\"");
mbsc.getAttribute(on, "Catalina:type=DataSource,path=/appdb,host=localhost,class=javax.sql.DataSource,name=\"jdbc/appdb\""));
The above is wrong because the second parameter for mbsc.getAttribute is supposed to be the attribute in the Mbean not the String name.
This gave me the correct attribute values
MBeanServerConnection conn = jmxc.getMBeanServerConnection();
ObjectName on = new ObjectName("Catalina:type=DataSource,path=/appdb,host=localhost,class=javax.sql.DataSource,name=\"jdbc/appdb\"");
mbsc.getAttribute(on, "numIdle")
And i also changed the MBeanServer i was using from getPlatformMbeanServer() to getMBeanserverConnection(). I must admit i dont quite understand the difference because since Tomcat is running on the same JVM as the one returned by getPlatformMbeanServer(). Does it mean that getPlatformMbeanServer() will only return sun specific Mbeans? and getMBeanserverConnection() will include both?
Thanks
That's because you are getting instance of JMX server for your client JVM, not the Tomcat one.
This is right:
JMXServiceURL url = new JMXServiceURL(
"service:jmx:rmi:///jndi/rmi://localhost:9004/jmxrmi");
JMXConnector jmxc = JMXConnectorFactory.connect(url, null);
But you should continue with something like:
MBeanServerConnection conn = jmxc.getMBeanServerConnection();
Set result = conn.queryMBeans(null,
"Catalina:type=DataSource,path=/appdb,host=localhost,class=javax.sql.DataSource");
To test your query string use some tool like JMX console.