Develop Reference

What is remoteObjectPort in enCache

I have been following the ehCache documentation regarding replication on https://www.ehcache.org/documentation/2.8/replication/rmi-replicated-caching.html
EhCache is already implemented in our project, but going through the code base, I am seeing at multiple places regarding remoteObjectPort being added in properties while configuring the listener.
Here is our code to add peer information to the peerProviderConfig
FactoryConfiguration peerProviderConfig = new FactoryConfiguration();
peerProviderConfig.setClass(RMICacheManagerPeerProviderFactory.class
.getName());
for (CacheHost remoteHost : remoteHosts) {
// create a RMI URL
rmiUrls += separator
createRmiUrl(remoteHost, cacheName);
separator = "|";
}
// and set it in configuration
peerProviderConfig.setProperties("peerDiscovery=manual, rmiUrls=" + rmiUrls);
I understand this piece of code. This will build and append the url for all the peers.
The createRmiUrl() method returns url as follows: //<hostname>:<hostport>/cacheName
For all the hosts, the hostport is same that is 40001. This means that on all the different hosts, ehCache will be running on port 40001. According to my understanding following things happen:
Whenever a peer comes alive, it will start the ehCache on port 40001
It will also configure all the peers in the cluster as we are using manual discovery
It will listen for some events and then update/ replicate its own data in the cache.
Is my understanding right?
Our PeerListenerProperties are as follows:
String peerListenerProps = "port=" + rmiPort;
peerListenerProps += ", remoteObjectPort=40002";
peerListenerProps += ", socketTimeoutMillis=" + socketTimeoutMillis;
FactoryConfiguration peerListenerConfig = new FactoryConfiguration();
peerListenerConfig.setClass(RMICacheManagerPeerListenerFactory.class.getName());
peerListenerConfig.setProperties(peerListenerProps);
So in the listener properties, we are adding port = rmiPort (which is 40001 and the same will be used by all the peers/hosts). What I don't understand is what is the usage of remoteObjectPort in the listener properties?

Test containers: ignore parent `EXPOSE` instruction from Dockerfile

I'm trying to run Couchbase v.5.1.1 docker container for test purposes via Test container with fixed exposed ports, like:
trait CouchbaseTestEnvironment extends ForAllTestContainer {
this: Suite =>
def couchbaseContainer: FixedHostPortGenericContainer = {
val consumer = new Slf4jLogConsumer(LoggerFactory.getLogger(getClass))
/*
* Couchbase should know which ports are exposed for client, because this is how it exposes services.
* E.g. client ask only for on port - say 8091. And query service port is 8093. So client, won't ask for every port,
* instead CB will tell client on which port query service exposed, that's why CB should be aware about port mapping.
* That's why we need to give CB port mappings
*
* See for more details:
* https://stackoverflow.com/questions/59277436/couchbase-in-docker-for-integration-tests-make-the-ports-8092-8093-8094-and-8
*/
def randomPort: Int = {
val (from, to) = (32768, 35000) //linux private port range
from + Random.nextInt(to - from)
}
val random8091Port = randomPort
val random8092Port = randomPort
val random8093Port = randomPort
val random8094Port = randomPort
val random11210Port = randomPort
val container = FixedHostPortGenericContainer(
imageName = "couchbase:community-5.0.1",
exposedHostPort = random8091Port,
exposedContainerPort = random8091Port,
env = Map(
"COUCHBASE_RANDOM_PORT_8091" -> random8091Port.toString,
"COUCHBASE_RANDOM_PORT_8092" -> random8092Port.toString,
"COUCHBASE_RANDOM_PORT_8093" -> random8093Port.toString,
"COUCHBASE_RANDOM_PORT_8094" -> random8094Port.toString,
"COUCHBASE_RANDOM_PORT_11210" -> random11210Port.toString
)
)
container.container.withFixedExposedPort(random8092Port, random8092Port)
container.container.withFixedExposedPort(random8093Port, random8093Port)
container.container.withFixedExposedPort(random8094Port, random8094Port)
container.container.withFixedExposedPort(random11210Port, random11210Port)
container.container.withLogConsumer(consumer)
container
}
}
So as you can see 5 FIXED ports should be exposed.
But, when I'm running tests I actually can see, that instead other ports exposed with random ports:
docker ps
f4fc1ce06544 couchbase:community-5.0.1 "/entrypoint.sh /opt…" 59 seconds ago Up 1 second 0.0.0.0:55264->8091/tcp, 0.0.0.0:55263->8092/tcp, 0.0.0.0:55262->8093/tcp, 0.0.0.0:55261->8094/tcp, 0.0.0.0:55260->11207/tcp, 0.0.0.0:55259->11210/tcp, 0.0.0.0:55258->11211/tcp, 0.0.0.0:55257->18091/tcp, 0.0.0.0:55256->18092/tcp, 0.0.0.0:55255->18093/tcp, 0.0.0.0:55254->18094/tcp unruffled_mendel
03b491ac2ea8 testcontainersofficial/ryuk:0.3.0
So as you can see another ports was exposed, and mapped to random ports instead fixed.
As far as I understand, test containers, ignores ports I gave, and instead exposes ports from Couchbase Dockerfile: https://github.com/couchbase/docker/blob/master/community/couchbase-server/5.1.1/Dockerfile#L74
EXPOSE 8091 8092 8093 8094 8095 8096 11207 11210 11211 18091 18092 18093 18094 18095 18096
Can I somehow force Test containers to ignore EXPOSE instruction?
Partially helped question: Couchbase in docker for integration tests: Make the ports 8092, 8093, 8094 and 8095 configurable to be able to use docker’s random ports

Can I somehow force Test containers to ignore EXPOSE instruction?
I don't know if there is a simple configuration option for this, but a workaround solution I found is to use an advanced feature of the docker-java create container command customization. I'm providing an example in Java, translate it to Scala yourself, please. Apply it as the last command before returning a container object from your function:
container.withCreateContainerCmdModifier(
cmd -> cmd.getHostConfig().withPublishAllPorts(false)
);
The main point here is the usage of .withPublishAllPorts(false). From my understanding, this is the same as --publish-all (or -P) arguments of the docker run command. Testcontainers library sets this value to true by default. This modification overrides it to false.
With this configuration no ports are published at all for your example, not the 5 fixed as expected:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
2ee4fb91b97c couchbase:community-5.0.1 "/entrypoint.sh couc…" 33 seconds ago Up 32 seconds 8091-8094/tcp, 11207/tcp, 11210-11211/tcp, 18091-18094/tcp trusting_keldysh
This is because in the answer you provided, the author created a special custom docker image of couchbase, which "understands" environment variables like COUCHBASE_RANDOM_PORT_8091. Your code uses the standard couchbase image couchbase:community-5.0.1, which basically just ignores these environment variables. So in order to run counchbase on not standard internal ports, you need to build a custom image with the "magic" configure-node.sh script, which tunes couchbase config using values provided in environment variables.
I hope it helps anyhow :)

JCS LTCP AUX cache configuration and usage

Background:
We have 4 physical servers (4 IPS), each one running in JBOSS 6 EAP running on port 80.All requests are redirected to any one of these servers via Load balancer.
Now I tried to implement Java cache system for such distributed env so that our properties gets updated in each servers cache.
POC:
For that we did a small POC on our local systems implementing JCS v1.3 lateral caching.
Enabled it in our maven project. The following config is used in .ccf file :
jcs.default=
jcs.default.cacheattributes=org.apache.jcs.engine.CompositeCacheAttributes
jcs.default.cacheattributes.MaxObjects=1000
jcs.default.cacheattributes.MemoryCacheName=org.apache.jcs.engine.memory.lru.LRUMemoryCache
# PRE-DEFINED CACHE REGION
##############################################################
##### AUXILIARY CACHES
# LTCP AUX CACHE
jcs.auxiliary.LTCP=org.apache.commons.jcs.auxiliary.lateral.socket.tcp.LateralTCPCacheFactory
jcs.auxiliary.LTCP.attributes=org.apache.commons.jcs.auxiliary.lateral.socket.tcp.TCPLateralCacheAttributes
#jcs.auxiliary.LTCP.attributes.TcpServers=152.144.219.209:8080
jcs.auxiliary.LTCP.attributes.TcpListenerPort=1118
jcs.auxiliary.LTCP.attributes.UdpDiscoveryAddr=228.5.6.8
jcs.auxiliary.LTCP.attributes.UdpDiscoveryPort=6780
jcs.auxiliary.LTCP.attributes.UdpDiscoveryEnabled=true
jcs.auxiliary.LTCP.attributes.Receive=true
jcs.auxiliary.LTCP.attributes.AllowGet=true
jcs.auxiliary.LTCP.attributes.IssueRemoveOnPut=false
jcs.auxiliary.LTCP.attributes.FilterRemoveByHashCode=false
jcs.auxiliary.LTCP.attributes.SocketTimeoOt=1001
jcs.auxiliary.LTCP.attributes.OpenTimeOut=2002
jcs.auxiliary.LTCP.attributes.ZombieQueueMaxSize=2000
And implementing the getter and setter methods for saving a string attribute in cache and getting it from cache
public void addProp(String propId)
throws PimsAppException {
try {
configMSCache.put(propId, propId);
} catch (CacheException e) {
e.printStackTrace();
}
}
#Override
public String testProp(String propId) throws PimsAppException {
if(configMSCache!=null){
return (String) configMSCache.get(propId);
}else{
return "It dint work";
}
}
The application is deployed fine no error in getting it up.
TEST METHOD:
deployed the project.war in my local server and in a remote server with different IP. Both machines are in same network, so no firewall issue in accessing each others IP.
Have saved a property in my server and get it. (Worked fine)
Tried to get the saved property via my local by the remote machine. (It returns blank response).
Means the distributed cache feature is NOT achieved.
Doubts :
1. Does the auxiliary caches set up properly? I mean the configurations
2. Am I testing it properly or how can I test it in dev environment.
3. As JCS UDP Discovery,lets us support the same config on multiple machines, then why it dint work on remote machine?
4. Or is there any caching mechanism, with good examples and documentation can suffice my application needs(as mentioned in background section).
Thanks in advance.

This reply might be too late. But I will suggest in case, to log the stats on both servers and see. As could be possible that it is propagating the cache but just in the processing time, there is an issue reading it.
For example:
JCSAdminBean admin = new JCSAdminBean();
LinkedList linkedList = admin.buildCacheInfo();
ListIterator iterator = linkedList.listIterator();
while (iterator.hasNext()) {
CacheRegionInfo info = (CacheRegionInfo)iterator.next();
CompositeCache compCache = info.getCache();
System.out.println("Cache Name: " + compCache.getCacheName());
System.out.println("Cache Type: " + compCache.getCacheType());
System.out.println("Cache Misses (not found): " + compCache.getMissCountNotFound());
System.out.println("Cache Misses (expired): " + compCache.getMissCountExpired());
System.out.println("Cache Hits (memory): " + compCache.getHitCountRam());
System.out.println("Cache value: " + compCache.get(propId));
}

Get ip of consumers who connected to IBM MQ queue by PCF request from Java

I know that MQExplorer GUI could show who connected to some queue by some channel and other information about this connection, but I didn't find anything in PCF to do that from Java.
Thanks in advance for commands and examples if they exist!

This shows sample snippet shows how to retrieve the conname.
// Create the PCF message type for the inquire.
PCFMessage pcfCmd = new PCFMessage(MQConstants.MQCMD_INQUIRE_Q_STATUS);
// Add queue name
pcfCmd.addParameter(MQConstants.MQCA_Q_NAME, "MYQ");
// We want Q HANDLE attributes
pcfCmd.addParameter(MQConstants.MQIACF_Q_STATUS_TYPE, MQConstants.MQIACF_Q_HANDLE);
// We want to retrieve only the connection name
pcfCmd.addParameter(MQConstants.MQIACF_Q_STATUS_ATTRS, MQConstants.MQCACH_CONNECTION_NAME);
// Execute the command. The returned object is an array of PCF messages.
PCFMessage[] pcfResponse = pcfCM.agent.send(pcfCmd);
try{
for(int i = 0; i < pcfResponse.length;i++){
String name = (String) pcfResponse[i].getParameterValue(MQConstants.MQCACH_CONNECTION_NAME);
System.out.println("Connection Name: " + name);
}
}catch(Exception ex) {
System.out.print(ex);
}
You can modify the snippet per your need. Hope it helps.

What you are seeing in MQ Explorer is the MQSC command:
DISPLAY QSTATUS(<QName>) TYPE(HANDLE)
So, you need to look up the equivalent PCF command.

How to programmatically get all running jobs in a Hadoop cluster using the new API?

I have a software component which submits MR jobs to Hadoop. I now want to check if there are other jobs running before submitting it. I found out that there is a Cluster object in the new API which can be used to query the cluster for running jobs, get their configurations and extract the relevant information from them. However I am having problems using this.
Just doing new Cluster(conf) where conf is a valid Configuration which can be used to access this cluster (e.g., to submit jobs to it) leaves the object unconfigured, and the getAllJobStatuses() method of Cluster returns null.
Extracting mapreduce.jobtracker.address from the configuration, constructing an InetSocketAddress from it and using the other constructor of Cluster throws Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses..
Using the old api, doing something like new JobClient(conf).getAllJobs() throws an NPE.
What am I missing here? How can I programmatically get the running jobs?

I investigated even more, and I solved it. Thomas Jungblut was right, it was because of the mini cluster. I used the mini cluster following this blog post which turned out to work for MR jobs, but set up the mini cluster in a deprecated way with an incomplete configuration. The Hadoop Wiki has a page on how to develop unit tests which also explains how to correctly set up a mini cluster.
Essentially, I do the mini cluster setup the following way:
// Create a YarnConfiguration for bootstrapping the minicluster
final YarnConfiguration bootConf = new YarnConfiguration();
// Base directory to store HDFS data in
final File hdfsBase = Files.createTempDirectory("temp-hdfs-").toFile();
bootConf.set(MiniDFSCluster.HDFS_MINIDFS_BASEDIR, hdfsBase.getAbsolutePath());
// Start Mini DFS cluster
final MiniDFSCluster hdfsCluster = new MiniDFSCluster.Builder(bootConf).build();
// Configure and start Mini MR YARN cluster
bootConf.setInt(YarnConfiguration.RM_SCHEDULER_MINIMUM_ALLOCATION_MB, 64);
bootConf.setClass(YarnConfiguration.RM_SCHEDULER, FifoScheduler.class, ResourceScheduler.class);
final MiniMRYarnCluster yarnCluster = new MiniMRYarnCluster("test-cluster", 1);
yarnCluster.init(bootConf);
yarnCluster.start();
// Get the "real" Configuration to use from now on
final Configuration conf = yarnCluster.getConfig();
// Get the filesystem
final FileSystem fs = new Path ("hdfs://localhost:" + hdfsCluster.getNameNodePort() + "/").getFileSystem(conf);
Now, I have conf and fs I can use to submit jobs and access HDFS, and new Cluster(conf) and cluster.getAllJobStatuses works as expected.
When everything is done, to shut down and clean up, I call:
yarnCluster.stop();
hdfsCluster.shutdown();
FileUtils.deleteDirectory(hdfsBase); // from Apache Commons IO
Note: JAVA_HOME must be set for this to work. When building on Jenkins, make sure JAVA_HOME is set for the default JDK. Alternatively you can explicitly state a JDK to use, Jenkins will then set up JAVA_HOME automatically.

I tried it like this, it worked for me, but it is after submitting the job
JobClient jc = new JobClient(job.getConfiguration());
for(JobStatus js: jc.getAllJobs())
{
if(js.getState().getValue() == State.RUNNING.getValue())
{
}
}
jc.close();
or else we can get the cluster from job api and there are methods which return all the jobs, jobs status
cluster.getAllJobStatuses();