Develop Reference

Hazelcast Jet not allowing Tomcat to stop

I am using Hazelcast jet to some aggregation and grouping but after being idle for sometime and when I tried to stop my tomcat it is not allowing to stop my tomcat and I have restart my PC. Below is the error which I am getting. Please anyone can guide me what it exactly error is showing and how to shutdown it gracefully?
Sending multicast datagram failed. Exception message saying the operation is not permitted
usually means the underlying OS is not able to send packets at a given pace. It can be caused by starting several hazelcast members in parallel when the members send their join message nearly at the same time.
java.net.NoRouteToHostException: No route to host: Datagram send failed
at java.net.TwoStacksPlainDatagramSocketImpl.send(Native Method)
at java.net.DatagramSocket.send(DatagramSocket.java:693)
at com.hazelcast.internal.cluster.impl.MulticastService.send(MulticastService.java:291)
at com.hazelcast.internal.cluster.impl.MulticastJoiner.searchForOtherClusters(MulticastJoiner.java:113)
at com.hazelcast.internal.cluster.impl.SplitBrainHandler.searchForOtherClusters(SplitBrainHandler.java:75)
at com.hazelcast.internal.cluster.impl.SplitBrainHandler.run(SplitBrainHandler.java:42)
at com.hazelcast.spi.impl.executionservice.impl.DelegateAndSkipOnConcurrentExecutionDecorator$DelegateDecorator.run(DelegateAndSkipOnConcurrentExecutionDecorator.java:77)
at com.hazelcast.internal.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:217)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
at com.hazelcast.internal.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76)
at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102)
Code is quite huge but I tried show you some sample may it won't work as it is just glimpse of a code:
Class Abc{
// It Create Jet Instance
JetConfig jetConfig = new JetConfig();
jetConfig.getHazelcastConfig().setProperty( "hazelcast.logging.type", "log4j" );
jetConfig.getInstanceConfig().setCooperativeThreadCount(5);
jetConfig.configureHazelcast(c -> {
c.getNetworkConfig().setReuseAddress(true);
c.setClusterName("DATA" + UUID.randomUUID().toString());
c.getNetworkConfig().setPort(9093);
c.getNetworkConfig().setPublicAddress("localhost");
c.getNetworkConfig().setPortAutoIncrement(true);
});
JetInstance jetInstance= Jet.newJetInstance(jetConfig);
public Pipeline createPipeline() {
return Pipeline.create();
}
// To Add Job to pipeline
public void joinPipeToJet(Pipeline pl, String name) {
JobConfig j = new JobConfig();
//j.setProcessingGuarantee(ProcessingGuarantee.EXACTLY_ONCE);
j.setName(name);
jetInstance.newJob(pl,j).join();
}
public void readJsonFile(final Map<String, Object> data) {
// Random Id for job so I can separate two jobs Imaps
String jobid = UUID.randomUUID().toString();
try {
Pipeline pl = createPipeline();
UUID idOne = UUID.randomUUID();
final IMap<Object, Object> abc = jetInstance.getMap(idOne.toString());
abc.putAll(data);
// Reading data from file and sending data to next
final BatchSource batchSource = Sources.map(abc);
pl.readFrom(batchSource)
.writeTo(Sinks.map(this.uid));
joinPipeToJet(pl, jobid);
abc.destroy();
} catch (Exception e) {
Job j1 = jetInstance.getJob(jobid);
if (j1 != null) {
j1.cancel();
}
} finally {
Job j1 = jetInstance.getJob(jobid);
if (j1 != null) {
j1.cancel();
}
}
}
//Process to mainplate data and returning it using BatchStage to Map
public Map<String, Object> runProcess(final Pipeline pl) {
String jobid = UUID.randomUUID().toString();
UID idOne = UUID.randomUUID();
BatchStage<Object> bd1 = ;//get data by calling method
bd1.writeTo(Sinks.list(idOne.toString()));
joinPipeToJet(pl, jobid);
IList<Object> abc = jetInstance.getList(idOne.toString());
List<Object> result = new ArrayList(abc);
final Map<String, Object> finalresult =new HashMap<String, Object>();
finalresult.put("datas", result.get(0));
abc.destroy();
return finalresult;
}
public static void main(String...args) {
Map<String, Object> p = new HashMap<String, Object>();
p.putAll("Some Data");
readJsonFile(p);
Pipeline pl = createPipeline();
runProcess(pl);
}
}

How to query the Alfresco Audit Service in Java

I can easily query the Alfresco audit log in REST using this query:
http://localhost:8080/alfresco/service/api/audit/query/audit-custom?verbose=true
But how to perform the same request in Java within Alfresco module?
It must be synchronous.
A lazy solution would be to call the REST URL in Java, but it would probably be inefficient, and more importantly it would require me to store an admin's password somewhere.
I noticed AuditService has a auditQuery method so I am trying to call it. Unfortunately it seems to be for asynchronous operations? I don't need callbacks, as I need to wait until the queried data is ready before going on to the next step.

Here is my implementation, mostly copied from the source code of the REST API:
int maxResults = 10000;
if (!auditService.isAuditEnabled(AUDIT_APPLICATION, ("/" + AUDIT_APPLICATION))) {
throw new WebScriptException(
"Auditing for " + AUDIT_APPLICATION + " is disabled!");
}
final List<Map<String, Object>> entries =
new ArrayList<Map<String,Object>>(limit);
AuditQueryCallback callback = new AuditQueryCallback() {
#Override
public boolean valuesRequired() {
return true; // true = verbose
}
#Override
public boolean handleAuditEntryError(
Long entryId, String errorMsg, Throwable error) {
return true;
}
#Override
public boolean handleAuditEntry(
Long entryId,
String applicationName,
String user,
long time,
Map<String, Serializable> values) {
// Convert values to Strings
Map<String, String> valueStrings =
new HashMap<String, String>(values.size() * 2);
for (Map.Entry<String, Serializable> mapEntry : values.entrySet()) {
String key = mapEntry.getKey();
Serializable value = mapEntry.getValue();
try {
String valueString = DefaultTypeConverter.INSTANCE.convert(
String.class, value);
valueStrings.put(key, valueString);
}
catch (TypeConversionException e) {
// Use the toString()
valueStrings.put(key, value.toString());
}
}
entry.put(JSON_KEY_ENTRY_VALUES, valueStrings);
}
entries.add(entry);
return true;
}
};
AuditQueryParameters params = new AuditQueryParameters();
params.setApplicationName(AUDIT_APPLICATION);
params.setForward(true);
auditService.auditQuery(callback, params, maxResults);
Though the callback might it look asynchronous, it is not.

Calling different Object's Constructor on each index of Array/Collection

I'm trying to tidy up my code as I have many different classes that need to be initialized before my program is ready to do it's tasks.
They are all optional and can fail, but one of them has to succeed.
All these classes implement an interface called Hook.
Each initialization is put into a HashMap, here is an example:
HashMap<String, Hook> hooks = new HashMap<>();
String key = "Fish";
if (isEnabled(key)) {
try {
hooks.put(key, new FishStoreHook());
} catch {
logError(key);
}
}
But now I have to have another one for, say Bread:
key = "Bread";
if (isEnabled(key)) {
try {
hooks.put(key, new BreadStoreHook());
} catch {
logError(key);
}
}
Is there a way to put something in an array or Collection so that they can be called with a for each statement?
(Reason they are put into a HashMap is because their methods are run with a for each HashMap.keySet(), and I need to log possible errors)
Thank you in advance!

I'd suggest using reflection:
String className = "package.name." + key + "StoreHook";
Class<Hook> clazz = (Class<Hook>) YourClass.class.forName(className);
Hook hook = clazz.newInstance();
OR If you are using Spring, you can also get all sub types of Hook.
ClassPathScanningCandidateComponentProvider provider = new ClassPathScanningCandidateComponentProvider(false);
provider.addIncludeFilter(new AssignableTypeFilter(Hook.class));
Set<BeanDefinition> definitions = provider.findCandidateComponents("package/name");
Map<String, Hook> hooks = new HashMap<>();
for (BeanDefinition definition : definitions) {
try {
Class clazz = Class.forName(definition.getBeanClassName());
hooks.put(clazz.getSimpleName(), clazz.newInstance());
} catch (ClassNotFoundException e) {
logger.error("Unable to get class: ", e);
}
}
And then you can get your instance from hashmap.

You can add all Hook objects to a List and then use a for-each as shown in the below code with inline comments:
List<Hook> hooks = new ArrayList<>();
hooks.add(new FishStoreHook("Fish"));//Pass HookName as a constructor arg
hooks.add(new BreadStoreHook("Bread"));//Pass HookName as a constructor arg
//add other hook objects
//Now iterate all the hook objects using foreach
for(Hook hook : hooks) {
if(isEnabled(hook.getName())) {//get HookName (set through constrcutor above)
try {
hooks.put(key, hook);
} catch {
logError(key);
}
}
}

#SuppressWarnings( "serial" )
final Map<String, Callable<Hook>> init =
new LinkedHashMap<String, Callable<Hook>>() {{
put( "Fish", new Callable<Hook>() {
#Override public Hook call() throws Exception {
return new FishStoreHook(); }} );
put( "Bread", new Callable<Hook>() {
#Override public Hook call() throws Exception {
return new BreadStoreHook(); }} );
}};
final Map<String, Hook> hooks = new HashMap<>();
for( Map.Entry<String, Callable<Hook>> e: init.entrySet() )
try {
if( isEnabled( e.getKey() ) )
hooks.put( e.getKey(), e.getValue().call() );
} catch( Exception ex ) {
logError( e.getKey() );
}
You need the LinkedHashMap in case you care about the order, otherwise just any kind of Map would do.

Why I am seeing lot of TimeoutException if any one server goes down?

Here is my DataClientFactory class.
public class DataClientFactory {
public static IClient getInstance() {
return ClientHolder.INSTANCE;
}
private static class ClientHolder {
private static final DataClient INSTANCE = new DataClient();
static {
new DataScheduler().startScheduleTask();
}
}
}
Here is my DataClient class.
public class DataClient implements IClient {
private ExecutorService service = Executors.newFixedThreadPool(15);
private RestTemplate restTemplate = new RestTemplate();
// for initialization purpose
public DataClient() {
try {
new DataScheduler().callDataService();
} catch (Exception ex) { // swallow the exception
// log exception
}
}
#Override
public DataResponse getDataSync(DataKey dataKeys) {
DataResponse response = null;
try {
Future<DataResponse> handle = getDataAsync(dataKeys);
response = handle.get(dataKeys.getTimeout(), TimeUnit.MILLISECONDS);
} catch (TimeoutException e) {
// log error
response = new DataResponse(null, DataErrorEnum.CLIENT_TIMEOUT, DataStatusEnum.ERROR);
} catch (Exception e) {
// log error
response = new DataResponse(null, DataErrorEnum.ERROR_CLIENT, DataStatusEnum.ERROR);
}
return response;
}
#Override
public Future<DataResponse> getDataAsync(DataKey dataKeys) {
Future<DataResponse> future = null;
try {
DataTask dataTask = new DataTask(dataKeys, restTemplate);
future = service.submit(dataTask);
} catch (Exception ex) {
// log error
}
return future;
}
}
I get my client instance from the above factory as shown below and then make a call to getDataSync method by passing DataKey object. DataKey object has userId and Timeout values in it. Now after this, call goes to my DataTask class to call method as soon as handle.get is called.
IClient dataClient = DataClientFactory.getInstance();
long userid = 1234l;
long timeout_ms = 500;
DataKey keys = new DataKey.Builder().setUserId(userid).setTimeout(timeout_ms)
.remoteFlag(false).secondaryFlag(true).build();
// call getDataSync method
DataResponse dataResponse = dataClient.getDataSync(keys);
System.out.println(dataResponse);
Here is my DataTask class which has all the logic -
public class DataTask implements Callable<DataResponse> {
private DataKey dataKeys;
private RestTemplate restTemplate;
public DataTask(DataKey dataKeys, RestTemplate restTemplate) {
this.restTemplate = restTemplate;
this.dataKeys = dataKeys;
}
#Override
public DataResponse call() {
DataResponse dataResponse = null;
ResponseEntity<String> response = null;
int serialId = getSerialIdFromUserId();
boolean remoteFlag = dataKeys.isRemoteFlag();
boolean secondaryFlag = dataKeys.isSecondaryFlag();
List<String> hostnames = new LinkedList<String>();
Mappings mappings = ClientData.getMappings(dataKeys.whichFlow());
String localPrimaryAdress = null;
String remotePrimaryAdress = null;
String localSecondaryAdress = null;
String remoteSecondaryAdress = null;
// use mappings object to get above Address by using serialId and basis on
// remoteFlag and secondaryFlag populate the hostnames linked list
if (remoteFlag && secondaryFlag) {
hostnames.add(localPrimaryHostIPAdress);
hostnames.add(localSecondaryHostIPAdress);
hostnames.add(remotePrimaryHostIPAdress);
hostnames.add(remoteSecondaryHostIPAdress);
} else if (remoteFlag && !secondaryFlag) {
hostnames.add(localPrimaryHostIPAdress);
hostnames.add(remotePrimaryHostIPAdress);
} else if (!remoteFlag && !secondaryFlag) {
hostnames.add(localPrimaryHostIPAdress);
} else if (!remoteFlag && secondaryFlag) {
hostnames.add(localPrimaryHostIPAdress);
hostnames.add(localSecondaryHostIPAdress);
}
for (String hostname : hostnames) {
// If host name is null or host name is in local block host list, skip sending request to this host
if (hostname == null || ClientData.isHostBlocked(hostname)) {
continue;
}
try {
String url = generateURL(hostname);
response = restTemplate.exchange(url, HttpMethod.GET, dataKeys.getEntity(), String.class);
// make DataResponse
break;
} catch (HttpClientErrorException ex) {
// make DataResponse
return dataResponse;
} catch (HttpServerErrorException ex) {
// make DataResponse
return dataResponse;
} catch (RestClientException ex) {
// If it comes here, then it means some of the servers are down.
// Add this server to block host list
ClientData.blockHost(hostname);
// log an error
} catch (Exception ex) {
// If it comes here, then it means some weird things has happened.
// log an error
// make DataResponse
}
}
return dataResponse;
}
private String generateURL(final String hostIPAdress) {
// make an url
}
private int getSerialIdFromUserId() {
// get the id
}
}
Now basis on userId, I will get the serialId and then get the list of hostnames, I am suppose to make a call depending on what flag is passed. Then I iterate the hostnames list and make a call to the servers. Let's say, if I have four hostnames (A, B, C, D) in the linked list, then I will make call to A first and if I get the data back, then return the DataResponse back. But suppose if A is down, then I need to add A to block list instantly so that no other threads can make a call to A hostname. And then make a call to hostname B and get the data back and return the response (or repeat the same thing if B is also down).
I have a background thread as well which runs every 10 minutes and it gets started as soon we get the client instance from the factory and it parses my another service URL to get the list of block hostnames that we are not supposed to make a call. Since it runs every 10 minutes so any servers which are down, it will get the list after 10 minutes only, In general suppose if A is down, then my service will provide A as the block list of hostnames and as soon as A becomes up, then that list will be updated as well after 10 minutes.
Here is my background thread code DataScheduler-
public class DataScheduler {
private RestTemplate restTemplate = new RestTemplate();
private static final Gson gson = new Gson();
private final ScheduledExecutorService scheduler = Executors.newScheduledThreadPool(1);
public void startScheduleTask() {
scheduler.scheduleAtFixedRate(new Runnable() {
public void run() {
try {
callDataService();
} catch (Exception ex) {
// log an error
}
}
}, 0, 10L, TimeUnit.MINUTES);
}
public void callDataService() throws Exception {
String url = null;
// execute the url and get the responseMap from it as a string
parseResponse(responseMap);
}
private void parseResponse(Map<FlowsEnum, String> responses) throws Exception {
// .. some code here to calculate partitionMappings
// block list of hostnames
Map<String, List<String>> coloExceptionList = gson.fromJson(response.split("blocklist=")[1], Map.class);
for (Map.Entry<String, List<String>> entry : coloExceptionList.entrySet()) {
for (String hosts : entry.getValue()) {
blockList.add(hosts);
}
}
if (update) {
ClientData.setAllMappings(partitionMappings);
}
// update the block list of hostnames
if (!DataUtils.isEmpty(responses)) {
ClientData.replaceBlockedHosts(blockList);
}
}
}
And here is my ClientData class which holds all the information for block list of hostnames and partitionMappings details (which is use to get the list of valid hostnames).
public class ClientData {
private static final AtomicReference<ConcurrentHashMap<String, String>> blockedHosts = new AtomicReference<ConcurrentHashMap<String, String>>(
new ConcurrentHashMap<String, String>());
// some code here to set the partitionMappings by using CountDownLatch
// so that read is blocked for first time reads
public static boolean isHostBlocked(String hostName) {
return blockedHosts.get().contains(hostName);
}
public static void blockHost(String hostName) {
blockedHosts.get().put(hostName, hostName);
}
public static void replaceBlockedHosts(List<String> blockList) {
ConcurrentHashMap<String, String> newBlockedHosts = new ConcurrentHashMap<>();
for (String hostName : blockList) {
newBlockedHosts.put(hostName, hostName);
}
blockedHosts.set(newBlockedHosts);
}
}
Problem Statement:-
When all the servers are up (A,B,C,D as an example) above code works fine and I don't see any TimeoutException happening at all from the handle.get but if let's say one server (A) went down which I was supposed to make a call from the main thread then I start seeing lot of TimeoutException, by lot I mean, huge number of client timeouts happening.
And I am not sure why this is happening? In general this won't be happening right since as soon as the server goes down, it will get added to blockList and then no thread will be making a call to that server, instead it will try another server in the list? So it should be smooth process and then as soon as those servers are up, blockList will get updated from the background thread and then you can start making a call.
Is there any problem in my above code which can cause this problem? Any suggestions will be of great help.
In general, what I am trying to do is - make a hostnames list depending on what user id being passed by using the mappings object. And then make a call to the first hostname and get the response back. But if that hostname is down, then add to the block list and make a call to the second hostname in the list.
Here is the Stacktrace which I am seeing -
java.util.concurrent.TimeoutException\n\tat java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:258)
java.util.concurrent.FutureTask.get(FutureTask.java:119)\n\tat com.host.client.DataClient.getDataSync(DataClient.java:20)\n\tat
NOTE: For multiple userId's, we can have same server, meaning server A can get resolve to multiple userId's.

In DataClient class, at the below line:
public class DataClient implements IClient {
----code code---
Future<DataResponse> handle = getDataAsync(dataKeys);
//BELOW LINE IS PROBLEM
response = handle.get(dataKeys.getTimeout(), TimeUnit.MILLISECONDS); // <--- HERE
} catch (TimeoutException e) {
// log error
response = new DataResponse(null, DataErrorEnum.CLIENT_TIMEOUT, DataStatusEnum.ERROR);
} catch (Exception e) {
// log error
response = new DataResponse(null, DataErrorEnum.ERROR_CLIENT, DataStatusEnum.ERROR);
----code code-----
You have assigned a timeout to handle.get(...), which is timing out before your REST connections can respond. The rest connections themselves may or may not be timing out, but since you are timing out of get method of future before the completion of the execution of the thread, the blocking of hosts has no visible effect, while the code inside the call method of DataTask may be performing as expected. Hope this helps.

You asked about suggestions, so here are some suggestions:
1.) Unexpected return value
Method returns unexpectedly FALSE
if (ClientData.isHostBlocked(hostname)) //this may return always false! please check
2.) Exception-Handling
Are you really sure, that a RestClientException occurs?
Only when this exception occured, the host will be added to blocked list!
Your posted code seems to ignore logging (it is commented out!)
...catch (HttpClientErrorException ex) {
// make DataResponse
return dataResponse;
} catch (HttpServerErrorException ex) {
// make DataResponse
return dataResponse;
} catch (RestClientException ex) {
// If it comes here, then it means some of the servers are down.
// Add this server to block host list
ClientData.blockHost(hostname);
// log an error
} catch (Exception ex) {
// If it comes here, then it means some weird things has happened.
// log an error
// make DataResponse
}

Get all the data from the database first time and if any entries got changed then print out only the change

I am working on a project in which I need to print out the data from the database. Let's take an example, suppose in my database, I have below entries only-
Hello 1.0.0
World 1.0.0
Then my Java method that will make a call to the database will return me a map of above data.
My map will have above data as below-
Key as Hello, Value as 1.0.0
Key as World, Value as 1.0.0
Suppose, I started my program for the first time, then it will print out like this with the below code I have, which is perfectly fine.
{Hello=1.0.0, World=1.0.0}
And then I am running background thread every 2 seconds that will make a call to the database and get all the data again from the database. And every two seconds, it will print out the same thing- (and my code is working fine according to that)
{Hello=1.0.0, World=1.0.0}
Now what is interesting is, suppose my application is running and somebody changed the database entry like this-
Hello 1.0.1
World 1.0.0
meaning, version got changed for Hello and now it is 1.0.1 and as my application is running, so now what it will print out basis on my below code? Something like this-
{Hello=1.0.1, World=1.0.0}
But that's not what I am looking for. It should print out like below, meaning only the entry that got changed
{Hello=1.0.1}
I hope the questions is clear enough. Below is my code that I have so far. It only prints out all the entry from the database every time which is not what I am looking for.
I want to print out everything only when the program is getting started for the first time but after if it is started, it should print out only the things that have changed or any new entry.
Below is my code:-
public class Graph {
public static Map<String, String> bundleList = new LinkedHashMap<String, String>();
public static void main(String[] args) {
getAttributesFromDatabase();
printOutBundleInformation();
loggingAfterEveryXMilliseconds();
}
private static void printOutBundleInformation() {
System.out.println(bundleList);
}
private static void getAttributesFromDatabase() {
Map<String, String> bundleInformation = new LinkedHashMap<String, String>();
bundleInformation = getFromDatabase();
if(!bundleInformation.isEmpty()) {
bundleList = bundleInformation;
}
}
private static Map<String, String> getFromDatabase() {
Map<String, String> hello = new LinkedHashMap<String, String>();
// In actual scenario, I will have a database call here
hello.put("Hello", "1.0.0");
hello.put("World", "1.0.0");
return hello;
}
private static void loggingAfterEveryXMilliseconds() {
new Thread() {
public void run() {
while (true) {
try {
Thread.sleep(2000);
} catch (InterruptedException ex) {
}
getAttributesFromDatabase();
printOutBundleInformation();
}
}
}.start();
}
}
Any help will be appreciated on this.

MapDifference.html#entriesDiffering() might be what you need.
Quick and dirty:
public class Graph {
public static Map<String, String> bundleList = new LinkedHashMap<String, String>();
private static Map<String, String> oldBundleList = new LinkedHashMap<String, String>();
// ...
private static void getAttributesFromDatabase() {
Map<String, String> bundleInformation = new LinkedHashMap<String, String>();
bundleInformation = getFromDatabase();
if(!bundleInformation.isEmpty()) {
oldBundleList = bundleList;
bundleList = bundleInformation;
}
}
// ...
private static void loggingAfterEveryXMilliseconds() {
new Thread() {
public void run() {
while (true) {
try {
Thread.sleep(2000);
} catch (InterruptedException ex) {
}
getAttributesFromDatabase();
final Map<String, ValueDifference<String>> entriesDiffering = Maps.difference(oldBundleList, bundleList).entriesDiffering();
if (!entriesDiffering.isEmpty()) {
for (String key : entriesDiffering.keySet()) {
System.out.println("{" + key + "=" + bundleList.get(key) + "}");
}
}
// printOutBundleInformation();
}
}
}.start();
}
}

You need to add extra column in the table called last modified(date with time). Every time querying to the database you pass the last access time(for example getFromDatabase(last access time) ). You need to keep your last access time in you program. In the query you need to verify the last access time against the last modified field.