Develop Reference

Apache Flink Kafka Stream Processing based on conditions

Am building a wrapper library using apache-flink where I am listening(consuming) from multiple topics and I have a set of applications that want to process the messages from those topics.
Example :
I have 10 applications app1, app2, app3 ... app10 (each of them is a java library part of the same on-prem project, ie., all 10 jars are part of same .war file)
out of which only 5 are supposed to consume the messages coming to the consumer group. I am able to do filtering for 5 apps with the help of filter function.
The challenge is in the strStream.process(executionServiceInterface) function, where app1 provides an implementation class for ExceucionServiceInterface as ExecutionServiceApp1Impl and similary app2 provides ExecutionServiceApp2Impl.
when there are multiple implementations available spring wants us to provide #Qualifier annotation or #Primary has to be marked on the implementations (ExecutionServiceApp1Impl , ExecutionServiceApp2Impl).
But I don't really want to do this. As am building a generic wrapper library that should support any no of such applications (app1, app2 etc) and all of them should be able to implement their own implementation logic(ExecutionServiceApp1Impl , ExecutionServiceApp2Impl).
Can someone help me here ? how to solve this ?
Below is the code for reference.
#Autowired
private ExceucionServiceInterface executionServiceInterface;
public void init(){
StreamExecutionEnvironment environment = StreamExecutionEnvironment.getExecutionEnvironment();
FlinkKafkaConsumer011<String> consumer = createStringConsumer(topicList, kafkaAddress, kafkaGroup);
if (consumer != null) {
DataStream<String> strStream = environment.addSource(consumer);
strStream.filter(filterFunctionInterface).process(executionServiceInterface);
}
}
public FlinkKafkaConsumer011<String> createStringConsumer(List<String> listOfTopics, String kafkaAddress, String kafkaGroup) throws Exception {
FlinkKafkaConsumer011<String> myConsumer = null;
try {
Properties props = new Properties();
props.setProperty("bootstrap.servers", kafkaAddress);
props.setProperty("group.id", kafkaGroup);
myConsumer = new FlinkKafkaConsumer011<>(listOfTopics, new SimpleStringSchema(), props);
} catch(Exception e) {
throw e;
}
return myConsumer;
}
Many thanks in advance!!

Solved this problem by using Reflection, below is the code that solved the issue.
Note : this requires me to know the list of fully qualified classNames and method names along with their parameters.
#Component
public class SampleJobExecutor extends ProcessFunction<String, String> {
#Autowired
MyAppProperties myAppProperties;
#Override
public void processElement(String inputMessage, ProcessFunction<String, String>.Context context,
Collector<String> collector) throws Exception {
String className = null;
String methodName = null;
try {
Map<String, List<String>> map = myAppProperties.getMapOfImplementors();
JSONObject json = new JSONObject(inputMessage);
if (json != null && json.has("appName")) {
className = map.get(json.getString("appName")).get(0);
methodName = map.get(json.getString("appName")).get(1);
}
Class<?> forName = Class.forName(className);
Object job = forName.newInstance();
Method method = forName.getDeclaredMethod(methodName, String.class);
method.invoke(job , inputMessage);
} catch (Exception e) {
e.printStackTrace();
}
}

How to get Workflow event type and Processstep name or Arguments in to java service in AEM?

I created a workflow for whenever I upload or remove asset from /content/dam/MyAsset folder I am able to Triggering JavaServices. I create two launcher with Created Event Type and Removed Event Type.
I need to get That event Type and Process step name or arguments in to Services
inside Triggering execute function
Here is My Code:
public void execute(WorkItem arg0, WorkflowSession arg1, MetaDataMap arg2)
throws WorkflowException
{
log.info("Workflow created ::::: ");
}
Is there any way to get Launcher event type and process arguments]to Services ?

You can not get launcher event type as that information is not passed down to the workflow. What you could do is check the payload to identify if the node exists or not. The workflow called on removal will have path to the node that is removed and if you try to resolve that path it will give you exception (yeah, bad approach but thats the only possibility).
For the process step arguments passed, all that information is available in meta data. Look into arg0.getWorkflowData().getMetaDataMap().get("MyIDkey",<Type>); where arg0 is your WorkItem instance.

Why don't you use a org.osgi.service.event.EventHandler instead? You will have all the information you need. Here is a small code snippet that waits for an event and adds the relevant data into a map that is then passed to the JobManager to generate a new Job that is executed by a matching JobConsumer:
import org.apache.sling.event.jobs.Job;
import org.apache.sling.event.jobs.JobManager;
import org.apache.sling.event.jobs.consumer.JobConsumer;
// and more imports...
#Service
#Component(immediate = true, policy = ConfigurationPolicy.OPTIONAL, description = "Listen to page modification events.")
#Properties(value = { #Property(name = "event.topics", value = { PageEvent.EVENT_TOPIC, DamEvent.EVENT_TOPIC}, propertyPrivate = true),
#Property(name = JobConsumer.PROPERTY_TOPICS, value = ModificationEventHandler.JOB_TOPICS, propertyPrivate = true) })
public class ModificationEventHandler implements EventHandler, JobConsumer {
/**
* Modification Job Topics.
*/
public static final String JOB_TOPICS = "de/rockware/aem/modification";
#Reference
private JobManager jobManager;
#Override
public void handleEvent(Event event) {
logger.trace("Checking event.");
PageEvent pageEvent = PageEvent.fromEvent(event);
DamEvent damEvent = DamEvent.fromEvent(event);
Map<String, Object> properties = new HashMap<>();
if (damEvent != null) {
// DamEvent is not serializable, so we cannot add the complete event to the map.
logger.trace("Event on {} is a dam event ({}).", damEvent.getAssetPath(), damEvent.getType().name());
properties.put(DAM_EVENT_ASSET_PATH, damEvent.getAssetPath());
}
if (pageEvent != null) {
logger.trace("Event is a page event.");
properties.put(PAGE_EVENT, pageEvent);
}
logger.trace("Adding new job.");
jobManager.addJob(JOB_TOPICS, properties);
}
#Override
public JobResult process(Job job) {
if (job.getProperty(PAGE_EVENT) != null) {
PageEvent pageEvent = (PageEvent) job.getProperty(PAGE_EVENT);
if(pageEvent.isLocal()) {
Iterator<PageModification> modificationsIterator;
modificationsIterator = pageEvent.getModifications();
while (modificationsIterator.hasNext()) {
PageModification modification = modificationsIterator.next();
logger.info("Handling modification {} on path {}.", modification.getType(), modification.getPath());
if(isRelevantModification(modification.getType())) {
logger.info(modification.getPath());
}
}
} else {
logger.trace("Page event is not local.");
}
} else if (job.getProperty(DAM_EVENT_ASSET_PATH) != null) {
String assetPath = (String) job.getProperty(DAM_EVENT_ASSET_PATH);
logger.trace(assetPath);
backupService.trackModification(assetPath);
} else {
logger.trace("Invalid event type. Cannot help.");
}
return JobResult.OK;
}
}

How can single instance of an OSGI factory configuration be read from Java in CQ

I need to read the specific child instance of an OSGi factory configuration.
I believe it can't be accessed with the Service PID of the factory configuration so there should be a way to reference the child configuration via Java.
Can anyone please help in providing a sample code or a way to do this?

Below is an example. "WSConnection" is an OSGI config where we can configure multiple configs. and the Helper class will help you pick the one you wanted. "configuration.id" is one of the properties for each of the OSGI config. Let me know if you need more details.
#Service(value = WSConnection.class)
#Component(immediate = true, label = "WS Factory", description = "WS
Connection Factory", configurationFactory = true, policy =
ConfigurationPolicy.REQUIRE, metatype = true)
#Properties({
#Property(name = "configuration.id", value = "", label = "Configuration ID", description = "Configuration ID to reference this configuration")
})
public class WebServiceConnection {
....
....
}
public class WSHelper extends WCMUse {
...
...
#Override
public void activate() throws Exception {
setProperties();
}
private void setProperties() {
BundleContext bundleContext = FrameworkUtil.getBundle(WSConnection.class).getBundleContext();
ServiceReference configurationAdminReference = bundleContext.getServiceReference(ConfigurationAdmin.class.getName());
if (configurationAdminReference != null) {
ConfigurationAdmin confAdmin = (ConfigurationAdmin) bundleContext.getService(configurationAdminReference);
try {
Configuration conf[] = confAdmin.listConfigurations("("+ConfigurationAdmin.SERVICE_FACTORYPID+"="+WSConnection.class.getName()+")");
for (Configuration c : conf){
Dictionary<String,Object> props = c.getProperties();
this.configurationId = props.get("configuration.id").toString();
break;
}
}
} catch (Exception e) {
log.error("Error getting Web Service URL", e);
}
}
}

Updating Dropwizard config at runtime

Is it possible to have my app update the config settings at runtime? I can easily expose the settings I want in my UI but is there a way to allow the user to update settings and make them permanent ie save them to the config.yaml file? The only way I can see it to update the file by hand then restart the server which seems a bit limiting.

Yes. It is possible to reload the service classes at runtime.
Dropwizard by itself does not have the way to reload the app, but jersey has.
Jersey uses a container object internally to maintain the running application. Dropwizard uses the ServletContainer class of Jersey to run the application.
How to reload the app without restarting it -
Get a handle to the container used internally by jersey
You can do this by registering a AbstractContainerLifeCycleListener in Dropwizard Environment before starting the app. and implement its onStartup method as below -
In your main method where you start the app -
//getting the container instance
environment.jersey().register(new AbstractContainerLifecycleListener() {
#Override
public void onStartup(Container container) {
//initializing container - which will be used to reload the app
_container = container;
}
});
Add a method to your app to reload the app. It will take in the list of string which are the names of the service classes you want to reload. This method will call the reload method of the container with the new custom DropWizardConfiguration instance.
In your Application class
public static synchronized void reloadApp(List<String> reloadClasses) {
DropwizardResourceConfig dropwizardResourceConfig = new DropwizardResourceConfig();
for (String className : reloadClasses) {
try {
Class<?> serviceClass = Class.forName(className);
dropwizardResourceConfig.registerClasses(serviceClass);
System.out.printf(" + loaded class %s.\n", className);
} catch (ClassNotFoundException ex) {
System.out.printf(" ! class %s not found.\n", className);
}
}
_container.reload(dropwizardResourceConfig);
}
For more details see the example documentation of jersey - jersey example for reload
Consider going through the code and documentation of following files in Dropwizard/Jersey for a better understanding -
Container.java
ContainerLifeCycleListener.java
ServletContainer.java
AbstractContainerLifeCycleListener.java
DropWizardResourceConfig.java
ResourceConfig.java

No.
Yaml file is parsed at startup and given to the application as Configuration object once and for all. I believe you can change the file after that but it wouldn't affect your application until you restart it.
Possible follow up question: Can one restart the service programmatically?
AFAIK, no. I've researched and read the code somewhat for that but couldn't find a way to do that yet. If there is, I'd love to hear that :).

I made a task that reloads the main yaml file (it would be useful if something in the file changes). However, it is not reloading the environment. After researching this, Dropwizard uses a lot of final variables and it's quite hard to reload these on the go, without restarting the app.
class ReloadYAMLTask extends Task {
private String yamlFileName;
ReloadYAMLTask(String yamlFileName) {
super("reloadYaml");
this.yamlFileName = yamlFileName;
}
#Override
public void execute(ImmutableMultimap<String, String> parameters, PrintWriter output) throws Exception {
if (yamlFileName != null) {
ConfigurationFactoryFactory configurationFactoryFactory = new DefaultConfigurationFactoryFactory<ReportingServiceConfiguration>();
ValidatorFactory validatorFactory = Validation.buildDefaultValidatorFactory();
Validator validator = validatorFactory.getValidator();
ObjectMapper objectMapper = Jackson.newObjectMapper();
final ConfigurationFactory<ServiceConfiguration> configurationFactory = configurationFactoryFactory.create(ServiceConfiguration.class, validator, objectMapper, "dw");
File confFile = new File(yamlFileName);
configurationFactory.build(new File(confFile.toURI()));
}
}
}

You can change the configuration in the YAML and read it while your application is running. This will not however restart the server or change any server configurations. You will be able to read any changed custom configurations and use them. For example, you can change the logging level at runtime or reload other custom settings.
My solution -
Define a custom server command. You should use this command to start your application instead of the "server" command.
ArgsServerCommand.java
public class ArgsServerCommand<WC extends WebConfiguration> extends EnvironmentCommand<WC> {
private static final Logger LOGGER = LoggerFactory.getLogger(ArgsServerCommand.class);
private final Class<WC> configurationClass;
private Namespace _namespace;
public static String COMMAND_NAME = "args-server";
public ArgsServerCommand(Application<WC> application) {
super(application, "args-server", "Runs the Dropwizard application as an HTTP server specific to my settings");
this.configurationClass = application.getConfigurationClass();
}
/*
* Since we don't subclass ServerCommand, we need a concrete reference to the configuration
* class.
*/
#Override
protected Class<WC> getConfigurationClass() {
return configurationClass;
}
public Namespace getNamespace() {
return _namespace;
}
#Override
protected void run(Environment environment, Namespace namespace, WC configuration) throws Exception {
_namespace = namespace;
final Server server = configuration.getServerFactory().build(environment);
try {
server.addLifeCycleListener(new LifeCycleListener());
cleanupAsynchronously();
server.start();
} catch (Exception e) {
LOGGER.error("Unable to start server, shutting down", e);
server.stop();
cleanup();
throw e;
}
}
private class LifeCycleListener extends AbstractLifeCycle.AbstractLifeCycleListener {
#Override
public void lifeCycleStopped(LifeCycle event) {
cleanup();
}
}
}
Method to reload in your Application -
_ymlFilePath = null; //class variable
public static boolean reloadConfiguration() throws IOException, ConfigurationException {
boolean reloaded = false;
if (_ymlFilePath == null) {
List<Command> commands = _configurationBootstrap.getCommands();
for (Command command : commands) {
String commandName = command.getName();
if (commandName.equals(ArgsServerCommand.COMMAND_NAME)) {
Namespace namespace = ((ArgsServerCommand) command).getNamespace();
if (namespace != null) {
_ymlFilePath = namespace.getString("file");
}
}
}
}
ConfigurationFactoryFactory configurationFactoryFactory = _configurationBootstrap.getConfigurationFactoryFactory();
ValidatorFactory validatorFactory = _configurationBootstrap.getValidatorFactory();
Validator validator = validatorFactory.getValidator();
ObjectMapper objectMapper = _configurationBootstrap.getObjectMapper();
ConfigurationSourceProvider provider = _configurationBootstrap.getConfigurationSourceProvider();
final ConfigurationFactory<CustomWebConfiguration> configurationFactory = configurationFactoryFactory.create(CustomWebConfiguration.class, validator, objectMapper, "dw");
if (_ymlFilePath != null) {
// Refresh logging level.
CustomWebConfiguration webConfiguration = configurationFactory.build(provider, _ymlFilePath);
LoggingFactory loggingFactory = webConfiguration.getLoggingFactory();
loggingFactory.configure(_configurationBootstrap.getMetricRegistry(), _configurationBootstrap.getApplication().getName());
// Get my defined custom settings
CustomSettings customSettings = webConfiguration.getCustomSettings();
reloaded = true;
}
return reloaded;
}

Although this feature isn't supported out of the box by dropwizard, you're able to accomplish this fairly easy with the tools they give you.
Before I get started, note that this isn't a complete solution for the question asked as it doesn't persist the updated config values to the config.yml. However, this would be easy enough to implement yourself simply by writing to the config file from the application. If anyone would like to write this implementation feel free to open a PR on the example project I've linked below.
Code
Start off with a minimal config:
config.yml
myConfigValue: "hello"
And it's corresponding configuration file:
ExampleConfiguration.java
public class ExampleConfiguration extends Configuration {
private String myConfigValue;
public String getMyConfigValue() {
return myConfigValue;
}
public void setMyConfigValue(String value) {
myConfigValue = value;
}
}
Then create a task which updates the config:
UpdateConfigTask.java
public class UpdateConfigTask extends Task {
ExampleConfiguration config;
public UpdateConfigTask(ExampleConfiguration config) {
super("updateconfig");
this.config = config;
}
#Override
public void execute(Map<String, List<String>> parameters, PrintWriter output) {
config.setMyConfigValue("goodbye");
}
}
Also for demonstration purposes, create a resource which allows you to get the config value:
ConfigResource.java
#Path("/config")
public class ConfigResource {
private final ExampleConfiguration config;
public ConfigResource(ExampleConfiguration config) {
this.config = config;
}
#GET
public Response handleGet() {
return Response.ok().entity(config.getMyConfigValue()).build();
}
}
Finally wire everything up in your application:
ExampleApplication.java (exerpt)
environment.jersey().register(new ConfigResource(configuration));
environment.admin().addTask(new UpdateConfigTask(configuration));
Usage
Start up the application then run:
$ curl 'http://localhost:8080/config'
hello
$ curl -X POST 'http://localhost:8081/tasks/updateconfig'
$ curl 'http://localhost:8080/config'
goodbye
How it works
This works simply by passing the same reference to the constructor of ConfigResource.java and UpdateConfigTask.java. If you aren't familiar with the concept see here:
Is Java "pass-by-reference" or "pass-by-value"?
The linked classes above are to a project I've created which demonstrates this as a complete solution. Here's a link to the project:
scottg489/dropwizard-runtime-config-example
Footnote: I haven't verified this works with the built in configuration. However, the dropwizard Configuration class which you need to extend for your own configuration does have various "setters" for internal configuration, but it may not be safe to update those outside of run().
Disclaimer: The project I've linked here was created by me.

Manually change task on running process-instance

in order of developing a custom BPM Application there is one feature we used with another BPM engine provider and like to use it with camunda too.
The targeted functionality is about setting/reset running process instances to a specified task other than the current active one. From our perspective necessary when e.g.:
authoring process-instances due to process-version migration
resolving incidents
resolving accidentially wrong usage by an user
Finally I didn't really found a simple function to do this but worked out some custom code which worked with some limitations. There are some weaknesses and uncertainities within this code so that I have the following question:
Did I miss an alternative way to achieve this or is the following approach correct or is it even fully unsupported at the moment ?
The current weaknesses imho:
first and most important: no historic task instance is stored. This
causes that it's not traceable who or even when the task was triggered/activate/started.
I found the following post on camunda google group (post) which says that it's
correct at this point because it’s a task out of the process definition scope but
by using a task definition from the underlying process definition I should
be "in scope" ?!
the code is based on internal implementation and not on official interface
at this point a lot of "bootstrap"/initialization have to be done manually
but as user (not developer of camunda) I am not fully aware of what is required and
what is optional
some parts like parsing expressions from task definition failed (see code commented out)
but that may be caused by wrong usage
Here's the code (experimental snippet of our camunda service facade) :
#Inject
protected HistoryService histService;
#Inject
protected TaskService taskService;
#Inject
protected ManagementService managementService;
#Inject
protected RuntimeService runtimeService;
#Inject
protected IdentityService identityService;
#Inject
protected RepositoryService repositoryService;
#Inject
protected FormService formService;
#Inject
protected ProcessEngine processEngine;
public void startTask(String processInstanceId, String taskKey) {
Collection<TaskDefinition> taskDefs = getAvailableTasks(
processInstanceId);
TaskEntity newTask = null;
TaskDefinition taskDef = null;
for (TaskDefinition taskDefinition : taskDefs) {
if (taskDefinition.getKey().equals(taskKey)) {
taskDef = taskDefinition;
break;
}
}
boolean taskDefExists = taskDef != null;
List<Task> runningTasksByKey = getTasksByKey(taskKey, processInstanceId);
boolean taskIsAlreadyRunning = runningTasksByKey != null
&& runningTasksByKey.size() > 0;
if (taskDefExists && !taskIsAlreadyRunning) {
newTask = (TaskEntity) taskService.newTask();
ProcessInstance procInst = getProcessInstance(processInstanceId);
ExecutionEntity procInstEntity = (ExecutionEntity) procInst;
String taskName = (String) taskDef.getNameExpression().
getExpressionText();
// String taskAssigne = (String) taskDef.getAssigneeExpression().
// getValue(
// procInstEntity);
// newTask.setAssignee(taskAssigne);
newTask.setTaskDefinitionKey(taskDef.getKey());
newTask.setProcessInstance(procInstEntity);
newTask.setTaskDefinition(taskDef);
newTask.setName(taskName);
newTask.setProcessInstanceId(processInstanceId);
newTask.setProcessDefinitionId(procInstEntity.
getProcessDefinitionId());
taskService.saveTask(newTask);
TaskServiceImpl taskServiceImpl = (TaskServiceImpl) BpmPlatform.
getProcessEngineService().getDefaultProcessEngine().
getTaskService();
CommandExecutor commandExecutor = taskServiceImpl.
getCommandExecutor();
ExecutionEntity executionEntity = commandExecutor.execute(
new SaveTaskActivityInstanceCmd(newTask,
procInstEntity));
// commandExecutor.execute(new `SaveTaskHistoricActivityInstanceCmd(executionEntity, newTask));`
}
}
public Collection<TaskDefinition> getAvailableTasks(String processInstanceId) {
Map<String, TaskDefinition> taskDefs = null;
Collection<TaskDefinition> taskDefObjects = null;
if (processInstanceId != null) {
ProcessInstanceQuery procInstQuery = runtimeService.
createProcessInstanceQuery().processInstanceId(
processInstanceId);
ProcessDefinitionEntity procDefEntity = getProcessDefinitionEager(
processInstanceId);
taskDefs = procDefEntity.getTaskDefinitions();
}
taskDefObjects = (Collection<TaskDefinition>) (taskDefs != null ? taskDefs.
values() : new ArrayList<TaskDefinition>());
return taskDefObjects;
}
public ProcessDefinitionEntity getProcessDefinitionEager(
String processInstanceId) {
ProcessInstanceQuery procInstQuery = runtimeService.
createProcessInstanceQuery().processInstanceId(
processInstanceId);
ProcessInstance procInst = procInstQuery.singleResult();
String procDefId = procInst.getProcessDefinitionId();
return (ProcessDefinitionEntity) repositoryService.getProcessDefinition(
procDefId);
}
public List<Task> getTasksByKey(String taskKey, String processInstanceId) {
List<Task> tasks = taskService.createTaskQuery().processInstanceId(
processInstanceId).taskDefinitionKey(taskKey).list();
return tasks;
}
public class SaveTaskActivityInstanceCmd implements Command<ExecutionEntity>,
Serializable {
private TaskEntity newTask;
private ExecutionEntity procInstEntity;
public SaveTaskActivityInstanceCmd(TaskEntity newTaskInit,
ExecutionEntity procInstEntityInit) {
this.newTask = newTaskInit;
this.procInstEntity = procInstEntityInit;
}
public ExecutionEntity execute(CommandContext commandContext) {
ActivityImpl actImpl = new ActivityImpl(newTask.
getTaskDefinitionKey(),
procInstEntity.getProcessDefinition());
actImpl.setActivityBehavior(new UserTaskActivityBehavior(
new CdiExpressionManager(), newTask.getTaskDefinition()));
ExecutionEntity execEntity = new ExecutionEntity();
execEntity.setActivity(actImpl);
execEntity.setActivityInstanceId(newTask.getTaskDefinitionKey()
+ ":" + newTask.getId());
execEntity.setEventName(newTask.getEventName());
execEntity.setProcessDefinitionId(newTask.getProcessDefinitionId());
execEntity.setActive(true);
execEntity.setProcessInstance(procInstEntity);
commandContext.getExecutionManager().insert(execEntity);
return execEntity;
}
}
I appreciate any hint or advice :-)

Beginning with Camunda 7.3, you can use process instance modification to start any activity in a process and cancel any active activity instance.
Example:
runtimeService.createProcessInstanceModification(processInstanceId)
.startBeforeActivity("someActivityId")
.cancelActivityInstance("someActivityInstanceId")
.execute();
See http://docs.camunda.org/7.3/guides/user-guide/#process-engine-process-instance-modification for documentation.

I wouldn't mess with the process instance on that level, as you already noticed, you are bypassing camundas services. When faced with a similar problem, we went with the following:
cancel the process instance of the old process version
start a new instance of the extended process and forward it programmatically to the desired state ...
Another option: model an entry point (message start event) inside the new process version. Then, instead of programmatically forwarding the instance to the desired state, just start the new instance via the event and pass all process variables of the old instance ...