Develop Reference

RxJava valve use case

Is there an operator in RxJava, an external library or a way I'm missing to create a flowable/observable that recieves a function that controls the emission of data, like a valve?
I have a huge json file I need to process but I have to get a portion of the file, a list of entities, process it and then get another portion, I have tried using windows(), buffer() but the BiFunction I pass to Flowable.generate() keeps executing after I recieved the first list and I haven't finished processing it. I also tried FlowableTransformers.valve() from hu.akarnokd.rxjava3.operators but it just piles up the items before the flatMap() function that process the list
private Flowable<T> flowable(InputStream inputStream) {
return Flowable.generate(() -> jsonFactory.createParser(new GZIPInputStream(inputStream)), (jsonParser, emitter) -> {
final var token = jsonParser.nextToken();
if (token == null) {
emitter.onComplete();
}
if (JsonToken.START_ARRAY.equals(token) || JsonToken.END_ARRAY.equals(token)) {
return jsonParser;
}
if (JsonToken.START_OBJECT.equals(token)) {
emitter.onNext(reader.readValue(jsonParser));
}
return jsonParser;
}, JsonParser::close);
}
Edit: I need to control de emission of items to don't overload the memory and the function that process the data, because that function reads and writes to database, also the processing needs to be sequentially. The function that process the data it's not entirely mine and it's written in RxJava and it's expected that I use Rx.
I managed to solve it like this but if there is another way let me know please:
public static <T> Flowable<T> flowable(InputStream inputStream, JsonFactory jsonFactory, ObjectReader reader, Supplier<Boolean> booleanSupplier) {
return Flowable.generate(() -> jsonFactory.createParser(new GZIPInputStream(inputStream)), (jsonParser, emitter) -> {
if (booleanSupplier.get()) {
final var token = jsonParser.nextToken();
if (token == null) {
emitter.onComplete();
}
if (JsonToken.START_ARRAY.equals(token) || JsonToken.END_ARRAY.equals(token)) {
return jsonParser;
}
if (JsonToken.START_OBJECT.equals(token)) {
emitter.onNext(reader.readValue(jsonParser));
}
}
return jsonParser;
}, JsonParser::close);
}
Edit2: This is one of the ways I'm currently consuming the function
public Flowable<List<T>> paging(Function<List<T>, Single<List<T>>> function) {
final var atomicInteger = new AtomicInteger(0);
final var atomicBoolean = new AtomicBoolean(true);
return flowable(inputStream, jsonFactory, reader, atomicBoolean::get)
.buffer(pageSize)
.flatMapSingle(list -> {
final var counter = atomicInteger.addAndGet(1);
if (counter == numberOfPages) {
atomicBoolean.set(false);
}
return function.apply(list)
.doFinally(() -> {
if (atomicInteger.get() == numberOfPages) {
atomicInteger.set(0);
atomicBoolean.set(true);
}
});
});
}

Managed to solve it like this
public static Flowable<Object> flowable(JsonParser jsonParser, ObjectReader reader, PublishProcessor<Boolean> valve) {
return Flowable.defer(() -> {
final var token = jsonParser.nextToken();
if (token == null) {
return Completable.fromAction(jsonParser::close)
.doOnError(Throwable::printStackTrace)
.onErrorComplete()
.andThen(Flowable.empty());
}
if (JsonToken.START_OBJECT.equals(token)) {
final var value = reader.readValue(jsonParser);
final var just = Flowable.just(value).compose(FlowableTransformers.valve(valve, true));
return Flowable.concat(just, flowable(jsonParser, reader, valve));
}
return flowable(jsonParser, reader, valve);
});
}

Reindex part of Elasticsearch index onto new index via Jest

I have a test ElasticSearch 6.0 index populated with millions of records, likely to be in the billions in production. I need to search for a subset of these records, then save this subset of the original set into a secondary index for later searching. I have proven this out via querying ES on Kibana, the challenge is to find appropriate APIs in Java 8 using my Jest client (searchbox.io, version 5.3.3) to do the same. The ElasticSearch cluster is on AWS, so using a transport client is out.
POST _reindex?slices=10&wait_for_completion=false
{ "conflicts": "proceed",
"source":{
"index": "my_source_idx",
"size": 5000,
"query": { "bool": {
"filter": { "bool" : { "must" : [
{ "nested": { "path": "test", "query": { "bool": { "must":[
{ "terms" : { "test.RowKey": ["abc"]} },
{ "range" : { "test.dates" : { "lte": "2018-01-01", "gte": "2010-08-01"} } },
{ "range" : { "test.DatesCount" : { "gte": 2} } },
{ "script" : { "script" : { "id": "my_painless_script",
"params" : {"min_occurs" : 1, "dateField": "test.dates", "RowKey": ["abc"], "fromDate": "2010-08-01", "toDate": "2018-01-01"}}}}
]}}}}
]}}
}}
},
"dest": {
"index": "my_dest_idx"
},
"script": {
"source": <My painless script>
} }
I am aware I can perform a search on the source index, then create and bulk load the response records onto the new index, but I want to be able to do this all in one shot, as I do have a painless script to glean off some information that is pertinent to the queries that will search the secondary index. Performance is a concern, as the application will be chaining subsequent queries together using the destination index to query against. Does anyone know how to do accomplish this using Jest?

It appears as if this particular functionality is not yet supported in Jest. The Jest API It has a way to pass in a script (not a query) as a parameter, but I even was having problems with that.
EDIT:
After some hacking with a coworker, we found a way around this...
Step 1) Extend the GenericResultAbstractionAction class with edits to the script:
public class GenericResultReindexActionHack extends GenericResultAbstractAction {
GenericResultReindexActionHack(GenericResultReindexActionHack.Builder builder) {
super(builder);
Map<String, Object> payload = new HashMap<>();
payload.put("source", builder.source);
payload.put("dest", builder.dest);
if (builder.conflicts != null) {
payload.put("conflicts", builder.conflicts);
}
if (builder.size != null) {
payload.put("size", builder.size);
}
if (builder.script != null) {
Script script = (Script) builder.script;
// Note the script parameter needs to be formatted differently to conform to the ES _reindex API:
payload.put("script", new Gson().toJson(ImmutableMap.of("id", script.getIdOrCode(), "params", script.getParams())));
}
this.payload = ImmutableMap.copyOf(payload);
setURI(buildURI());
}
#Override
protected String buildURI() {
return super.buildURI() + "/_reindex";
}
#Override
public String getRestMethodName() {
return "POST";
}
#Override
public String getData(Gson gson) {
if (payload == null) {
return null;
} else if (payload instanceof String) {
return (String) payload;
} else {
// We need to remove the incorrect formatting for the query, dest, and script fields:
// TODO: Need to consider spaces in the JSON
return gson.toJson(payload).replaceAll("\\\\n", "")
.replace("\\", "")
.replace("query\":\"", "query\":")
.replace("\"},\"dest\"", "},\"dest\"")
.replaceAll("\"script\":\"","\"script\":")
.replaceAll("\"}","}")
.replaceAll("},\"script\"","\"},\"script\"");
}
}
public static class Builder extends GenericResultAbstractAction.Builder<GenericResultReindexActionHack , GenericResultReindexActionHack.Builder> {
private Object source;
private Object dest;
private String conflicts;
private Long size;
private Object script;
public Builder(Object source, Object dest) {
this.source = source;
this.dest = dest;
}
public GenericResultReindexActionHack.Builder conflicts(String conflicts) {
this.conflicts = conflicts;
return this;
}
public GenericResultReindexActionHack.Builder size(Long size) {
this.size = size;
return this;
}
public GenericResultReindexActionHack.Builder script(Object script) {
this.script = script;
return this;
}
public GenericResultReindexActionHack.Builder waitForCompletion(boolean waitForCompletion) {
return setParameter("wait_for_completion", waitForCompletion);
}
public GenericResultReindexActionHack.Builder waitForActiveShards(int waitForActiveShards) {
return setParameter("wait_for_active_shards", waitForActiveShards);
}
public GenericResultReindexActionHack.Builder timeout(long timeout) {
return setParameter("timeout", timeout);
}
public GenericResultReindexActionHack.Builder requestsPerSecond(double requestsPerSecond) {
return setParameter("requests_per_second", requestsPerSecond);
}
public GenericResultReindexActionHack build() {
return new GenericResultReindexActionHack(this);
}
}
}
Step 2) Use of this class with a query then requires you to pass in the query as part of the source, then remove the '\n' characters:
ImmutableMap<String, Object> sourceMap = ImmutableMap.of("index", sourceIndex, "query", qb.toString().replaceAll("\\\\n", ""));
ImmutableMap<String, Object> destMap = ImmutableMap.of("index", destIndex);
GenericResultReindexActionHack reindex = new GenericResultReindexActionHack.Builder(sourceMap, destMap)
.waitForCompletion(false)
.conflicts("proceed")
.size(5000L)
.script(reindexScript)
.setParameter("slices", 10)
.build();
JestResult result = handleResult(reindex);
String task = result.getJsonString();
return (task);
Note the reindexScript parameter is of type org.elasticsearch.script.
This is a messy, hack-y way of getting around the limitations of Jest, but it seems to work. I understand that by doing it this way there may be some limitations to what may be acceptable in the input formatting...

Processing a list of rx.Observables

I want to process a List<Observable<MyResponse>>. How can I do this?Had a look at Observable.zip() but unable to arrive at a conclusion.
Code snippet of what I wanna do:
List<String> myList = myMethod(); // myMethod() returns a List<String> of any possible length.
List<Observable<MyResponse>> observables = new ArrayList<>();
for (String value : myList) {
observableList.add(myAPI.getData(value))
}
// How to? : For each Observable<MyResponse> in observables call a method myProcessor(MyResponse, ...)

You can flatMap the myList:
Flowable.fromIterable(myList)
.flatMap(value -> myAPI.getData(value))
.doOnNext(myResponse -> { /* ... */ })
...

I tried again & the following worked for me:
List<Observable<MyResponse>> observables = new ArrayList<>();
List<String> values = getValues();
for (String value : values) {
observables.add(myAPI.myMethod());
}
Observable.zip(observables, new FuncN() {
#Override
public Object call(Object... args) {
for (Object arg : args) {
process((MyResponse(arg));
}
return null;
}
}).toBlocking().first();

How to get Workflow event type and Processstep name or Arguments in to java service in AEM?

I created a workflow for whenever I upload or remove asset from /content/dam/MyAsset folder I am able to Triggering JavaServices. I create two launcher with Created Event Type and Removed Event Type.
I need to get That event Type and Process step name or arguments in to Services
inside Triggering execute function
Here is My Code:
public void execute(WorkItem arg0, WorkflowSession arg1, MetaDataMap arg2)
throws WorkflowException
{
log.info("Workflow created ::::: ");
}
Is there any way to get Launcher event type and process arguments]to Services ?

You can not get launcher event type as that information is not passed down to the workflow. What you could do is check the payload to identify if the node exists or not. The workflow called on removal will have path to the node that is removed and if you try to resolve that path it will give you exception (yeah, bad approach but thats the only possibility).
For the process step arguments passed, all that information is available in meta data. Look into arg0.getWorkflowData().getMetaDataMap().get("MyIDkey",<Type>); where arg0 is your WorkItem instance.

Why don't you use a org.osgi.service.event.EventHandler instead? You will have all the information you need. Here is a small code snippet that waits for an event and adds the relevant data into a map that is then passed to the JobManager to generate a new Job that is executed by a matching JobConsumer:
import org.apache.sling.event.jobs.Job;
import org.apache.sling.event.jobs.JobManager;
import org.apache.sling.event.jobs.consumer.JobConsumer;
// and more imports...
#Service
#Component(immediate = true, policy = ConfigurationPolicy.OPTIONAL, description = "Listen to page modification events.")
#Properties(value = { #Property(name = "event.topics", value = { PageEvent.EVENT_TOPIC, DamEvent.EVENT_TOPIC}, propertyPrivate = true),
#Property(name = JobConsumer.PROPERTY_TOPICS, value = ModificationEventHandler.JOB_TOPICS, propertyPrivate = true) })
public class ModificationEventHandler implements EventHandler, JobConsumer {
/**
* Modification Job Topics.
*/
public static final String JOB_TOPICS = "de/rockware/aem/modification";
#Reference
private JobManager jobManager;
#Override
public void handleEvent(Event event) {
logger.trace("Checking event.");
PageEvent pageEvent = PageEvent.fromEvent(event);
DamEvent damEvent = DamEvent.fromEvent(event);
Map<String, Object> properties = new HashMap<>();
if (damEvent != null) {
// DamEvent is not serializable, so we cannot add the complete event to the map.
logger.trace("Event on {} is a dam event ({}).", damEvent.getAssetPath(), damEvent.getType().name());
properties.put(DAM_EVENT_ASSET_PATH, damEvent.getAssetPath());
}
if (pageEvent != null) {
logger.trace("Event is a page event.");
properties.put(PAGE_EVENT, pageEvent);
}
logger.trace("Adding new job.");
jobManager.addJob(JOB_TOPICS, properties);
}
#Override
public JobResult process(Job job) {
if (job.getProperty(PAGE_EVENT) != null) {
PageEvent pageEvent = (PageEvent) job.getProperty(PAGE_EVENT);
if(pageEvent.isLocal()) {
Iterator<PageModification> modificationsIterator;
modificationsIterator = pageEvent.getModifications();
while (modificationsIterator.hasNext()) {
PageModification modification = modificationsIterator.next();
logger.info("Handling modification {} on path {}.", modification.getType(), modification.getPath());
if(isRelevantModification(modification.getType())) {
logger.info(modification.getPath());
}
}
} else {
logger.trace("Page event is not local.");
}
} else if (job.getProperty(DAM_EVENT_ASSET_PATH) != null) {
String assetPath = (String) job.getProperty(DAM_EVENT_ASSET_PATH);
logger.trace(assetPath);
backupService.trackModification(assetPath);
} else {
logger.trace("Invalid event type. Cannot help.");
}
return JobResult.OK;
}
}

Some informations of how handle the main() args in Java [duplicate]

This question already has answers here:
How to make IntelliJ prompt me for command line arguments
(3 answers)
Closed 8 years ago.
I have to develop a command line Java application in which the main() method accept 2 String parameters named respetivelly partitaIVA and nomePDF.
So, as starting point, I created this simple Main class:
public class Main {
public static void main(String[] args) {
System.out.println("Hello World !!!");
}
}
I think that I can perform this minimalistic application from the Windows console and that I can perform my application passion these parameters to it doing something like this in the Windows console (or in the Linux shell):
java Main 123456789 myDocument.pdf
and I think that I can retrieve it inside my application modifying the original code in this way:
public class Main {
public static void main(String[] args) {
System.out.println("Hello World !!!");
String partitaIVA = args[0];
String nomePDF = args[1];
}
}
So now I have 2 doubts about this topic:
1) I know that I can perform this application specifying my 2 parameters using the Windows command line or the Linux shell but can I do the same thing into my IDE console? Specifically in the Run tab of IntelliJ?
2) Can I specify in some way that the parameters that the user can specify are only 2?

1) There is something called run/debug configuration https://www.jetbrains.com/idea/help/creating-and-editing-run-debug-configurations.html (here are also sone details about the specific options you have: https://www.jetbrains.com/idea/help/creating-and-editing-run-debug-configurations.html#d1628194e152)
2) No, you can only print an error and guide the user

You should invest the time in learning a modern CLI argument parser:
I prefer JewelCli
<dependency>
<groupId>com.lexicalscope.jewelcli</groupId>
<artifactId>jewelcli</artifactId>
<version>0.8.9</version>
</dependency>
Here is an example that can be used as a base class:
public class Main
{
private static final Logger LOG;
static
{
LOG = LoggerFactory.getLogger(Main.class);
}
private static Args init(#Nonnull final String[] args)
{
final Cli<Args> cli = CliFactory.createCli(Args.class);
try
{
return cli.parseArguments(args);
}
catch (final ArgumentValidationException e)
{
for (final ValidationFailure vf : e.getValidationFailures())
{
LOG.error(vf.getMessage());
}
LOG.info(cli.getHelpMessage());
System.exit(2); // Bash standard for arg parsing errors
return null; // This is to make the compiler happy!
}
}
private static List<String> parseKey(#Nonnull final String key)
{
return new ArrayList<String>(Arrays.asList(key.toLowerCase().split("\\.")));
}
#SuppressWarnings("unchecked")
private static Map<String, Object> addNode(#Nonnull Map<String, Object> node, #Nonnull final List<String> keys, #Nonnull final String value)
{
if (keys.isEmpty())
{
return node;
}
else if (keys.size() == 1)
{
node.put(keys.remove(0), value.trim());
return node;
}
else if (node.containsKey(keys.get(0)))
{
return addNode((Map<String, Object>) node.get(keys.remove(0)), keys, value);
}
else
{
final Map<String, Object> map = new HashMap<String, Object>();
node.put(keys.remove(0), map);
return addNode(map, keys, value);
}
}
public static void main(final String[] args)
{
try
{
final Args a = init(args);
final Properties p = new Properties();
p.load(new FileInputStream(a.getInputFile()));
final HashMap<String, Object> root = new HashMap<String, Object>();
for (final String key : p.stringPropertyNames())
{
addNode(root, parseKey(key), p.getProperty(key));
}
switch (a.getFormat().toLowerCase().charAt(0))
{
case 'j': LOG.info(mapToJson(root)); break;
case 'b' : LOG.info(Strings.bytesToHex(mapToCbor(root))); break;
case 'x' : LOG.error("XML not implemented at this time!"); break;
default : LOG.error("Invalid format {}", a.getFormat());
}
}
catch (IOException e)
{
throw new RuntimeException(e);
}
}
interface Args
{
#Option(shortName = "i", longName = "input", description = "Properties file to read from.")
File getInputFile();
#Option(shortName = "o", longName = "output", description = "JSON file to output to.")
File getOutputFile();
#Option(shortName = "f", longName = "format", description = "Format of output Json|Binary|Xml")
String getFormat();
#Option(helpRequest = true, description = "Display Help", shortName = "h")
boolean getHelp();
}
}

In Intellij (Linux) you do:
Press Alt + Shift + F10 (the run shortcut)
Press right key
Go down to Edit
Then press Tab to go to "Program arguments".
This is where you pass the arugments in IntelliJ. After that just hit run.