Develop Reference

How to poll a directory and if the any xlsx file copied in the directory call an REST api to load the feed

I have a use case, i want to poll a directory, if the any .*xlsx file get pasted in that directory, i want to call a post rest API that will load the data.
I'm not able to find the my way, please suggest some way to do this.

I believe that you are looking for fully working sample and I doubt that is going to be one since your business task might not be the same what other people are doing.
Although we won't mind if you contribute back such a sample: https://github.com/spring-projects/spring-integration-samples.
So, to build a logic we need to provide an IntegrationFlow: https://docs.spring.io/spring-integration/docs/current/reference/html/dsl.html#java-dsl.
To read files from a dir we need to use a Files.inboundAdapter() with respective polling policy.
You may do some transformation (.transform()) about polled file content or so.
Call the REST service via Http.outboundGateway()
Do the post-process.
Something like this:
#Bean
public IntegrationFlow fileReadingFlow() {
return IntegrationFlows
.from(Files.inboundAdapter(new File("myDir"))
.patternFilter("*.xlsx"),
e -> e.poller(Pollers.fixedDelay(1000)))
.transform(...)
.handle(Http.outboundGateway("")
.expectedResponseType(String.class))
.transform(...)
.get();
}
(Haven't checked as working since I don't know what is your XSLT content and how you call the REST service.)
This sample does something with files reading to gather some ideas: https://github.com/spring-projects/spring-integration-samples/tree/main/applications/file-split-ftp

We have a system that does (almost) the same (just xml instead of xlsx) and we use Apache Camel https://camel.apache.org/
Integration is good with Spring Boot. You just need to define your route from("file:///<path").to("http:<host>:port/<path>) and it will do probably what you need.
Might need to tweek the line of code to get filtering and maybe add some transformation but it is a nice peace of software.

Apache camel losing trace id and span id after camel split eip

I need to have trace id and span id available in all my logs. However I am observing that after the first splitter in my camel route, I can no longer see the trace id and span id in my logs.
[traceId: spanId:] INFO ---
Is there any way to enable back the tracing information?
From the Camel Documentation I have tried to start the tracing after the split by using
context.setTracing(true)
But looks like this is not working.
Am I missing anything, please help.

You probably have the traceId and spanId stored in the exchange message headers which are lost after the split.
A solution is to store them in the exchange properties(before the split) which are stored for the entire processing of the exchange(see Passing values between processors in apache camel).
If you are using the Java DSL you can use:
.setProperty("traceId ", constant("traceIdValue"))
.setProperty("spanId", constant("spanIdValue"))
You can use the Simple Expression Language(https://camel.apache.org/manual/latest/simple-language.html) to access the properties after the split using exchangeProperty.property_name.
Example:
.log(LoggingLevel.INFO, "[traceId:${exchangeProperty.traceId} spanId:${exchangeProperty.spanId}]")

When you use split, a new and old exchange will be created and to pass exchange properties downstream, you would need to use an aggregator to do so.
Example:
.split().tokenize(System.lineSeparator()).aggregationStrategy(new YourAggregationStrategyClass())

Iterating over user user defined collections in Apache came route

I got a requirement that i have on main route and child route. In the main route will get the list of objects, there i need to make a call to child route for each individual elements in the list.
Then in Child route will make a call to web-service by appending that element as one of the parameter.
Code:
from("direct:SupplierRoute")
.choice()
.when(header(IS_SUPPLIER_AVAILABLE).isEqualTo(true))
.split(body())
.parallelProcessing()
.streaming()
.to("direct:SUPGetHotelAggregatorRatesRQ")
.bean(parallelProcessingRequestProcessor)
.end()
.end()
.end();
from("direct:SUPGetHotelAggregatorRatesRQ")
.process(startOperation(DISTRIBUTION, GET_HOTEL_AGGREGATOR_RATES_API_GENERATE_VM_REQUEST))
.to("velocity:velocity/GetHotelAggregatorRatesRQToGetHotelSupplierRatesRQ.vm")
.process(endOperation(DISTRIBUTION, GET_HOTEL_AGGREGATOR_RATES_API_GENERATE_VM_REQUEST))
.end();
I'm setting the value to the exchange body as below,
public static final List<HotelRefs.HotelRef> supplierHotelRefs = new ArrayList();
exchange.getIn().setBody(supplierHotelRefs);
But the above code sample was not working,If anyone let us know if there is any approach in camel to iterate over user defined collections at route level.
Thanks,
Raghavan

Put a log statement after the Splitter and run the code.
.split(body())
.log("How many log lines do you receive?")
If you get just one log line, the Splitter does not work as expected. You could then try to use another Camel version.
If you get as many log lines as your ArrayList contains elements, the iteration works fine and you've got another problem. You should then find out the real problem and ask a new question.
If you get no log line at all, your condition in when is not satisfied.

Testing HLS using JMeter

I am using JMeter to test HLS playback from a Streaming Server. So, the first HTTP request is for a master manifest file(m3u8). Say,
http://myserver/application1/subpath1/file1.m3u8
The reply to this will result in a playlist something like,
subsubFolder/360p/file1.m3u8
subsubFolder/480p/file1.m3u8
subsubFolder/720p/file1.m3u8
So, next set of URLs become
http://myserver/application1/subpath1/subsubFolder/360p/file1.m3u8
http://myserver/application1/subpath1/subsubFolder/480p/file1.m3u8
http://myserver/application1/subpath1/subsubFolder/720p/file1.m3u8
Now, individual reply to these further will be an index of chunks, like
0/file1.ts
1/file1.ts
2/file2.ts
3/file3.ts
Again, we have next set of URLs as
http://myserver/application1/subpath1/subsubFolder/360p/0/file1.ts
http://myserver/application1/subpath1/subsubFolder/360p/1/file1.ts
http://myserver/application1/subpath1/subsubFolder/360p/2/file1.ts
http://myserver/application1/subpath1/subsubFolder/360p/3/file1.ts
This is just the case of one set(360p). There will be 2 more sets like these(for 480p, 720p).
I hope the requirement statement is clear uptill this.
Now, the problem statement.
Using http://myserver/application1 as static part, regex(.+?).m3u8 is applied at 1st reply which gives subpath1/subsubFolder/360p/file1. This, is then added to the static part again, to get http://myserver/application1/subpath1/subsubFolder/360p/file1 + .m3u8
The problem comes at the next stage. As, you can see, with parts extracted previously, all I'm getting is
http://myserver/application1/subpath1/subsubFolder/360p/file1/0/file1.ts
The problem is obvious, an extra file1, 360p/file1 in place of 360p/0.
Any suggestions, inputs or alternate approaches appreciated.

If I understood the problem correctly, all you need is the file name as the other URLs can be constructed with it. Rather than using http://myserver/application1 as static part of your regex, I would try to get the filename directly:
([^\/.]+)\.m3u8$
# match one or more characters that are not a forward slash or a period
# followed by a period
# followed by the file extension (m3u8)
# anchor the whole match to the end
Now consider your urls, e.g. http://myserver/application1/subpath1/subsubFolder/360p/file1.m3u8, the above regex will capture file1, see a working demo here. Now you can construct the other URLs, e.g. (pseudo code):
http://myserver/application1/subpath1/subsubFolder/360p/ + filename + .m3u8
http://myserver/application1/subpath1/subsubFolder/360p/ + filename + /0/ + filename + .ts
Is this what you were after?

Make sure you use:
(.*?) - as Regular Expression (change plus to asterisk in your regex)
-1 - as Match No.
$1$- as template
See How to Load Test HTTP Live Media Streaming (HLS) with JMeter article for detailed instructions.

If you are ready to pay for a commercial plugin, then there is an easy and much more realistic solution which is a plugin for Apache JMeter provided by UbikLoadPack:
Besides doing this job for you, it will simulate the way a player would read the file. It will also scale much better than any custom script or player solution.
It supports VOD and Live which are quite difficult to script.
See:
http://www.ubik-ingenierie.com/blog/easy-and-realistic-load-testing-of-http-live-streaming-hls-with-apache-jmeter/
http://www.ubik-ingenierie.com/blog/ubikloadpack-http-live-streaming-plugin-jmeter-videostreaming-mpegdash/
Disclaimer, we are the providers of this solution

Apache camel Composed Message Processor

I am using Apache Camel in my Application. I am trying to use Composed Message Processor. I have exchange whose body contains some URLs to hit and by using split(body(), MyAggregationStrategy()), I am trying to get the data from urls and using Aggregation Strategy want to combine each data. But there is a problem where I am stuck. If there is some invalid url on the first line of the body then it happens that aggregation is working fine but it is not moving to the next processor and if invalid url is anywhere else except first line than it is working fine..
please help,
Here is the code for reference
onException(HttpOperationFailedException.class).handled(true)
.retryAttemptedLogLevel(LoggingLevel.DEBUG)
.maximumRedeliveries(5).redeliveryDelay(3000)
.process(new HttpExceptionProcessor(exceptions));
from("jms:queue:supplier")
.process(
new RequestParserProcessor(payloadDetailsMap,
metaDataDetailsPOJO, routesEndpointNamePOJO))
.choice().when(new AggregateStrategy(metaDataDetailsPOJO))
.to("direct:aggregate").otherwise().to("direct:single");
from("direct:aggregate").process(new SplitBodyProcessor())
.split(body(), new AggregatePayload(aggregatePayload))
.to("direct:aggregatepayloadData").end()
.to("direct:payloadDataAggregated").end();
from("direct:aggregatepayloadData").process(basicProcessor)
.recipientList(header(ApplicationConstants.URL));
from("direct:payloadDataAggregated")
.process(
new AggregateJsonGenerator(aggregatePayload,
payloadDetailsMap, metaDataDetailsPOJO)).
In this code AggregateJsonProcessor is never called if there some invalid url on the first hit..

You probably need to set continue(true) in your OnException code. See here:
http://camel.apache.org/exception-clause.html