When introducing parallel processing to an application where multiple save entity calls are being made, I see prior dev has chosen to do it via Spring Integration using split().channel(MessageChannels.executor(Executors.newfixedThreadPool(10))).handle("saveClass","saveMethod").aggregate().get() - where this method is mapped to a requestChannel using #Gateway annotation. My question is this task seems to be simpler to do using the parallelStream() and forEach() method. Does IntergrationFlow provide any benefit in this case?
If you really do a plain in-memory data processing where Java's Stream API is enough, then indeed you don't need the whole messaging solution like Spring Integration. But if you deal with distributed requirements to process data from different systems, like from HTTP to Apache Kafka or DB, then it is better to use some tool which allows you smoothly to connect everything together. Also: no one stops you to use Stream API in the Spring Integration application. In the end all your code is Java anyway. Please, learn more what is EIP and why we would need a special framework to implement this messaging-based solutions: https://www.enterpriseintegrationpatterns.com/
Related
I am new to Redis and planning to use it as in memory cache. I am using Lettuce 5.2 client for it.
I have multiple applications which will use redis as in memory cache. My idea is to write library using lettuce like wrapper which can be used by multiple application in order to interact with Redis. That library will manage connection pooling, redis failover cases and command execution etc. so that application writer should not worry about all this and just need to use my library.
Now for this library i am confused on below points :
1) Should i use Spring data redis (it also supports lettuce)? If my objective is to create library then first of all, can i use spring data redis ?
2) What all advantage Spring data redis will give me. I have checked documentation https://docs.spring.io/spring-data/data-redis/docs/current/reference/html/#reference
3) If i don't use Spring data redis then I will just use only lettuce and create client, connention pool etc myself.
I am confused whether i should use spring data redis for creating library or not ?
Can you please help me to clear my confusion ?
You are able to implement custom Repository methods in Spring Data, which has been outlined in other answers on SO such as here: How to add custom method to Spring Data JPA.
So you can easily combine both the out of the box Spring Data Redis functionality with custom Lettuce method code for a Spring Data Repository, I would suggest starting with Spring Data, and if you need to fine tune anything beyond that then write a custom methods with Lettuce.
So long as you can use the same connection pool in Lettuce as Spring Data Redis, you should be able to share that as a resource, the same way you can consider Threads as a resource.
No one can really give you a yes no answer as to what libraries you should or shouldn't use, hopefully you have enough information now to make progress going forward.
Best Architecture for implementing a WebService that takes requests from one side, save and enhance that and then call another service with new parameters.
is there any special Design Pattern for this?
There's not a lot to go on, but from what you've said it sounds like a job for "pipes and filters"!
To get a more precise answer, you might want to ask yourself some more detailed questions:
If you need to do any validation or transformation of the incoming message? Will you want to handle all requests the same way, or are there different types? Are the external services likely to change, and if so, will they do this frequently? What do you want to do if the final web service call fails (should you rollback the database record?)? How do you want to report failures/responses - do you need to report these back? Do you need a mechanism to track the progress of a particular request?
Since you are looking for a design pattern, I think you might want to compare the pros and cons of using microservices orchestration vs choreography in the context of your project.
If you do not need an immediate response to the calling system I would suggest to you to use event-driven approach if that's feasible. So instead of REST services, you will have a message broker and your services will be subscribed for certain events. This will hide your consumers behind the message broker which will make your system less coupled.
This can be implemented via Spring Cloud Stream, where you will have a Sink (microservice producing events, transformer - microservice that makes intermediate transformations possible and a source - microservice that receives a final result for further processing).
Another possible case could be Camel. It has basically all the integration patterns built in, so it should not be a problem to implement the solution either based on REST APIs or events.
I'd like to trace my async application with some key checkpoint.
Is there any popular framework I can use?
For example, I may choose to use vert.x or any other java async framework. For each request/response cycle, I'd make some checkpoint to log something while these points might happen in different threads.
I'd like to see an aggregated view of 1 request to see what's going on. Supporting distributed case would be better but single JVM is good to go.
What you are looking for is OpenTracing. It's an API that allows you to have distributed tracing features in a way that is vendor agnostic.
For your specific case, you'd have to handle the context propagation yourself, as there's no other (reliable) way to do that on OpenTracing yet for teh async case. For other cases (sync JAX-RS, Servlets, Spring Boot, ...), it would be safe to use the native framework integration and/or the Java agent rules.
For Vert.x, you'll need to inject the span context into Vert.x' message, and extract this context later on.
There's an example of OpenTracing + Vert.x on the Hawkular APM example directory, that might help you get started. Note, however, that you might want to use another backend should you decide to move forward, as we (Hawkular APM team) decided to join forces with Jaeger for the OpenTracing backend.
I'm working on an ETL project. I've been using spring integration for a long time. The data source is currently files or chronicle but it may change to live streams and volumes are likely to grow. There is a potential to move on to big data solutions (hadoop, spark etc) in the future.
Based on this I need a comparison between spring integration and reactive streams? Why would anyone use one over the other (or am I wrong in the first place trying to compare the two)? Scenarios (if any) where you think they could be used together?
Actually, both of them can be used together. Check out the documentation for Reactive Spring Integration.
I have been messing around with Spring Integration, and I'd like to use its capabilities in a project. I have two different applications, one is a business-sided one, which when users do certain actions, should send messages to another application. The other application should receive these messages (using some kind of queue (rabbitmq or another) to handle big loads) and store them, so it can use it to create realtime statistics of a number of applications running.
These messages will just contain information of the actions of the users, for example "bought N of product X" or "Used searchbar for Y".
This scenario is of course pretty simple, but I don't want to use any kind of XML configuration in my Spring applications. The examples I have seen so far all rely on XML, but I want to use some kind of annotations instead.
Spring Integration java dsl is already released which would serve your purpose.
A gentle introduction can be found at
https://dzone.com/articles/spring-integration-java-dsl
You can also checkout Camel (has a fluent dsl for EIP) which also gels well with spring