I want to implement a feature that user connects his account with external applications (similar feature is in Facebook). User has to log in to external application and grant permission to access data by my application.
Once user connected an external app, data will be exchanged in background using access and refresh tokens.
Application architecture is:
SPA front-end (Angular)
REST API (Spring), multiple nodes
ScyllaDB
Envoy proxy (with JWT verification)
The first idea is to use Spring OAuth2 Client. However, some changes need to be made:
there is no Principal because JWT is verified by Envoy proxy and X-USER-ID header is added
REST API is stateless and we shouldn't store authorization code in session
even with sessions, there are multiple nodes and we need to share authorization code between nodes
custom URL, e.g. /app_name/connect instead of /oauth2/authorization/app_name
redirect URL may be invalid (but it's verified by Spring's filter)
How this could work:
user click "Connect with app" in SPA
SPA redirects user to /oauth2/authorization/app_name (or custom URL)
Spring redirects user to external app's authentication server
user authenticates and grants permissions
external app redirects user back to Spring (or straight to SPA?)
Spring redirects user back to SPA (or SPA sends access token to REST API?)
Despite Spring Security components can be replaced, many of them are coupled and you need to rewrite OAuth2 Client flow almost from scratch. Maybe I'm doing something wrong and it can be achieved easier.
What I already did:
http
.cors().and()
.csrf().disable()
.authorizeRequests().anyRequest().permitAll().and()
.oauth2Client(); // get rid of these two filters?
#Configuration
#RequiredArgsConstructor
public class OAuth2ClientConfig {
private final CassandraTemplate cassandraTemplate;
// overriding original client service - we need to store tokens in database
#Bean
public OAuth2AuthorizedClientService authorizedClientService(
CassandraTemplate cassandraTemplate,
ClientRegistrationRepository clientRegistrationRepository) {
return new ScyllaOAuth2AuthorizedClientService(cassandraTemplate, clientRegistrationRepository);
}
// configure client provider to use authorization code with refresh token
#Bean
public OAuth2AuthorizedClientManager authorizedClientManager(
ClientRegistrationRepository clientRegistrationRepository,
OAuth2AuthorizedClientRepository authorizedClientRepository) {
var authorizedClientProvider =
OAuth2AuthorizedClientProviderBuilder.builder()
.authorizationCode()
.refreshToken()
.build();
var authorizedClientManager = new DefaultOAuth2AuthorizedClientManager(
clientRegistrationRepository,
authorizedClientRepository);
authorizedClientManager.setAuthorizedClientProvider(authorizedClientProvider);
return authorizedClientManager;
}
// the specs recommend to use WebClient for exchanging data instead of RestTemplate
#Bean
public WebClient webClient(OAuth2AuthorizedClientManager authorizedClientManager) {
ServletOAuth2AuthorizedClientExchangeFilterFunction oauth2Client =
new ServletOAuth2AuthorizedClientExchangeFilterFunction(authorizedClientManager);
return WebClient.builder()
.apply(oauth2Client.oauth2Configuration())
.build();
}
// override request repository - and I'm stuck there
#Bean
public AuthorizationRequestRepository<OAuth2AuthorizationRequest> authorizationRequestRepository() {
return new ScyllaOAuth2AuthorizationRequestRepository(cassandraTemplate);
}
}
Because there are multiple nodes of REST API, we can't use sessions. We need to store request somewhere, e.g. ScyllaDB, Redis, Hazelcast, etc. I decided to store it as JSON in ScyllaDB but I ran into trouble.
#Slf4j
#RequiredArgsConstructor
public final class ScyllaOAuth2AuthorizationRequestRepository implements AuthorizationRequestRepository<OAuth2AuthorizationRequest> {
private final CassandraTemplate cassandraTemplate;
private final ObjectMapper objectMapper = new ObjectMapper();
#Override
public OAuth2AuthorizationRequest loadAuthorizationRequest(HttpServletRequest request) {
Assert.notNull(request, "request cannot be null");
var stateParameter = this.getStateParameter(request);
if (stateParameter == null) {
return null;
}
return this.getAuthorizationRequest(request, stateParameter);
}
#Override
public void saveAuthorizationRequest(OAuth2AuthorizationRequest authorizationRequest, HttpServletRequest request,
HttpServletResponse response) {
Assert.notNull(request, "request cannot be null");
Assert.notNull(response, "response cannot be null");
if (authorizationRequest == null) {
this.removeAuthorizationRequest(request, response);
return;
}
var state = authorizationRequest.getState();
var userId = UUID.fromString(request.getHeader(Constants.USER_ID));
Assert.hasText(state, "authorizationRequest.state cannot be empty");
try {
// serialization of Auth2AuthorizationRequest to JSON works
cassandraTemplate.getCqlOperations().execute("insert into oauth2_requests (user_id,state,data) values (?,?,?)",
userId, state, objectMapper.writeValueAsString(authorizationRequest));
} catch (JsonProcessingException e) {
log.warn("Unable to save authorization request", e);
}
}
#Override
public OAuth2AuthorizationRequest removeAuthorizationRequest(HttpServletRequest request) {
Assert.notNull(request, "request cannot be null");
var stateParameter = this.getStateParameter(request);
if (stateParameter == null) {
return null;
}
var userId = UUID.fromString(request.getHeader(Constants.USER_ID));
var originalRequest = this.getAuthorizationRequest(request, stateParameter);
cassandraTemplate.getCqlOperations().execute("delete from oauth2_requests where user_id=? and state=?",
userId, stateParameter);
return originalRequest;
}
private String getStateParameter(HttpServletRequest request) {
return request.getParameter(OAuth2ParameterNames.STATE);
}
private UUID getUserId(HttpServletRequest request) {
return UUID.fromString(request.getHeader(Constants.USER_ID));
}
private OAuth2AuthorizationRequest getAuthorizationRequest(HttpServletRequest request, String state) {
var userId = getUserId(request);
var jsonRequest = cassandraTemplate.getCqlOperations().queryForObject(
"select data from oauth2_requests where user_id=? and state=?", String.class, userId, state);
if (StringUtils.isNotBlank(jsonRequest)) {
try {
// trying to mess with OAuth2ClientJackson2Module
var objectMapper = new Jackson2ObjectMapperBuilder().autoDetectFields(true)
.autoDetectGettersSetters(true)
.modules(new OAuth2ClientJackson2Module())
.visibility(PropertyAccessor.ALL, JsonAutoDetect.Visibility.ANY)
.build();
return objectMapper.readValue(jsonRequest, OAuth2AuthorizationRequest.class);
} catch (JsonProcessingException e) {
log.warn("Error decoding authentication request", e);
}
}
return null;
}
}
I get error when trying to deserialize JSON to OAuth2AuthorizationRequest:
Missing type id when trying to resolve subtype of [simple type, class org.springframework.security.oauth2.core.endpoint.OAuth2AuthorizationRequest]: missing type id property '#class'
Without adding OAuth2ClientJackson2Module there is another error:
Cannot construct instance of `org.springframework.security.oauth2.core.endpoint.OAuth2AuthorizationResponseType` (although at least one Creator exists): cannot deserialize from Object value (no delegate- or property-based Creator)
By the way, OAuth2ClientJackson2Module seems never used in original filters.
Maybe it's better to serialize this object Java way and store it as BLOB or do not store request in database but somewhere other.
Another part is the controller action:
// it had to be /apps/app_name/connect but in Spring OAuth2 Client it's hardcoded to append provider name at the end
#GetMapping("/apps/connect/app_name")
public void connect(HttpServletRequest request, HttpServletResponse response) throws IOException {
userAppService.authorize(request, response, "app_name");
}
To get rid of filters which verify redirect URL and have many things hardcoded:
#Service
#RequiredArgsConstructor
public class UserAppService {
private final HttpSecurity httpSecurity;
private final AuthenticationDetailsSource<HttpServletRequest, ?> authenticationDetailsSource = new WebAuthenticationDetailsSource();
private final AuthorizationRequestRepository<OAuth2AuthorizationRequest> authorizationRequestRepository;
private final ClientRegistrationRepository clientRegistrationRepository;
private final OAuth2AuthorizedClientManager authorizedClientManager;
private final OAuth2AuthorizedClientRepository authorizedClientRepository;
private final RedirectStrategy redirectStrategy = new DefaultRedirectStrategy();
public void authorize(HttpServletRequest request, HttpServletResponse response, String appName) throws IOException {
var userId = UUID.fromString(request.getHeader(Constants.USER_ID));
var authorizeRequest = OAuth2AuthorizeRequest
.withClientRegistrationId(appName)
.principal(UUIDPrincipal.fromUserId(userId))
.build();
if (isAuthorizationResponse(request)) {
var authorizationRequest = this.authorizationRequestRepository.loadAuthorizationRequest(request);
if (authorizationRequest != null) {
processAuthorizationRequest(request, response);
}
} else {
try {
OAuth2AuthorizedClient authorizedClient = authorizedClientManager.authorize(authorizeRequest);
if (authorizedClient != null) {
OAuth2AccessToken accessToken = authorizedClient.getAccessToken();
System.out.println(accessToken);
}
} catch (ClientAuthorizationException e) {
// in this URL provider name is appended at the end and no way to change this behavior
var authorizationRequestResolver = new DefaultOAuth2AuthorizationRequestResolver(clientRegistrationRepository,
"/apps/connect");
var authorizationRequest = authorizationRequestResolver.resolve(request);
this.authorizationRequestRepository.saveAuthorizationRequest(authorizationRequest, request, response);
this.redirectStrategy.sendRedirect(request, response, authorizationRequest.getAuthorizationRequestUri());
}
}
}
private void processAuthorizationRequest(HttpServletRequest request, HttpServletResponse response) throws IOException {
var authorizationRequest = this.authorizationRequestRepository.removeAuthorizationRequest(request, response);
var registrationId = (String) authorizationRequest.getAttribute(OAuth2ParameterNames.REGISTRATION_ID);
var clientRegistration = this.clientRegistrationRepository.findByRegistrationId(registrationId);
var params = toMultiMap(request.getParameterMap());
var redirectUri = UrlUtils.buildFullRequestUrl(request);
var authorizationResponse = convert(params, redirectUri);
var authenticationRequest = new OAuth2AuthorizationCodeAuthenticationToken(
clientRegistration, new OAuth2AuthorizationExchange(authorizationRequest, authorizationResponse));
authenticationRequest.setDetails(this.authenticationDetailsSource.buildDetails(request));
OAuth2AuthorizationCodeAuthenticationToken authenticationResult;
try {
var authenticationManager = httpSecurity.getSharedObject(AuthenticationManager.class);
authenticationResult = (OAuth2AuthorizationCodeAuthenticationToken) authenticationManager
.authenticate(authenticationRequest);
} catch (OAuth2AuthorizationException ex) {
OAuth2Error error = ex.getError();
UriComponentsBuilder uriBuilder = UriComponentsBuilder.fromUriString(authorizationRequest.getRedirectUri())
.queryParam(OAuth2ParameterNames.ERROR, error.getErrorCode());
if (!StringUtils.hasText(error.getDescription())) {
uriBuilder.queryParam(OAuth2ParameterNames.ERROR_DESCRIPTION, error.getDescription());
}
if (!StringUtils.hasText(error.getUri())) {
uriBuilder.queryParam(OAuth2ParameterNames.ERROR_URI, error.getUri());
}
this.redirectStrategy.sendRedirect(request, response, uriBuilder.build().encode().toString());
return;
}
// just copy-paste of original filter - trying to understand what's happening there
Authentication currentAuthentication = SecurityContextHolder.getContext().getAuthentication();
String principalName = (currentAuthentication != null) ? currentAuthentication.getName() : "anonymousUser";
OAuth2AuthorizedClient authorizedClient = new OAuth2AuthorizedClient(
authenticationResult.getClientRegistration(), principalName, authenticationResult.getAccessToken(),
authenticationResult.getRefreshToken());
this.authorizedClientRepository.saveAuthorizedClient(authorizedClient, currentAuthentication, request,
response);
String redirectUrl = authorizationRequest.getRedirectUri();
this.redirectStrategy.sendRedirect(request, response, redirectUrl);
}
private static boolean isAuthorizationResponse(HttpServletRequest request) {
return isAuthorizationResponseSuccess(request) || isAuthorizationResponseError(request);
}
private static boolean isAuthorizationResponseSuccess(HttpServletRequest request) {
return StringUtils.hasText(request.getParameter(OAuth2ParameterNames.CODE))
&& StringUtils.hasText(request.getParameter(OAuth2ParameterNames.STATE));
}
private static boolean isAuthorizationResponseError(HttpServletRequest request) {
return StringUtils.hasText(request.getParameter(OAuth2ParameterNames.ERROR))
&& StringUtils.hasText(request.getParameter(OAuth2ParameterNames.STATE));
}
// copy paste - not tested this code yet
static MultiValueMap<String, String> toMultiMap(Map<String, String[]> map) {
MultiValueMap<String, String> params = new LinkedMultiValueMap<>(map.size());
map.forEach((key, values) -> {
if (values.length > 0) {
for (String value : values) {
params.add(key, value);
}
}
});
return params;
}
static OAuth2AuthorizationResponse convert(MultiValueMap<String, String> request, String redirectUri) {
String code = request.getFirst(OAuth2ParameterNames.CODE);
String errorCode = request.getFirst(OAuth2ParameterNames.ERROR);
String state = request.getFirst(OAuth2ParameterNames.STATE);
if (StringUtils.hasText(code)) {
return OAuth2AuthorizationResponse.success(code).redirectUri(redirectUri).state(state).build();
}
String errorDescription = request.getFirst(OAuth2ParameterNames.ERROR_DESCRIPTION);
String errorUri = request.getFirst(OAuth2ParameterNames.ERROR_URI);
return OAuth2AuthorizationResponse.error(errorCode)
.redirectUri(redirectUri)
.errorDescription(errorDescription)
.errorUri(errorUri)
.state(state)
.build();
}
}
Client service to stored authorized clients in database:
#RequiredArgsConstructor
public class ScyllaOAuth2AuthorizedClientService implements OAuth2AuthorizedClientService {
private final CassandraTemplate cassandraTemplate;
private final ClientRegistrationRepository clientRegistrationRepository;
#Override
#SuppressWarnings("unchecked")
public OAuth2AuthorizedClient loadAuthorizedClient(String clientRegistrationId, String principal) {
var id = BasicMapId.id("userId", principal).with("appCode", clientRegistrationId);
var userApp = cassandraTemplate.selectOneById(id, UserApp.class);
if (userApp != null) {
var clientRegistration = getClientRegistration(clientRegistrationId);
var accessToken = getAccessToken(userApp);
var refreshToken = getRefreshToken(userApp);
return new OAuth2AuthorizedClient(clientRegistration, principal, accessToken, refreshToken);
} else {
return null;
}
}
#Override
public void saveAuthorizedClient(OAuth2AuthorizedClient authorizedClient, Authentication principal) {
Assert.notNull(authorizedClient, "authorizedClient cannot be null");
Assert.notNull(principal, "principal cannot be null");
var userApp = new UserApp();
userApp.setUserId((UUID) principal.getPrincipal());
userApp.setAppCode(authorizedClient.getClientRegistration().getClientId());
if (authorizedClient.getAccessToken() != null) {
userApp.setAccessToken(authorizedClient.getAccessToken().getTokenValue());
userApp.setAccessTokenType(OAuth2AccessToken.TokenType.BEARER.getValue());
userApp.setAccessTokenScopes(authorizedClient.getAccessToken().getScopes());
userApp.setAccessTokenIssuedAt(authorizedClient.getAccessToken().getIssuedAt());
userApp.setAccessTokenExpiresAt(authorizedClient.getAccessToken().getExpiresAt());
}
if (authorizedClient.getRefreshToken() != null) {
userApp.setRefreshToken(authorizedClient.getRefreshToken().getTokenValue());
userApp.setRefreshTokenIssuedAt(authorizedClient.getRefreshToken().getIssuedAt());
userApp.setRefreshTokenExpiresAt(authorizedClient.getRefreshToken().getExpiresAt());
}
cassandraTemplate.insert(userApp);
}
#Override
public void removeAuthorizedClient(String clientRegistrationId, String principal) {
var id = BasicMapId.id("userId", principal).with("appCode", clientRegistrationId);
cassandraTemplate.deleteById(id, UserApp.class);
}
private ClientRegistration getClientRegistration(String clientRegistrationId) {
var clientRegistration = this.clientRegistrationRepository.findByRegistrationId(clientRegistrationId);
if (clientRegistration == null) {
throw new DataRetrievalFailureException(
"The ClientRegistration with id '" + clientRegistrationId + "' exists in the data source, "
+ "however, it was not found in the ClientRegistrationRepository.");
}
return clientRegistration;
}
private OAuth2AccessToken getAccessToken(UserApp userApp) {
return new OAuth2AccessToken(
OAuth2AccessToken.TokenType.BEARER,
userApp.getAccessToken(),
userApp.getAccessTokenIssuedAt(),
userApp.getAccessTokenExpiresAt(),
userApp.getAccessTokenScopes());
}
private OAuth2RefreshToken getRefreshToken(UserApp userApp) {
return new OAuth2RefreshToken(userApp.getRefreshToken(), userApp.getRefreshTokenIssuedAt());
}
}
Too much code overwrite. I need to make it as simple as possible.
Currently I'm struggling with storing authorize request in database.
How to do it Spring way but to keep the app architecture given at the beginning of this question?
Any way to configure OAuth2 Client without hardcoded URL like /oauth2/authorization/provider_name?
Maybe it's better to do the whole OAuth2 flow client-side (within SPA) and the SPA should send access and request token to REST API (to store the tokens in order to be able to exchange data with external app)?
In OAuth2 wording, REST APIs are resource-servers, not clients.
What you can do is have
your proxy be transparent to OAuth2 (forward requests with their JWT access-token authorization header and responses status code)
configure each REST API as resource-server. Tutorials there: https://github.com/ch4mpy/spring-addons/tree/master/samples/tutorials.
add an OAuth2 client library to your Angular app to handle tokens and authorize requests. My favorite is angular-auth-oidc-client
probably use an intermediate authorization-server for identity federation (Google, Facebook, etc., but also internal DB, LDAP, or whatever is needed), roles management, MFA,... Keycloak is a famous "on premise" solution, but you can search for "OIDC authorization-server" in your favorite search engine and have plenty of alternate choices, including SaaS like Auth0 or Amazon Cognito.
This is fully compatible with distributed architectures and micro-services (session-less is the default configuration for resource-servers in the tutorials I linked).
Two cases for a micro-service delegating some of its processing to another resource-server:
the "child" request is made on behalf of the user who initiated the request => retrieve original access-token from Authentication instance in security-context and forward it (set it as Bearer authorization header for the sub-request)
the "child" request is not made on behalf of a user => client-credentials must be used (the micro-services acquires a new access-token in its own name to authorize the sub request). Refer to spring-boot-oauth2-client and your preferred REST client docs for details (WebClient, #FeignClient, RestTemplate).
I'm trying to send a formData post request (using axios) to my backend (springboot) but I'm not sure of the proper way to do it. My plan is to pass the data through the controller to a service that will utilize it.
Axios call -
startStreamLocation() {
const location = new FormData();
location.set("accuracy", this.accuracy)
location.set("lat", this.lat)
location.set("lng", this.lng)
location.set("timeStamp", this.timeStamp)
axios.post("http://localhost:8080/api/v1/location/request-location", location,
{headers: {'Content-Type': 'application/json'}})
},
Controller -
#PostMapping(value = "request-location")
public ResponseEntity<?> requestLocation() {
connectionRequestService.addDataToStream();
return new ResponseEntity<Authenticator.Success>(HttpStatus.OK);
}
Service -
public void addDataToStream() {
BasicAWSCredentials awsCredentials = new BasicAWSCredentials(awsAccessKey, awsSecretKey);
AmazonKinesis kinesisClient = AmazonKinesisClient.builder()
.withCredentials(new AWSStaticCredentialsProvider(awsCredentials))
.withRegion(awsRegion)
.build();
PutRecordsRequest putRecordsRequest = new PutRecordsRequest();
putRecordsRequest.setStreamName("location-stream");
List <PutRecordsRequestEntry> putRecordsRequestEntryList = new ArrayList<>();
PutRecordsRequestEntry putRecordsRequestEntry = new PutRecordsRequestEntry();
putRecordsRequestEntry.setData(ByteBuffer.wrap(( INJECT DATA HERE ).getBytes()));
putRecordsRequestEntry.setPartitionKey(String.format("partitionKey-%d"));
putRecordsRequestEntryList.add(putRecordsRequestEntry);
putRecordsRequest.setRecords(putRecordsRequestEntryList);
PutRecordsResult putRecordsResult = kinesisClient.putRecords(putRecordsRequest);
System.out.println("\nData sent successfully... \n" + putRecordsResult);
try {
Thread.sleep(1000);
}
catch (InterruptedException exception) {
throw new RuntimeException(exception);
}
}
Since you want to send form data to the server, you would need to change the Content-Type header in your Axios call to multipart/form-data. This helps the server understand the resource type being sent by the client.
On the server end, you'll want to read this form data. I can think of the following two ways to do that
Use #RequestParam to read individual form keys. For example, if my form data contains a key named Foo, I'd read it on the server end as this
#PostMapping(value = "/form-data")
public void readFormData( #RequestParam(value = "Foo") String foo )
Use #RequestBody to map the form data to a MultiValueMap which can be then read from like a normal map. Here's the code snippet for the same
#PostMapping(value = "/form-data")
public void readFormData( #RequestBody MultiValueMap<String,String> formData )
Please have a look at my codes below. The Java codes seemed to work just fine, but localhost:8080 gives me the error code 404 when I try to access it. I want to make localhost 8080 work. Please let me know if you need further information.
Application
#SpringBootApplication(exclude = { ErrorMvcAutoConfiguration.class })
// exclude part is to elimnate whitelabel error
#EnableScheduling
public class Covid19TrackerApplication {
public static void main(String[] args) {
SpringApplication.run(Covid19TrackerApplication.class, args);
}
}
Controller
#Controller
public class HomeController {
CovidDataService covidDataService;
#RequestMapping("/")
public #ResponseBody String home(Model model) {
model.addAttribute( "locationStats", covidDataService.getAllStats());
return "home";
}
}
Main Code
#Service
public class CovidDataService {
private static String Covid_Data_URL = "https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_time_series/time_series_covid19_confirmed_global.csv";
private List<LocationStats> allStats = new ArrayList<>();
public List<LocationStats> getAllStats() {
return allStats;
}
#PostConstruct//?
#Scheduled(cron = "* * 1 * * *") //????
// * sec * min *hour and so on
public void fetchCovidData() throws IOException, InterruptedException {
List<LocationStats> newStats = new ArrayList<>(); // why we are adding this? To prevent user get an error while we are working on new data.
HttpClient client = HttpClient.newHttpClient();
HttpRequest request = HttpRequest.newBuilder()
.uri(URI.create(Covid_Data_URL))
.build(); // uri = uniform resource identifier
HttpResponse<String> httpResponse = client.send(request, HttpResponse.BodyHandlers.ofString());
StringReader csvBodyReader = new StringReader(httpResponse.body()); //StringReader needs to be imported
Iterable<CSVRecord> records = CSVFormat.DEFAULT.withFirstRecordAsHeader().parse(csvBodyReader); // parse(in) had error, we needed a "reader" instance.
for (CSVRecord record : records) {
LocationStats locationStat = new LocationStats(); //create an instance
locationStat.setState(record.get("Province/State"));
locationStat.setCountry(record.get("Country/Region"));
locationStat.setLatestTotalCase(Integer.parseInt(record.get(record.size()-1)));
System.out.println(locationStat);
newStats.add(locationStat);
}
this.allStats = newStats;
}
}
The problem may come from this piece of code
#RequestMapping("/")
public #ResponseBody String home(Model model) {
model.addAttribute( "locationStats", covidDataService.getAllStats());
return "home";
}
it returns "home" which should be existing view, normally, the view will be a jsp file which is placed somewhere in WEB-INF, please see this tutorial: https://www.baeldung.com/spring-mvc-view-resolver-tutorial
In the case of wrong mapping, it may returns 404 error
when you run the server, you should be able to see which port it's taken in the console.
Also, is server.port=8080 in the src/main/resources/application.properties file?
In the controller, the RequestMapping annotation is missing the method type and header
#RequestMapping(
path="/",
method= RequestMethod.GET,
produces=MediaType.APPLICATION_JSON_VALUE)
public String home(Model model) {
model.addAttribute( "locationStats", covidDataService.getAllStats());
return "home";
}
make sure to add consumes for POST or PUT methods
A bit unrelated to the question but the line in the controller is missing #Autowired annotation
CovidDataService covidDataService;
Preferrably, add the #Autowired in the constructor
#Autowired
public HomeController(CovidDataService covidDataService) {
this.covidDataService = covidDataService;
}
I am developing prototype for a new project. The idea is to provide a Reactive Spring Boot microservice to bulk index documents in Elasticsearch. Elasticsearch provides a High Level Rest Client which provides an Async method to bulk process indexing requests. Async delivers callbacks using listeners are mentioned here. The callbacks receive index responses (per requests) in batches. I am trying to send this response back to the client as Flux. I have come up with something based on this blog post.
Controller
#RestController
public class AppController {
#SuppressWarnings("unchecked")
#RequestMapping(value = "/test3", method = RequestMethod.GET)
public Flux<String> index3() {
ElasticAdapter es = new ElasticAdapter();
JSONObject json = new JSONObject();
json.put("TestDoc", "Stack123");
Flux<String> fluxResponse = es.bulkIndex(json);
return fluxResponse;
}
ElasticAdapter
#Component
class ElasticAdapter {
String indexName = "test2";
private final RestHighLevelClient client;
private final ObjectMapper mapper;
private int processed = 1;
Flux<String> bulkIndex(JSONObject doc) {
return bulkIndexDoc(doc)
.doOnError(e -> System.out.print("Unable to index {}" + doc+ e));
}
private Flux<String> bulkIndexDoc(JSONObject doc) {
return Flux.create(sink -> {
try {
doBulkIndex(doc, bulkListenerToSink(sink));
} catch (JsonProcessingException e) {
sink.error(e);
}
});
}
private void doBulkIndex(JSONObject doc, BulkProcessor.Listener listener) throws JsonProcessingException {
System.out.println("Going to submit index request");
BiConsumer<BulkRequest, ActionListener<BulkResponse>> bulkConsumer =
(request, bulkListener) ->
client.bulkAsync(request, RequestOptions.DEFAULT, bulkListener);
BulkProcessor.Builder builder =
BulkProcessor.builder(bulkConsumer, listener);
builder.setBulkActions(10);
BulkProcessor bulkProcessor = builder.build();
// Submitting 5,000 index requests ( repeating same JSON)
for (int i = 0; i < 5000; i++) {
IndexRequest indexRequest = new IndexRequest(indexName, "person", i+1+"");
String json = doc.toJSONString();
indexRequest.source(json, XContentType.JSON);
bulkProcessor.add(indexRequest);
}
System.out.println("Submitted all docs
}
private BulkProcessor.Listener bulkListenerToSink(FluxSink<String> sink) {
return new BulkProcessor.Listener() {
#Override
public void beforeBulk(long executionId, BulkRequest request) {
}
#SuppressWarnings("unchecked")
#Override
public void afterBulk(long executionId, BulkRequest request, BulkResponse response) {
for (BulkItemResponse bulkItemResponse : response) {
JSONObject json = new JSONObject();
json.put("id", bulkItemResponse.getResponse().getId());
json.put("status", bulkItemResponse.getResponse().getResult
sink.next(json.toJSONString());
processed++;
}
if(processed >= 5000) {
sink.complete();
}
}
#Override
public void afterBulk(long executionId, BulkRequest request, Throwable failure) {
failure.printStackTrace();
sink.error(failure);
}
};
}
public ElasticAdapter() {
// Logic to initialize Elasticsearch Rest Client
}
}
I used FluxSink to create the Flux of Responses to send back to the Client. At this point, I have no idea whether this correct or not.
My expectation is that the calling client should receive the responses in batches of 10 ( because bulk processor processess it in batches of 10 - builder.setBulkActions(10); ). I tried to consume the endpoint using Spring Webflix Client. But unable to work it out. This is what I tried
WebClient
public class FluxClient {
public static void main(String[] args) {
WebClient client = WebClient.create("http://localhost:8080");
Flux<String> responseFlux = client.get()
.uri("/test3")
.retrieve()
.bodyToFlux(String.class);
responseFlux.subscribe(System.out::println);
}
}
Nothing is printing on console as I expected. I tried to use System.out.println(responseFlux.blockFirst());. It prints all the responses as a single batch at the end and not in batches at .
If my approach is correct, what is the correct way to consume it? For the solution in my mind, this client will reside is another Webapp.
Notes: My understanding of Reactor API is limited. The version of elasticsearch used is 6.8.
So made the following changes to your code.
In ElasticAdapter,
public Flux<Object> bulkIndex(JSONObject doc) {
return bulkIndexDoc(doc)
.subscribeOn(Schedulers.elastic(), true)
.doOnError(e -> System.out.print("Unable to index {}" + doc+ e));
}
Invoked subscribeOn(Scheduler, requestOnSeparateThread) on the Flux, Got to know about it from, https://github.com/spring-projects/spring-framework/issues/21507
In FluxClient,
Flux<String> responseFlux = client.get()
.uri("/test3")
.headers(httpHeaders -> {
httpHeaders.set("Accept", "text/event-stream");
})
.retrieve()
.bodyToFlux(String.class);
responseFlux.delayElements(Duration.ofSeconds(1)).subscribe(System.out::println);
Added "Accept" header as "text/event-stream" and delayed Flux elements.
With the above changes, was able to get the response in real time from the server.
In my spring boot project, one of my Service depends on external service like Amazon. I am writing the integration testing of the Controller classes. So, I want to mock the method in the AmazonService class(as it depends on third party API). The method is void with a single Long argument and can throw a custom application-specific exceptions.
The method is as follows:-
class AmazonService{
public void deleteMultipleObjects(Long enterpriseId) {
String key = formApplicationLogokey(enterpriseId,null);
List<S3ObjectSummary> objects = getAllObjectSummaryByFolder(key);
List<DeleteObjectsRequest.KeyVersion> keys = new ArrayList<>();
objects.stream().forEach(object->keys.add(new DeleteObjectsRequest.KeyVersion(object.getKey())));
try{
DeleteObjectsRequest deleteObjectsRequest = new DeleteObjectsRequest(this.bucket).withKeys(keys);
this.s3client.deleteObjects(deleteObjectsRequest);
log.debug("All the Application logos deleted from AWS for the Enterprise id: {}",enterpriseId);
}
catch(AmazonServiceException e){
throw new AppScoreException(AppScoreErrorCode.OBJECT_NOT_DELETED_FROM_AWS);
}
}}
class Test
class Test
{
#Autowired
AmazonServiceImpl amazonService;
#Autowired
EnterpriseService enterpriseService;
#Before
public void init()
{
amazonService = Mockito.mock(AmazonServiceImpl.class);
Mockito.doNothing().when(amazonService).deleteMultipleObjects(isA(Long.class));
}
#Test
public void testDeleteEnterprise(){
setHeaders();
EnterpriseDTO enterpriseDTO = createEnterpriseEntity(null,"testDeleteEnterpriseName3",null,null,null);
String postUrl = TestUrlUtil.createURLWithPort(TestConstants.ADD_ENTERPRISE,port);
HttpEntity<EnterpriseDTO> request1 = new HttpEntity<>(enterpriseDTO,headers);
ResponseEntity<EnterpriseDTO> response1 = restTemplate.postForEntity(postUrl,request1,EnterpriseDTO.class);
assert response1 != null;
Long enterpriseId = Objects.requireNonNull(response1.getBody()).getId();
String url = TestUrlUtil.createURLWithPort(TestConstants.DELETE_ENTERPRISE,port)+File.separator+enterpriseId;
HttpEntity<EnterpriseDTO> request = new HttpEntity<>(null, headers);
ResponseEntity<Object> response = restTemplate.exchange(url,HttpMethod.DELETE,request,Object.class);
Assert.assertEquals(Constants.ENTERPRISE_DELETION_SUCCESS_MESSAGE,response.getBody());
}
}
class EnterpriseResource
class EnterpriseResource
{
#DeleteMapping("/enterprises/{enterpriseId}")
public ResponseEntity<Object> deleteEnterprise(#PathVariable Long enterpriseId) {
log.debug("REST request to delete Enterprise : {}", enterpriseId);
enterpriseService.delete(enterpriseId);
return ResponseEntity.badRequest().body(Constants.ENTERPRISE_DELETION_SUCCESS_MESSAGE);
}
}
class EnterpriseServiceImpl
class EnterpriseServiceImpl
{
#Override
public void delete(Long enterpriseId) {
log.debug("Request to delete Enterprise : {}", enterpriseId);
enterpriseRepository.deleteById(enterpriseId);
amazonService.deleteMultipleObjects(enterpriseId);
}
}
I have tried various approaches to Mock this method but it didn't work and control is going inside this method during debugging. I want to do nothing in this method during testing.
I have tried the various approaches like throw(), doNothing(), spy() etc.
Please help what is missing here?
Thanks