I am using RestTemplate as my HttpClient to execute URL and the server will return back a json string as the response. Customer will call this library by passing DataKey object which has userId in it.
Using the given userId, I will find out what are the machines that I can hit to get the data and then store those machines in a LinkedList, so that I can execute them sequentially.
After that I will check whether the first hostname is in block list or not. If it is not there in the block list, then I will make a URL with the first hostname in the list and execute it and if the response is successful then return back the response. But let's say if that first hostname is in the block list, then I will try to get the second hostname in the list and make the url and execute it, so basically, first find the hostname which is not in block list before making the URL.
Now, let's say if we selected first hostname which was not in the block list and executed the URL and somehow server was down or not responding, then I will execute the second hostname in the list and keep doing this until you get a successful response. But make sure they were not in the block list as well so we need to follow above point.
If all servers are down or in block list, then I can simply log and return the error that service is unavailable.
Below is my DataClient class which will be called by customer and they will pass DataKey object to getData method.
public class DataClient implements Client {
private RestTemplate restTemplate = new RestTemplate(new HttpComponentsClientHttpRequestFactory());
private ExecutorService service = Executors.newFixedThreadPool(15);
public Future<DataResponse> getData(DataKey key) {
DataExecutorTask task = new DataExecutorTask(key, restTemplate);
Future<DataResponse> future = service.submit(task);
return future;
}
}
Below is my DataExecutorTask class:
public class DataExecutorTask implements Callable<DataResponse> {
private DataKey key;
private RestTemplate restTemplate;
public DataExecutorTask(DataKey key, RestTemplate restTemplate) {
this.restTemplate = restTemplate;
this.key = key;
}
#Override
public DataResponse call() {
DataResponse dataResponse = null;
ResponseEntity<String> response = null;
MappingsHolder mappings = ShardMappings.getMappings(key.getTypeOfFlow());
// given a userId, find all the hostnames
// it can also have four hostname or one hostname or six hostname as well in the list
List<String> hostnames = mappings.getListOfHostnames(key.getUserId());
for (String hostname : hostnames) {
// If host name is null or host name is in local block list, skip sending request to this host
if (ClientUtils.isEmpty(hostname) || ShardMappings.isBlockHost(hostname)) {
continue;
}
try {
String url = generateURL(hostname);
response = restTemplate.exchange(url, HttpMethod.GET, key.getEntity(), String.class);
if (response.getStatusCode() == HttpStatus.NO_CONTENT) {
dataResponse = new DataResponse(response.getBody(), DataErrorEnum.NO_CONTENT,
DataStatusEnum.SUCCESS);
} else {
dataResponse = new DataResponse(response.getBody(), DataErrorEnum.OK,
DataStatusEnum.SUCCESS);
}
break;
// below codes are duplicated looks like
} catch (HttpClientErrorException ex) {
HttpStatusCodeException httpException = (HttpStatusCodeException) ex;
DataErrorEnum error = DataErrorEnum.getErrorEnumByException(httpException);
String errorMessage = httpException.getResponseBodyAsString();
dataResponse = new DataResponse(errorMessage, error, DataStatusEnum.ERROR);
return dataResponse;
} catch (HttpServerErrorException ex) {
HttpStatusCodeException httpException = (HttpStatusCodeException) ex;
DataErrorEnum error = DataErrorEnum.getErrorEnumByException(httpException);
String errorMessage = httpException.getResponseBodyAsString();
dataResponse = new DataResponse(errorMessage, error, DataStatusEnum.ERROR);
return dataResponse;
} catch (RestClientException ex) {
// if it comes here, then it means some of the servers are down so adding it into block list
ShardMappings.blockHost(hostname);
}
}
if (ClientUtils.isEmpty(hostnames)) {
dataResponse = new DataResponse(null, DataErrorEnum.PERT_ERROR, DataStatusEnum.ERROR);
} else if (response == null) { // either all the servers are down or all the servers were in block list
dataResponse = new DataResponse(null, DataErrorEnum.SERVICE_UNAVAILABLE, DataStatusEnum.ERROR);
}
return dataResponse;
}
}
My block list keeps-on getting updated from another background thread every 1 minute. If any server is down and not responding, then I need to block that server by using this -
ShardMappings.blockHost(hostname);
And to check whether any server is in block list or not, I use this -
ShardMappings.isBlockHost(hostname);
I am returning SERVICE_UNAVAILABLE if servers are down or in block list,on the basis of response == null check, not sure whether it's a right approach or not.
I am not following Single Responsibility Principle here I guess at all.
Can anyone provide an example what is the best way to use SRP principle here.
After thinking a lot, I was able to extract hosts class like given below but not sure what is the best way to use this in my above DataExecutorTask class.
public class Hosts {
private final LinkedList<String> hostsnames = new LinkedList<String>();
public Hosts(final List<String> hosts) {
checkNotNull(hosts, "hosts cannot be null");
this.hostsnames.addAll(hosts);
}
public Optional<String> getNextAvailableHostname() {
while (!hostsnames.isEmpty()) {
String firstHostname = hostsnames.removeFirst();
if (!ClientUtils.isEmpty(firstHostname) && !ShardMappings.isBlockHost(firstHostname)) {
return Optional.of(firstHostname);
}
}
return Optional.absent();
}
public boolean isEmpty() {
return hostsnames.isEmpty();
}
}
Your concern is valid. First, let's see what the original data executor do:
First, it is getting list of hostnames
Next, it loops through every hostnames that do the following things:
It checks whether the hostname is valid to send request.
If not valid: skip.
Else continue.
Generate the URL based on hostname
Send the request
Translate the request response to domain response
Handle exceptions
If the hostnames is empty, generate an empty response
Return response
Now, what can we do to follow SRP? As I can see, we can group those operations into some groups. What I can see is, these operations can be split into:
HostnameValidator: checks whether the hostname is valid to send request
--------------
HostnameRequestSender: Generate the URL
Send the request
--------------
HttpToDataResponse: Translate the request response to domain response
--------------
HostnameExceptionHandler: Handle exceptions
That is, one approach to de-couple your operations and to follow SRP. There is also other approach, for example to simplify your operations:
First, it is getting list of hostnames
If the hostnames is empty, generate an empty response
Next, it loops through every hostnames that do the following things:
It checks whether the hostname is valid to send request
If not valid: remove hostname
Else: Generate the URL based on hostname
Next, it loops through every valid hostnames that do the following things:
Send the request
Translate the request response to domain response
Handle exceptions
Return response
Then it can also be split into:
HostnameValidator: checks whether the hostname is valid to send request
--------------
ValidHostnameData: Getting list of hostnames
Loops through every hostnames that do the following things:
Checks whether the hostname is valid to send request
If not valid: remove hostname
Else: Generate the URL based on hostname
--------------
HostnameRequestSender: Send the request
--------------
HttpToDataResponse: Translate the request response to domain response
--------------
HostnameExceptionHandler: Handle exceptions
Of course there are also other way to do it. And I leave the implementation details blank because there is many way to implement it.
Related
I have a requirement where I am connecting one microservice to other microservice via Vertx client. In the code I am checking if another microservice is down then on failure it should create some JsonObject with solrError as key and failure message as value. If there is a solr error I mean if other microservice is down which is calling solr via load balancing then it should throw some error response. But Vertx client is taking some time to check on failure and when condition is checked that time there is no solrError in jsonobject as Vertx client is taking some time to check for failure so condition fails and resp is coming as null. In order to avoid this what can be done so that Vertx client fails before the condition to check for solrError and returns Internal server error response?
Below is the code :
solrQueryService.executeQuery(query).subscribe().with(jsonObject -> {
ObjectMapper objMapper = new ObjectMapper();
SolrOutput solrOutput = new SolrOutput();
List<Doc> docs = new ArrayList<>();
try {
if(null != jsonObject.getMap().get("solrError")){
resp = Response.status(Response.Status.INTERNAL_SERVER_ERROR)
.entity(new BaseException(
exceptionService.processSolrDownError(request.header.referenceId))
.getResponse()).build();
}
solrOutput = objMapper.readValue(jsonObject.toString(), SolrOutput.class);
if (null != solrOutput.getResponse()
&& CollectionUtils.isNotEmpty(solrOutput.getResponse().getDocs())) {
docs.addAll(solrOutput.getResponse().getDocs());
uniDocList = Uni.createFrom().item(docs);
}
} catch (JsonProcessingException e) {
e.printStackTrace();
}
});
if(null!=resp && resp.getStatus() !=200) {
return resp ;
}
SolrQueryService is preparing query and send out URL and query to Vertx web client as below :
public Uni<JsonObject> search(URL url, SolrQuery query,Integer timeout) {
int port = url.getPort();
if (port == -1 && "https".equals(url.getProtocol())) {
port = 443;
}
if (port == -1 && "http".equals(url.getProtocol())) {
port = 80;
}
HttpRequest<Buffer> request = client.post(port, url.getHost(), url.getPath()).timeout(timeout);
return request.sendJson(query).map(resp -> {
return resp.bodyAsJsonObject();
}).onFailure().recoverWithUni(f -> {
return Uni.createFrom().item(new JsonObject().put("solrError", f.getMessage()));
});
}
I have not used the Vertx client but assume its reactive and non-blocking. Assuming this is the case, your code seems to be mixing imperative and reactive constructs. The subscribe in the first line is reactive and the lambda you provide will be called when the server responds to the client request. However, after the subscribe, you have imperative code which runs before the lambda even has a chance to be called so your checks and access to the "resp" object will never be a result of what happened in the lambda itself.
You need to move all the code into the lambda or at least make subsequent code chain onto the result of the subscribe.
I am calling a File Server's REST API using POST method and sending it file content to be uploaded. The REST API should ideally save the file and send a response which contains fileName.
My code is something like this.
public String uploadFile() {
UploadResponse response = restTemplate.postForObject(
FILE_SERVER_URL/upload,
new HttpEntity<>(fileContent, headers),
UploadResponse.class);
return response.getFileName();
}
In the above code, the compiler complains that UploadResponse response could be null, and I should handle that.
I plan to handle it with the below code.
public String uploadFile() {
UploadResponse response = restTemplate.postForObject(
FILE_SERVER_URL/upload,
new HttpEntity<>(fileContent, headers),
UploadResponse.class);
if(response != null) {
return response.getFileServiceId();
}
else {
throw new RuntimeException("File upload failed");
}
}
However, I am not sure if it is the right way to handle this. I don't feel this is a Runtime Exception. Please guide me as how should I handle the case that response could be null.
You can use Optional to avoid null checks in your code.
You can read this really insightful Q/A here - https://softwareengineering.stackexchange.com/questions/309134/why-is-using-an-optional-preferential-to-null-checking-the-variable
I'm trying to obtain data from Botify's REST API to use it inside a project, which is also a REST API. I'm using an instance of Spring's RestTemplate class to make the actual requests to Botify, specifically the .exchange method as I need to pass Botify's key as a header parameter.
My problem comes when I need to call to a method of the endpoint which takes a URL as a part of the request's URI (not a parameter). Documentation of this endpoint is in https://developers.botify.com/api/reference/#!/Analysis/getUrlDetail
Basically the structure of the requests is like this:
/analyses/{username}/{project_slug}/{analysis_slug}/urls/{url}
The last part of that URI is a URL address, which needs to be encoded in UTF-8 to make it possible to separate it from the actual request.
The problem is (I believe) that the .exchange method always encodes the request, so what I try to send like this:
/analyses/myusername/myprojectname/myprojectslug/urls/https%3A%2F%2Fwww.example.com
...ends up like this:
/analyses/myusername/myprojectname/myprojectslug/urls/https%253A%252F%252Fwww.example.com'
Which obviously doesn't work. This is an excerpt from the method that makes the call to Botify:
public String callBotifyEndpoint(String reportType, String parameters) throws UnsupportedEncodingException {
String request = this.baseUri + "/analyses/myusername/myprojectname/myprojectslug/urls/https%3A%2F%2Fwww.example.com"
HttpHeaders headers = new HttpHeaders();
headers.set("Authorization", "Token " + this.apiKey);
HttpEntity<String> entity = new HttpEntity<>(headers);
UriComponentsBuilder botifyQueryBuilder = UriComponentsBuilder.fromUriString(request);
String queryStringBuild = botifyQueryBuilder.build(true).toUriString();
String botifyResult = null;
try {
System.out.println("Calling Botify API: " + queryStringBuild);
ResponseEntity<String> response = botifyTemplate.exchange(queryStringBuild, HttpMethod.GET, entity, String.class);
if(response.hasBody()) {
botifyResult = response.getBody();
}
} catch(RestClientException ex) {
ex.printStackTrace();
}
try {
} catch (Exception e) {
// TODO: handle exception
}
return botifyResult;
}
In this line:
botifyQueryBuilder.build(true).toUriString();
The "true" parameter indicates whether the data is already encoded or not. I've tried to disable it but the result is the same.
I've removed actual request generation process (along with my user and project's name) to simplify things, but this should return a response from Botify with the existing data for that URL.
Instead, it returns a 400 bad request error (which makes sense, because the URL is not correct).
I'm feeling like this may be a bug in RestTemplate's .exchange method, but maybe I'm not using it properly. Any suggestions?
Don't encode prematurly as you do here:
String request = this.baseUri + "/analyses/myusername/myprojectname/myprojectslug/urls/https%3A%2F%2Fwww.example.com";
Use parameter placeholders feature in RestTemplate instead of text concatenation.
Refer to:
Spring RestTemplate GET with parameters
I am developing an API REST using Spring WebFlux, but I have problems when uploading files. They are stored but I don't get the expected return value.
This is what I do:
Receive a Flux<Part>
Cast Part to FilePart.
Save parts with transferTo() (this return a Mono<Void>)
Map the Mono<Void> to Mono<String>, using file name.
Return Flux<String> to client.
I expect file name to be returned, but client gets an empty string.
Controller code
#PostMapping(value = "/muscles/{id}/image")
public Flux<String> updateImage(#PathVariable("id") String id, #RequestBody Flux<Part> file) {
log.info("REST request to update image to Muscle");
return storageService.saveFiles(file);
}
StorageService
public Flux<String> saveFiles(Flux<Part> parts) {
log.info("StorageService.saveFiles({})", parts);
return
parts
.filter(p -> p instanceof FilePart)
.cast(FilePart.class)
.flatMap(file -> saveFile(file));
}
private Mono<String> saveFile(FilePart filePart) {
log.info("StorageService.saveFile({})", filePart);
String filename = DigestUtils.sha256Hex(filePart.filename() + new Date());
Path target = rootLocation.resolve(filename);
try {
Files.deleteIfExists(target);
File file = Files.createFile(target).toFile();
return filePart.transferTo(file)
.map(r -> filename);
} catch (IOException e) {
throw new RuntimeException(e);
}
}
FilePart.transferTo() returns Mono<Void>, which signals when the operation is done - this means the reactive Publisher will only publish an onComplete/onError signal and will never publish a value before that.
This means that the map operation was never executed, because it's only given elements published by the source.
You can return the name of the file and still chain reactive operators, like this:
return part.transferTo(file).thenReturn(part.filename());
It is forbidden to use the block operator within a reactive pipeline and it even throws an exception at runtime as of Reactor 3.2.
Using subscribe as an alternative is not good either, because subscribe will decouple the transferring process from your request processing, making those happen in different execution sequences. This means that your server could be done processing the request and close the HTTP connection while the other part is still trying to read the file part to copy it on disk. This is likely to fail in subtle ways at runtime.
FilePart.transferTo() returns Mono<Void> that is a constant empty. Then, map after that was never executed. I solved it by doing this:
private Mono<String> saveFile(FilePart filePart) {
log.info("StorageService.saveFile({})", filePart);
String filename = DigestUtils.sha256Hex(filePart.filename() + new Date());
Path target = rootLocation.resolve(filename);
try {
Files.deleteIfExists(target);
File file = Files.createFile(target).toFile();
return filePart
.transferTo(file)
.doOnSuccess(data -> log.info("do something..."))
.thenReturn(filename);
} catch (IOException e) {
throw new RuntimeException(e);
}
}
I'm trying to create a dynamic Rest client, where I can set the HTTP Method(GET-POST-PUT-DELETE), Query Params and body(Json, plain, XML), this is basically what I need, for the request I think i know how I can do it, but my concern is for reading the answer, since I know what I should get ( format) but I dont know how to read it properly, so far I return an object, below the code (only for POST, but the idea is the same):
Response responseRest = null;
Client client = null;
try {
client = new ResteasyClientBuilder().establishConnectionTimeout(TIME_OUT, TimeUnit.MILLISECONDS).socketTimeout(TIME_OUT, TimeUnit.MILLISECONDS).build();
WebTarget target = client.target(request.getUrlTarget());
MediaType type = assignResponseType(request.getTypeResponse());
switch (request.getProtocol()) {
case POST: {
if (request.getParamQuery() != null) {
for (VarRequestDTO varRequest : request.getParamQuery()) {
target = target.queryParam(varRequest.getName(), varRequest.getValue());
}
}
responseRest = target.request().post(Entity.entity(new ResponseWrapper(), type));
break;
}
default:
//HTTP METHOD No supported
}
Object result = responseRest.readEntity(Object.class);
}
catch (Exception e) {
response.setError(Boolean.TRUE);
response.setMessage(e.getMessage());
e.printStackTrace();
} finally {
if (responseRest != null) {
responseRest.close();
}
if (client != null) {
client.close();
}
}
What I basically I need is to return the object in the format needed, and where is called it's supposed to do a cast to the correct format, I just need it to be dynamic and used for any service.
Thanks
Every request that a ReST client makes to a ReST service, it passes an "Accept" header.
This is to indicate to the service the MIME-type of the resource the client is willing to accept.
In the above case, what are the acceptable formats (json/ plain text/ etc.) for you?
Depending on the "accept" format you choose, and the "Content-type" header that you receive, you can write a deserializer to accept that data and process.
Also, instead of returning an Object which is too generic, consider returning a readable Stream to the caller.