I using spring (boot) integration in order to connect to a restful WebService and retrieve data from that, Here are some part of my codes:
#Bean
public PollerMetadata downloadTrigger()
{
PeriodicTrigger trigger = new PeriodicTrigger(config.getDownloadInterval());
trigger.setFixedRate(true);
PollerMetadata pollerMetadata = new PollerMetadata();
pollerMetadata.setTrigger(trigger);
pollerMetadata.setMaxMessagesPerPoll(1);
return pollerMetadata;
}
#Bean(name = PollerMetadata.DEFAULT_POLLER)
public PollerMetadata poller()
{
PeriodicTrigger trigger = new PeriodicTrigger(10);
trigger.setFixedRate(true);
PollerMetadata pollerMetadata = new PollerMetadata();
pollerMetadata.setTrigger(trigger);
return pollerMetadata;
}
#InboundChannelAdapter(value = "channel1", poller = #Poller("downloadTrigger"))
public ResponseEntity<AppsItem[]> download()
{
String url = "https://example.com/?page{pageNumber}";
try
{
SSLUtil.turnOffSslChecking();
}
catch (Exception e)
{
}
HttpHeaders requestHeaders = new HttpHeaders();
requestHeaders.setAccept(Collections.singletonList(new MediaType("application","json")));
HttpEntity<?> requestEntity = new HttpEntity<Object>(requestHeaders);
template.getMessageConverters().add(new MappingJackson2HttpMessageConverter());
ResponseEntity<AppsItem[]> responseEntity = template.exchange(url, HttpMethod.GET, requestEntity, AppsItem[].class, 10, 1);
return responseEntity;
}
#Splitter(inputChannel = "channel1", outputChannel = "channel2")
public List<AppsItem> scrape(ResponseEntity<AppsItem[]> payload)
{
//do something;
}
As you can see, I have a url that accept a variable with pageNum name, i want to increase page number while the response body be empty, I don't know how can i implement it with spring integration.But let me explain what it need:
1 - Poller calling inbound method once peer hour
2 - Inbound method start calling WebService from page 1 to page n.
3 - Inbound method should pass data to next channel before fetching next page
I need something like below:
#InboundChannelAdapter(value = "channel1", poller = #Poller("downloadTrigger"))
public ResponseEntity<AppsItem[]> download()
{
String url = "https://example.com/?page{pageNumber}";
try
{
SSLUtil.turnOffSslChecking();
}
catch (Exception e)
{
}
HttpHeaders requestHeaders = new HttpHeaders();
requestHeaders.setAccept(Collections.singletonList(new MediaType("application","json")));
HttpEntity<?> requestEntity = new HttpEntity<Object>(requestHeaders);
template.getMessageConverters().add(new MappingJackson2HttpMessageConverter());
ResponseEntity<AppsItem[]> responseEntity;
int i = 0;
do
{
i++;
responseEntity = template.exchange(url, HttpMethod.GET, requestEntity, AppsItem[].class, 10, i);
scraper.parse(responseEntity);
LOG.info("entry={}", "");
//Send response to the next channel in this loop
}
while (responseEntity.hasBody());
return responseEntity;
}
Remove setMaxMessagesPerPoll(1) from downloadTrigger and the poller (on each poll) will keep calling the download() until it returns null.
Increment the page count on each call and reset it when no more pages exist; the poller will next run after the interval.
Related
I have a controller in gateway microservice that accepts the MultipartFile and resends to the service behind it
#PostMapping
public ResponseEntity upload(#ApiParam(name = "file", value = "File", required = true) MultipartFile file)
throws BaseException {
if (Objects.isNull(file)){
throw new CheckFieldException("file", MultipartFile.class);
}
if (megabyte * maxFileSize - file.getSize() < 0){
return ResponseEntity.accepted().body(new DocumentResponseDTO(false, "File size exceeds " + maxFileSize + "MB"));
}
DiscoveryConfig.CashTracking config = discoveryConfig.getCashTracking();
UriComponents uriStatementUpload = UriComponentsBuilder.newInstance().scheme(config.getScheme())
.host(config.getHost()).port(config.getPort()).path(config.getExcelNominalOperationsPath()).build(true);
try {
HttpEntity<byte[]> fileEntity = new HttpEntity(file.getBytes());
ResponseEntity<DocumentResponseDTO> entity = restTemplate.postForEntity(uriStatementUpload.toUri(), fileEntity, DocumentResponseDTO.class);
return entity;
} catch (HttpClientErrorException e) {
return ResponseEntity.status(e.getStatusCode()).body(e.getResponseBodyAsString());
} catch (IOException e) {
return ResponseEntity.status(500).body("IOException while getting bytes stream from file");
}
}
and in CashTracking service there is also file upload like that:
#PostMapping(value = "/upload")
public ResponseEntity uploadExcelNominalOperationsFile(#ApiParam(name = "file", value = "File", required = true) MultipartFile file) throws IOException {
try (InputStream is = file.getInputStream()) {
log.info("Processing incoming Excel file with nominal operations");
Workbook workbook = new XSSFWorkbook(is);
log.info("Processing workbook");
Sheet sheet = workbook.getSheetAt(0);
log.info("Processing the first sheet");
List<NominalOperationVO> nominalOperationVOs = new ArrayList<>();
List<String> fileHeaders = new ArrayList<>();
And when the file is actually uploaded to the gateway service, the service behind it starts processing the file upload, but the MultipartFile file is null. I have explicitly put it in the Entity I have sent to the service behind the gateway, the question, what I'm doing wrong if it is null? If I do upload to that microservice directly, it process the request correctly.
The main stuff I was missing was putting the Http headers per specific multipart form's parts. They should be identical to what has been sent to the gateway service.
public ResponseEntity upload(#ApiParam(name = "file", value = "Файл", required = true) MultipartFile file)
throws BaseException {
if (Objects.isNull(file)){
throw new CheckFieldException("file", MultipartFile.class);
}
if (megabyte * maxFileSize - file.getSize() < 0){
return ResponseEntity.accepted().body(new DocumentResponseDTO(false, "File size exceeds " + maxFileSize + "MB"));
}
DiscoveryConfig.CashTracking config = discoveryConfig.getCashTracking();
UriComponents uriStatementUpload = UriComponentsBuilder.newInstance().scheme(config.getScheme())
.host(config.getHost()).port(config.getPort()).path(config.getExcelNominalOperationsPath()).build(true);
try {
HttpHeaders headers = new HttpHeaders();
headers.setContentType(MediaType.MULTIPART_FORM_DATA);
MultipartBodyBuilder multipartBodyBuilder = new MultipartBodyBuilder();
//here is the really needed stuff with 2 headers
Resource resource = new ByteArrayResource(file.getBytes());
multipartBodyBuilder.part("file", resource)
.header("Content-Type",file.getContentType())
.header("Content-Disposition","form-data; name=\"file\"; filename=\""+file.getOriginalFilename()+"\"");
// multipart/form-data request body
MultiValueMap<String, HttpEntity<?>> body = multipartBodyBuilder.build();
HttpEntity<MultiValueMap<String, HttpEntity<?>>> requestEntity
= new HttpEntity<>(body, headers);
ResponseEntity<DocumentResponseDTO> entity = restTemplate.postForEntity(uriStatementUpload.toUri(), requestEntity, DocumentResponseDTO.class);
return entity;
} catch (HttpClientErrorException e) {
return ResponseEntity.status(e.getStatusCode()).body(e.getResponseBodyAsString());
} catch (IOException e) {
return ResponseEntity.status(500).body("IOException while getting bytes stream from file");
}
}
Right now I have 2 Spring App.
App A will have a controller that will receive a video file ad Multipart file and sending the file to App B via rest template.
Some Code from App A that handle sending request to App B.
#RestController
public class AppAController {
#Autowired
private final AppBService service;
#PostMapping("/sendToB")
public ResponseEntity<String> contoller(#RequestParam("file") MultipartFile file) {
String result = service.sendToB(file);
return new ResponseEntity<>(result, HttpStatus.OK);
}
}
#Service
public class AppBService {
public String sendToB(MultipartFile file) throws ResponseStatusException {
HttpHeaders headers = new HttpHeaders();
headers.setContentType(MediaType.MULTIPART_FORM_DATA);
MultiValueMap<String, Object> body = new LinkedMultiValueMap<>();
body.add("file", file);
HttpEntity<MultiValueMap<String, Object>> requestEntity = new HttpEntity<>(body, headers);
String serverUrl = "http://127.0.0.1:8090/makeFrames";
RestTemplate restTemplate = new RestTemplate();
ResponseEntity<String> response = restTemplate.exchange(
serverUrl,
HttpMethod.POST,
requestEntity,
String.class
);
if (response.getStatusCode() != HttpStatus.OK) {
throw new ResponseStatusException(response.getStatusCode(), response.getBody());
}
return response.getBody();
}
}
And for App B, it will receive a video and extracting key frames from a video using JavaCV.
#RestController
public class ProcessorController {
#PostMapping("/makeFrames")
public ResponseEntity<String> framesExtractorController(#RequestParam("file") MultipartFile file) {
try {
File result = FramesExtractor.grabFrames(file);
return new ResponseEntity<>(result.getAbsolutePath(), HttpStatus.OK);
} catch (IOException e) {
return new ResponseEntity<>(e.getMessage(), HttpStatus.NOT_FOUND);
}
}
}
public class FramesExtractor {
private static final Logger LOG = LoggerFactory.getLogger(FramesExtractor.class);
private FramesExtractor() {
}
public static File grabFrames(MultipartFile video) throws IOException {
LOG.info("Extracting Frames from the video " + video.getName());
String directoryName = video.getOriginalFilename() + "-frames-result";
directoryAssurance(directoryName);
try (FFmpegFrameGrabber frameGrabber = new FFmpegFrameGrabber(video.getInputStream());
Java2DFrameConverter converter = new Java2DFrameConverter()) {
frameGrabber.setImageWidth(480);
frameGrabber.setImageHeight(360);
frameGrabber.start();
Frame frame;
int i = 0;
while ((frame = frameGrabber.grabKeyFrame()) != null) {
BufferedImage bi = converter.getBufferedImage(frame);
ImageIO.write(bi, "png", new File(directoryName + "/" + String.format("%03d", i) + ".png"));
i++;
}
frameGrabber.stop();
LOG.info("Finish Extracting Frames");
return new File(directoryName);
} catch (Exception e) {
LOG.error(e.getMessage());
throw e;
}
}
}
Both app A and B have these properties set to their application.properties files.
server.port={There respective port}
spring.servlet.multipart.enabled=true
spring.servlet.multipart.max-file-size=100MB
spring.servlet.multipart.max-request-size=100MB
And when i testing the application by using PostMan to send post request with video file in form-data body to App A /sendToB api i receive this error message.
"Type definition error: [simple type, class java.io.FileDescriptor]; nested exception is com.fasterxml.jackson.databind.exc.InvalidDefinitionException: No serializer found for class java.io.FileDescriptor and no properties discovered to create BeanSerializer (to avoid exception, disable SerializationFeature.FAIL_ON_EMPTY_BEANS) (through reference chain: org.springframework.web.multipart.support.StandardMultipartHttpServletRequest$StandardMultipartFile[\"inputStream\"]->java.io.FileInputStream[\"fd\"])"
So I want to ask what cause this problems and how I can fix it or sending video file to other Spring App running on other port properly.
I edited my previous answer because you can actually get parameters from a response body in a POST with #RequestParam annotation.
I would use JSON as content type. This is my solution:
public String sendToB(MultipartFile file) throws ResponseStatusException {
HttpHeaders headers = new HttpHeaders();
headers.setContentType(MediaType.APPLICATION_JSON);
byte[] bytesFile = file.getBytes();
String base64String = Base64.getEncoder().encodeToString(bytesFile);
Map<String, Object> body = new HashMap<>();
body.put("file", base64String);
body.put("directoryName", file.getOriginalFilename());
body.put("videoName", file.getName())
HttpEntity<Map<String, Object>> requestEntity = new HttpEntity<>(body, headers);
String serverUrl = "http://127.0.0.1:8090/makeFrames";
RestTemplate restTemplate = new RestTemplate();
ResponseEntity<String> response = restTemplate.exchange(
serverUrl,
HttpMethod.POST,
requestEntity,
String.class
);
if (response.getStatusCode() != HttpStatus.OK) {
throw new ResponseStatusException(response.getStatusCode(), response.getBody());
}
return response.getBody();
}
For App B, the code would be:
#PostMapping("/makeFrames")
public ResponseEntity<String> framesExtractorController(#RequestBody Map<String, Object> requestBody) {
try {
String base64String = requestBody.get("file").toString();
byte[] bytesFile = Base64.getDecoder().decode(base64String);
InputStream inputStream = new ByteArrayInputStream(bytesFile);
String directoryName = requestBody.get("directoryName").toString();
String videoName = requestBody.get("videoName").toString();
File result = FramesExtractor.grabFrames(inputStream, directoryName, videoName);
return new ResponseEntity<>(result.getAbsolutePath(), HttpStatus.OK);
} catch (IOException e) {
return new ResponseEntity<>(e.getMessage(), HttpStatus.NOT_FOUND);
}
}
}
public static File grabFrames(InputStream inputStream, String directoryName, String videoName) throws IOException {
//Your code
}
I am using Spring RestTemplate and want to make a call to another service that doesn't return any response body. So, I don't want to wait for the response. So, it's just fire and forget, and continue with the remaining code. I am thinking of creating a new Thread to do this but really not sure what's the correct approach.
If you use Java 11, java support asynchronous HTTP Client. Asynchronous client using CompletableFuture in the back. You can see javadoc.
HttpRequest request = HttpRequest.newBuilder()
.uri(URI.create("http://openjdk.java.net/"))
.timeout(Duration.ofMinutes(1))
.header("Content-Type", "application/json")
.POST(BodyPublishers.ofFile(Paths.get("file.json")))
.build();
client.sendAsync(request, BodyHandlers.ofString())
.thenApply(response -> { System.out.println(response.statusCode());
return response; } )
.thenApply(HttpResponse::body)
.thenAccept(System.out::println);
The correct approach is to execute the async with a callback (using DeferredResult, like this (assuming we have a class someClass that we want to retrieve from the API:
#GetMapping(path = "/testingAsync")
public DeferredResult<String> value() throws ExecutionException, InterruptedException, TimeoutException {
AsyncRestTemplate restTemplate = new AsyncRestTemplate();
String baseUrl = "http://someUrl/blabla";
HttpHeaders requestHeaders = new HttpHeaders();
requestHeaders.setAccept(Arrays.asList(MediaType.APPLICATION_JSON));
String value = "";
HttpEntity entity = new HttpEntity("parameters", requestHeaders);
final DeferredResult<String> result = new DeferredResult<>();
ListenableFuture<ResponseEntity<someClass>> futureEntity = restTemplate.getForEntity(baseUrl, someClass.class);
futureEntity.addCallback(new ListenableFutureCallback<ResponseEntity<someClass>>() {
#Override
public void onSuccess(ResponseEntity<someClass> result) {
System.out.println(result.getBody().getName());
result.setResult(result.getBody().getName());
}
#Override
public void onFailure(Throwable ex) {
result.setErrorResult(ex.getMessage());
}
});
return result;
}
There are many ways you can use to fire the request using the AsyncRestTemplate
The simplest way is just like restTemplate and call exchange method:
AsyncRestTemplate asyncRestTemplate = new AsyncRestTemplate();
JSONObject json = new JSONObject();
json.put("firstName","testUser");
HttpHeaders headers = new HttpHeaders();
headers.setContentType(MediaType.APPLICATION_JSON);
HttpEntity<String> requestEntity = new HttpEntity<String>(json.toString(), headers);
Class<String> responseType = String.class;
ListenableFuture<ResponseEntity<String>> future = asyncRestTemplate.exchange("https://xxxxx.com/", HttpMethod.POST, requestEntity,responseType );
// If you want for the result then you can use
try {
//waits for the result
ResponseEntity<String> entity = future.get();
//prints body source code for the given URL
log.info(entity.getBody());
} catch (InterruptedException e) {
e.printStackTrace();
} catch (Exception e) {
e.printStackTrace();
}
If we want to play with the failure (fallback scenario) or success in that case we can use the below code :
AsyncRestTemplate asyncRestTemplate = new AsyncRestTemplate();
JSONObject json = new JSONObject();
json.put("firstName","testUser");
HttpHeaders headers = new HttpHeaders();
headers.setContentType(MediaType.APPLICATION_JSON);
HttpEntity<String> requestEntity = new HttpEntity<String>(json.toString(), headers);
//final DeferredResult<String> result = new DeferredResult<>();
ListenableFuture<ResponseEntity<String>> future =
asyncRestTemplate.postForEntity("https://xxxx.com", requestEntity, String.class);
future.addCallback(new ListenableFutureCallback<ResponseEntity<String>>() {
#Override
public void onFailure(Throwable ex) {
// insert into the table or log or some other decision
log.info(ex.getMessage());
}
#Override
public void onSuccess(ResponseEntity<String> result) {
log.info(result.getBody());
log.info("Sucess");
}
});
I have this method
public HTTPResult post(String url, String requestBody) throws Exception {
return HTTPPostPut(url, requestBody, HttpMethod.POST);
}
public HTTPResult HTTPPostPut(String url, String requestBody,HttpMethod httpMethod) throws Exception {
MultiValueMap<String, String> headers = new LinkedMultiValueMap<>();
headers.add("content-type","application/json");
HttpEntity requestEntity = new HttpEntity(requestBody,headers);
try {
ResponseEntity<String> response = this.restTemplate.exchange(url, httpMethod, requestEntity, String.class);
return new HTTPResult((String) response.getBody(), response.getStatusCode().value());
} catch (ResourceAccessException var8) {
String responseBody = var8.getCause().getMessage();
JSONObject obj = new JSONObject(responseBody);
return new HTTPResult(obj.getString("responseBody"), Integer.parseInt(obj.getString("statusCode")));
}
}
Which I created for it mock and getting null pointer exception:
public void testPost() throws Exception{
MultiValueMap<String, String> headers = new LinkedMultiValueMap<>();
headers.add("content-type","application/json");
HttpEntity requestEntity = new HttpEntity("{blbl}",headers);
ResponseEntity<String> response = new ResponseEntity("{blbl}", HttpStatus.OK);
RestTemplate mockRestTemplate = mock(RestTemplate.class);
when(mockRestTemplate.exchange(baseUrl, HttpMethod.POST, requestEntity, String.class)).thenReturn(response);
RestAPI api = new RestAPI(mockRestTemplate);
HTTPResult res = null;
try {
res = api.post(baseUrl,"{blbl}");
} catch (IOException e) {
e.printStackTrace();
}
assertEquals(res.getResponseBody(), "{blbl}");
assertEquals(res.getStatusCode(), HttpStatus.OK.value());
}
I am getting null pointer exception when calling:
res = api.post(baseUrl,"{blbl}");
This is because the response is null.
Use an argument matcher when arranging the mock as the instance being passed to the mocked dependency is different to what is passed when the test is exercised.
This will cause the mock to return null response as expected instances do not match
Refactor the test
public void testPost() throws Exception {
//Arrange
String expected = "{blbl}";
ResponseEntity<String> response = new ResponseEntity(expected, HttpStatus.OK);
RestTemplate mockRestTemplate = mock(RestTemplate.class);
when(mockRestTemplate.exchange(eq(baseUrl), eq(HttpMethod.POST), any(HttpEntity.class), eq(String.class)))
.thenReturn(response);
RestAPI api = new RestAPI(mockRestTemplate);
//Act
HTTPResult res = api.post(baseUrl, expected);
//Assert
assertEquals(res.getResponseBody(), expected);
assertEquals(res.getStatusCode(), HttpStatus.OK.value());
}
Note the use of the any(HttpEntity.class) matcher which will allow the passed HttpEntity to be matched when invoked.
Since the use of argument matches is none or all, the eq() matcher is used for the remaining constant arguments.
I try to simulate restful server:
private void btnPostActionPerformed(java.awt.event.ActionEvent evt) {
RestTemplate restTemplate = new RestTemplate();
Issuer issuer = new Issuer();
issuer.setCountry("Teacher 1");
issuer.setIssuerName("Department 1");
String url = txtHost.getText()+txtGet.getText();
restTemplate.postForObject(url, issuer, Issuer.class) ;
}
Controller code:
#RequestMapping(value = "/issuer/addIssuer", method = RequestMethod.POST)
#ResponseBody
public Issuer addIssuer(#ModelAttribute("issuer") Issuer issuer) {
if (issuer != null) {
logger.info("Inside addIssuer, adding: " + issuer.toString());
} else {
logger.info("Inside addIssuer...");
}
issuers.put(issuer.getTicker(), issuer);
return issuer;
}
I have fill some attributes, but when I debug the server, all values is null.
INFO : com.avaldes.tutorial.RestController - Inside addIssuer, adding: [null, null, null, null]
IssuerName and country is null too..
What is wrong with my code?
You are using #ModelAttribute in your controller. In that case you'll need to send your data as application/x-www-form-urlencoded:
MultiValueMap<String, Object> variables = new LinkedMultiValueMap<>();
variables.add("country", "Teacher 1");
variables.add("issuerName", "Department 1");
RestTemplate restTemplate = new RestTemplate();
HttpHeaders requestHeaders = new HttpHeaders();
requestHeaders.setContentType(MediaType.APPLICATION_FORM_URLENCODED);
HttpEntity<MultiValueMap<String, Object>> requestEntity = new HttpEntity<>(variables, requestHeaders);
String url = txtHost.getText()+txtGet.getText();
ResponseEntity<String> response = restTemplate.exchange(url, HttpMethod.POST, requestEntity, Issuer.class);