Develop Reference

getEntity() returns null, readEntity(InputStream.class) loads everything to memory

I have problem downloading file (getting the input stream) from server (quarkus implementation) in client (quarkus implementation)
I am using 2.9.2.Final version of Quarkus for both client and server,
dependency - quarkus-rest-client-reactive-jackson
I have a server which returns InputStream inside Response:
#Override
public Response downloadFileById(String id) {
var inputStream = repo.downloadFileByFileId(id);
return Response.ok(inputStream, MediaType.APPLICATION_OCTET_STREAM)
.header("Content-Disposition", "attachment; filename = \"" + fileName + "\"")
.header("Content-Length", repo.getFileLengthByFileById(id))
.build();
}
On client side, I've tried these variations:
This works but loads everything into memory. It uses ByteArrayInputStream inside ClientSendRequestHandler.java from package org.jboss.resteasy.reactive.client.handlers; (see line 340 in https://github.com/quarkusio/quarkus/blob/b0019c087880f9fd1371776b8c23c1b49129dcb3/independent-projects/resteasy-reactive/client/runtime/src/main/java/org/jboss/resteasy/reactive/client/handlers/ClientSendRequestHandler.java)
var stream = service.downloadFileById(id)
.await()
.indefinitely()
.readEntity(InputStream.class);
I expected this to work but it returns null
var stream = service
.downloadFileById(id)
.await()
.indefinitely()
.getEntity();
As I am going through following method from the ClientSendRequestHandler.java
private void attachSentHandlers(Future<HttpClientResponse> sent,
HttpClientRequest httpClientRequest,
RestClientRequestContext requestContext)
I am wondering, is it not supported?
I see cases for Multipart data, File, and an else branch
Not sure what am I missing.

iText 7 - How to send to an API and receive/render on the other end

I am trying to send a PDF that will be created in memory from one API to another API which will then render it in an HTML page. Info about my technologies:
Springboot
Microservices Architecture (with Eureka)
Java
Thymeleaf
What I have so far is a microservice which receives a String input from an input field (through an API) and then I get it here and I believe I prepare a pdf in memory(haven't tested it yet):
public InputStream convert(String input) throws FileNotFoundException {
ByteArrayOutputStream out = new ByteArrayOutputStream();
PdfWriter writer = new PdfWriter(out);
PdfDocument pdf = new PdfDocument(writer);
Document document = new Document(pdf);
document.add(new Paragraph(input));
document.close();
return new ByteArrayInputStream(out.toByteArray());
}
Here is my sending controller currently:
#RequestMapping("/cnv")
public InputStream doConversion(#RequestParam(defaultValue = "0") String input) {
try {
return textToPDFService.c2f(input);
} catch (FileNotFoundException e) {
throw new RuntimeException("Exception thrown while writing file: " + e);
}
}
I don't have any code to show for the web server that receives it yet, but you can expect since this is Springboot I am going to have a relevant endpoint in a #Controller and a method that communicates with my microservice of type #Service.
Question is, how do I receive this in InputStream in my web server's service and render it? Helpful resources are also welcome.
PS: I have never used iText before, Springboot or microservices prior to this. Also, never had a requirement about PDFs (yeah, I know I am a big noob).

It seems I got my answer. Make sure you return in your controller that performs the conversions a byte[] or even better ResponseEntity<byte[]>. Then, you should add headers to your request like so:
return ResponseEntity.ok()
.header(headerKey, headerValue)
.contentType(MediaType.APPLICATION_PDF)
.body(res);
On your receiving microservice you need a service which will do the following:
// Setup Headers & URL
String url = blah blah..;
HttpHeaders headers = new HttpHeaders();
headers.setAccept(Collections.singletonList(MediaType.APPLICATION_PDF));
HttpEntity<String> entity = new HttpEntity<>("body", headers);
// Get Result from Microservice
return restTemplate.exchange(url, HttpMethod.GET, entity, byte[].class, input);
Of course, this will return to you this ResponseEntity<byte[]>. You can just pass this to the controller endpoint and you are done:
return textToPDFService.textToPDFRequest(input);
Keep in mind you should mind exceptions and HTTP codes, but this will do as a minimal solution.

How to get InputStream via Spring-Feign?

I want to download and save a file in local directory from server by Spring-OpenFeign with zero-copy.
Naive download method as following:
import org.apache.commons.io.FileUtils
#GetMapping("/api/v1/files")
ResponseEntity<byte[]> getFile(#RequestParam(value = "key") String key) {
ResponseEntity<byte[]> resp = getFile("filename.txt")
File fs = new File("/opt/test")
FileUtils.write(file, resp.getBody())
}
In this code, data flow will be like this feign Internal Stream -> Buffer -> ByteArray -> Buffer -> File
How can I downalod and save a file memory-efficiently and faster way?

TL;DR. Use ResponseEntity<InputStreamResource> and Java NIO
According to SpringDecoder, Spring decode response using HttpMessageConverters
ResourceHttpMessageConverter which is one of HttpMesageConverters return InputStreamResource which contain InputStream and filename derived from Content-Disposition.
But, ResourceHttpMessageConverter must be initialized supportsReadStreaming = true (default value)
If you have further interests on this implementation, check this code.
So, changed code are as followings:
#GetMapping("/api/v1/files")
ResponseEntity<InputStreamResource> getFile(#RequestParam(value = "key") String key)
JDK9
try (OutputStream os = new FileOutputStream("filename.txt")) {
responeEntity.getBody().getInputStream().transferTo(os);
}
JDK8 or less
Use Guava ByteStreams.copy()
Path p = Paths.get(responseEntity.getFilename())
ReadableByteChannel rbc = Channels.newChannel(responeEntity.getBody().getInputStream())
try(FileChannel fc = FileChannel.open(p, StandardOpenOption.WRITE)) {
ByteStreams.copy(rbc, fc)
}
Now, Feign Internal Stream -> File

Jackson error "Illegal character... only regular white space allowed" when parsing JSON

I am trying to retrieve JSON data from a URL but get the following error:
Illegal character ((CTRL-CHAR, code 31)):
only regular white space (\r, \n,\t) is allowed between tokens
My code:
final URI uri = new URIBuilder(UrlConstants.SEARCH_URL)
.addParameter("keywords", searchTerm)
.addParameter("count", "50")
.build();
node = new ObjectMapper().readTree(new URL(uri.toString())); <<<<< THROWS THE ERROR
The url constructed is i.e https://www.example.org/api/search.json?keywords=iphone&count=50
What is going wrong here? And how can I parse this data successfully?
Imports:
import com.google.appengine.repackaged.org.codehaus.jackson.JsonNode;
import com.google.appengine.repackaged.org.codehaus.jackson.map.ObjectMapper;
import com.google.appengine.repackaged.org.codehaus.jackson.node.ArrayNode;
import org.apache.http.client.utils.URIBuilder;
example response
{
meta: {
indexAllowed: false
},
products: {
products: [
{
id: 1,
name: "Apple iPhone 6 16GB 4G LTE GSM Factory Unlocked"
},
{
id: 2,
name: "Apple iPhone 7 8GB 4G LTE GSM Factory Unlocked"
}
]
}
}

I got this same issue, and I found that it was caused by the Content-Encoding: gzip header. The client application (where the exception was being thrown) was not able to handle this content-encoding. FWIW the client application was using io.github.openfeign:feign-core:9.5.0, and this library appears to have some issues around compression (link).
You might try adding the header Accept-Encoding: identity to your request, however, not all web servers/web applications are configured properly, and some seem to disregard this header. See this question for more details about how to prevent gzipped content.

I had a similar issue. After some research, I found of that restTemplate uses the SimpleClientHttpRequestFactory which does not support gzip encoding. To enable gzip encoding for your response, you will need to set a new request factory for the rest template object - HttpComponentsClientHttpRequestFactory.
restTemplate.setRequestFactory(new HttpComponentsClientHttpRequestFactory());

The message should be pretty self-explanatory:
There is an illegal character (in this case character code 31, i.e. the control code "Unit Separator") in the JSON you are processing.
In other words, the data you are receiving is not proper JSON.
Background:
The JSON spec (RFC 7159) says:
JSON Grammar
A JSON text is a sequence of tokens. The set of tokens includes six
tructural characters, strings, numbers, and three literal names.
[...]
Insignificant whitespace is allowed before or after any of the
six structural characters.
ws = *(
%x20 / ; Space
%x09 / ; Horizontal tab
%x0A / ; Line feed or New line
%x0D ) ; Carriage return
In other words: JSON may contain whitespace between the tokens ("tokens" meaning the part of the JSON, i.e. lists, strings etc.), but "whitespace" is defined to only mean the characters Space, Tab, Line feed and Carriage return.
Your document contains something else (code 31) where only whitespace is allowed, hence is not valid JSON.
To parse this:
Unfortunately, the Jackson library you are using does not offer a way to parse this malformed data. To parse this successfully, you will have to filter the JSON before it is handled by Jackson.
You will probably have to retrieve the (pseudo-)JSON yourself from the REST service, using standard HTTP using, e.g. java.net.HttpUrlConnection. Then suitably filter out "bad" characters, and pass the resulting string to Jackson. How to do this exactly depends on how you use Jackson.
Feel free to ask a separate questions if you are having trouble :-).

I had the same problem. After setting Gzip it was fixed. Please refer my code
public String sendPostRequest(String req) throws Exception {
// Create connection
URL urlObject = new URL(mURL);
HttpURLConnection connection = (HttpURLConnection) urlObject.openConnection();
connection.setRequestMethod("POST");
connection.setRequestProperty("Content-Type", "application/json");
connection.setRequestProperty("Content-Length", Integer.toString(req.getBytes().length));
connection.setRequestProperty("Content-Language", "en-US");
connection.setUseCaches(false);
connection.setDoOutput(true);
// Send request
DataOutputStream wr = new DataOutputStream(connection.getOutputStream());
wr.writeBytes(req);
wr.close();
//Response handling
InputStream responseBody = null;
if (isGzipResponse(connection)) {
responseBody = new GZIPInputStream(connection.getInputStream());
}else{
responseBody = connection.getInputStream();
}
convertStreamToString(responseBody);
return response.toString();
}
protected boolean isGzipResponse(HttpURLConnection con) {
String encodingHeader = con.getHeaderField("Content-Encoding");
return (encodingHeader != null && encodingHeader.toLowerCase().indexOf("gzip") != -1);
}
public void convertStreamToString(InputStream in) throws Exception {
if (in != null) {
ByteArrayOutputStream baos = new ByteArrayOutputStream();
byte[] buffer = new byte[4096];
int length = 0;
while ((length = in.read(buffer)) != -1) {
baos.write(buffer, 0, length);
}
response = new String(baos.toByteArray());
baos.close();
} else {
response = null;
}
}

I had the same issue with zalando logbook in my spring boot application, and after reading the answers in here carefully, I realized, that the response interceptor must be applied after whatever takes care for decompression:
#Configuration
public class RestTemplateConfig {
[....]
#Bean
public RestTemplate restTemplate() {
return new RestTemplateBuilder()
.requestFactory(new MyRequestFactorySupplier())
.build();
}
class MyRequestFactorySupplier implements Supplier<ClientHttpRequestFactory> {
#Override
public ClientHttpRequestFactory get() {
CloseableHttpClient client = HttpClientBuilder.create()
.addInterceptorFirst(logbookHttpRequestInterceptor)
// wrong: .addInterceptorFirst(logbookHttpResponseInterceptor)
.addInterceptorLast(logbookHttpResponseInterceptor)
.build();
HttpComponentsClientHttpRequestFactory clientHttpRequestFactory =
new HttpComponentsClientHttpRequestFactory(client);
return clientHttpRequestFactory;
}
}
}

We had the same issue in out integration tests recently. We have a spring boot application and we use wiremock to mock a integrated microservice server. For one of the test get requests that we had implemented we started getting this error. We had to downgrade wiremock from 2.18.0 to 2.17.0 and it worked fine. Due to some bug the jackson parser and the that particular version of wiremock didn't work together. We didnt have time to figure out what actually the bug was in those libraries.

Those who use FeignClient, please refer to this answer spring-feign-not-compressing-response
Spring is not able to Decode the response on the fly, so you need to define a custom GZip Decoder.
Solved for me.

How to add an input stream to a multipart http request? (Java Spring)

I am creating a REST service using Spring 4's rest controller and testing with Spring's RestTemplate.
I would like to accept a request to create a new file (POST).
The concept of a file in my system contains two parts - (1) Metadata, (2) Content. Therefore I have defined my rest controller to accept a MultipartHttpServletRequest. I want to pull the metadata map from one part and a file input stream from the second part, but I'm not sure how to make this work.
Having an input stream instead of the entire file in the request is important because the files may be very large and I don't want to stress the network by making users send giant http requests with all of the file data.
Below I have a non-working implementation of my rest controller method and my test I use to send a test Http Request. When I run the below code I get a 500 Server Error exception because I believe that the request isn't matching correctly to how I have configured the RestController.
I have tried, in the controller, to make the #RequestParam's generic Objects instead of a Map and InputStream. This will run without a 500 exception, but then the objects are then interpreted as Strings, and that's not what I want (this would be okay for dealing with the metadata, but I want an InputStream for the file!).
Create a request:
public void createAndSendRequest() throws IOException {
MultiValueMap<String,Object> requestMap = new LinkedMultiValueMap<String, Object>();
url = "http:/my.domain/someService/document";
method = HttpMethod.POST;
returnType = String.class;
// add metadata to the request
Map<String, String> metadata = new HashMap<>();
metadata.put("somedata1", "alpha");
metadata.put("somedata2", "beta");
metadata.put("somedata3", "gamma");
requestMap.add("metadata", metadata);
// add the file to the request (as a stream?)
URL fileUrl = this.getClass().getClassLoader().getResource("test.txt");
File file = new File(fileUrl.getFile());
InputStream stream = new FileInputStream(file);
InputStreamResource resource = new InputStreamResource(stream);
requestMap.add("file", resource);
response = restTemplate.exchange(url, method, new HttpEntity<Object>(requestMap, headers), returnType);
handleResponse(response);
}
Rest Controller method to handle the request:
#RequestMapping(value = "/document", method = RequestMethod.POST)
public ResponseEntity<String> createDocument(MultipartHttpServletRequest request,
#RequestParam("file") InputStream fileStream,
#RequestParam("metadata") Map metadata) throws Exception {
// do some file stream work
int ch;
ch = fileStream.read();
while (ch != -1) {
System.out.println((char) ch);
ch = fileStream.read();
}
// do application stuff
return new ResponseEntity(HttpStatus.CREATED);
}