Develop Reference

POST multipart/form-data vs GET multipart/form-data [duplicate]

I have a java controller which have to send me some text data and different byte arrays. So I am building n multipart request and writing it to stream from HttpServletResponse.
Now my problem is how to parse the response at client side and extract the multiple parts.
SERVER CODE SNIPPET:-
MultipartEntityBuilder builder = MultipartEntityBuilder.create();
// Prepare payload
builder.addBinaryBody("document1", file);
builder.addBinaryBody("document2", file2);
builder.addPart("stringData", new StringBody(jsonData, ContentType.TEXT_PLAIN));
// Set to request body
HttpEntity entity = builder.build();
postRequest.setEntity(entity);
CLIENT CODE SNIPPET:-
HttpPost httpPost = new HttpPost(finalUrl);
StringEntity entity = new StringEntity(json);
httpPost.setEntity(entity);
httpPost.setHeader("Content-type", APPLICATION_JSON_TYPE);
ByteArrayOutputStream baos = new ByteArrayOutputStream();
CloseableHttpResponse response = httpClient.execute(httpPost);
InputStream in = new BufferedInputStream(response.getEntity().getContent());
I checked CloseableHttpResponse and HttpEntity but none of them is providing method to parse multipart request.
EDIT 1:
This is my sample response I am receiving at client side stream:-
--bvRi5oZum37DUldtLgQGSbc5RRVZxKpjZMO4SYDe
Content-Disposition: form-data; name="numeric"
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit
01010110
--bvRi5oZum37DUldtLgQGSbc5RRVZxKpjZMO4SYDe
Content-Disposition: form-data; name="stringmessage"
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding:8bit
testmessage
--bvRi5oZum37DUldtLgQGSbc5RRVZxKpjZMO4SYDe
Content-Disposition: form-data; name="binarydata"; filename="file1"
Content-Type: application/octet-stream
Content-Transfer-Encoding: binary
HI, THIS IS MY BINARY DATA
--bvRi5oZum37DUldtLgQGSbc5RRVZxKpjZMO4SYDe
Content-Disposition: form-data; name="ending"
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit
ending
--bvRi5oZum37DUldtLgQGSbc5RRVZxKpjZMO4SYDe--

I have finally got a workaround for it.
I will be using javax mail MimeMultipart.
Below is a code snipped for the solution:-
ByteArrayDataSource datasource = new ByteArrayDataSource(in, "multipart/form-data");
MimeMultipart multipart = new MimeMultipart(datasource);
int count = multipart.getCount();
log.debug("count " + count);
for (int i = 0; i < count; i++) {
BodyPart bodyPart = multipart.getBodyPart(i);
if (bodyPart.isMimeType("text/plain")) {
log.info("text/plain " + bodyPart.getContentType());
processTextData(bodyPart.getContent());
} else if (bodyPart.isMimeType("application/octet-stream")) {
log.info("application/octet-stream " + bodyPart.getContentType());
processBinaryData(bodyPart.getInputStream()));
} else {
log.warn("default " + bodyPart.getContentType());
}
}
Please let me know if anybody else have any standard solution.

TL;DR
If you know what HttpMessageConverters are, then skip the "In General" part.
If your client is reactive, than you most likely don't have a big problem. Go to the part named "Reactive Client (Spring Webflux)" for details.
If your client is non-reactive i.e. you're using Spring MVC and RestTemplate then the last section is for you. In short, it is not possible out of the box and you need to write custom code.
In General
When we want to read multipart data, then we are at the serialization/marshalling layer of our application. This is basically the same layer as when we transform a JSON or an XML document to a POJO via Jackson for example. What I want to emphasize here is that the logic for parsing multipart data should not take place in a service but rather much earlier.
Our hook to transform multipart data comes as early, as to when an HTTP response enters our application in form of an HttpInputMessage. Out of the box, Spring provides a set of HttpMessageConverters, that are able to transform our HTTP response to an object with which we can work. For example, the MappingJackson2HttpMessageConverter is used to read and write all request/responses that have the MediaType "application/Json".
If the application is reactive, then Spring uses HttpMessageReader
and HttpMessageWriter instead of HttpMessageConverters. They save the same purpose.
The following two sections show how to read (download) a multipart response via the two different paradigms.
Reactive Client (Spring Webflux)
This would be the easiest use case and the only thing we need, is already available in Spring Webflux out of the box.
The class MultipartHttpMessageReader would be doing all the heavy lifting. In case it does not behave exactly how you need it to, you can easily extend it and overwrite the methods to your liking. Your custom Reader can then be registered as a bean like so:
#Configuration
public class MultipartMessageConverterConfiguration {
#Bean
public CodecCustomizer myCustomMultipartHttpMessageWriter() {
return configurer -> configurer.customCodecs()
.register(new MyCustomMultipartHttpMessageWriter());
}
}
Non-Reactive Client (Spring MVC/RestTemplate)
If you have a "classic" application that uses RestTemplate to communicate via HTTP, then you need to rely on the aforementioned HttpMessageConverters. Unfortunately, the MessageConverter that is responsible for reading multipart data, does not support reading/downloading data:
Implementation of HttpMessageConverter to read and write 'normal' HTML forms and also to write (but not read) multipart data (e.g. file uploads)
Source: FormHttpMessageConverter Documentation
So what we need to do, is write our own MessageConverter, which is able to download multipart data. An easy way to do that, would be to make use of the DefaultPartHttpMessageReader that is internally used by MultipartHttpMessageReader. We don't even need Webflux for that, as it is already shipped with spring-web.
First let us define 2 classes in which we save the several parts the we read:
public class MyCustomPart {
public MyCustomPart(byte[] content, String filename, MediaType contentType) {
//assign to corresponding member variables; here omitted
}
}
/**
* Basically a container for a list of objects of the class above.
*/
public class MyCustomMultiParts {
public MyCustomMultiParts(List<MyCustomPart> parts){
//assign to corresponding member variable; here omitted
}
}
Later on, you can always take each Part and convert it to whatever is appropriate. The MyCustomPart represents a single block of your multipart data response. The MyCustomMultiParts represent the whole multipart data.
Now we come to the meaty stuff:
public class CustomMultipartHttpMessageConverter implements HttpMessageConverter<MyCustomMultiParts> {
private final List<MediaType> supportedMediaTypes = new ArrayList<>();
private final DefaultPartHttpMessageReader defaultPartHttpMessageReader;
public CustomMultipartHttpMessageConverter() {
this.supportedMediaTypes.add(MediaType.MULTIPART_FORM_DATA);
this.defaultPartHttpMessageReader = new DefaultPartHttpMessageReader();
}
#Override
public boolean canRead(final Class<?> clazz, #Nullable final MediaType mediaType) {
if (!MyCustomMultiParts.class.isAssignableFrom(clazz)) {
return false;
}
if (mediaType == null) {
return true;
}
for (final MediaType supportedMediaType : getSupportedMediaTypes()) {
if (supportedMediaType.includes(mediaType) && mediaType.getParameter("boundary") != null) {
return true;
}
}
return false;
}
/**
* This wraps the input message into a "reactive" input message, that the reactive DefaultPartHttpMessageReader uses.
*/
private ReactiveHttpInputMessage wrapHttpInputMessage(final HttpInputMessage message) {
return new ReactiveHttpInputMessage() {
#Override
public HttpHeaders getHeaders() {
return message.getHeaders();
}
#SneakyThrows //Part of lombok. Just use a try catch block if you're not using it
#Override
public Flux<DataBuffer> getBody() {
final DefaultDataBuffer wrappedBody = new DefaultDataBufferFactory()
.wrap(message.getBody().readAllBytes());
return Flux.just(wrappedBody);
}
};
}
#Override
public MyCustomMultiParts read(#Nullable final Class<? extends MyCustomMultiParts> clazz,
final HttpInputMessage message) throws IOException, HttpMessageNotReadableException {
final ReactiveHttpInputMessage wrappedMessage = wrapHttpInputMessage(message);
final ResolvableType resolvableType = ResolvableType.forClass(byte[].class); //plays no role
List<Part> rawParts = defaultPartHttpMessageReader.read(resolvableType, wrappedMessage, Map.of())//
.buffer()//
.blockFirst();
//You can check here whether the result exists or just continue
final List<MyCustomPart> customParts = rawParts.stream()// Now we convert to our customPart
.map(part -> {
//Part consists of a DataBuffer, we make a byte[] so we can convert it to whatever we want later
final byte[] content = Optional.ofNullable(part.content().blockFirst())//
.map(DataBuffer::asByteBuffer)//
.map(ByteBuffer::array)//
.orElse(new byte[]{});
final HttpHeaders headers = part.headers();
final String filename = headers.getContentDisposition().getFilename();
final MediaType contentType = headers.getContentType();
return new MyCustomPart(content, filename, contentType);
}).collect(Collectors.toList());
return new MyCustomMultiParts(customParts);
}
#Override
public void write(final MyCustomMultiParts parts, final MediaType contentType,
final HttpOutputMessage outputMessage) {
// we're just interested in reading
throw new UnsupportedOperationException();
}
#Override
public boolean canWrite(final Class<?> clazz, final MediaType mediaType) {
// we're just interested in reading
return false;
}
#Override
public List<MediaType> getSupportedMediaTypes() {
return this.supportedMediaTypes;
}
}
From here on, you should know better what to do with your "CustomPart". Whether it is a JSON, a bitmap or a PDF. From the byte array you can convert it into anything.
Now if you want to test it, you only have to add your CustomConverter to a RestTemplate and then "await" the MyCustomMultiParts that we defined:
// This could also be inside your #Bean definition of RestTemplate of course
final RestTemplate restTemplate = new RestTemplate();
final List<HttpMessageConverter<?>> messageConverters = restTemplate.getMessageConverters();
messageConverters.add(new CustomMultipartHttpMessageConverter());
String url = "http://server.of.choice:8080/whatever-endpoint-that-sends-multiparts/";
final HttpHeaders headers = new HttpHeaders();
headers.setAccept(List.of(MediaType.MULTIPART_FORM_DATA));
final HttpEntity<Void> requestEntity = new HttpEntity<>(headers);
//here we await our MyCustomMultiParts
final MyCustomMultiParts entity = restTemplate.exchange(url, GET, requestEntity, MyCustomMultiParts.class);

Mime4j from Apache is one way to parse the responses from client-side. Its a common practice to use a tool like this.
You can refer this link - http://www.programcreek.com/java-api-examples/index.php?api=org.apache.james.mime4j.MimeException
You can download the jar from this link -
http://james.apache.org/download.cgi#Apache_Mime4J

The request body is sent as json even though the content type is set as application/x-www-form-urlencoded

This is related to an existing spring boot question raised by me(Request Body is not properly encoded and hidden when using spring form encoder in Feign Client).
According to this question, we can add either content type in headers or add during request mapping itself as consumes.
So what I did was added content type in headers in the client configuration class
public class EmailClientConfiguration {
#Bean
public RequestInterceptor requestInterceptor(Account<Account> account) {
return template -> {
template.header("Content-Type", "application/x-www-form-urlencoded");
};
}
#Bean
public OkHttpClient client() {
return new OkHttpClient();
}
#Bean
Logger.Level feignLoggerLevel() {
return Logger.Level.FULL;
}
#Bean
public Decoder feignDecoder() {
return new JacksonDecoder();
}
#Bean
public Encoder feignFormEncoder () {
return new SpringFormEncoder(new JacksonEncoder());
}
}
and I see in the headers the content type is correctly set as application/x-www-form-urlencoded when the request is sent. But the request body is still sent in json format and also not hidden.
Request Body:
Map<String, String> requestBody = new HashMap<>();
requestBody.put("username", "xyz");
requestBody.put("email", "xyz#gmail.com");
requestBody.put("key", "xxx");
Request Body received in server end:
{"{\n \"key\" : \"xxx\",\n \"email\" : \"xyz#gmail.com\",\n \"username\" : \"xyz\"\n}"
When I add consumes in my request mapping as application/x-www-form-urlencoded
#FeignClient(name = "email", url = "localhost:3000",
configuration = EmailClientConfiguration.class)
public interface EmailClient {
#PostMapping(value = "/email/send", consumes = "application/x-www-form-urlencoded")
ResponseDto sendEmail(#RequestBody Map<String, String> requestBody);
}
it works fine(request body is hidden in server end and also properly encoded). And when I removed the header in the configuration class and adding only consumes works fine without no issues but the vice versa has this problem.
I searched in internet for this and couldn't find any answer.

Feign encodes the request body and parameters before passing the request to any RequestInterceptor (and rightly so). If you do not declare consumes = "application/x-www-form-urlencoded", SprinFormEncoder doesn't know that you're trying to send form data, so it delegates serialization to the inner JacksonEncoder which only does JSON (see for yourself by printing template.body() before setting the header).
Handling such a well-supported header in the interceptor doesn't seem like a good idea, when you already have consumes. If you insist on doing so, you have to provide your own encoder which doesn't rely on the header value and always outputs form-urlencoded data.

How to pass complex object with file data via REST

I need to download files from one source and pass this data to another application via rest. The file types are: .txt .csv and .zip for now. The file size could be up to 500Mb - 1 Gb.
What is the optimal way to do it. Should I convert java File objects to byte array at first? Will the Multipart content type be the most appropriate one for this purpose?
I stacked a bit because the class I am going to transfer can contain different file types.
There is not code needed from your side just to have a clue how to handle it in a better way! ;)
The class below to transfer is:
public class FileEventsRequest {
private File originalFile;
private int rowCount;
private String md5;
private String cobDate;
private File controlFile; }

I worked on a task almost identical to the one you describe fairly recently!
To download the file I used -
#PutMapping("/{originalFileName}")
public ResponseEntity<ImmutableDocument> send(#PathVariable String originalFileName, InputStream payload) {
LOG.info("Receiving: {}", originalFileName);
sendPayload(payload, originalFileName);
return ResponseEntity.ok().build();
}
I chose InputStream as it's generally a bad idea to handle massive files 1GB in memory with a byte array, you could potentially blow the stack!
As for sending that file on to the target -
#Autowired
private RestTemplate sendTemplate;
private ResponseEntity<Void> sendPayload(final InputStream payload, final String originalFileName) throws IOException {
// You can send any other bits of information you need on the headers too
HttpHeaders headers = new HttpHeaders();
headers.put("originalFileName", originalFileName);
headers.setContentType(asMediaType(MimeType.valueOf({desired mimetype})));
headers.setAccept(singletonList(APPLICATION_OCTET_STREAM));
HttpEntity<Resource> requestEntity = new HttpEntity<>(new InputStreamResource(payload), headers);
UriComponentsBuilder builder = fromUriString({someurl});
UriComponents uriComponents = builder.build().encode();
return sendTemplate.exchange(uriComponents.toUri(), HttpMethod.POST, requestEntity, Void.class);
}
}
I also used my own Configuration class for the ResTemplate with a message converter for sending the binary InputStream -
#Configuration
public class RestClientConfiguration {
#Bean
public RestTemplate sendTemplate(ClientHttpRequestFactory clientHttpRequestFactory) {
return new RestTemplateBuilder()
.requestFactory(() -> clientHttpRequestFactory)
.messageConverters(new ResourceHttpMessageConverter())
.build();
}
}

How to modify part of the Content-Disposition in java rest template client

I'm struggling with creating valid request to some internal service using java spring. The problem is with proper payload for multipart/form-data boundary.
Environment:
java server -> (rest) http multipart/form-data -> some service
(there is no browser in the middle)
Valid payload should look like this:
------WebKitFormBoundaryp8mrQWOb5GiyC90y
Content-Disposition: form-data; name="files"; filename="0000.png"
Content-Type: image/png
[binary data]
------WebKitFormBoundaryp8mrQWOb5GiyC90y--
Unfortunately I'm unable to change this "headers" and I'm getting something like this:
--fkGT7CJaQB9-2aa8G1ePv17iHKnWSsd
Content-Disposition: form-data; name="files"
Content-Length: 170096
[binary data]
--fkGT7CJaQB9-2aa8G1ePv17iHKnWSsd--
I've searched many stackoverlow questions, but nothing seems to work.
This is what I've done till now (generates above payload):
HashMap<String, List<String>> additionalHeaders = new HashMap<>();
String fileMd5 = "tgrlfG0pjblWZB6g1f7j5w=="; //#todo
File file = new File(systemFile.getAbsoluteFileLocation());
Path filePath = Paths.get(systemFile.getAbsoluteFileLocation());
try{
DiskFileItem fileItem = new DiskFileItem("file", "image/png", false, file.getName(), (int) file.length() , file.getParentFile());
InputStream input = new FileInputStream(file);
OutputStream os = fileItem.getOutputStream();
int ret = input.read();
while ( ret != -1 )
{
os.write(ret);
ret = input.read();
}
os.flush();
MultipartFile multipartFile = new CommonsMultipartFile(fileItem);
MultiValueMap<String, Object> parts =
new LinkedMultiValueMap<>();
ByteArrayResource resource = new ByteArrayResource(multipartFile.getBytes());
parts.add("files", resource);
additionalHeaders.put("Content-MD5", Collections.singletonList(fileMd5));
additionalHeaders.put("Content-Disposition", Collections.singletonList("attachment; filename=\""+systemFile.getFilenameWithExtension()+"\""));
ResponseEntity<FrpFileServer> responseEntity = formDataRestClient.post(this, parts, FrpFileServer.class, isServerSide, frpToken.getTokenId(), additionalHeaders, MediaType.MULTIPART_FORM_DATA);
return responseEntity.getBody();
} catch (IOException e) {
return null;
}
formDataRestClient builds the rest of the request via RestTemplate
public <K, T> ResponseEntity<T> post(RestClientInterface reference, K requestClass, Class<T> responseClass, boolean isServerSide, String resourceId, HashMap<String, List<String>> additionalHeaders, MediaType contentType) {

Ok, I've fixed my problem with replacing resource creation with this:
ByteArrayResource resource = new ByteArrayResource(multipartFile.getBytes()){
#Override
public String getFilename() {
return systemFile.getFilenameWithExtension();
}
};
Thanks to that, restTemplate handles it's magic in proper way :)

I wrote one generic REST client which you can use in any java based application or framework, see GIT source of delete method how to load anything in request header where i am passing Authorization- https://github.com/gajeralalji/JAVA-REST-Client/wiki
check source of REST-client.java class and let me know if you are still facing any issue.

Why RestTemplate GET response is in JSON when should be in XML?

I struggled with an extrange spring behavior using RestTemplate (org.springframework.web.client.RestTemplate) without success.
I use in my hole application below code and always receive an XML response, which I parse and evaluate its result.
String apiResponse = getRestTemplate().postForObject(url, body, String.class);
But can't figure out why a server response is in JSON format after executing:
String apiResponse = getRestTemplate().getForObject(url, String.class);
I've debugged at low level RestTemplate and the content type is XML, but have no idea why the result is in JSON.
When I access from a browser the response is also in XML, but in apiResponse I got JSON.
I tried many options after reading Spring documentation
http://docs.spring.io/spring/docs/3.0.x/api/org/springframework/web/client/RestTemplate.html
Also tried to modify explicitly the headers but still can't figure it out.
I debugged RestTemplate class and noticed that this method is always setting application/json:
public void doWithRequest(ClientHttpRequest request) throws IOException {
if (responseType != null) {
List<MediaType> allSupportedMediaTypes = new ArrayList<MediaType>();
for (HttpMessageConverter<?> messageConverter : getMessageConverters()) {
if (messageConverter.canRead(responseType, null)) {
List<MediaType> supportedMediaTypes = messageConverter.getSupportedMediaTypes();
for (MediaType supportedMediaType : supportedMediaTypes) {
if (supportedMediaType.getCharSet() != null) {
supportedMediaType =
new MediaType(supportedMediaType.getType(), supportedMediaType.getSubtype());
}
allSupportedMediaTypes.add(supportedMediaType);
}
}
}
if (!allSupportedMediaTypes.isEmpty()) {
MediaType.sortBySpecificity(allSupportedMediaTypes);
if (logger.isDebugEnabled()) {
logger.debug("Setting request Accept header to " + allSupportedMediaTypes);
}
request.getHeaders().setAccept(allSupportedMediaTypes);
}
}
}
Could you give an idea?

I could solve my issue with RC.'s help. I'll post the answer to help other people.
The problem was that Accept header is automatically set to APPLICATION/JSON so I had to change the way to invoke the service in order to provide the Accept header I want.
I changed this:
String response = getRestTemplate().getForObject(url, String.class);
To this in order to make the application work:
// Set XML content type explicitly to force response in XML (If not spring gets response in JSON)
HttpHeaders headers = new HttpHeaders();
headers.setAccept(Arrays.asList(MediaType.APPLICATION_XML));
HttpEntity<String> entity = new HttpEntity<String>("parameters", headers);
ResponseEntity<String> response = getRestTemplate().exchange(url, HttpMethod.GET, entity, String.class);
String responseBody = response.getBody();