Develop Reference

Transfer file and store file id (elasticsearch) via webflux (spring-boot)

I'm trying to transfer files to a third party service via webflux and store the file ids in a elasticsearch. Files are transferred and saved, but the id is not attached to the entity.
controller:
#PostMapping(value = "upload", consumes = MediaType.MULTIPART_FORM_DATA_VALUE, produces = MediaType.APPLICATION_JSON_VALUE)
public Flux<String> store(#RequestParam(required = false) String orderId, #RequestPart("file") Flux<FilePart> files){
return imageService.store(orderId, files);
}
service:
public Flux<String> store(String orderId, Flux<FilePart> files) {
return marketService.findById(orderId)
.filter(Objects::nonNull)
.flatMapMany(order -> {
return files.ofType(FilePart.class).flatMap(file -> save(orderId, file));
});
}
private Mono<String> save(String orderId, FilePart file) {
return file.content()
.flatMap(dataBuffer -> {
byte[] bytes = new byte[dataBuffer.readableByteCount()];
dataBuffer.read(bytes);
String image = storeApi.upload(bytes, file.filename());
DataBufferUtils.release(dataBuffer);
return Mono.just(image);
})
.doOnNext(image -> marketService.addImages(orderId, image))
.last();
}
marketService.addImages:
public Mono<Order> addImages(String id, String image){
log.info("addImages: id={}, image={}", id, image);
return orderRepository
.findById(id)
.doOnNext(order -> {
if(order.getProduct().getImages() == null){
order.getProduct().setImages(new ArrayList<>());
}
order.getProduct().getImages().add(image);
})
.flatMap(this::create);
}
The code in the doOnNext and flatMap block in method (addImages) does not work. In doing so, calling the method (addImages) from the controller works fine. Tell me please what i'm missing.

I think I have found a solution. I changed the operation from doOnNext to flatMap as per this solution: Spring Webflux Proper Way To Find and Save
public Mono<OrderPostgres> store(String orderId, Flux<FilePart> files) {
return marketService.findById(orderId)
.filter(Objects::nonNull)
.flatMapMany(order -> {
return files.ofType(FilePart.class)
.flatMap(file -> save(orderId, file))
.collectList();
})
.flatMap(image -> marketService.addImages(orderId, image))
.then(Mono.empty());
}
private Flux<String> save(String orderId, FilePart file) {
return file.content()
.log()
.flatMap(dataBuffer -> {
byte[] bytes = new byte[dataBuffer.readableByteCount()];
dataBuffer.read(bytes);
String image = storeApi.upload(bytes, file.filename());
DataBufferUtils.release(dataBuffer);
return Flux.just(image);
}).switchIfEmpty(Flux.empty());
}
public Mono<OrderPostgres> addImages(String id, List<String> images){
log.info("### addImage: id={}, images={}", id, images);
Mono<OrderPostgres> byId = orderRepository.findById(id);
return byId
.flatMap(order -> {
if (order.getImages() == null) {
order.setImages(new ArrayList<>());
}
order.getImages().addAll(images);
return orderRepository.save(order);
}).then(Mono.empty());
}

Spring WebClient corrupts binary data

I'm sending files containing binary data from service A to service B. When the number of files is relatively small (let's say 5) everything works well. However, when I try to send more files (let's say several hundred) it sometimes fails. I tried to check what is happening with this binary data, and it looks like WebClient corrupts it in some way (weird padding appears at the end).
I created a minimal reproducible example to illustrate this issue.
Endpoint in service B (consuming binary files):
#RestController
class FilesController {
#PostMapping(value = "/files")
Mono<List<String>> uploadFiles(#RequestBody Flux<Part> parts) {
return parts
.filter(FilePart.class::isInstance)
.map(FilePart.class::cast)
.flatMap(part -> DataBufferUtils.join(part.content())
.map(buffer -> {
byte[] data = new byte[buffer.readableByteCount()];
buffer.read(data);
DataBufferUtils.release(buffer);
return Base64.getEncoder().encodeToString(data);
})
)
.collectList();
}
}
Tests illustrating how the service A sends data:
public class BinaryUploadTest {
private final CopyOnWriteArrayList<String> sentBytes = new CopyOnWriteArrayList<>();
#BeforeEach
void before() {
sentBytes.clear();
}
/**
* this test passes all the time
*/
#Test
void shouldUpload5Files() {
// given
MultiValueMap<String, HttpEntity<?>> body = buildResources(5);
// when
List<String> receivedBytes = sendPostRequest(body);
// then
assertEquals(sentBytes, receivedBytes);
}
/**
* this test fails most of the time
*/
#Test
void shouldUpload1000Files() {
// given
MultiValueMap<String, HttpEntity<?>> body = buildResources(1000);
// when
List<String> receivedBytes = sendPostRequest(body);
// then
assertEquals(sentBytes, receivedBytes);
}
private List<String> sendPostRequest(MultiValueMap<String, HttpEntity<?>> body) {
return WebClient.builder().build().post()
.uri("http://localhost:8080/files")
.contentType(MediaType.MULTIPART_FORM_DATA)
.body(BodyInserters.fromMultipartData(body))
.retrieve()
.bodyToMono(new ParameterizedTypeReference<List<String>>() {
})
.block();
}
private MultiValueMap<String, HttpEntity<?>> buildResources(int numberOfResources) {
MultipartBodyBuilder builder = new MultipartBodyBuilder();
for (int i = 0; i < numberOfResources; i++) {
builder.part("item-" + i, buildResource(i));
}
return builder.build();
}
private ByteArrayResource buildResource(int index) {
byte[] bytes = randomBytes();
sentBytes.add(Base64.getEncoder().encodeToString(bytes)); // keeps track of what has been sent
return new ByteArrayResource(bytes) {
#Override
public String getFilename() {
return "filename-" + index;
}
};
}
private byte[] randomBytes() {
byte[] bytes = new byte[ThreadLocalRandom.current().nextInt(16, 32)];
ThreadLocalRandom.current().nextBytes(bytes);
return bytes;
}
}
What could be the reason for this data corruption?

It turned out to be a bug in the Spring Framework (in the MultipartParser class to be more precise). I have created a GitHub issue which will be fixed in the next version (5.3.16). The bug is fixed by this commit.

How to stream file from Multipart/form-data in Spring WebFlux

I want to receive Multipart/form-data from a client (frontend for example). And then stream file content of form-data to another backend service.
For now i can read the whole file and pass it somewhere via byte[] (base64 string) like this:
#PostMapping(consumes = MediaType.MULTIPART_FORM_DATA_VALUE)
public Mono<ResponseType> upload(#RequestPart("document") FilePart document,
#RequestPart("stringParam") String stringParam) {
return service.upload(document, stringParam);
}
// Casually convert to single byte array...
private Mono<byte[]> convertFilePartToByteArray(FilePart filePart) {
return Mono.from(filePart
.content()
.map(dataBuffer -> {
byte[] bytes = new byte[dataBuffer.readableByteCount()];
dataBuffer.read(bytes);
DataBufferUtils.release(dataBuffer);
return bytes;
}));
}
There're a few problems with this approach:
I don't want to read the whole file into memory;
Array size in limited to Integer.MAX_VALUE;
Array encodes as base64 String, which takes extra memory;
Since i put the whole array in Mono - "spring.codec.max-in-memory-size" must be bigger than array size.
I've already tried sending file via asyncPart of WebClientBuilder:
MultipartBodyBuilder builder = new MultipartBodyBuilder();
builder.asyncPart("document", document.content(), DataBuffer.class);
But i'm getting an error:
java.lang.IllegalStateException: argument type mismatch
Method [public reactor.core.publisher.Mono<> upload(**org.springframework.http.codec.multipart.FilePart**,java.lang.String)] with argument values:
[0] [type=**org.springframework.http.codec.multipart.DefaultParts$DefaultFormFieldPart**]
UPD: full code, which generates error
// External controller for client.
#PostMapping(consumes = MediaType.MULTIPART_FORM_DATA_VALUE, value = "/v2")
public Mono<DocumentUploadResponse> uploadV2(#RequestPart("document") FilePart document,
#RequestPart("stringParam") String stringParam) {
MultipartBodyBuilder builder = new MultipartBodyBuilder();
builder.asyncPart("document", document.content(), DataBuffer.class);
builder.part("stringParam", stringParam);
WebClient webClient = webClientBuilder.build();
return webClient.post()
.uri("URL_TO_ANOTHER_SERVICE")
.contentType(MediaType.MULTIPART_FORM_DATA)
.body(BodyInserters.fromMultipartData(builder.build()))
.retrieve()
.bodyToMono(FileMetaDto.class)
.map(DocumentUploadResponse::new);
}
// Internal service controller.
#PostMapping(path = "/upload/v2", consumes = MediaType.MULTIPART_FORM_DATA_VALUE)
public Mono<FileMetaDto> upload(#RequestPart("document") FilePart document,
#RequestPart("stringParam") String stringParam) {
return ...;
}

Looks like i was managed to stream file, working code below:
// External controller for client.
#PostMapping(consumes = MediaType.MULTIPART_FORM_DATA_VALUE, value = "/v2")
public Mono<DocumentUploadResponse> uploadV2(#RequestPart("document") FilePart document,
#RequestPart("stringParam") String stringParam) {
MultipartBodyBuilder builder = new MultipartBodyBuilder();
builder.asyncPart("document", document.content(), DataBuffer.class).filename(document.filename());
builder.part("stringParam", stringParam);
WebClient webClient = webClientBuilder.build();
return webClient.post()
.uri("URL_TO_ANOTHER_SERVICE")
.contentType(MediaType.MULTIPART_FORM_DATA)
.body(BodyInserters.fromMultipartData(builder.build()))
.retrieve()
.bodyToMono(FileMetaDto.class)
.map(DocumentUploadResponse::new);
}
// Internal service controller.
#PostMapping(path = "/upload/v2", consumes = MediaType.MULTIPART_FORM_DATA_VALUE)
public Mono<FileMetaDto> upload(#RequestPart("document") FilePart document,
#RequestPart("stringParam") String stringParam) {
return ...;
}
In the original question code i've been missing:
builder.asyncPart("document", document.content(), DataBuffer.class).filename(document.filename());

Spring Web Reactive client

I'm trying to use the Spring Reactive WebClient to upload a file to a spring controller. The controller is really simple and looks like this:
#PostMapping(value = "/upload", consumes = MediaType.MULTIPART_FORM_DATA_VALUE)
public ResponseEntity<String> uploadFile(
#RequestParam("multipartFile") MultipartFile multipartFile,
#RequestParam Map<String, Object> entityRequest
) {
entityRequest.entrySet().forEach(System.out::println);
System.out.println(multipartFile);
return ResponseEntity.ok("OK");
}
When I use this controller with cURL everything works fine
curl -X POST http://localhost:8080/upload -H 'content-type: multipart/form-data;' -F fileName=test.txt -F randomKey=randomValue -F multipartFile=#document.pdf
The multipartFile goes to the correct parameter and the other parameters go in to the Map.
When I try to do the same from the WebClient I get stuck. My code looks like this:
WebClient client = WebClient.builder().baseUrl("http://localhost:8080").build();
MultiValueMap<String, Object> map = new LinkedMultiValueMap<>();
map.set("multipartFile", new ByteArrayResource(Files.readAllBytes(Paths.get("/path/to/my/document.pdf"))));
map.set("fileName", "test.txt");
map.set("randomKey", "randomValue");
String result = client.post()
.uri("/upload")
.contentType(MediaType.MULTIPART_FORM_DATA)
.syncBody(map)
.exchange()
.flatMap(response -> response.bodyToMono(String.class))
.flux()
.blockFirst();
System.out.println("RESULT: " + result);
This results in an 400-error
{
"timestamp":1510228507230,
"status":400,
"error":"Bad Request",
"message":"Required request part 'multipartFile' is not present",
"path":"/upload"
}
Does anyone know how to solve this issue?

So i found a solution myself. Turns out that Spring really needs the Content-Disposition header to include a filename for a upload to be serialized to a MultipartFile in the Controller.
To do this i had to create a subclass of ByteArrayResource that supports setting the filename
public class MultiPartResource extends ByteArrayResource {
private String filename;
public MultiPartResource(byte[] byteArray) {
super(byteArray);
}
public MultiPartResource(byte[] byteArray, String filename) {
super(byteArray);
this.filename = filename;
}
#Nullable
#Override
public String getFilename() {
return filename;
}
public void setFilename(String filename) {
this.filename = filename;
}
}
Which can then be used in the client with this code
WebClient client = WebClient.builder().baseUrl("http://localhost:8080").build();
MultiValueMap<String, Object> map = new LinkedMultiValueMap<>();
map.set("fileName", "test.txt");
map.set("randomKey", "randomValue");
ByteArrayResource resource = new MultiPartResource(Files.readAllBytes(Paths.get("/path/to/my/document.pdf")), "document.pdf");
String result = client.post()
.uri("/upload")
.contentType(MediaType.MULTIPART_FORM_DATA)
.body(BodyInserters.fromMultipartData(map))
.exchange()
.flatMap(response -> response.bodyToMono(String.class))
.flux()
.blockFirst();
System.out.println("RESULT: " + result);

You need include a filename to file part to upload success, in combination with asyncPart() to avoid buffering all file content, then you can write the code like this:
WebClient client = WebClient.builder().baseUrl("http://localhost:8080").build();
Mono<String> result = client.post()
.uri("/upload")
.contentType(MediaType.MULTIPART_FORM_DATA)
.body((outputMessage, context) ->
Mono.defer(() -> {
MultipartBodyBuilder bodyBuilder = new MultipartBodyBuilder();
Flux<DataBuffer> data = DataBufferUtils.read(
Paths.get("/tmp/file.csv"), outputMessage.bufferFactory(), 4096);
bodyBuilder.asyncPart("file", data, DataBuffer.class)
.filename("filename.csv");
return BodyInserters.fromMultipartData(bodyBuilder.build())
.insert(outputMessage, context);
}))
.exchange()
.flatMap(response -> response.bodyToMono(String.class));
System.out.println("RESULT: " + result.block());

Easier way to provide the Content-Disposition
MultipartBodyBuilder builder = new MultipartBodyBuilder();
String header = String.format("form-data; name=%s; filename=%s", "paramName", "fileName.pdf");
builder.part("paramName", new ByteArrayResource(<file in byte array>)).header("Content-Disposition", header);
// in the request use
webClient.post().body(BodyInserters.fromMultipartData(builder.build()))

Using a ByteArrayResource in this case is not efficient, as the whole file content will be loaded in memory.
Using a UrlResource with the "file:" prefix or a ClassPathResource should solve both issues.
UrlResource resource = new UrlResource("file:///path/to/my/document.pdf");

Java Swagger (Springfox) annotations for streaming multipart file upload

We are using spring controllers to handle file uploads:
For example:
#RequestMapping(value = "/scan", method = RequestMethod.POST, consumes = {MediaType.MULTIPART_FORM_DATA_VALUE})
public ScanResult scan(HttpServletRequest request) throws IOException, FileUploadException {
return scanService.scanFile(parseMultipart(request));
}
But we are not using any multipart resolver, we are streaming the files from the servlet request input stream. We need to start processing the file immediately for performance reasons.
When doing this this way, we can't seem to use the typical detection/configuration for multipart files. I know Springfox (which we use to generate our swagger docs) will generate the appropriate swagger controls if it sees a MultipartFile as a controller parameter, which will not be the case for us.
Are there any other config options available to hint to springfox that we want a file upload here?

Regarding breaking changes in Springfox v2.7.0:
You need to use dataType = "__file" instead of file as commented in https://github.com/springfox/springfox/issues/1285

Found my answer here: https://github.com/springfox/springfox/issues/1285
The following implicit params give me what I need:
#ApiImplicitParams (value = {
#ApiImplicitParam(dataType = "file", name = "file", required = true,paramType = "form")}
#RequestMapping(value = "/scan", method = RequestMethod.POST, consumes = {MediaType.MULTIPART_FORM_DATA_VALUE})
public ScanResult scan(HttpServletRequest request) throws IOException, FileUploadException {
return scanService.scanFile(parseMultipart(request));
}
This adds a simple file picker to the API. To make things more confusing, turns out this functionality was broken in Springfox 2.4 - the version I was using. Adding that annotation and updating versions was all I needed to do.

That's right
https://stackoverflow.com/a/44385675/3810914
In Controller It should be:
#ApiOperation(value = "Upload file", response = String.class)
#ApiResponses({
#ApiResponse(code = 500, message = "Internal Server Error"),
#ApiResponse(code = 400, message = "Bad request")
})
#ApiImplicitParams (value = {
#ApiImplicitParam(dataType = "__file", name = "fileData", required = true,paramType = "form")})
#PostMapping(value = "/upload", consumes = {MediaType.MULTIPART_FORM_DATA_VALUE})
public ResponseEntity<?> uploadFileSimple(UploadFile form) {
// Create folder to save file if not exist
File uploadDir = new File(UPLOAD_DIR);
if (!uploadDir.exists()) {
uploadDir.mkdirs();
}
MultipartFile fileData = form.getFileData();
String name = fileData.getOriginalFilename();
if (name != null && name.length() > 0) {
try {
// Create file
File serverFile = new File(UPLOAD_DIR + "/" + name);
BufferedOutputStream stream = new BufferedOutputStream(new FileOutputStream(serverFile));
stream.write(fileData.getBytes());
stream.close();
return ResponseEntity.ok("/file/" + name);
} catch (Exception e) {
return ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR).body("Error when uploading");
}
}
return ResponseEntity.status(HttpStatus.BAD_REQUEST).body("Bad request");
}
And In Model:
package com.xxx.xxx.request;
import lombok.*;
import org.springframework.web.multipart.MultipartFile;
#Setter
#Getter
#NoArgsConstructor
#AllArgsConstructor
public class UploadFile {
private MultipartFile fileData;
}