I'm trying to transfer files to a third party service via webflux and store the file ids in a elasticsearch. Files are transferred and saved, but the id is not attached to the entity.
controller:
#PostMapping(value = "upload", consumes = MediaType.MULTIPART_FORM_DATA_VALUE, produces = MediaType.APPLICATION_JSON_VALUE)
public Flux<String> store(#RequestParam(required = false) String orderId, #RequestPart("file") Flux<FilePart> files){
return imageService.store(orderId, files);
}
service:
public Flux<String> store(String orderId, Flux<FilePart> files) {
return marketService.findById(orderId)
.filter(Objects::nonNull)
.flatMapMany(order -> {
return files.ofType(FilePart.class).flatMap(file -> save(orderId, file));
});
}
private Mono<String> save(String orderId, FilePart file) {
return file.content()
.flatMap(dataBuffer -> {
byte[] bytes = new byte[dataBuffer.readableByteCount()];
dataBuffer.read(bytes);
String image = storeApi.upload(bytes, file.filename());
DataBufferUtils.release(dataBuffer);
return Mono.just(image);
})
.doOnNext(image -> marketService.addImages(orderId, image))
.last();
}
marketService.addImages:
public Mono<Order> addImages(String id, String image){
log.info("addImages: id={}, image={}", id, image);
return orderRepository
.findById(id)
.doOnNext(order -> {
if(order.getProduct().getImages() == null){
order.getProduct().setImages(new ArrayList<>());
}
order.getProduct().getImages().add(image);
})
.flatMap(this::create);
}
The code in the doOnNext and flatMap block in method (addImages) does not work. In doing so, calling the method (addImages) from the controller works fine. Tell me please what i'm missing.
I think I have found a solution. I changed the operation from doOnNext to flatMap as per this solution: Spring Webflux Proper Way To Find and Save
public Mono<OrderPostgres> store(String orderId, Flux<FilePart> files) {
return marketService.findById(orderId)
.filter(Objects::nonNull)
.flatMapMany(order -> {
return files.ofType(FilePart.class)
.flatMap(file -> save(orderId, file))
.collectList();
})
.flatMap(image -> marketService.addImages(orderId, image))
.then(Mono.empty());
}
private Flux<String> save(String orderId, FilePart file) {
return file.content()
.log()
.flatMap(dataBuffer -> {
byte[] bytes = new byte[dataBuffer.readableByteCount()];
dataBuffer.read(bytes);
String image = storeApi.upload(bytes, file.filename());
DataBufferUtils.release(dataBuffer);
return Flux.just(image);
}).switchIfEmpty(Flux.empty());
}
public Mono<OrderPostgres> addImages(String id, List<String> images){
log.info("### addImage: id={}, images={}", id, images);
Mono<OrderPostgres> byId = orderRepository.findById(id);
return byId
.flatMap(order -> {
if (order.getImages() == null) {
order.setImages(new ArrayList<>());
}
order.getImages().addAll(images);
return orderRepository.save(order);
}).then(Mono.empty());
}
Related
the example is https://www.vinsguru.com/spring-webflux-file-upload/
#PostMapping("file/multi")
public Mono<Void> upload(#RequestPart("files") Flux<FilePart> partFlux){
return partFlux
.doOnNext(fp -> System.out.println(fp.filename()))
.flatMap(fp -> fp.transferTo(basePath.resolve(fp.filename())))
.then();
}
but i want to return to upload file name
............................
You can use .then(Mono.just(fileName)) or .thenReturn(fileName) to continue the flow and return file name.
#PostMapping("file/multi")
public Mono<String> upload(#RequestPart("files") Flux<FilePart> partFlux){
return partFlux
.flatMap(fp -> {
var fileName = basePath.resolve(fp.filename());
return fp.transferTo(fileName)
.thenReturn(fileName);
});
}
Trying to add some enhancements to this app,
private void parseCsv(CsvMapReader csvMapReader) throws IOException {
String[] header = csvMapReader.getHeader(true);
List<String> headers = Arrays.asList(header);
verifySourceColumn(headers);
verifyPovColumn(headers);
final CellProcessor[] processors = getProcessors(headers);
Map<String, Object> csvImportMap = null;
while ((csvImportMap = csvMapReader.read(header, processors)) != null) {
CsvImportDTO csvImportDto = new CsvImportDTO(csvImportMap);
if ( activationTypeP(csvImportDto) ){
AipRolloutVO aipRolloutVO = new AipRolloutVO(csvImportDto.getSource(),
csvImportDto.getPov(),
csvImportDto.getActivationType(),
csvImportDto.getActivationDate(),
csvImportDto.getDeactivationDate(),
csvImportDto.getMssValue());
aipRolloutRepository.updateAipRollout(aipRolloutVO.getDc(),
aipRolloutVO.getPov(),
aipRolloutVO.getActivationType(),
aipRolloutVO.getMssValue());
}
}
}
When it goes to the repo I get:
cannot find local variable 'csvImportMap'
5 times and then:
((CsvParserService)this).aipRolloutService = inconvertiible types; cannont cast 'org.spring.....
my controller method
#PostMapping(value = "/updatecsv", produces = MediaType.APPLICATION_JSON_VALUE)
#ResponseBody
public ResponseEntity<?> processCsv( #RequestParam("csvFile") MultipartFile csvFile) throws IOException {
if (csvFile.isEmpty()) return new ResponseEntity(
responceJson("please select a file!"),
HttpStatus.NO_CONTENT
);
csvParserService.parseCsvFile(csvFile);
return new ResponseEntity(
responceJson("Successfully uploaded - " + csvFile.getOriginalFilename()),
new HttpHeaders(),
HttpStatus.CREATED
);
}
repo im trying to reuse to update these values
public int updateAipRollout(String dc, String pov, String activationType, int mssValue) {
String query = some query
logger.debug("Rollout update query: " + query);
int num =jdbcTemplate.update(query);
return num;
}
Do I need to autowire the other service class that this repo is in and then call that service? But when I did that it also didn't fix the error....
I'm sending files containing binary data from service A to service B. When the number of files is relatively small (let's say 5) everything works well. However, when I try to send more files (let's say several hundred) it sometimes fails. I tried to check what is happening with this binary data, and it looks like WebClient corrupts it in some way (weird padding appears at the end).
I created a minimal reproducible example to illustrate this issue.
Endpoint in service B (consuming binary files):
#RestController
class FilesController {
#PostMapping(value = "/files")
Mono<List<String>> uploadFiles(#RequestBody Flux<Part> parts) {
return parts
.filter(FilePart.class::isInstance)
.map(FilePart.class::cast)
.flatMap(part -> DataBufferUtils.join(part.content())
.map(buffer -> {
byte[] data = new byte[buffer.readableByteCount()];
buffer.read(data);
DataBufferUtils.release(buffer);
return Base64.getEncoder().encodeToString(data);
})
)
.collectList();
}
}
Tests illustrating how the service A sends data:
public class BinaryUploadTest {
private final CopyOnWriteArrayList<String> sentBytes = new CopyOnWriteArrayList<>();
#BeforeEach
void before() {
sentBytes.clear();
}
/**
* this test passes all the time
*/
#Test
void shouldUpload5Files() {
// given
MultiValueMap<String, HttpEntity<?>> body = buildResources(5);
// when
List<String> receivedBytes = sendPostRequest(body);
// then
assertEquals(sentBytes, receivedBytes);
}
/**
* this test fails most of the time
*/
#Test
void shouldUpload1000Files() {
// given
MultiValueMap<String, HttpEntity<?>> body = buildResources(1000);
// when
List<String> receivedBytes = sendPostRequest(body);
// then
assertEquals(sentBytes, receivedBytes);
}
private List<String> sendPostRequest(MultiValueMap<String, HttpEntity<?>> body) {
return WebClient.builder().build().post()
.uri("http://localhost:8080/files")
.contentType(MediaType.MULTIPART_FORM_DATA)
.body(BodyInserters.fromMultipartData(body))
.retrieve()
.bodyToMono(new ParameterizedTypeReference<List<String>>() {
})
.block();
}
private MultiValueMap<String, HttpEntity<?>> buildResources(int numberOfResources) {
MultipartBodyBuilder builder = new MultipartBodyBuilder();
for (int i = 0; i < numberOfResources; i++) {
builder.part("item-" + i, buildResource(i));
}
return builder.build();
}
private ByteArrayResource buildResource(int index) {
byte[] bytes = randomBytes();
sentBytes.add(Base64.getEncoder().encodeToString(bytes)); // keeps track of what has been sent
return new ByteArrayResource(bytes) {
#Override
public String getFilename() {
return "filename-" + index;
}
};
}
private byte[] randomBytes() {
byte[] bytes = new byte[ThreadLocalRandom.current().nextInt(16, 32)];
ThreadLocalRandom.current().nextBytes(bytes);
return bytes;
}
}
What could be the reason for this data corruption?
It turned out to be a bug in the Spring Framework (in the MultipartParser class to be more precise). I have created a GitHub issue which will be fixed in the next version (5.3.16). The bug is fixed by this commit.
I'm trying to decode a multipart-related request that is just a simple multi files download but with a specific content type by part (application/dicom and not application/octet-stream).
Since the structure of the response body might be identical, I could just tell the "multipart codec" to treat that content type as an octet-stream.
public Flux<FilePart> getDicoms(String seriesUri) {
return webClient.get()
.uri(seriesUri)
.accept(MediaType.ALL)
.retrieve()
.bodyToFlux(FilePart.class);
}
How can I do that?
An easier way of reading a multipart response:
private Mono<ResponseEntity<Flux<Part>>> queryForFiles(String uri)
final var partReader = new DefaultPartHttpMessageReader();
partReader.setStreaming(true);
return WebClient.builder()
.build()
.get()
.uri(wadoUri)
.accept(MediaType.ALL)
.retrieve()
.toEntityFlux((inputMessage, context) -> partReader.read(ResolvableType.forType(DataBuffer.class), inputMessage, Map.of())))
This is what I've done to make it work. I used directly the DefaultPartHttpMessageReader class to do it cleanly (spring 5.3).
public Flux<Part> getDicoms(String wadoUri) {
final var partReader = new DefaultPartHttpMessageReader();
partReader.setStreaming(true);
return WebClient.builder()
.build()
.get()
.uri(wadoUri)
.accept(MediaType.ALL)
//.attributes(clientRegistrationId("keycloak"))
.exchange()
.flatMapMany(clientResponse -> {
var message = new ReactiveHttpInputMessage() {
#Override
public Flux<DataBuffer> getBody() {
return clientResponse.bodyToFlux(DataBuffer.class);
}
#Override
public HttpHeaders getHeaders() {
return clientResponse.headers().asHttpHeaders();
}
};
return partReader.read(ResolvableType.forType(DataBuffer.class), message, Map.of());
});
}
How to upload multiple files using Webflux?
I send request with content type: multipart/form-data and body contains one part which value is a set of files.
To process single file I do it as follow:
Mono<MultiValueMap<String, Part> body = request.body(toMultipartData());
body.flatMap(map -> FilePart part = (FilePart) map.toSingleValueMap().get("file"));
But how to done it for multiple files?
PS. Is there another way to upload a set of files in webflux ?
I already found some solutions.
Let's suppose that we send an http POST request with an parameter files which contains our files.
Note responses are arbitrary
RestController with RequestPart
#PostMapping("/upload")
public Mono<String> process(#RequestPart("files") Flux<FilePart> filePartFlux) {
return filePartFlux.flatMap(it -> it.transferTo(Paths.get("/tmp/" + it.filename())))
.then(Mono.just("OK"));
}
RestController with ModelAttribute
#PostMapping("/upload-model")
public Mono<String> processModel(#ModelAttribute Model model) {
model.getFiles().forEach(it -> it.transferTo(Paths.get("/tmp/" + it.filename())));
return Mono.just("OK");
}
class Model {
private List<FilePart> files;
//getters and setters
}
Functional way with HandlerFunction
public Mono<ServerResponse> upload(ServerRequest request) {
Mono<String> then = request.multipartData().map(it -> it.get("files"))
.flatMapMany(Flux::fromIterable)
.cast(FilePart.class)
.flatMap(it -> it.transferTo(Paths.get("/tmp/" + it.filename())))
.then(Mono.just("OK"));
return ServerResponse.ok().body(then, String.class);
}
You can iterate hashmap with Flux and return Flux
Flux.fromIterable(hashMap.entrySet())
.map(o -> hashmap.get(o));
and it will be send as an array with filepart
the key is use toParts instead of toMultipartData, which is more simpler. Here is the example that works with RouterFunctions.
private Mono<ServerResponse> working2(final ServerRequest request) {
final Flux<Void> voidFlux = request.body(BodyExtractors.toParts())
.cast(FilePart.class)
.flatMap(filePart -> {
final String extension = FilenameUtils.getExtension(filePart.filename());
final String baseName = FilenameUtils.getBaseName(filePart.filename());
final String format = LocalDateTime.now().format(DateTimeFormatter.BASIC_ISO_DATE);
final Path path = Path.of("/tmp", String.format("%s-%s.%s", baseName, format, extension));
return filePart.transferTo(path);
});
return ServerResponse
.ok()
.contentType(APPLICATION_JSON_UTF8)
.body(voidFlux, Void.class);
}
希望对你有帮助
#PostMapping(value = "/upload", consumes = MediaType.MULTIPART_FORM_DATA_VALUE)
public JSON fileUpload(#RequestPart FilePart file)throws Exception{
OSS ossClient = new OSSClientBuilder().build(APPConfig.ENDPOINT, APPConfig.ALI_ACCESSKEYID, APPConfig.ALI_ACCESSSECRET);
File f = null;
String url;
try {
String suffix = file.filename();
String fileName = "images/" + file.filename();
Path path = Files.createTempFile("tempimg", suffix.substring(1, suffix.length()));
file.transferTo(path);
f = path.toFile();
ossClient.putObject(APPConfig.BUCKETNAME, fileName, new FileInputStream(f));
Date expiration = new Date(System.currentTimeMillis() + 3600L * 1000 * 24 * 365 * 10);
url = ossClient.generatePresignedUrl(APPConfig.BUCKETNAME, fileName, expiration).toString();
}finally {
f.delete();
ossClient.shutdown();
}
return JSONUtils.successResposeData(url);
}
Following is the working code for uploading multiple files using WebFlux.
#RequestMapping(value = "upload", method = RequestMethod.POST)
Mono<Object> upload(#RequestBody Flux<Part> parts) {
return parts.log().collectList().map(mparts -> {
return mparts.stream().map(mmp -> {
if (mmp instanceof FilePart) {
FilePart fp = (FilePart) mmp;
fp.transferTo(new File("c:/hello/"+fp.filename()));
} else {
// process the other non file parts
}
return mmp instanceof FilePart ? mmp.name() + ":" + ((FilePart) mmp).filename() : mmp.name();
}).collect(Collectors.joining(",", "[", "]"));
});
};