In the following snippet, I'm trying to extract a file's content (sent to a given service) using Spring's Part object and convert it into a String.
The issue is that it skips the mapper function and the code inside the mapper function doesn't execute like the filePartMono's content is empty, but when I inspect the object at runtime , its storage field has the file's data.
public void parseFilePart(Part filePartMono) {
filePartMono.content().map(dataBuffer -> {
byte[] bytes = new byte[dataBuffer.readableByteCount()];
dataBuffer.read(bytes);
DataBufferUtils.release(dataBuffer);
String fileContent = new String(bytes, StandardCharsets.UTF_8);
});
}
org.springframework.http.codec.multipart.Part.content() returns a Flux<DataBuffer>, meaning nothing happens until you subscribe to this Publisher.
If your code can be executed in a blocking way without causing errors, you can refactor it like this to get the String result:
public void parseFilePart(Part filePartMono) {
List<String> parts =
filePartMono.content()
.map(dataBuffer -> {
byte[] bytes = new byte[dataBuffer.readableByteCount()];
dataBuffer.read(bytes);
DataBufferUtils.release(dataBuffer);
return new String(bytes, StandardCharsets.UTF_8);
})
.collectList()
.block();
//do what you want here with the Strings you retrieved
}
If you're sure that the Flux<DataBuffer> will always emit 1 single DataBuffer, you can replace .collectList().block() with .blockFirst() and obtain a String result instead of List<String>.
If your code can't be executed in a blocking fashion, then you could refactor it like this:
public void parseFilePart(Part filePartMono) {
filePartMono.content()
.map(dataBuffer -> {
byte[] bytes = new byte[dataBuffer.readableByteCount()];
dataBuffer.read(bytes);
DataBufferUtils.release(dataBuffer);
return new String(bytes, StandardCharsets.UTF_8);
})
.subscribe(resultString -> {
//do what you want with the result String here
});
}
P.S. I didn't test your implementation to convert DataBuffer to String, so you might have to double-check that now that it's actually invoked
Related
I am trying to download a file (or multiple files), based on the result of a previous webrequest. After downloading the file I need to send the previous Mono result (dossier and obj) and the file to another system. So far I have been working with flatMaps and Monos. But when reading large files, I cannot use the Mono during the file download, as the buffer is too small.
Simplified the code looks something like this:
var filePath = Paths.get("test.pdf");
this.dmsService.search()
.flatMap(result -> {
var dossier = result.getObjects().get(0).getProperties();
var objectId = dossier.getReferencedObjectId();
return Mono.zip(this.dmsService.getById(objectId), Mono.just(dossier));
})
.flatMap(tuple -> {
var obj = tuple.getT1();
var dossier = tuple.getT2();
var media = this.dmsService.getDocument(objectId);
var writeMono = DataBufferUtils.write(media, filePath);
return Mono.zip(Mono.just(obj), Mono.just(dossier), writeMono);
})
.flatMap(tuple -> {
var obj = tuple.getT1();
var dossier = tuple.getT2();
var objectId = dossier.getReferencedObjectId();
var zip = zipService.createZip(objectId, obj, dossier);
return zipService.uploadZip(Flux.just(zip));
})
.flatMap(newWorkItemId -> {
return updateMetadata(newWorkItemId);
})
.subscribe(() -> {
finishItem();
});
dmsService.search(), this.dmsService.getById(objectId), zipService.uploadZip() all return Mono of a specific type.
dmsService.getDocument(objectId) returns a Flux due to support for large files. With a DataBuffer Mono it was worked for small files if I simply used a Files.copy:
...
var contentMono = this.dmsService.getDocument(objectId);
return contentMono;
})
.flatMap(content -> {
Files.copy(content.asInputStream(), Path.of("test.pdf"));
...
}
I have tried different approaches but always ran into problems.
Based on https://www.amitph.com/spring-webclient-large-file-download/#Downloading_a_Large_File_with_WebClient
DataBufferUtils.write(dataBuffer, destination).share().block();
When I try this, nothing after .block() is ever executed. No download is made.
Without the .share() I get an exception, that I may not use block:
java.lang.IllegalStateException: block()/blockFirst()/blockLast() are blocking, which is not supported in thread reactor-http-nio-5
Since DataBufferUtils.write returns a Mono my next assumption was, that instead of calling block, I can Mono.zip() this together with my other values, but this never returns either.
var media = this.dmsService.getDocument(objectId);
var writeMono = DataBufferUtils.write(media, filePath);
return Mono.zip(Mono.just(obj), Mono.just(dossier), writeMono);
Any inputs on how to achieve this are greatly appreachiated.
I finally figured out that if I use a WritableByteChannel which returns a Flux<DataBuffer> instead of a Mono<Void> I can map the return value to release the DataBufferUtils, which seems to do the trick. I found the inspiration for this solution here: DataBuffer doesn't write to file
var media = this.dmsService.getDocument(objectId);
var file = Files.createTempFile(objectId, ".tmp");
WritableByteChannel filechannel = Files.newByteChannel(file, StandardOpenOption.WRITE);
var writeMono = DataBufferUtils.write(media, filechannel)
.map(DataBufferUtils::release)
.then(Mono.just(file));
return Mono.zip(Mono.just(obj), Mono.just(dossier), writeMono);
I'm trying to make a live video stream where on the backend chunks are received and stored into a DataBuffer which then is continuously writen to a File. But instead of a bigger getting DataBuffer and file, I rather want a DataBuffer which is limited in its size and (hopefully) works like FIFO, so that if new chunks are pushed into the DataBuffer, older chunks are pushed out.
What I tried is to create a DataBuffer with the DataBufferFactory and allocateBuffer(), but this always gives me type missmatch errors because I have the Flux<DataBuffer> videoDataFlux which contains the chunk data on the one hand, and a <DataBufferFactory> on the other hand, which somehow can't be combined.
So, whats the way to go?
What is working so far:
#Override
public Mono<Void> handle(WebSocketSession webSocketSession) {
String filename = "strm";
Path path = FileSystems.getDefault().getPath("C:\\Program Files (x86)\\Apache Software Foundation\\
Tomcat 9.0\\webapps\\stream\\videos");
Flux<DataBuffer> videoDataFlux = webSocketSession.receive()
.map(WebSocketMessage::getPayload);
try{
Path file = Files.createTempFile(path, filename, ".webm");
AsynchronousFileChannel channel = AsynchronousFileChannel.open(file, StandardOpenOption.WRITE);
return (DataBufferUtils.write(videoDataFlux, channel, 0)
.then()
.doOnNext(s -> {
if((!videoDataFlux.equals(null)) && (!webSocketSession.equals(null))){
DataBufferUtils.write(videoDataFlux, channel, 0).subscribe();
}
}));
} catch(IOException e){
}
return null;
}
Initially I had the following code:
Attempt 1
try (var output = new ByteArrayOutputStream();
var printer = new CSVPrinter(new OutputStreamWriter(output), CSVFormat.DEFAULT)) {
printer.printRecord(EMAIL);
for (MyBean mb : items) {
printer.printRecord(mb.getEmail());
}
externalHttpCall(output.toByteArray());
}
Here I found out that sometimes the byte array is not written fully.
I understand that it is because of the fact that stream is not flushed during externalHttpCall invocations.
To fix it I wrote the following:
Attempt 2
try (var output = new ByteArrayOutputStream();
var printer = new CSVPrinter(new OutputStreamWriter(output), CSVFormat.DEFAULT)) {
printer.printRecord(EMAIL);
for (MyBean mb : items) {
printer.printRecord(mb.getEmail());
}
printer.flush();
log.info("Printer was flushed");
externalHttpCall(output.toByteArray());
}
It solved the problem, but here I was lost in a thought that it is really bad idea to close stream only after externalHttpCall. So I came up with the following solution:
Attempt 3
externalHttpCall(convertToByteArray(items);
public byte[] convertToByteArray(List<MyBean> items){
try (var output = new ByteArrayOutputStream();
var printer = new CSVPrinter(new OutputStreamWriter(output), CSVFormat.DEFAULT)) {
printer.printRecord(EMAIL);
for (MyBean mb : items) {
printer.printRecord(mb.getEmail());
}
return output.toByteArray();
}
}
I expected that flush will happen before stream close. But based on my experiments it doesn't work. Looks like it happens because of flush happens before stream close but after toByteArray invocation.
How could I fix it?
Given the three code snippets in the question I'd assume that this should work:
externalHttpCall(convertToByteArray(items);
public byte[] convertToByteArray(List<MyBean> items){
try (var output = new ByteArrayOutputStream();
var printer = new CSVPrinter(new OutputStreamWriter(output), CSVFormat.DEFAULT)) {
printer.printRecord(EMAIL);
for (MyBean mb : items) {
printer.printRecord(mb.getEmail());
}
printer.flush()
log.info("Printer was flushed");
return output.toByteArray();
}
}
Depending on the CSVFormat the CSVPrinter is flushed automatically on close (CSVFormat.DEFAULT will not be flushed automatically...). You can use CSVFormat's builder like pattern to make the format flush on close with CSVFormat.DEFAULT.withAutoFlush(true) (thanks to #PetrBodnár for this hint). This will however probably make no difference in the above example.
If you translate the try-with-resource to the actual call order you will get something like this:
var output = new ByteArrayOutputStream();
var printer = new CSVPrinter(new OutputStreamWriter(output), CSVFormat.DEFAULT)
printer.printRecord(EMAIL);
...
var result = output.toByteArray();
printer.close(); // might call flush
output.close();
return result;
As the close operations will be called in the finally-block, they will take place after creation of the byte array. If flush is needed, you will need to do it prior to calling toByteArray.
The following is a correct usage:
var output = new ByteArrayOutputStream();
try (var printer = new CSVPrinter(
new OutputStreamWriter(output, StandardCharsets.UTF_8), CSVFormat.DEFAULT)) {
printer.printRecord(EMAIL);
for (MyBean mb : items) {
printer.printRecord(mb.getEmail());
}
}
// Everything flushed and closed.
externalHttpCall(output.toByteArray());
This error behavior might stem from something else.
For instance the externalHttpCall not flushing . Or writing the bytes as text (using a Writer i.o. OutputStream), and expecting UTF-8, whose multi-byte sequences are brittle, maybe raising an exception. Or setting the HTTP header Content-Length wrong, as String.length().
An other cause: items containing a null or getEmail throwing an exception that is not detected.
Available is also:
String s = output.toString(StandardCharsets.UTF_8);
I am developing an API REST using Spring WebFlux, but I have problems when uploading files. They are stored but I don't get the expected return value.
This is what I do:
Receive a Flux<Part>
Cast Part to FilePart.
Save parts with transferTo() (this return a Mono<Void>)
Map the Mono<Void> to Mono<String>, using file name.
Return Flux<String> to client.
I expect file name to be returned, but client gets an empty string.
Controller code
#PostMapping(value = "/muscles/{id}/image")
public Flux<String> updateImage(#PathVariable("id") String id, #RequestBody Flux<Part> file) {
log.info("REST request to update image to Muscle");
return storageService.saveFiles(file);
}
StorageService
public Flux<String> saveFiles(Flux<Part> parts) {
log.info("StorageService.saveFiles({})", parts);
return
parts
.filter(p -> p instanceof FilePart)
.cast(FilePart.class)
.flatMap(file -> saveFile(file));
}
private Mono<String> saveFile(FilePart filePart) {
log.info("StorageService.saveFile({})", filePart);
String filename = DigestUtils.sha256Hex(filePart.filename() + new Date());
Path target = rootLocation.resolve(filename);
try {
Files.deleteIfExists(target);
File file = Files.createFile(target).toFile();
return filePart.transferTo(file)
.map(r -> filename);
} catch (IOException e) {
throw new RuntimeException(e);
}
}
FilePart.transferTo() returns Mono<Void>, which signals when the operation is done - this means the reactive Publisher will only publish an onComplete/onError signal and will never publish a value before that.
This means that the map operation was never executed, because it's only given elements published by the source.
You can return the name of the file and still chain reactive operators, like this:
return part.transferTo(file).thenReturn(part.filename());
It is forbidden to use the block operator within a reactive pipeline and it even throws an exception at runtime as of Reactor 3.2.
Using subscribe as an alternative is not good either, because subscribe will decouple the transferring process from your request processing, making those happen in different execution sequences. This means that your server could be done processing the request and close the HTTP connection while the other part is still trying to read the file part to copy it on disk. This is likely to fail in subtle ways at runtime.
FilePart.transferTo() returns Mono<Void> that is a constant empty. Then, map after that was never executed. I solved it by doing this:
private Mono<String> saveFile(FilePart filePart) {
log.info("StorageService.saveFile({})", filePart);
String filename = DigestUtils.sha256Hex(filePart.filename() + new Date());
Path target = rootLocation.resolve(filename);
try {
Files.deleteIfExists(target);
File file = Files.createFile(target).toFile();
return filePart
.transferTo(file)
.doOnSuccess(data -> log.info("do something..."))
.thenReturn(filename);
} catch (IOException e) {
throw new RuntimeException(e);
}
}
I need to write upload file code using vertx and then save it into PostgreSQL table. but as file is uploaded in multipart and asynchronously I am unable to get byte complete array. Following is my code
public static void uploadLogo(RoutingContext routingContext) {
HttpServerRequest request = routingContext.request();
HttpServerResponse response = routingContext.response();
request.setExpectMultipart(true);
request.uploadHandler(upload -> {
upload.handler(chunk -> {
byte[] fileBytes = chunk.getBytes();
});
upload.endHandler(endHandler -> {
System.out.println("uploaded successfully");
});
upload.exceptionHandler(cause -> {
request.response().setChunked(true).end("Upload failed");
});
});
}
Here I get byte array in fileBytes but only part at a time. I dont understand how to add next byte array to it as it works asynchronously. Is there any way to get byte array of entire file
Hi I am able to extract bytes by using below code :
router.post("/upload").handler(ctx->{
ctx.request().setExpectMultipart(true);
ctx.request().bodyHandler(buffer->{
byte[] bytes=buffer.getBytes();
//transfer bytes to whichever service you want from here
});
ctx.response().end();
});
Request context has .fileUploads() method for that.
See here for the full example: https://github.com/vert-x3/vertx-examples/blob/master/web-examples/src/main/java/io/vertx/example/web/upload/Server.java#L42
If you want to access uploaded files:
Vertx vertx = Vertx.vertx();
Router router = Router.router(vertx);
router.post("/upload").handler(ctx -> {
for (FileUpload fu : ctx.fileUploads()) {
vertx.fileSystem().readFile(fu.uploadedFileName(), fileHandler -> {
// Do something with buffer
});
}
});
to get the uploaded files you have to use fileUploads() method after you can get the byte array.
JsonArray attachments = new JsonArray();
for (FileUpload f : routingContext.fileUploads()) {
Buffer fileUploaded = routingContext.vertx().fileSystem().readFileBlocking(f.uploadedFileName());
attachments.add(new JsonObject().put("body",fileUploaded.getBytes()).put("contentType",f.contentType())
.put("fileName",f.fileName()));
}
You need to build it manually by appending incoming parts in upload.handler into Buffer. Once upload.endHandler is called the upload process has ended, and you can get the resulting buffer and its byte array.
request.uploadHandler(upload -> {
Buffer cache = null;
upload.handler(chunk -> {
if (cache == null) {
cache = chunk;
} else {
cache.appendBuffer(chunk);
}
});
upload.endHandler(end -> {
byte[] result = cache.get().getBytes();
});
});