I'm using spring-test with spring-boot in a small Scala application. (Apologies for the long intro/short question!)
So far, everything has worked out fine until I decided to modify one of the endpoints to support streaming. To do this, I added the HttpServletResponse object to my request handler and copy the source data using Apache Commons' IOUtils.copy.
#RequestMapping(value = Array("/hello"), method = Array(RequestMethod.GET))
def retrieveFileForVersion(response:HttpServletResponse) = {
val is = getAnInputStream
val os = response.getOutputStream
try {
IOUtils.copy(is, os)
} finally {
IOUtils.closeQuietly(is)
os.flush()
IOUtils.closeQuietly(os)
}
}
}
This seems to work rather well. I can retrieve binary data from the endpoint and verify that its MD5 checksum matches the source data's MD5 checksum.
However, I noticed this is no longer the case when using the REST controller in spring-test's MockMvc. In fact, when the request is performed through MockMvc, the response is actually four bytes bigger than usual. Thus, some simple assertions fail:
#Test
def testHello() = {
// ... snip ... read the binary file into a byte array
val bytes = IOUtils.toByteArray(...)
val result = mockMvc.perform(get("/hello")).andExpect(status.isOk).andReturn
val responseLength = result.getResponse.getContentAsByteArray.length
// TODO - find out why this test fails!
assert(responseLength == bytes.length, s"Response size: $responseLength, file size: ${bytes.length}")
assert(Arrays.equals(result.getResponse.getContentAsByteArray, bytes))
}
Using the debugger, I was able to determine that MockMvc is appending to the response OutputStream even though it is already closed using IOUtils.closeQuietly. In fact, it is appending the return value of the request handler which is the number of bytes in the OutputStream (from IOUtils.closeQuietly in fact).
Why is MockMvc appending to the OutputStream after it's already closed? Is this a bug, or am I using the library incorrectly?
The return value from a controller method can be interpreted in different ways depending on the return type, the method annotations, and in some cases the input arguments.
This is exhaustively listed on the #RequestMapping annotation and in the reference documentation. For your streaming case, taking the combination of HttpServletResponse as input argument (you could also take OutputStream by the way) and void as return type, indicates to Spring MVC that you're handled the response yourself.
Related
I am trying to upload a Flux object into azure blob storage, but I'm not sure how to send a Flux pojo using BlobAsyncClient. BlobAsyncClient has upload methods that take Flux or BinaryData but I have no luck trying to convert CombinedResponse to BYteBuffer or BinaryData. Does anyone have any suggestions or know how to upload a flux object to blob storage?
You will need an asynch blob container client:
#Bean("blobServiceClient")
BlobContainerAsyncClient blobServiceClient(ClientSecretCredential azureClientCredentials, String storageAccount, String containerName) {
BlobServiceClientBuilder blobServiceClientBuilder = new BlobServiceClientBuilder();
return blobServiceClientBuilder
.endpoint(format("https://%s.blob.core.windows.net/", storageAccount))
.credential(azureClientCredentials)
.buildAsyncClient()
.getBlobContainerAsyncClient(containerName);
}
And in your code you can use it to get a client, and save your Flux to it:
Flux<ByteBuffer> content = getContent();
blobServiceClient.getBlobAsyncClient(id)
.upload(content, new ParallelTransferOptions(), true);
I get that the getContent() step is the part you are struggling with. You can save either a BinaryData object or a Flux<ByteBuffer> stream.
To turn your object into a BinaryData object, use the static helper method:
BinaryData foo = BinaryData.fromObject(myObject);
BinaryData is meant for exactly what the name says: binary data. For example the content of an image file.
If you want to turn it into a ByteBuffer, keep in mind that you're trying to turn an object into a stream of data here. You will probably want to use some standardized way of doing that, so it can be reliably reversed, so rather than a stream of bytes that may break if you ever load the data in a different client, or even just a different version of the same, we usually save a json or xml representation of the object.
My go-to tool for this is Jackson:
byte[] myBytes = new ObjectMapper().writeValueAsBytes(myObject);
var myByteBuffer = ByteBuffer.wrap(myBytes);
And return it as a Flux:
Flux<ByteBuffer> myFlux = Flux.just(myByteBuffer);
By the way, Azure uses a JSON serializer under the hood in the BinaryData.fromObject() method. From the JavaDoc:
Creates an instance of BinaryData by serializing the Object using the default JsonSerializer.
Note: This method first looks for a JsonSerializerProvider
implementation on the classpath. If no implementation is found, a
default Jackson-based implementation will be used to serialize the object
I am reading parts of large file via a Java FileInputStream and would like to stream it's content back to the client (in the form of an akka HttpResponse). I am wondering if this is possible, and how I would do this?
From my research, EntityStreamingSupport can be used but only supports json or csv data. I will be streaming raw data from the file, which will not be in the form of json or csv.
Assuming you use akka-http and Scala you may use getFromFile to stream the entire binary file from a path to the HttpResponse like this:
path("download") {
get {
entity(as[FileHandle]) { fileHandle: FileHandle =>
println(s"Server received download request for: ${fileHandle.fileName}")
getFromFile(new File(fileHandle.absolutePath), MediaTypes.`application/octet-stream`)
}
}
}
Taken from this file upload/download roundtrip akka-http example:
https://github.com/pbernet/akka_streams_tutorial/blob/f246bc061a8f5a1ed9f79cce3f4c52c3c9e1b57a/src/main/scala/akkahttp/HttpFileEcho.scala#L52
Streaming the entire file eliminates the need for "manual chunking", thus the example above will run with limited heap size.
However, if needed manual chunking could be done like this:
val fileInputStream = new FileInputStream(fileHandle.absolutePath)
val chunked: Source[ByteString, Future[IOResult]] = akka.stream.scaladsl.StreamConverters
.fromInputStream(() => fileInputStream, chunkSize = 10 * 1024)
chunked.map(each => println(each)).runWith(Sink.ignore)
I want to create a custom NiFi processor which can read ESRi ASCII grid files and return CSV like representation with some metadata per file and geo-referenced user data in WKT format.
Unfortunately, the parsed result is not written back as an updated flow file.
https://github.com/geoHeil/geomesa-nifi/blob/rasterAsciiGridToWKT/geomesa-nifi-processors/src/main/scala/org/geomesa/nifi/geo/AsciiGrid2WKT.scala#L71-L107 is my try at making this happen in NiFi.
Unfortunately, only the original files are returned. The converted output is not persisted.
When trying to adapt it to manually serialize some CSV strings like:
val lineSep = System.getProperty("line.separator")
val csvResult = result.map(p => p.productIterator.map{
case Some(value) => value
case None => ""
case rest => rest
}.mkString(";")).mkString(lineSep)
var output = session.write(flowFile, new OutputStreamCallback() {
#throws[IOException]
def process(outputStream: OutputStream): Unit = {
IOUtils.write(csvResult, outputStream, "UTF-8")
}
})
still no flowflies are written. Either the issue from above persists or I get Stream not closed exceptions for the outputStream.
It must be a tiny bit which is missing, but I can't seem to find the missing bit.
Each session method that changes flow file like session.write() returns a new version of file and you have to transfer this new version.
If you change your file in converterIngester() function, you have to return this new version to caller function to transfer to relationship.
I have written a service that uses Jersey. In the last class that returns a Response object, I have:
Stream<Revision> revisionss = dataSourceHandler.getRevisions( .... );
StreamingOutput stream = os -> objectHistoryWorker.revisionsTransfer(revisions, os);
return Response.ok(stream).build();
This gets a Stream of Revision objects, converts the stream into a StreamingOutput, and then sends it out in the Response.
I'm trying to write an integration test to test this, and I want to see what contents are actually inside of the Response. In other words, I want to know information such as
How many Revision objects exist
Does a Revision object contain the correct information
The issue I'm having is that it is an OutboundJaxrsResponse, and the readEntity() method is not supported for it.
It has methods that will return whether it passed or not (i.e. status code 200), but I can't seem to figure out a way to actually read the contents of the Response.
Is there a way to get that information?
(The expected response content will be in Json format)
The class org.glassfish.jersey.message.internal.OutboundJaxrsResponse does support getEntity().
How to get the response contents as a Java String. From this, I'm sure you can work out how to read it as Json or XML ....
private String entity( Response response ) throws WebApplicationException, IOException
{
ByteArrayOutputStream baos = new ByteArrayOutputStream();
StreamingOutput output = (StreamingOutput) response.getEntity();
output.write( baos );
return baos.toString( "UTF-8" );
}
I have a binary value being URL Encoded, and then POSTed to an HttpServlet. The following code shows how I first attempted to extract this data. Very simple except that the result is a String, not bytes.
This seemed to work at first, except that an extra byte appeared three bytes from the end. What I eventually figured out was that my data was being treated as Unicode and converted from one Unicode encoding to UTF-8.
So, other that getting the entire post body and parsing it myself, how can I extract my data without treating it as a string after the url encoding is decoded? Have I misunderstood the specs for posted data in general, or is this a Java/Tomcat specific issue?
protected void doPost(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
// Receive/Parse the request
String requestStr = request.getParameter("request");
byte[] rawRequestMsg = requestStr.getBytes();
Here is a snippet of the Python test script I'm using for the request:
urlRequest = urllib.urlencode( {'request': rawRequest} )
connection = urllib.urlopen(self.url, data = urlRequest)
result = connection.readlines()
connection.close()
There are two possible solutions:
ASCII-encode your data before POSTing it. Base64 would be a sensible choice. Decode it in your servlet and you have your original binary again.
Use form content type multipart/form-data ( http://www.w3.org/TR/html401/interact/forms.html#h-17.13.4 ) to encode your binary data as a stream of bytes; then your servlet can do servletRequest.getReader() to read the data in, again as a binary stream.
I think this should work (it treats request as a single-byte encoding, so transformation to String is completely reversible):
String someSingleByteEncoding = "ISO-8859-1";
request.setCharacterEncoding(someSingleByteEncoding);
String requestStr = request.getParameter("request");
byte[] rawRequestMsg = requestStr.getBytes(someSingleByteEncoding);
you can do this with a servlet wrapper (HttpServletRequestWrapper)... catch the request and snatch the request body before its decoded
but the best way is probably to send the data as a file upload (multipart/form-data content type)