Develop Reference

camel REST DSL streaming file

We are trying to retrieve data from a Database and stream the result back to the REST client in a ZipOutputStream. The code is implemented in Camel
The data is quite large. Therefore loading all of it at once would cause memory problems.
We get the data in the sql camel component. Then split the result. Each splitted exchange creates a File stream (InputStream). All files should get zipped into 1 file and returned. How is this possible with Camel?
rest("/api/v1/request/{Id}/data")
.get().outType(StreamingOutput.class)
.route().routeId("getDataRoute")
.to("sql:SELECT * FROM Data WHERE ID_ID=:#id?outputType=StreamList")
.split(body()).parallelProcessing().stopOnException()
.bean(new FileStreamTranslator)
//Generate a filestream per split. Zip the files and return the ZipStream. how can this be achieved?

IOUtils.copyLarge not working for compressed input

I am attempting to transfer a gzipped file using IOUtils.copyLarge. When I transfer from a GZIPInputStream to a non-compressed output, it works fine, but when I transfer the original InputStream (attempting to leave it compressed) the end result is 0 bytes.
I have verified the input file is correct. Here is an example of what works
IOUtils.copyLarge(new GZIPInputStream(inputStream), out)
This of course results in an uncompressed file being written out. I would like to keep the file compressed as it is in the original input.
When I try val countBytes = IOUtils.copyLarge(inputStream, out) the result is 0, and the resulting file is empty. The desired result is simply copying the already compressed gzip file to a new destination maintaining compression.
Reading the documentation for the API, I should be using this correctly. Any ideas on what is preventing it from working?

Consume JSON / Base64 encoded file in Android / Java

I have a web service capable of returning PDF files in two ways:
RAW: The file is simply included in the response body. For example:
HTTP/1.1 200 OK
Content-Type: application/pdf
<file_contents>
JSON: The file is encoded (Base 64) and served as a JSON with the following structure:
HTTP/1.1 200 OK
Content-Type: application/json
{
"base64": <file_contents_base64>
}
I want to be able to consume both services on Android / Java by using the following architecture:
// Get response body input stream (OUT OF THE SCOPE OF THIS QUESTION)
InputStream bodyStream = getResponseBodyInputStream();
// Get PDF file contents input stream from body stream
InputStream fileStream = getPDFFileContentsInputStream(bodyStream);
// Write stream to a local file (OUT OF THE SCOPE OF THIS QUESTION)
saveToFile(fileStream);
For the first case (RAW response), the response body will the file itself. This means that the getPDFFileContentsInputStream(InputStream) method implementation is trivial:
#NonNull InputStream getPDFFileContentsInputStream(#NonNull InputStream bodyStream) {
// Return the input
return bodyStream;
}
The question is: how to implement the getPDFFileContentsInputStream(InputStream) method for the second case (JSON response)?

You can use any json parser (like Jackson or Gson), and then use Base64InputStream from apache-commons codec.
EDIT: You can obtain an input stream from string using ByteArrayInputStream, i.e.
InputStream stream = new ByteArrayInputStream(exampleString.getBytes(StandardCharsets.UTF_8));
as stated here.
EDIT 2: This will cause 2 pass over the data, and if the file is big, you might have memory problems. To solve it, you can use Jackson and parse the content yourself like this example instead of obtaining the whole object through reflection. you can wrap original input stream in another one, say ExtractingInputStream, and this will skip the data in the underlying input stream until the encoded part. Then you can wrap this ExtractingInputStream instance in a Base64InputStream. A simple algorithm to skip unnecessary parts would be like this: In the constructor of ExtractingInputStream, skip until you have read three quotation marks. In read method, return what underlying stream returns except return -1 if the underlying stream returns quotation mark, which corresponds to the end of base 64 encoded data.

Java NIO File Channel writing to file

I am sending file over internet and the receiver receives the position of the file it was sent from and of course the data, I am supposed to write the data received on the file starting from the position sent but it looks like NIO likes to over write any data before the position.
So what I am doing is
fc = new FileOutputStream(new File(file)).getChannel();
then i write on it the buffer starting from pos thats defined before
fc.write(buffer, pos);
When i send file that is splitted for more than one piece and writing the data to file like I showed for some reason the data written before gets overwritten by zeros
I've also tried debugging in such ways as checking that its actually starting to write from the position and it worked correctly and having a delay between transfers so they couldn't write on the file concurrent which shouldn't be problem anyway.
I am also closing the channel
fc.close();
because I am sending the file in pieces I can't keep the fc in memory.
Whats wrong with my code or how can I possibly write in file starting from x position without emptying the data before the position?

The canonical way to copy between channels is as follows:
while (in.read(buffer) > 0 || buffer.hasRemaining())
{
buffer.flip();
out.write(buffer);
buffer.compact();
}
If you're closing and reopening the file you will need to open it in append mode after the first time.

How to convert byte array to file

I have connected to an ftp location using;
URL url = new URL("ftp://user:password#mydomain.com/" + file_name +";type=i");
I read the content into a byte array as shown below;
byte[] buffer = new byte[1024];
int count = 0;
while((count = fis.read(buffer)) > 0)
{
//check if bytes in buffer is a file
}
I want to be able to check if the bytes in buffer is a file without explicitly passing a specific file to write to it like;
File xfile= new File("dir1/");
FileOutputStream fos = new FileOutputStream(xfile);
fos.write(bytes);
if(xfile.isFile())
{
}
In an Ideal world something like this;
File xfile = new File(buffer);//Note: you cannot do this in java
if(xfile.isFile())
{
}
isFile() is to check if the bytes read from the ftp is file. I don't want to pass an explicit file name as I do not know the name of the file on the ftp location.
Any solutions available?

What is a file?
A computer file is a block of arbitrary information [...] which is available to a computer program and is usually based on some kind of durable storage. A file is durable in the sense that it remains available for programs to use after the current program has finished.
Your bytes that are stored in the byte array will be a part of a file if you write them on some kind of durable storage.
Sure, we often say that we read a file or write a file, but basically we read bytes from a file and write bytes to a file.
So we can't test a byte array whether it's content is a file or not. Simply because every byte array can be used to create a file (even an empty array).
BTW - the ftp server does not send a file, it (1) reads bytes and (2) a filename and (3) sends the bytes and (4) the filename so that a client can (5) read the bytes and (6) the filename and use both datasets to (7) create a file. The ftp server doesn't have to access a file, it can take bytes and names from a database or create both in memory...

I guess you cannot check if the byte[] array is a file or not. Why dont' you just use already written and tested library like maybe for example: http://commons.apache.org/net/

There is no way to do that easily.
A file is a byte array on a disk and a byte array will be a file if you write it to disk. There is no reliable way of telling what is in the data you just received, without parsing the data and checking if you can find a valid file header in it.

Where is isFile() file means the content fetched from from the ftp stream is a file.
The answer to that is simple. You can't do it because it doesn't make any sense.
What you have read from the stream IS a sequence of bytes stored in memory.
A file is a sequence of bytes stored on a disk (typically).
These are not the same thing. (Or if you want to get all theoretical / philosophical you have to answer the question "when is a sequence of bytes a file, and when is it not a file".
Now a more sensible question to ask might be:
How do I know if the stuff I fetched by FTP is the contents of a file on the FTP server.
(... as distinct from a rendering of a directory or something).
The answer is that you can't be sure if you fetched the file by opening an URLConnection to the FTP server ... like you have done. It is like asking "is '(123) 555-5555' a phone number?". It could be a phone number, or it could just be a sequence of characters that look like a phone number.