I need to load and process a template in freemarker. I am using a piped stream to read back the generated result by freemarker.
Sample code:
PipedInputStream pi = new PipedInputStream();
PipedOutputStream po = new PipedOutputStream(pi);
Writer writer = new OutputStreamWriter(po);
configuration.getTemplate("xx").process(rootMap, writer);
The issue is that sometimes it's freezing inside freemarker procsss method.
No Error, no Exception, but it's not returning back from the process method.
If I convert the piped stream to a ByteArray stream, it works fine.
Am I using piped stream in correct way?
No, piped streams are designed to pass data between two threads. There is only a small buffer between the ends of the pipe. If you write into the piped output stream, your thread will be blocked if the buffer is full until another thread will read from the corresponding piped input stream. This will not work with just one thread.
From the Javadoc:
Typically, data is read from a
PipedInputStream object by one
thread and data is written to the
corresponding PipedOutputStream by
some other thread.
So for small templates just use a StringWriter, for large ones you may use a FileWriter on a temp file created by File.createTempFile().
As Arne writes, the amount of buffer space in a piped stream is fairly small. If you can't use a buffer that can hold all of the data (whether in memory or on disk) then one thing you could try is to see if you can run the template processing in another thread with a piped stream sending the results back to the main thread where you're doing this.
PipedInputStream pi = new PipedInputStream();
final Writer writer = new OutputStreamWriter(new PipedOutputStream(pi));
Thread worker = new Thread(new Runnable() {
public void run() {
configuration.getTemplate("xx").process(rootMap, writer);
}
});
worker.start();
You might need to add final keywords to other variables to make this work in your real code. It depends on whether the variable configuration, the argument to getTemplate or the rootMap variable are local variables or instance (or class) variables.
(I could have subclassed Thread when specifying the thread's behavior of course, but I prefer to instantiate an interface – Runnable in this case – for such things.)
This is how I made it work.
final String uploadReportAsCsv = FreeMarkerTemplateUtils.processTemplateIntoString(
fileUploadReportTemplate, modelMap);
message.addAttachment("fileUploadProcessedReport.csv",
new InputStreamSource() {
//InputStreamResource from Spring is always returning an open stream,
// thus we need to create this anonymous class.
#Override
public InputStream getInputStream() throws IOException {
return new StringInputStream(uploadReportAsCsv);
}
}
);
Related
I am writing code to asynchronously copy file from one to another while handling backpressure.
I know that there is a convenient class Pump which does the copy job, but, I am doing copy work manually as I want to transform the data before I write it to destination.
To handle backpressure, I am using below code
fileRead.handler(buffer -> {
fileWrite.write(applyTransform(buffer));
//check if reader is out of capacity for taking more data
if(fileWrite.writeQueueFull()){
System.out.println("writing queue full:");
//this pauses file handler from being called
fileRead.pause();
//Once, the writer is ready to accept more data, let it take up more
fileWrite.drainHandler(done -> {
System.out.println("write queue drained, so resuming now...");
//allow file reader handler to resume so, that it accepts incoming data
fileRead.resume();
});
}
});
My question is only regarding fileWrite.writeQueueFull() as the api documentation of WriteStream says
This will return true if there are more bytes in the write queue than
the value set using setWriteQueueMaxSize(int)
The question is how do determine int value for setWriteQueueMaxSize(int)
I've been reading up a lot on Iteratees & Enumerators in order to implement a new module in my application.
I'm now at a point where I'm integrating with a 3rd party Java library, and am stuck at working with this method:
public Email addAttachment(String name, InputStream file) throws IOException {
this.attachments.put(name, file);
return this;
}
What I have in my API is the body returned from a WS HTTP call that is an Enumerator[Array[Byte]].
I am wondering now how to write an Iteratee that would process the chunks of Array[Bytes] and create an InputStream to use in this method.
(Side bar): There are other versions of the addAttachment method that take java.io.File however I want to avoid writing to the disk in this operation, and would rather deal with streams.
I attempted to start by writing something like this:
Iteratee.foreach[Array[Byte]] { bytes =>
???
}
However I'm not sure how to interact with the java InputStream here. I found something called a ByteArrayInputStream however that takes the entire Array[Byte] in its constructor, which I'm not sure would work in this scenario as I'm working with chunks ?
I probably need some Java help here!
Thanks for any help in advance.
If I'm following you, I think you want to work with PipedInputStream and PipedOutputStream:
https://docs.oracle.com/javase/8/docs/api/java/io/PipedInputStream.html
You always use them in pairs. You can construct the pair like so:
PipedInputStream in = new PipedInputStream(); //can also specify a buffer size
PipedOutputStream out = new PipedOutputSream(in);
Pass the input stream to the API, and in your own code iterate through your chucks and write your bytes.
The only caveat is that you need to read/write in separate threads. In your case, its probably good to do your iterating / writing in a separate thread. I'm sure you can handle it in Scala better than me, in Java it would be something like:
PipedInputStream in = new PipedInputStream(); //can also specify a buffer size
PipedOutputStream out = new PipedOutputSream(out);
new Thread(() -> {
// do your looping in here, write to 'out'
out.close();
}).run();
email.addAttachment(in);
email.send();
in.close();
(Leaving out exception handling & resource handling for clarity)
I have a small piece of code.
I am creating an InputStream using the byte array and putting some data on server.
public void putStreamGetBytes() {
try {
String key = getKey();
byte[] data = getTestData(getPayloadSize());
InputStream stream = new ByteArrayInputStream(data);
putStream(assetNWK, key, stream, true, RESPONSE_OK, VERSION_1_1, null, true);
validateBytes(assetNWK, key, data, RESPONSE_OK, VERSION_1_1, null, true);
} catch (Exception e) {
handleError(e);
}
}
I haven't opened any resource like a file or something. Do I need to close the stream to avoid memory leaks?
Sometimes you are not getting the expected output from your stream. So sometimes stream get block. To avoid these blockage you need to flush() the stream then close() it.
Here you are not using file or socket so memory leak would not happen but it's good practice to close stream after use.
To avoid memory leakage. you need to free the stream. to free the stream you need to call the close method in final block.
The only thing you are allocating for is the InputStream itself. Especially in cases of IOExceptions, depending on your usage of the code, this might be problematic anyway, i would suggest to use the try-with-resources syntax, if possible, like:
try(InputStream stream = new ByteArrayInputStream(data)){
......
}
I am trying to connect to a terminal emulator using a library in android, this will connect to a serial device and should show me sent/received data. To attach to a terminal session I need to provide an inputstream to setTermIn(InputStream) and an outputstream to setTermOut(OutputStream).
I initialize and attach some streams like so in onCreate(), these are just initial streams and are not attached to the data I want to be sending/receiving.
private OutputStream bos;
private InputStream bis;
...
byte[] a = new byte[4096];
bis = new ByteArrayInputStream(a);
bos = new ByteArrayOutputStream();
session.setTermIn(bis);
session.setTermOut(bos);
/* Attach the TermSession to the EmulatorView. */
mEmulatorView.attachSession(session);
I now want to assign the streams to data as I send and receive it, but I think I am doing it wrong. In The sendData() method, which I call every time I press enter, I have:
public void sendData(byte[] data)
{
bos = new ByteArrayOutputStream(data.length);
}
and in the onReceiveData() method, called every time data is received over serial:
public void onDataReceived(int id, byte[] data)
{
bis = new ByteArrayInputStream(data);
}
I am not seeing any data on my terminal screen, but I am sending and receiving it over serial successfully. So my question is, should I be setting the streams every single time I am sending and receiving data or just set them once. Also do I need to attach them to the terminal session again mEmulatorView.attachSession(session) somewhere or should the new streams automatically be sent to the screen?
My theory is that my terminal is attached to the old streams and that is why I can't see data on the terminal screen. Would this be correct?
I tried to set the new input/output streams in each method just once using a boolean and an if statement, but then I get warning messages in logcat
RuntimeException 'sending message to a Handler on a dead thread'
I've edited it to write and rad now based on answered but I notice that the library has it's own write method to feed data to the terminal, so I don't even know what the streams are for if that is that case, and I need this write to write to the emulator?
public void write(byte[] data,
int offset,
int count)
Write data to the terminal output. The written data will be consumed by the emulation client as input.
write itself runs on the main thread. The default implementation writes the data into a circular buffer and signals the writer thread to copy it from there to the OutputStream.
Subclasses may override this method to modify the output before writing it to the stream, but implementations in derived classes should call through to this method to do the actual writing.
Parameters:
data - An array of bytes to write to the terminal.
offset - The offset into the array at which the data starts.
count - The number of bytes to be written.
Objects in java are passed by reference, hence if you do
bos = new ByteArrayOutputStream(data.length)
You're essentially throwing away the previous outputstream and creating a new one.
Try keeping reference to your input and output stream and write data into it, eg:
bos.write(data);
I have to parse xml from the content of a Jetty buffer using SAX.
From my ContentExchange I can call getRequestContent, and then I get a Buffer
I need an InputStream, or an InputSoruce or a String or a File in order to parse it with SAX. How can I convert the buffer to one of those, and which way is the most efficient?
It looks like something obvious, but I can not find any information in the documentation.
Apologies for answering an old question, but someone (such as myself) may stumble upon this in the future.
Jetty's Buffer class implements a writeTo(OutputStream) method. A simple solution would be to do the following:
PipedInputStream is = new PipedInputStream();
PipedOutputStream os = new PipedOutputStream(is);
Then for each Buffer received, do:
void processBuffer(Buffer buf) {
buf.writeTo(os);
}
This way you can stream responses without need for caching them.
EDIT:
Of course, make sure that processBuffer() and readers of the PipedInputStream are running in separate threads to avoid potential deadlock.
Perhaps you could wrap the buffer in your own custom (anonymous?) InputStream since you only need to implement the read() method. For example:
public InputStream forBuffer(final Buffer buf) {
return new InputStream() {
#Override
public int read() /* throws IOException */ {
return buf.get();
}
};
}
From the Jetty docs it's hard to tell what happens when the Buffer#get() method hits the end but some simple testing should reveal it (and if it happens to return -1 then this example is complete!).