I am writing code to asynchronously copy file from one to another while handling backpressure.
I know that there is a convenient class Pump which does the copy job, but, I am doing copy work manually as I want to transform the data before I write it to destination.
To handle backpressure, I am using below code
fileRead.handler(buffer -> {
fileWrite.write(applyTransform(buffer));
//check if reader is out of capacity for taking more data
if(fileWrite.writeQueueFull()){
System.out.println("writing queue full:");
//this pauses file handler from being called
fileRead.pause();
//Once, the writer is ready to accept more data, let it take up more
fileWrite.drainHandler(done -> {
System.out.println("write queue drained, so resuming now...");
//allow file reader handler to resume so, that it accepts incoming data
fileRead.resume();
});
}
});
My question is only regarding fileWrite.writeQueueFull() as the api documentation of WriteStream says
This will return true if there are more bytes in the write queue than
the value set using setWriteQueueMaxSize(int)
The question is how do determine int value for setWriteQueueMaxSize(int)
Related
I'm trying to setup Aalto-xml (Woodstox) in a Async environment, but having troubles figuring out how to do it correctly!
My flow is:
Pre-allocate 'AsyncStreamReader'
xmlInputFactory.createAsyncForByteArray();
Receive XML stream from socket
Feed streamReader with data
streamReader.getInputFeeder().feedInput(bytes, 0, bufSize);
Parse the different XML tags in a loop:
while (streamReader.hasNext()) {...}
Upon Last XMLEvent.END_ELEMENT is parsed --> Invoke:
streamReader.getInputFeeder().endOfInput();
Upon XMLEvent.END_DOCUMENT do:
streamReader = xmlInputFactory.createAsyncForByteArray();
Upon AsyncXMLStreamReader.EVENT_INCOMPLETE --> Break the loop and wait for more data to feed the parser.
I do it like this although I know it's wrong because If I don't call:
'endOfInput()' then when feeding the reader with new buffer it throws an exception like I'm not done with the previous xml.
The point that fails me is that in order to get END_DOCUMENT you must call endOfInput() which closes the buffer so you cannot feed it with more input that comes from the socket...I'm stuck in a loop here !
How can I fix my flow?
Here is a Gist with the parser code:
https://gist.github.com/shvalb/ca9cd526aea31ccf280adf289e0991d7
Wait for end of stream;
* InputStream.read() to returns -1, or
* Stream is closed()) before calling endOfInput()
See this example, it also optionally stops feeding input if the EVENT_INCOMPLETE is not returned.
I've been reading up a lot on Iteratees & Enumerators in order to implement a new module in my application.
I'm now at a point where I'm integrating with a 3rd party Java library, and am stuck at working with this method:
public Email addAttachment(String name, InputStream file) throws IOException {
this.attachments.put(name, file);
return this;
}
What I have in my API is the body returned from a WS HTTP call that is an Enumerator[Array[Byte]].
I am wondering now how to write an Iteratee that would process the chunks of Array[Bytes] and create an InputStream to use in this method.
(Side bar): There are other versions of the addAttachment method that take java.io.File however I want to avoid writing to the disk in this operation, and would rather deal with streams.
I attempted to start by writing something like this:
Iteratee.foreach[Array[Byte]] { bytes =>
???
}
However I'm not sure how to interact with the java InputStream here. I found something called a ByteArrayInputStream however that takes the entire Array[Byte] in its constructor, which I'm not sure would work in this scenario as I'm working with chunks ?
I probably need some Java help here!
Thanks for any help in advance.
If I'm following you, I think you want to work with PipedInputStream and PipedOutputStream:
https://docs.oracle.com/javase/8/docs/api/java/io/PipedInputStream.html
You always use them in pairs. You can construct the pair like so:
PipedInputStream in = new PipedInputStream(); //can also specify a buffer size
PipedOutputStream out = new PipedOutputSream(in);
Pass the input stream to the API, and in your own code iterate through your chucks and write your bytes.
The only caveat is that you need to read/write in separate threads. In your case, its probably good to do your iterating / writing in a separate thread. I'm sure you can handle it in Scala better than me, in Java it would be something like:
PipedInputStream in = new PipedInputStream(); //can also specify a buffer size
PipedOutputStream out = new PipedOutputSream(out);
new Thread(() -> {
// do your looping in here, write to 'out'
out.close();
}).run();
email.addAttachment(in);
email.send();
in.close();
(Leaving out exception handling & resource handling for clarity)
I am trying to use protocol buffer to record a little market data. Each time I get a quote notification from the market, I take this quote and convert it into a protocol buffers object. Then I call "writeDelimitedTo"
Example of my recorder:
try {
writeLock.lock();
LimitOrder serializableQuote = ...
LimitOrderTransport gpbQuoteRaw = serializableQuote.serialize();
LimitOrderTransport gpbQuote = LimitOrderTransport.newBuilder(gpbQuoteRaw).build();
gpbQuote.writeDelimitedTo(fileStream);
csvWriter1.println(gpbQuote.getIdNumber() + DELIMITER+ gpbQuote.getSymbol() + ...);
} finally {
writeLock.unlock();
}
The reason for the locking is because quotes coming from different markets are handled by different threads, so I was trying to simplify and "serialize" the logging to the file.
Code that Reads the resulting file:
FileInputStream stream = new FileInputStream(pathToFile);
PrintWriter writer = new PrintWriter("quoteStream6-compare.csv", "UTF-8");
while(LimitOrderTransport.newBuilder().mergeDelimitedFrom(stream)) {
LimitOrderTransport gpbQuote= LimitOrderTransport.parseDelimitedFrom(stream);
csvWriter2.println(gpbQuote.getIdNumber()+DELIMITER+ gpbQuote.getSymbol() ...);
}
When I run the recorder, I get a binary file that seems to grow in size. When I use my reader to read from the file I also appear to get a large number of quotes. They are all different and appear correct.
Here's the issue: Many of the quotes appear to be "missing" - Not present when my reader reads from the file.
I tried an experiment with csvWriter1 and csvWriter2. In my writer, I write out a csv file then in my reader I write a second cvs file using the my protobufs file as a source.
The theory is that they should match up. They don't match up. The original csv file contains many more quotes in it than the csv that I generate by reading my protobufs recorded data.
What gives? Am I not using writeDelimitedTo/parseDelimitedFrom correctly?
Thanks!
Your problem is here:
while(LimitOrderTransport.newBuilder().mergeDelimitedFrom(stream)) {
LimitOrderTransport gpbQuote= LimitOrderTransport.parseDelimitedFrom(stream);
The first line constructs a new LimitOrderTransport.Builder and uses it to parse a message from the stream. Then that builder is discarded.
The second line parses a new message from the same stream, into a new builder.
So you are discarding every other message.
Do this instead:
while (true) {
LimitOrderTransport gpbQuote = LimitOrderTransport.parseDelimitedFrom(stream);
if (gpbQuote == null) break; // EOF
I am trying to connect to a terminal emulator using a library in android, this will connect to a serial device and should show me sent/received data. To attach to a terminal session I need to provide an inputstream to setTermIn(InputStream) and an outputstream to setTermOut(OutputStream).
I initialize and attach some streams like so in onCreate(), these are just initial streams and are not attached to the data I want to be sending/receiving.
private OutputStream bos;
private InputStream bis;
...
byte[] a = new byte[4096];
bis = new ByteArrayInputStream(a);
bos = new ByteArrayOutputStream();
session.setTermIn(bis);
session.setTermOut(bos);
/* Attach the TermSession to the EmulatorView. */
mEmulatorView.attachSession(session);
I now want to assign the streams to data as I send and receive it, but I think I am doing it wrong. In The sendData() method, which I call every time I press enter, I have:
public void sendData(byte[] data)
{
bos = new ByteArrayOutputStream(data.length);
}
and in the onReceiveData() method, called every time data is received over serial:
public void onDataReceived(int id, byte[] data)
{
bis = new ByteArrayInputStream(data);
}
I am not seeing any data on my terminal screen, but I am sending and receiving it over serial successfully. So my question is, should I be setting the streams every single time I am sending and receiving data or just set them once. Also do I need to attach them to the terminal session again mEmulatorView.attachSession(session) somewhere or should the new streams automatically be sent to the screen?
My theory is that my terminal is attached to the old streams and that is why I can't see data on the terminal screen. Would this be correct?
I tried to set the new input/output streams in each method just once using a boolean and an if statement, but then I get warning messages in logcat
RuntimeException 'sending message to a Handler on a dead thread'
I've edited it to write and rad now based on answered but I notice that the library has it's own write method to feed data to the terminal, so I don't even know what the streams are for if that is that case, and I need this write to write to the emulator?
public void write(byte[] data,
int offset,
int count)
Write data to the terminal output. The written data will be consumed by the emulation client as input.
write itself runs on the main thread. The default implementation writes the data into a circular buffer and signals the writer thread to copy it from there to the OutputStream.
Subclasses may override this method to modify the output before writing it to the stream, but implementations in derived classes should call through to this method to do the actual writing.
Parameters:
data - An array of bytes to write to the terminal.
offset - The offset into the array at which the data starts.
count - The number of bytes to be written.
Objects in java are passed by reference, hence if you do
bos = new ByteArrayOutputStream(data.length)
You're essentially throwing away the previous outputstream and creating a new one.
Try keeping reference to your input and output stream and write data into it, eg:
bos.write(data);
I need to load and process a template in freemarker. I am using a piped stream to read back the generated result by freemarker.
Sample code:
PipedInputStream pi = new PipedInputStream();
PipedOutputStream po = new PipedOutputStream(pi);
Writer writer = new OutputStreamWriter(po);
configuration.getTemplate("xx").process(rootMap, writer);
The issue is that sometimes it's freezing inside freemarker procsss method.
No Error, no Exception, but it's not returning back from the process method.
If I convert the piped stream to a ByteArray stream, it works fine.
Am I using piped stream in correct way?
No, piped streams are designed to pass data between two threads. There is only a small buffer between the ends of the pipe. If you write into the piped output stream, your thread will be blocked if the buffer is full until another thread will read from the corresponding piped input stream. This will not work with just one thread.
From the Javadoc:
Typically, data is read from a
PipedInputStream object by one
thread and data is written to the
corresponding PipedOutputStream by
some other thread.
So for small templates just use a StringWriter, for large ones you may use a FileWriter on a temp file created by File.createTempFile().
As Arne writes, the amount of buffer space in a piped stream is fairly small. If you can't use a buffer that can hold all of the data (whether in memory or on disk) then one thing you could try is to see if you can run the template processing in another thread with a piped stream sending the results back to the main thread where you're doing this.
PipedInputStream pi = new PipedInputStream();
final Writer writer = new OutputStreamWriter(new PipedOutputStream(pi));
Thread worker = new Thread(new Runnable() {
public void run() {
configuration.getTemplate("xx").process(rootMap, writer);
}
});
worker.start();
You might need to add final keywords to other variables to make this work in your real code. It depends on whether the variable configuration, the argument to getTemplate or the rootMap variable are local variables or instance (or class) variables.
(I could have subclassed Thread when specifying the thread's behavior of course, but I prefer to instantiate an interface – Runnable in this case – for such things.)
This is how I made it work.
final String uploadReportAsCsv = FreeMarkerTemplateUtils.processTemplateIntoString(
fileUploadReportTemplate, modelMap);
message.addAttachment("fileUploadProcessedReport.csv",
new InputStreamSource() {
//InputStreamResource from Spring is always returning an open stream,
// thus we need to create this anonymous class.
#Override
public InputStream getInputStream() throws IOException {
return new StringInputStream(uploadReportAsCsv);
}
}
);