I've been reading up a lot on Iteratees & Enumerators in order to implement a new module in my application.
I'm now at a point where I'm integrating with a 3rd party Java library, and am stuck at working with this method:
public Email addAttachment(String name, InputStream file) throws IOException {
this.attachments.put(name, file);
return this;
}
What I have in my API is the body returned from a WS HTTP call that is an Enumerator[Array[Byte]].
I am wondering now how to write an Iteratee that would process the chunks of Array[Bytes] and create an InputStream to use in this method.
(Side bar): There are other versions of the addAttachment method that take java.io.File however I want to avoid writing to the disk in this operation, and would rather deal with streams.
I attempted to start by writing something like this:
Iteratee.foreach[Array[Byte]] { bytes =>
???
}
However I'm not sure how to interact with the java InputStream here. I found something called a ByteArrayInputStream however that takes the entire Array[Byte] in its constructor, which I'm not sure would work in this scenario as I'm working with chunks ?
I probably need some Java help here!
Thanks for any help in advance.
If I'm following you, I think you want to work with PipedInputStream and PipedOutputStream:
https://docs.oracle.com/javase/8/docs/api/java/io/PipedInputStream.html
You always use them in pairs. You can construct the pair like so:
PipedInputStream in = new PipedInputStream(); //can also specify a buffer size
PipedOutputStream out = new PipedOutputSream(in);
Pass the input stream to the API, and in your own code iterate through your chucks and write your bytes.
The only caveat is that you need to read/write in separate threads. In your case, its probably good to do your iterating / writing in a separate thread. I'm sure you can handle it in Scala better than me, in Java it would be something like:
PipedInputStream in = new PipedInputStream(); //can also specify a buffer size
PipedOutputStream out = new PipedOutputSream(out);
new Thread(() -> {
// do your looping in here, write to 'out'
out.close();
}).run();
email.addAttachment(in);
email.send();
in.close();
(Leaving out exception handling & resource handling for clarity)
Related
I have to parse xml from the content of a Jetty buffer using SAX.
From my ContentExchange I can call getRequestContent, and then I get a Buffer
I need an InputStream, or an InputSoruce or a String or a File in order to parse it with SAX. How can I convert the buffer to one of those, and which way is the most efficient?
It looks like something obvious, but I can not find any information in the documentation.
Apologies for answering an old question, but someone (such as myself) may stumble upon this in the future.
Jetty's Buffer class implements a writeTo(OutputStream) method. A simple solution would be to do the following:
PipedInputStream is = new PipedInputStream();
PipedOutputStream os = new PipedOutputStream(is);
Then for each Buffer received, do:
void processBuffer(Buffer buf) {
buf.writeTo(os);
}
This way you can stream responses without need for caching them.
EDIT:
Of course, make sure that processBuffer() and readers of the PipedInputStream are running in separate threads to avoid potential deadlock.
Perhaps you could wrap the buffer in your own custom (anonymous?) InputStream since you only need to implement the read() method. For example:
public InputStream forBuffer(final Buffer buf) {
return new InputStream() {
#Override
public int read() /* throws IOException */ {
return buf.get();
}
};
}
From the Jetty docs it's hard to tell what happens when the Buffer#get() method hits the end but some simple testing should reveal it (and if it happens to return -1 then this example is complete!).
I am trying to write a server that accepts files and write it in certain directory using DataInputStream and BufferedInputStream.
The server gets 'user name(string)' 'number of files(int)' 'file name(string)' 'size of each file(long)' and 'contents of file which is uninterpreted bytes(byte[])'
and if everything is successful then, I am supposed to send boolean value.
But the problem is that it is not receiving file correctly.
From time to time I get 'broken pipe' error message or the file is corrupted after I receive.
I looked at my code for 4hrs and couldn't find the problem.
Would you please help me about this?
You can assume that client is working fine.
FIrst you don't have to close all of those streams. That's probably why you're seeing the broken pipe problem. You just need to close the input and output stream.
DataInputStream din = new DataInputStream( new BufferedInputStream( socket.getInputStream() ) );
DataOutputStream dout = new DateOutputStream( new BufferedOutputStream( socket.getOutputStream() );
try {
} finally {
din.close();
dout.close();
}
The reason you don't have to close all of those streams is because your Buffered*Streams and socket InputStream/OutStream will be closed when din/dout.close() is called. Those will close the streams through the reference they chain to. You can also get rid of all that if( blah != null ) junk on each of those because if you make it to the finally clause you know they are non-null. You know that's the case if you don't new up inside the try.
You're also leaking your FileOutputStream because you overwrote the fos variable with the second new FileOutputStream(). What are you doing there with the SUBMIT_DONE file? That's truly weird. It's pretty bad idea to do that. Don't use variable references twice like that. I would probably close the first file after your loop. Think about wrapping that loop with try {} finally { fos.close(); }.
And you might try using methods to break this up a little. Ditch the static.
Update:
What exactly do you think the following is doing?
while(c!='\0') {
userName += c;
c = din.readChar();
}
Depending on how you are sending the data from your client or server you could just use:
String userName = din.readUTF();
Remember with DataInputStream you are processing formatted BINARY data. You also have that exact loop code repeated again for the filenames. If you can't use readUTF() then create a method that wraps up that loop and returns a string and call it from those two places. You have all sorts of security issues allowing clients to upload raw filenames and files to you. I hope to sweet baby jeez this server you're building isn't being deployed in production.
You also need to flush and close each file you receive over the socket so the full amount of data sent is completely written to the files.
I have a file that contains bytes, chars, and an object, all of which need to be written then read. What would be the best way to utilize Java's different IO streams for writing and reading these data types? More specifically, is there a proper way to add delimiters and recognize those delimiters, then triggering what stream should be used? I believe I need some clarification on using multiple streams in the same file, something I have never studied before. A thorough explanation would be a sufficient answer. Thanks!
As EJP already suggested, use ObjectOutputStream and ObjectInputStream an0d wrap your other elements as an object(s). I'm giving as an answer so I could show an example (it's hard to do it in comment) EJP - if you want to embed it in your question, please do and I'll delete the answer.
class MyWrapedData implements serializeable{
private String string1;
private String string2;
private char char1;
// constructors
// getters setters
}
Write to file:
ObjectOutputStream out = new ObjectOutputStream(new FileOutputStream(fileName));
out.writeObject(myWrappedDataInstance);
out.flush();
Read from file
ObjectInputStream in = new ObjectInputStream(new FileInputStream(fileName));
Object obj = in.readObject();
MyWrapedData wraped = null;
if ((obj != null) && (obj instanceof MyWrappedData))
wraped = (MyWrapedData)obj;
// get the specific elements from the wraped object
see very clear example here: Read and Write
Redesign the file. There is no sensible way of implementing it as presently designed. For example the object presupposes an ObjectOutputStream, which has a header - where's that going to go? And how are you going to know where to switch from bytes to chars?
I would probably use an ObjectOutputStream for the whole thing and write everything as objects. Then Serialization solves all those problems for you. After all you don't actually care what's in the file, only how to read and write it.
Can you change the structure of the file? It is unclear because the first sentence of your question contradicts being able to add delineators. If you can change the file structure you could output the different data types into separate files. I would consider this the 'proper' way to delineate the data streams.
If you are stuck with the file the way it is then you will need to write an interface to the file's structure which in practice is a shopping list of read operations and a lot of exception handling. A hackish way to program because it will require a hex editor and a lot of trial and error but it works in certain cases.
Why not write the file as XML, possibly with a nice simple library like XSTream. If you are concerned about space, wrap it in gzip compression.
If you have control over the file format, and it's not an exceptionally large file (i.e. < 1 GiB), have you thought about using Google's Protocol Buffers?
They generate code that parses (and serializes) file/byte[] content. Protocol buffers use a tagging approach on every value that includes (1) field number and (2) a type, so they have nice properties such as forward/backward compatability with optional fields etc. They are fairly well optimized for both speed and file size, adding only ~2 bytes of overhead for a short byte[], with ~2-4 additional bytes to encode the length on larger byte[] fields (VarInt encoded lengths).
This could be overkill, but if you have a bunch of different fields & types, protobuf is really helpful. See: http://code.google.com/p/protobuf/.
An alternative is Thrift by Facebook, with support for a few more languages although possibly less use in the wild last I checked.
If the structure of your file is not fixed, consider using a wrapper per type. First you need to create the interface of your wrapper classes….
interface MyWrapper extends Serializable {
void accept(MyWrapperVisitor visitor);
}
Then you create the MyWrapperVisitor interface…
interface MyWrapperVisitor {
void visit(MyString wrapper);
void visit(MyChar wrapper);
void visit(MyLong wrapper);
void visit(MyCustomObject wrapper);
}
Then you create your wrapper classes…
class MyString implements MyWrapper {
public final String value;
public MyString(String value) {
super();
this.value = value;
}
#Override
public void accept(MyWrapperVisitor visitor) {
visitor.visit(this);
}
}
.
.
.
And finally you read your objects…
final InputStream in = new FileInputStream(myfile);
final ObjectInputStream objIn = new ObjectInputStream(in);
final MyWrapperVisitor visitor = new MyWrapperVisitor() {
#Override
public void visit(MyString wrapper) {
//your logic here
}
.
.
.
};
//loop over all your objects here
final MyWrapper wrapper = (MyWrapper) objIn.readObject();
wrapper.accept(visitor);
I need to load and process a template in freemarker. I am using a piped stream to read back the generated result by freemarker.
Sample code:
PipedInputStream pi = new PipedInputStream();
PipedOutputStream po = new PipedOutputStream(pi);
Writer writer = new OutputStreamWriter(po);
configuration.getTemplate("xx").process(rootMap, writer);
The issue is that sometimes it's freezing inside freemarker procsss method.
No Error, no Exception, but it's not returning back from the process method.
If I convert the piped stream to a ByteArray stream, it works fine.
Am I using piped stream in correct way?
No, piped streams are designed to pass data between two threads. There is only a small buffer between the ends of the pipe. If you write into the piped output stream, your thread will be blocked if the buffer is full until another thread will read from the corresponding piped input stream. This will not work with just one thread.
From the Javadoc:
Typically, data is read from a
PipedInputStream object by one
thread and data is written to the
corresponding PipedOutputStream by
some other thread.
So for small templates just use a StringWriter, for large ones you may use a FileWriter on a temp file created by File.createTempFile().
As Arne writes, the amount of buffer space in a piped stream is fairly small. If you can't use a buffer that can hold all of the data (whether in memory or on disk) then one thing you could try is to see if you can run the template processing in another thread with a piped stream sending the results back to the main thread where you're doing this.
PipedInputStream pi = new PipedInputStream();
final Writer writer = new OutputStreamWriter(new PipedOutputStream(pi));
Thread worker = new Thread(new Runnable() {
public void run() {
configuration.getTemplate("xx").process(rootMap, writer);
}
});
worker.start();
You might need to add final keywords to other variables to make this work in your real code. It depends on whether the variable configuration, the argument to getTemplate or the rootMap variable are local variables or instance (or class) variables.
(I could have subclassed Thread when specifying the thread's behavior of course, but I prefer to instantiate an interface – Runnable in this case – for such things.)
This is how I made it work.
final String uploadReportAsCsv = FreeMarkerTemplateUtils.processTemplateIntoString(
fileUploadReportTemplate, modelMap);
message.addAttachment("fileUploadProcessedReport.csv",
new InputStreamSource() {
//InputStreamResource from Spring is always returning an open stream,
// thus we need to create this anonymous class.
#Override
public InputStream getInputStream() throws IOException {
return new StringInputStream(uploadReportAsCsv);
}
}
);
What I want to do is log the output from an inputstream that I go using
org.apache.http.HttpEntity entity = response.getEntity();
org.apache.http.HttpResponse content =entity.getContent();
//Print the result to the screen for debugging
//puroposes
if(Logging.DEBUG) {
InputStream content =entity.getContent();
int i;
StringBuilder b = new StringBuilder();
while( (i=content.read()) != -1 ) {
b.append((char)i);
}
Log.d(TAG, b.toString());
}
Now after I have finished logging, I want to use the exact same stream through an XML parser. The problem is that it tells me that the steam has already been used.
I tried to the use mark() and reset() calls before and after debugging but it didn't work.
It depends whether the inputstream that is returned supports it. The default implementation in the InputStream class does nothing, as described in the API. So you can't be sure whether the returned Stream actually supports it. To be sure of this, you should wrap it in a BufferedInputStream, which does supports these methods.
In general mark() and reset() won't work on an arbitrary InputStream. They only work on subclasses like FileInputStream where the underlying data source supports these operations.
For something like a SocketInputStream or a console InputStream, your only option will be to read and buffer the entire stream contents somewhere; e.g. in memory or by writing it to a temporary file.