I currently have an application that inserts byte[] into our DB through the use of Spring JDBC [SqlLobValue]. The problem is, this is not a scalable way to take in data, as the server is buffering all the data in memory before writing to the database. I would like to stream the data from the HttpServletRequest Inputstream, but all the constructors I can find for any classes that take an Inputstream as an argument also require the content length as an argument. I do not, and will not, require the user to know the content length when POSTing data to my application. Is there a way around this limitation?
I can find no documentation about what happens if I pass -1 for content length, but my guess is it will throw an Exception. I'm not sure why they couldn't just have the stream keep reading until the read(...) returns -1, the required behavior of an InputStream.
I presume you meant "InputStream" rather than "OutputStream". I tried this out, but I was having bigger problems with my JDBC driver, so I am unsure if this actually works.
InputStream inputStream = httpServletRequest.getInputStream();
int contentLength = -1; // fake, will be ignored anyway
SqlLobValue sqlLobValue = new SqlLobValue(
inputStream,
contentLength,
new DefaultLobHandler() {
public LobCreator getLobCreator() {
return new DefaultLobHandler.DefaultLobCreator() {
public void setBlobAsBinaryStream(PreparedStatement ps, int paramIndex, InputStream binaryStream, int contentLength) throws SQLException {
// The contentLength parameter should be the -1 we provided earlier.
// You now have direct access to the PreparedStatement.
// Simply avoid calling setBinaryStream(int, InputStream, int)
// in favor of setBinaryStream(int, InputStream).
ps.setBinaryStream(paramIndex, binaryStream);
}
};
}
}
);
jdbcTemplate.update(
"INSERT INTO foo (bar) VALUES (?)",
new Object[]{ sqlLobValue }
);
Related
My Nifi processor implementation is generating a strange cache key and value, something like '\xac\xed\x00\x05t\x00\x06' is prepended to each key/ val.
For example, inserting 'key':'val' generates "\xac\xed\x00\x05t\x00\x03key":"\xac\xed\x00\x05t\x00\x03val".
Nifi uses spring data in the background, issue described here:
Spring boot + Redis - Generating a strange key
Spring boot caching with redis,key have \xac\xed\x00\x05t\x00\x06
Nifi processor is just java code that overrides provided method.
I don't have spring config available to edit or spring data classes, that's all done within Nifi's implementation.
All I have, and have to provide is serializer and deserializer:
void serialize(T value, OutputStream output) throws SerializationException, IOException;
T deserialize(byte[] input) throws DeserializationException, IOException;
For example getAndPutIfAbsent method looks like this:
cache.getAndPutIfAbsent(key, value, keySerializer, valueSerializer, valueDeserializer);
My byte serialization/ deserialization is simple:
public static byte[] serialize(Object obj) throws IOException {
ByteArrayOutputStream out = new ByteArrayOutputStream();
ObjectOutputStream os = new ObjectOutputStream(out);
os.writeObject(obj);
return out.toByteArray();
}
public static Object deserialize(byte[] data) {
if (data == null || data.length == 0) return null;
try {
ByteArrayInputStream in = new ByteArrayInputStream(data);
ObjectInputStream is = new ObjectInputStream(in);
return is.readObject();
} catch (Exception e) {
// log
}
return null;
}
This generates keys/ vals with prepended value. Put and get do work, I can get the value that I previously put using the same serializer and deserializer, except in one case.
I have few caches and they all use same object serializer/ deserializer pasted above. As I mentioned everything works except one cache where I use scan method (paging through all keys). On key deserialization it throws: java invalid stream header: EFBFBDEF. I don't get it why other caches work but this one throws exception, they are all serializing/ deserializing the same.
My goal is to get keys/ vals without prepended chars and working consistently. I can not tinker with spring configs and classes. How to proceed?
Using following de/serialization for strings fixed both issues:
value.getBytes(StandardCharsets.UTF_8)
new String(value, StandardCharsets.UTF_8);
Not sure why, but it works now.
In a Java (only) Play 2.3 project we need to send a non-chunked response of an InputStream directly to the client. The InputStream comes from a remote service from which we want to stream directly to the client, without blocking or buffering to a local file. Since we know the size before reading the input stream, we do not want a chunked response.
What is the best way to return a result for an input stream with a known size? (preferable without using Scala).
When looking at the default ok(file, ..) method for returning File objects it goes deep into play internals which are only accessible from scala, and it uses the play-internal execution context which can't even be accessed from outside. Would be nice if it would work identical, just with an InputStream.
FWIW I have now found a way to serve an InputStream, which basically duplicates the logic which the Results.ok(File) method to allow directly passing in an InputStream.
The key is to use the scala call to create an Enumerator from an InputStream: play.api.libs.iteratee.Enumerator$.MODULE$.fromStream
private final MessageDispatcher fileServeContext = Akka.system().dispatchers().lookup("file-serve-context");
protected void serveInputStream(InputStream inputStream, String fileName, long contentLength) {
response().setHeader(
HttpHeaders.CONTENT_DISPOSITION,
"attachment; filename=\"" + fileName + "\"");
// Set Content-Type header based on file extension.
scala.Option<String> contentType = MimeTypes.forFileName(fileName);
if (contentType.isDefined()) {
response().setHeader(CONTENT_TYPE, contentType.get());
} else {
response().setHeader(CONTENT_TYPE, ContentType.DEFAULT_BINARY.getMimeType());
}
response().setHeader(CONTENT_LENGTH, Long.toString(contentLength));
return new WrappedScalaResult(new play.api.mvc.Result(
new ResponseHeader(StatusCode.OK, toScalaMap(response().getHeaders())),
// Enumerator.fromStream() will also close the input stream once it is done.
play.api.libs.iteratee.Enumerator$.MODULE$.fromStream(
inputStream,
FILE_SERVE_CHUNK_SIZE,
fileServeContext),
play.api.mvc.HttpConnection.KeepAlive()));
}
/**
* A simple Result which wraps a scala result so we can call it from our java controllers.
*/
private static class WrappedScalaResult implements Result {
private play.api.mvc.Result scalaResult;
public WrappedScalaResult(play.api.mvc.Result scalaResult) {
this.scalaResult = scalaResult;
}
#Override
public play.api.mvc.Result toScala() {
return scalaResult;
}
}
Updates:
For now using a Map. Class that wants to send something to other instance sends the object, the routing string.
Use an object stream, use Java serializable to write the object to servlet.
Write String first and then the object.
Receiving servlet wraps input stream around a ObjectInputStream. Reads string first and then the Object. Routing string decides were it goes.
A more generic way might have been to send a class name and its declared method or a Spring bean name, but this was enough for us.
Original question
Know the basic way but want details of steps. Also know I can use Jaxb or RMI or EJB ... but would like to do this using pure serialization to a bytearray and then encode that send it from servlet 1 in jvm 1 to servlet 2 in jvm 2 (two app server instances in same LAN, same java versions and jars set up in both J2EE apps)
Basic steps are (Approcah 1) :-
serialize any Serializable object to a byte array and make a string. Exact code see below
Base64 output of 1. Is it required to base 64 or can skip step 2?
use java.util.URLEncode.encode to encode the string
use apache http components or URL class to send from servlet 1 to 2 after naming params
on Servlet 2 J2EE framework would have already URLDecoced it, now just do reverse steps and cast to object according to param name.
Since both are our apps we would know the param name to type / class mapping. Basically looking for the fastest & most convenient way of sending objects between JVMs.
Example :
POJO class to send
package tst.ser;
import java.io.Serializable;
public class Bean1 implements Serializable {
/**
* make it 2 if add something without default handling
*/
private static final long serialVersionUID = 1L;
private String s;
public String getS() {
return s;
}
public void setS(String s) {
this.s = s;
}
}
* Utility *
package tst.ser;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.ObjectInputStream;
import java.io.ObjectOutputStream;
import java.net.URLEncoder;
public class SerUtl {
public static String serialize(Object o) {
String s = null;
ObjectOutputStream os = null;
try {
os = new ObjectOutputStream(new ByteArrayOutputStream());
os.writeObject(o);
s = BAse64.encode(os.toByeArray());
//s = URLEncoder.encode(s, "UTF-8");//keep this for sending part
} catch (Exception e) {
// TODO: logger
e.printStackTrace();
return null;
} finally {
// close OS but is in RAM
try {
os.close();// not required in RAM
} catch (Exception e2) {// TODO: handle exception logger
}
os = null;
}
return s;
}
public static Object deserialize(String s) {
Object o = null;
ObjectInputStream is = null;
try {
// do base 64 decode if done in serialize
is = new ObjectInputStream(new ByteArrayInputStream(
Base64.decode(s)));
o = is.readObject();
} catch (Exception e) {
// TODO: logger
e.printStackTrace();
return null;
} finally {
// close OS but is in RAM
try {
is.close();// not required in RAM
} catch (Exception e2) {// TODO: handle exception logger
}
is = null;
}
return o;
}
}
**** sample sending servlet ***
Bean1 b = new Bean1(); b.setS("asdd");
String s = SerUtl.serialize(b);
//do UrlEncode.encode here if sending lib does not.
HttpParam p = new HttpParam ("bean1", s);
//http components send obj
**** sample receiving servlet ***
String s = request.getParameter("bean1");
Bean1 b1 = (Beean1)SerUtl.deserialize(s);
Serialize any Serializable object with to a byte array
Yes.
and make a string.
No.
Exact statements see below
os = new ObjectOutputStream(new ByteArrayOutputStream());
os.writeObject(o);
s = os.toString();
// s = Base64.encode(s);//Need this some base 64 impl like Apache ?
s = URLEncoder.encode(s, "UTF-8");
These statements don't even do what you have described, which is in any case incorrect. OutputStream.toString() doesn't turn any bytes into Strings, it just returns a unique object identifier.
Base64 output of 1.
The base64 output should use the byte array as the input, not a String. String is not a container for binary data. See below for corrected code.
ByteArrayOutputStream baos = new ByteArrayOutputStream();
os = new ObjectOutputStream(baos);
os.writeObject(o);
os.close();
s = Base64.encode(baos.toByeArray()); // adjust to suit your API
s = URLEncoder.encode(s, "UTF-8");
This at least accomplishes your objective.
Is it required to base 64 or can skip step 2?
If you want a String you must encode it somehow.
Use java.util.URLEncode.encode to encode the string
This is only necessary if you're sending it as a GET or POST parameter.
Use apache http components or URL class to send from servlet 1 to 2 after naming params
Yes.
On Servlet 2 J2EE framework would have already URLDecoded it, now just do reverse steps and cast to object according to param name.
Yes, but remember to go directly from the base64-encoded string to the byte array, no intermediate String.
Basically looking for the fastest & most convenient way of sending objects between JVMs.
These objectives aren't necessarily reconcilable. The most convenient these days is probably XML or JSON but I doubt that these are faster than Serialization.
os = null;
Setting references that are about to fall out of scope to null is pointless.
HttpParam p = new HttpParam ("bean1", s);
It's possible that HttpParam does the URLEncoding for you. Check this.
You need not convert to string. You can post the binary data straight to the servlet, for example by creating an ObjectOutputStream on top of a HttpUrlConnection's outputstream. Set the request method to POST.
The servlet handling the post can deserialize from an ObjectStream created from the HttpServletRequest's ServletInputStream.
I'd recommend JAXB any time over binary serialization, though. The frameworks are not only great for interoperability, they also speed up development and create more robust solutions.
The advantages I see are way better tooling, type safety, and code generation, keeping your options open so you can call your code from another version or another language, and easier debugging. Don't underestimate the cost of hard to solve bugs caused by accidentally sending the wrong type or doubly escaped data to the servlet. I'd expect the performance benefits to be too small to compensate for this.
Found this Base64 impl that does a lot of the heavy lifting for me : http://iharder.net/base64
Has utility methods :
String encodeObject(java.io.Serializable serializableObject, int options )
Object decodeToObject(String encodedObject, int options, final ClassLoader loader )
Using :
try {
String dat = Base64.encodeObject(srlzblObj, options);
StringBuilder data = new StringBuilder().append("type=");
data.append(appObjTyp).append("&obj=").append(java.net.URLEncoder.encode(dat, "UTF-8"));
Use the type param to tell the receiving JVM what type of object I'm sending. Each servlet/ jsps at most receives 4 types, usually 1. Again since its our own app and classes that we are sending this is quick (as in time to send over the network) and simple.
On the other end unpack it by :
String objData = request.getParameter("obj");
Object obj = Base64.decodeToObject(objData, options, null);
Process it, encode the result, send result back:
reply = Base64.encodeObject(result, options);
out.print("rsp=" + reply);
Calling servlet / jsp gets the result:
if (reply != null && reply.length() > 4) {
String objDataFromServletParam = reply.substring(4);
Object obj = Base64.decodeToObject(objDataFromServletParam, options, null);
options can be 0 or Base64.GZIP
You can use JMS as well.
Apache Active-MQ is one good solution. You will not have to bother with all this conversion.
/**
* #param objectToQueue
* #throws JMSException
*/
public void sendMessage(Serializable objectToQueue) throws JMSException
{
ObjectMessage message = session.createObjectMessage();
message.setObject(objectToQueue);
producerForQueue.send(message);
}
/**
* #param objectToQueue
* #throws JMSException
*/
public Serializable receiveMessage() throws JMSException
{
Message message = consumerForQueue.receive(timeout);
if (message instanceof ObjectMessage)
{
ObjectMessage objMsg = (ObjectMessage) message;
Serializable sobject = objMsg.getObject();
return sobject;
}
return null;
}
My point is do not write custom code for Serialization, iff it can be avoided.
When you use AMQ, all you need to do is make your POJO serializable.
Active-MQ functions take care of serialization.
If you want fast response from AMQ, use vm-transport. It will minimize n/w overhead.
You will automatically get benefits of AMQ features.
I am suggesting this because
You have your own Applications running on network.
You need a mechanism to transfer objects.
You will need a way to monitor it as well.
If you go for custom solution, you might have to solve above things yourselves.
I want to save my file on Oracle Instance. I am using Hibernate for data objects. How can I insert files into Oracle Blob. Is there any sample code for this?
In first you must annotate your entity field with #javax.persistance.Lob annotation.
Like this:
public class InfoMessage {
private byte[] body;
#Lob
public byte[] getBody() {
return body;
}
public void setBody(byte[] body) {
this.body = body;
}
}
and set it with bytes array. It's depends on wich File class you use. The first google result for java.io.File. I guess there's better solution for this operation.
public static byte[] getBytesFromFile(File file) throws IOException {
InputStream is = new FileInputStream(file);
// Get the size of the file
long length = file.length();
if (length > Integer.MAX_VALUE) {
// File is too large
}
// Create the byte array to hold the data
byte[] bytes = new byte[(int)length];
// Read in the bytes
int offset = 0;
int numRead = 0;
while (offset < bytes.length
&& (numRead=is.read(bytes, offset, bytes.length-offset)) >= 0) {
offset += numRead;
}
// Ensure all the bytes have been read in
if (offset < bytes.length) {
throw new IOException("Could not completely read file "+file.getName());
}
// Close the input stream and return bytes
is.close();
return bytes;
}
The #Lob annotation is not Hibernate's one. It's javax.persistence and you can use it in entity bean with any hibernate mapping.
Yes, the big file is obvious problem for such example. But i can't find any workaround for this case. FileInputStream using int type value to represent offset point.
I googled this one with similiar problem: http://www.coderanch.com/t/449055/Streams/java/JAVA-HEAP-SIZE-files-byte You can use solution with SQLPrepareSteatement if you use Java 1.6. Othwerwise you can try to use BufferedReader and somehow convert results to byteArray[] and try to beat another problem: you'll need so much memory as file size is.
EDITED: Another thing: Oracle can append data to it's clob\blob fields using dbms_lob.writeappend() procedure so you can avoid having all file in memory, but will GC perform clean as fast as BufferedReader read from file. And seems it's not a hibernate work to do this... jdbc and PreparedStatements back again.
I'm writing to the browser window using servletResponse.getWriter().write(String).
But how do I clear the text which was written previously by some other similar write call?
The short answer is, you cannot -- once the browser receives the response, there is no way to take it back. (Unless there is some way to abnormally stop a HTTP response to cause the client to reload the page, or something to that extent.)
Probably the last place a response can be "cleared" in a sense, is using the ServletResponse.reset method, which according to the Servlet Specification, will reset the buffer of the servlet's response.
However, this method also seems to have a catch, as it will only work if the buffer has not been committed (i.e. sent to the client) by the ServletOutputStream's flush method.
You cannot. The best thing is to write to a buffer (StringWriter / StringBuilder) and then you can replace the written data any time. Only when you know for sure what is the response you can write the buffer's content to the response.
In the same matter, and reason to write the response this way and not to use some view technology for your output such as JSP, Velocity, FreeMarker, etc.?
If you have an immediate problem that you need to solve quickly, you could work around this design problem by increasing the size of the response buffer - you'll have to read your application server's docs to see if this is possible. However, this solution will not scale as you'll soon run into out-of-memory issues if you site traffic peaks.
No view technology will protect you from this issue. You should design your application to figure out what you're going to show the user before you start writing the response. That means doing all your DB access and business logic ahead of time. This is a common issue I've seen with convoluted system designs that use proxy objects that lazily access the database. E.g. ORM with Entity relationships are bad news if accessed from your view layer! There's not much you can do about an exception that happens 3/4 of the way into a rendered page.
Thinking about it, there might be some way to inject a page redirect via AJAX. Anyone ever heard of a solution like that?
Good luck with re-architecting your design!
I know the post is pretty old, but just thought of sharing my views on this.
I suppose you could actually use a Filter and a ServletResponseWrapper to wrap the response and pass it along the chain.
That is, You can have an output stream in the wrapper class and write to it instead of writing into the original response's output stream... you can clear the wrapper's output stream as and when you please and you can finally write to the original response's output stream when you are done with your processing.
For example,
public class MyResponseWrapper extends HttpServletResponseWrapper {
protected ByteArrayOutputStream baos = null;
protected ServletOutputStream stream = null;
protected PrintWriter writer = null;
protected HttpServletResponse origResponse = null;
public MyResponseWrapper( HttpServletResponse response ) {
super( response );
origResponse = response;
}
public ServletOutputStream getOutputStream()
throws IOException {
if( writer != null ) {
throw new IllegalStateException( "getWriter() has already been " +
"called for this response" );
}
if( stream == null ) {
baos = new ByteArrayOutputStream();
stream = new MyServletStream(baos);
}
return stream;
}
public PrintWriter getWriter()
throws IOException {
if( writer != null ) {
return writer;
}
if( stream != null ) {
throw new IllegalStateException( "getOutputStream() has already " +
"been called for this response" );
}
baos = new ByteArrayOutputStream();
stream = new MyServletStream(baos);
writer = new PrintWriter( stream );
return writer;
}
public void commitToResponse() {
origResponse.getOutputStream().write(baos.toByteArray());
origResponse.flush();
}
private static class MyServletStream extends ServletOutputStream {
ByteArrayOutputStream baos;
MyServletStream(ByteArrayOutputStream baos) {
this.baos = baos;
}
public void write(int param) throws IOException {
baos.write(param);
}
}
//other methods you want to implement
}