In my Java app, I'm looking for a streaming version of URLEncoder.encode(String s, String enc). I'd like to stream a large HTTP post request using the "application/x-www-form-urlencoded" content type. Does such a thing exist either in a library, or an open source project? Or is there an easy way to implement it?
This was an early attempt, but is incorrect because it doesn't handle UTF codepoints larger than one byte:
// Incorrect attempt at creating a URLEncoder OutputStream
private class URLEncoderOutputStream extends FilterOutputStream
{
public URLEncoderOutputStream(OutputStream out)
{
super(out);
}
#Override
public void write(int b) throws IOException
{
String s = new String(new byte[] { (byte)b });
String enc = URLEncoder.encode(s, "UTF-8");
out.write(enc.getBytes("UTF-8"));
}
}
The problem is that OutputStreams don't know anything about characters, only bytes. What you really want is a Writer, e.g.
public class URLEncodedWriter extends FilterWriter {
public void write(int c) {
out.write(URLEncoder.encode((char)c, "UTF-8"));
}
... // Same for 2 other write() methods
}
I think the answer is I shouldn't be trying to do this. According to the HTML Specification:
The content type "application/x-www-form-urlencoded" is inefficient for sending large quantities of binary data or text containing non-ASCII characters. The content type "multipart/form-data" should be used for submitting forms that contain files, non-ASCII data, and binary data.
Most servers will reject HTTP headers that exceed a certain length in any case.
Related
Now that I have some spare time on my hands, I decided to create a Java program to connect my XBee (i.e. zigbee) chips to my new SmartThings hub. I found a nice tutorial on doing this by creating the packets by hand (https://nzfalco.jimdofree.com/electronic-projects/xbee-to-smartthings/). My next task is to create a set of Java routines to create, send, receive, and access the required packets (i.e. a sequence of bytes).
Having done similar in C for other projects, my first thought was to simple create a class with the packet structure and send it. Something like this:
class DeviceAnnounce {
public byte frameId;
public byte addr64[];
public byte addr16[];
public byte capability;
};
Problem is there does not appear to be a way to cast this "structure" to an array of bytes to send to the device.
Next I thought, we have a serialize capability built into the Java runtime. So I added Serializable to the class and used the writeObject() method to convert the instance into a byte stream. Problem here is that writeObject() converts not only your bytes, but includes the definition of the object in the encoding. Works great for reading and writing object to disk, but it's not creating the packet I need to send to the xbee device.
I finally coded it the hard way, explicitly adding a method to my class that creates the byte array.
class DeviceAnnounce {
public DeviceAnnounce(byte frameId, byte[] addr64, byte[] addr16, byte capability) {
super();
this.frameId = frameId;
this.addr64 = addr64;
this.addr16 = addr16;
this.capability = capability;
}
public byte frameId;
public byte addr64[];
public byte addr16[];
public byte capability;
byte[] getBytes() throws IOException {
byte[] data=new byte[12];
data[0]=frameId;
data[1]=addr64[7];
data[2]=addr64[6];
data[3]=addr64[5];
data[4]=addr64[4];
data[5]=addr64[3];
data[6]=addr64[2];
data[7]=addr64[1];
data[8]=addr64[0];
data[9]=addr16[1];
data[10]=addr16[0];
data[11]=capability;
return data;
}
#Override
public String toString() {
return "DeviceAnnounce [frameId=" + frameId + ", addr64=" + HexUtils.prettyHexString(addr64) + ", addr16="
+ HexUtils.prettyHexString(addr16) + ", capability=" + capability + "]";
}
}
It works, but I keep thinking there must be a better way. Now the 64 dollar (or maybe bit) question. Is there a way to convert a POJO into a simple byte stream/array?
To build a block of bytes for transmitting, I recommend using the built-in ByteBuffer, which e.g. has helpers for 16-, 32-, and 64-bit integers in big- or little-endian.
You should then store the values as you use them, e.g.
public byte frameId;
public long addr64;
public short addr16;
public byte capability;
byte[] getBytes() throws IOException {
ByteBuffer buf = ByteBuffer.allocate(12)
.order(ByteOrder.BIG_ENDIAN/*Network Byte Order*/);
buf.put(frameId);
buf.putLong(addr64);
buf.putShort(addr16);
buf.put(capability);
return buf.array(); // or return the ByteBuffer itself
}
I've recently developed my own file parsing class called the BufferedParseStream, and used this to decode PNG images. I've been comparing it's performance against the open source project PNGJ, and have seen that for smaller image sizes, PNGJ can be up to twice as fast as my own implementation. I assume this is associated with the implementation overhead when using the BufferedInputStream, as PNGJ roll their own equivalent instead.
Are there any existing design patterns which guide high performance file parsing, into primitives such as an int, float etc.?
public class BufferedParseStream extends BufferedInputStream {
private final ByteBuffer mByteBuffer;
public BufferedParseStream(final InputStream pInputStream, final int pBufferSize) {
super(pInputStream, pBufferSize);
/* Initialize the ByteBuffer. */
this.mByteBuffer = DataUtils.delegateNative(new byte[8]);
}
private final void buffer(final int pNumBytes) throws IOException {
/* Read the bytes into the ByteStorage. */
this.read(this.getByteBuffer().array(), 0, pNumBytes);
/* Reset the ByteBuffer Location. */
this.getByteBuffer().position(0);
}
public final char parseChar() throws IOException {
/* Read a single byte. */
this.buffer(DataUtils.BYTES_PER_CHAR);
/* Return the corresponding character. */
return this.getByteBuffer().getChar();
}
public final int parseInt() throws IOException {
/* Read four bytes. */
this.buffer(DataUtils.BYTES_PER_INT);
/* Return the corresponding integer. */
return this.getByteBuffer().getInt();
}
public final long parseLong() throws IOException {
/* Read eight bytes. */
this.buffer(DataUtils.BYTES_PER_LONG);
/* Return the corresponding long. */
return this.getByteBuffer().getLong();
}
public final void setParseOrder(final ByteOrder pByteOrder) {
this.getByteBuffer().order(pByteOrder);
}
private final ByteBuffer getByteBuffer() {
return this.mByteBuffer;
}
}
Java nio should be faster than using input streams, your class that you present seems odd to me (might just be me though :)) because it has an extra layer on top of ByteBuffer which I don't think is required.
You should use the byte buffer directly, it has a getInt, getFloat method which you can feed directly in to the required variables.
I think though your performance problems could be in the PNG decoder code as someone else has already mentioned. You should post that for further analysis
Consider a generic byte reader implementing the following simple API to read an unspecified number of bytes from a data structure that is otherwise inaccessible:
public interface ByteReader
{
public byte[] read() throws IOException; // Returns null only at EOF
}
How could the above be efficiently converted to a standard Java InputStream, so that an application using all methods defined by the InputStream class, works as expected?
A simple solution would be subclassing InputStream to
Call the read() method of the ByteReader as much as needed by the read(...) methods of the InputStream
Buffer the bytes retrieved in a byte[] array
Return part of the byte array as expected, e.g., 1 byte at a time whenever the InputStream read() method is called.
However, this requires more work to be efficient (e.g., for avoiding multiple byte array allocations). Also, for the application to scale to large input sizes, reading everything into memory and then processing is not an option.
Any ideas or open source implementations that could be used?
Create multiple ByteArrayInputStream instances around the returned arrays and use them in a stream that provides for concatenation. You could for instance use SequenceInputStream for this.
Trick is to implement a Enumeration<ByteArrayInputStream> that is can use the ByteReader class.
EDIT: I've implemented this answer, but it is probably better to create your own InputStream instance instead. Unfortunately, this solution does not let you handle IOException gracefully.
final Enumeration<ByteArrayInputStream> basEnum = new Enumeration<ByteArrayInputStream>() {
ByteArrayInputStream baos;
boolean ended;
#Override
public boolean hasMoreElements() {
if (ended) {
return false;
}
if (baos == null) {
getNextBA();
if (ended) {
return false;
}
}
return true;
}
#Override
public ByteArrayInputStream nextElement() {
if (ended) {
throw new NoSuchElementException();
}
if (baos.available() != 0) {
return baos;
}
getNextBA();
return baos;
}
private void getNextBA() {
byte[] next;
try {
next = byteReader.read();
} catch (IOException e) {
throw new IllegalStateException("Issues reading byte arrays");
}
if (next == null) {
ended = true;
return;
}
this.baos = new ByteArrayInputStream(next);
}
};
SequenceInputStream sis = new SequenceInputStream(basEnum);
I assume, by your use of "convert", that a replacement is acceptable.
The easiest way to do this is to just use a ByteArrayInputStream, which already provides all the features you are looking for (but must wrap an existing array), or to use any of the other already provided InputStream for reading data from various sources.
It seems like you may be running the risk of reinventing wheels here. If possible, I would consider scrapping your ByteReader interface entirely, and instead going with one of these options:
Replace with ByteInputStream.
Use the various other InputStream classes (depending on the source of the data).
Extend InputStream with your custom implementation.
I'd stick to the existing InputStream class everywhere. I have no idea how your code is structured but you could, for example, add a getInputStream() method to your current data sources, and have them return an appropriate already-existing InputStream (or a custom subclass if necessary).
By the way, I recommend avoiding the term Reader in your own IO classes, as Reader is already heavily used in the Java SDK to indicate stream readers that operate on encoded character data (as opposed to InputStream which generally operates on raw byte data).
i premiss that i am a newbie of netty; i am trying to create a client to an external server, this server outputs messages that terminates with 0x0d so i decide to use the DelimiterBasedFrameDecoder to handle these messages.
This is just a test of the handler:
public class TestHandler extends DelimiterBasedFrameDecoder {
public TestHandler(){
super(200, true, ChannelBuffers.wrappedBuffer(new byte[] { 0x0d }));
}
#Override
protected Object decode(ChannelHandlerContext ctx, Channel ch,
ChannelBuffer cbuf) throws Exception {
ByteBuffer buf = ByteBuffer.allocate(cbuf.readableBytes());
cbuf.readBytes(buf);
byte[] data = buf.array();
for(byte b : data){
System.out.print(b + " ");
}
System.out.println();
...... (some other code)
}
what i see wrong from this is that it doesn't strip the delimiter as i specified in the constructor; at the end of the byte[] data i always have the 0x0d;
So, just as test, i changed the delimiter in the constructor putting on it a test value like 0x55
super(200, true, ChannelBuffers.wrappedBuffer(new byte[] { 0x55 }));
and it works in the same way, there is not difference from before.
I think i am using it in the wrong way, or i am reading the data in the wrong way.
What is the right way to use this class?
To be clear, in the real code from this handler i create an object from the read data and i return this object from the decode() method, then i have another handler that extends SimpleChannelHandler that get this object ( it is similar to the example in the user guide ).
Thanks for the help
Bye
I think your decode method is not using the "stripBytes" part of the constructor properly.
If you check out the code in DelimiterBasedFrameDecoder of Netty, you will see the following if condition in code, which is missing in your overriden decode method. This is causing the bytes not to be stripped.
if (stripDelimiter) {
frame = buffer.readBytes(minFrameLength);
buffer.skipBytes(minDelimLength);
} else {
frame = buffer.readBytes(minFrameLength + minDelimLength);
}
This is a total beginner question, I've spent the past hour searching both stackoverflow and Google, but I haven't found what I'm looking for, hopefully someone here can point me in the right direction.
I'm trying to write a string to an OutputStream, which I will then use to write data to a MySQL database. I've successfully retrieved data from a MySQL (from a .php, implementing JSON and RESTful), so I have some idea of what I'm doing, I think. I'm creating a method which will take a string and return an output stream, and I'm having trouble writing to an output stream, because when I try to initialize one, it creates an anonymous inner class with the write(int oneByte) method. That's not what I want.
private static OutputStream convertStringtoStream(String string) {
byte[] stringByte = string.getBytes();
OutputStream os = new OutputStream() {
#Override
public void write(int oneByte) throws IOException {
/** I'd rather this method be something like
public void write(byte[] bytes), but it requires int oneByte*/
}
};
//return os here
}
As you can see, I want to write to my OutputStream with the buffer, not a single byte. I'm sure this is simple question, but I've not been able to find an answer, or even sample code which does what I want. If someone could point me in the right direction I'd really appreciate it. Thanks.
Your method could look like this, but I'm not sure what it would accomplish. How would you use the returned OutputStream?
private static OutputStream convertStringtoStream(String string) {
byte[] stringByte = string.getBytes();
ByteArrayOutputStream bos = new ByteArrayOutputStream(string.length());
bos.write(stringByte);
return bos;
}
Also, note that using String.getBytes() might get you into trouble in the long run because it uses the system's default encoding. It's better to choose an explicit encoding and use the String.getBytes(Charset) method.
Instead of using the abstract OutputStream class, you might want to use ByteArrayOutputStream which allows you to write a buffer. Even better perhaps would be ObjectOutputStream which would allow you to write string directly since string is serializable. Hope that helps.