Develop Reference

Constructing HTTP Chunks from ByteBuf in Netty 4

I have a Netty 3 codebase with an HTTP Message decoder which extends ReplayingDecoder, and I need to migrate code which analyzes the chunks of the message, which means I need to get the chunks.
protected Object decode(ChannelHandlerContext ctx, Channel channel,
ByteBuf buffer, State state) {
//...
HttpChunk chunk = new DefaultHttpChunk(buffer.readBytes(toRead));
//...
}
From what I've gathered, I need to use HttpChunkedInput instead, but creating one is surprisingly difficult.
//requires an InputStream...
HttpChunkedInput hc = new HttpChunkedInput(new ChunkedStream(...));
//but this seems really clunky/too awkward to be right.
HttpChunkedInput hc = new HttpChunkedInput(new ChunkedStream(new ByteArrayInputStream(buffer.array()));
ByteBuf doesn't seem to have a way to dump out a stream directly. Am I missing a Util class API that would be better? I did find there's a new EmbeddedChannel class which can simply readInbound() like here, but I'm not sure I should be changing the types just for that or casting the bare Channel to an EmbeddedChannel to get out of the problem.

Netty 4.x onwards comes with an out of the box HTTP codec which unless you use an HTTP aggregation handler will give you HTTP chunks as HttpContent objects.
This example shows how to write a handler that received such chunks:
https://github.com/netty/netty/blob/a329857ec20cc1b93ceead6307c6849f93b3f101/example/src/main/java/io/netty/example/http/snoop/HttpSnoopServerHandler.java#L60

RestTemplate: Is there a way to protect jvm agains huge response size?

When using a RestTemplate to talk to an external service, I've seen more than once OutOfMemory errors on our application because the service streams gigs of data (due to a bad implementation on their side, in case of errors they they were sending back big stacktraces in each element of the array, which usually contains few thousand). It was ending in about 6gb of data, serialized by jackson in our app and totally exploding the Xmx of the jvm.
I've looked around but there don't seem to be any way to protect against this kind of even, i.e. aborting the request when the streamed response exceed a given size.
Is there a solution to this? We are using apache's httpcomponents httpclient 4.5.5, but any other underlying implementation would be acceptable.
Besides RestTemplates, a solution for Spring's reactive WebClient would also be welcome.

This has to be enforced at the underlying HTTP client library (spring supports different ones like JDK client, apache client, okHTTP..)
Here you talking about apache-httpcomponent , did you check this HttpEntity.getContent() ?
It actually returns an InputStream that you can read by yourself and determine when the size has been exceeded..
https://hc.apache.org/httpcomponents-core-4.4.x/httpcore/apidocs/org/apache/http/HttpEntity.html

For the records, here is the final solution. The problematic was to load a list of objects that can be very big, using pagination (through elastic search scroll api).
ResponseExtractor<Car[]> responseExtractor = responseEntity -> {
long pageContentLengthInBytes = responseEntity.getHeaders().getContentLength();
long presumableFreeMemoryInBytes = this.getAvailableFreeMemoryAmount();
if (presumableFreeMemoryInBytes - TWENTY_MEGABYTES < pageContentLengthInBytes) {
log.error("Not enough memory to store the page ({} avaiable, content-length={}, trashing it", presumableFreeMemoryInBytes, pageContentLengthInBytes);
responseEntity.close();
return null;
}
return objectMapper.readValue(responseEntity.getBody(), Car[].class);
};
Car[] responseEntities = this.restTemplate.execute(uri, HttpMethod.GET, null, responseExtractor);
/**
* Returns the current amount of memory which may be allocated until an out-of-memory error occurs.
* see https://stackoverflow.com/a/12807848/8836232
*/
private long getAvailableFreeMemoryAmount() {
long allocatedMemory = (Runtime.getRuntime().totalMemory() - Runtime.getRuntime().freeMemory());
return Runtime.getRuntime().maxMemory() - allocatedMemory;
}

Downloading xml data on very slow 2G internet

i have successfully programmed an app that takes Traces of allot of system services (GPS location, Network Location, wifi, neighbouringcellinfo, sensors ....) in every 10 seconds which works very well, but as soon as i restrict the internet on my phone to use only 2G and turn off Wifi i get the app still works but start to lag.
I have tried to find out where the Problem is coming from and i have noticed that it comes from this code line:
XmlPullParser receivedData = XmlPullParserFactory.newInstance()
.newPullParser().setInput(xmlUrl.openStream(), null); return receivedData;
As soon as i delete this couple of code lines in my activity the app works without lagging, but seeing as they are essential for my app i would very much like to have them work (which they already do) but without causing lags.
Can anyone please help me?
I have printed the parsed result from the XML file and it is correct, so my only problem here is the lagging of the app.
A typical XML file that i would be dealing with looks like this:
<rsp stat="ok">
<cell lat="49.88415658974359" lon="8.637537076923078" mcc="262" mnc="7" lac="41146"
cellid="42404" averageSignalStrength="-79" samples="39" changeable="1"/>
</rsp>

2g is some really slow connection. Even worse is the "warm up" of the antenna. It may last up to 30 seconds, before teh first bit is received. (And there is not really something you can do against this, because it is all about physics).
So the only thing you could do is loading the File in an background-Thread. This will make the appp resonsive (if yo don't need the data in time).

Mayb explicitly a BufferedInputStream
XmlPullParser receivedData = XmlPullParserFactory.newInstance()
.newPullParser().setInput(xmlUrl.openStream(), null);
XmlPullParser receivedData = XmlPullParserFactory.newInstance()
.newPullParser().setInput(
new BufferedInputStream(xmlUrl.openStream()), null);
Maybe, maybe compression
As you know in HTTP a browser may declare in its headers that it can decompress compressed data; and then the server may send a compressed version of the HTML. This serves to put less load on the server side, and may speed up communication, depending.
The same one can do oneself.
For an external uncontrolled site one might try. Send a header
Accept-Encoding: gzip
And one is lucky when receiving a response header:
Content-Encoding: gzip
Doing boing sides oneselfby wrapping the streams:
outputStream = new GZipOutputStream(outputStream);
inputStream = new GZipImüputStream(inputStream);
Saving memory
To make the same string instancees unique reduces memory and might help, even if it costs considerable time itself. String.intern() is bad idea, as prior to java 8, the strings go into the permanent (unrecoverable) memory space. One might use a
private Map<String, String> identityMap = new HashMap<>();
public String unique(String s) {
if (s.length() >= 30) {
return s;
}
String t = identityMap.get(s);
if (t == null) {
t = s;
identityMap.put(s, t);
}
return t;
}
The hope is, that processing becomes faster.

A reliable way to read socket data

The application that I am working on has two parts. The server part runs on a Linux machine. The client part, an Android application, queries the server and gets necessary response. Both the parts are written in Java, use socket-based communication, and transfer textual data.
Right after sending the request, here is how the client receives the response:
public static String ReadAvailableTextFromSocket(BufferedReader input) throws IOException {
if (input.ready() == false) {
return null;
}
StringBuilder retVal = new StringBuilder();
while(input.ready()) {
char ch = (char) input.read();
retVal.append(ch);
}
return retVal.toString();
}
However, this doesn't seem to be that reliable. The input is not always ready because of server response time or transmission delays.
Looks like input.ready() is not the right way to wait for getting data.
I am wondering if there is a better way to accomplish this. Perhaps there is some standard practice that I could use.

Perhaps you should use Threads. Keep a listener thread in a while(true) loop that reads more data as it comes in, and simply buffers the data in a data structure (let's say a queue) shared with the main thread. That way, the main thread could simply dequeue data as needed. If the queue is empty, it can infer that no new data was received.
Edit: see this multithreaded chat server/client code as an example.

Here is how I solved this problem. As I am responsible for writing both, the client side as well as the server side, when a request comes to the server, the first line of information I send as the response is the number of bytes the client can expect. This way, the client first waits to read a line. Once the line is read, the client now knows how many bytes of data to expect from the server.
Hope this helps others.
Regards,Peter

ReadableByteChannel hangs on read(bytebuffer)

Im working on Instant messenger using java 1.6. IM uses multithreading - main thread, receiving, and ping. For tcp/ip communication I used SocketChannel. And it seems there is a problem with receiving bigger packages from server. Server instead of one sends a couple of packages and thats where the problem begins. Every first 8 bytes is telling what is the type of package and how big it is. This is how I managed reading:
public void run(){
while(true){
try{
Headbuffer.clear();
bytes = readChannel.read(Headbuffer); //ReadableByteChannel
Headbuffer.flip();
if(bytes != -1){
int head = Headbuffer.getInt();
int size = Headbuffer.getInt();
System.out.println("received pkg: 0x" + Integer.toHexString(head)+" with size "+ size+" bytes);
switch(head){
case incoming.Pkg1: ReadWelcome(); break;
case incoming.Pkg2: ReadLoginFail();break;
case incoming.Pkg3: ReadLoginOk();break;
case incoming.Pkg4: ReadUserList();break;
case incoming.Pkg5: ReadUserData();break;
case incoming.Pkg6: ReadMessage();break;
case incoming.Pkg7: ReadTypingNotify();break;
case incoming.Pkg8: ReadListStatus();break;
case incoming.Pkg9: ChangeStatus();break;
}
}
}catch(Exception e){
e.printStackTrace();
}
}
}
And during the tests everything was fine until i logged on my account and import my buddylist. I send request to server for statuses and he send me back about 10 out of 80 contacts. So I came up with something like this:
public synchronized void readInStatus(ByteBuffer headBuffer){
byteArray.add(headBuffer); //Store every buffer in ArrayList
int buddies = MainController.controler.getContacts().getSize();
while(buddies>0){
readStuff();
readDescription();
--buddies;
}
}
and each readStuff() and readDescription() are checking each parameter size with remaining bytes in the buffer:
if(byteArray.get(current).remaining() >= 4){
uin = byteArray.get(current).getInt();
}else{
byteArray.add(Receiver.receiver.read());
current = current +1;
uin = byteArray.get(current).getInt();
}
and Receiver.receiver.read() is:
public ByteBuffer read(){
try {
ByteBuffer bb = ByteBuffer.allocate(40000);
bb.order(ByteOrder.LITTLE_ENDIAN);
bytes = readChannel.read(bb);
bb.flip();
return bb;
} catch (Exception e) {
e.printStackTrace();
}
return null;
}
So application is lunched, logged and then sends contacts. Server send me back just a piece of my list. But in the method readInStatus(ByteBuffer headBuffer) I try to force the rest of the list. And now the fun part - after some time it gets to the Receiver.receiver.read() and on bytes = readChannel.read(bb) it just stops and I dont know why , no errors no nothing even after some time and Im out of the ideas. Im fighting with this whole week and i dont get anywhere near the solution. I will appreciate any suggestions. Thanks.
Thanks for response. Yes, I'm using blocking SocketChannel, I tried non-blocking but it goes wild and out of control so I skipped the idea. About the bytes I expect - this is kind of weird, because its giving me size only once in head but its size of the first part not the whole package, other parts is not containing header bytes at all. I can't predict how much bytes it would be, the reason is - descriptions with 255 bytes capacity. This is exactly why I've created variable buddies in : public synchronized void readInStatus(ByteBuffer headBuffer)
wich is basically length of my buddy list and before reading each field I'm checking if there is enough bytes left if its not, I do read(). But last field before description is integer with the length of the incoming description. But its impossible to determine how long package is, until some processing is done. #robert do you think I should try again switching to non-blocking SocketChannel in that situation ?

The problem is most likely that you are sending fewer bytes than you are trying to read. You might have missed writing something, written things in the wrong order, misread a size field or something like that.
I think I'd attack this problem by adding tracing code to count and log the number of bytes read and written, notional packect sizes and so on. Then run, and compare the traces to see where things start to get out of sync.

If you are using a blocking SocketChannel, then read will block until the buffer is filled or the server delivers end of stream. For a server with connection keep-alive, the server does not send end of stream - it will simply stop sending data, and the read will hang indefinitely or until timeout.
You could either:
(i) try using a non-blocking SocketChannel, repeatedly reading until the read delivers 0 bytes (but beware 0 bytes does not necessarily mean end of stream - it could mean an interruption) or
(ii) if you have to use the blocking version, and you know how many bytes you were expecting from the server e.g. from a header, when the number of bytes left to read is less than buffer.capacity(), move position and/or limit on the buffer so as to leave only the required space in the buffer before the read. I am working this solution now. If it works for you, please let me know!
So far as I can work out, if you have to use a blocking SocketChannel and you do not know how many bytes you are expecting, and the server does not send end of stream, there is no solution.