I'm trying to construct a proxy service with Netty that will support streaming calls and HTTP/1 traffic.
I've managed to do this successfully but ran into problems when I tried adding an HttpObjectAggregator to the pipeline. I need the Aggregator in order to construct a FullHttpResponse for reporting.
My current setup uses 2 ChannelInitializers and 2 business logic handlers
To initiate the proxy service:
ServerBootstrap serverBootstrap = new ServerBootstrap();
serverBootstrap.group(eventLoopGroup)
.channel(NioServerSocketChannel.class)
.handler(new LoggingHandler(LogLevel.INFO))
.childHandler(new SourceTransportHandlerInitializer())
.childOption(ChannelOption.AUTO_READ, false);
Channel channel;
try {
channel = serverBootstrap.bind(localPort).sync().channel();
channel.closeFuture().sync();
}catch (InterruptedException e){
// oh no
}
SourceTransportHandlerInitializer.java
public class SourceTransportHandlerInitializer extends ChannelInitializer<SocketChannel>{
#Override
protected void initChannel(SocketChannel socketChannel) {
ChannelPipeline pipeLine = socketChannel.pipeline();
pipeLine.addLast(new HttpServerCodec(102400,102400,102400));
pipeLine.addLast(new HttpObjectAggregator(1048576));
pipeLine.addLast(new SourceHandler());
pipeLine.addLast(new LoggingHandler(LogLevel.INFO));
}
}
SourceHandler.java
public class SourceHandler extends ChannelInboundHandlerAdapter {
private volatile Channel outboundChannel;
#Override
public void channelActive(ChannelHandlerContext context) {
final Channel inChannel = context.channel();
Bootstrap bootstrap = new Bootstrap();
bootstrap.group(inChannel.eventLoop())
.channel(context.channel().getClass())
.handler(new TargetTransportHandlerInitializer(inChannel))
.option(ChannelOption.AUTO_READ, false);
ChannelFuture channelFuture = bootstrap.connect(Constants.host, Constants.hostPort);
outboundChannel = channelFuture.channel();
channelFuture.addListener((ChannelFutureListener) channelFuture1 -> {
if (channelFuture1.isSuccess()) {
inChannel.read();
} else {
inChannel.close();
}
});
}
#Override
public void channelRead(final ChannelHandlerContext ctx, Object msg) {
if (msg instanceof FullHttpRequest) {
//record request
}
if (outboundChannel.isActive()) {
outboundChannel.writeAndFlush(msg).addListener((ChannelFutureListener) channelFuture -> {
if (channelFuture.isSuccess()) {
// flush and read data
ctx.channel().read();
} else {
channelFuture.channel().close();
}
});
} else {
LOG.debug("SourceHandler did not read. Outbound Channel not active");
}
}
...
}
TargetTransportHandlerInitializer.java
public class TargetTransportHandlerInitializer extends ChannelInitializer<SocketChannel>{
protected final Channel inChannel;
public TargetTransportHandlerInitializer (final Channel inChannel){
this.inChannel = inChannel;
}
#Override
protected void initChannel(SocketChannel socketChannel) {
ChannelPipeline pipeLine = socketChannel.pipeline();
pipeLine.addLast("codec", new HttpClientCodec(102400, 102400, 102400));
//pipeLine.addLast(new HttpObjectAggregator(1048576));
pipeLine.addLast(new TargetHandler(inChannel));
}
}
TargetHandler.java
public class TargetHandler extends ChannelInboundHandlerAdapter {
private final Channel inChannel;
public TargetHandler(Channel inChannel) {
this.inChannel = inChannel;
}
#Override
public void channelActive(ChannelHandlerContext ctx) {
ctx.read();
}
#Override
public void channelRead(final ChannelHandlerContext ctx, Object msg) {
if (msg instanceof FullHttpResponse ) {
//record response
} else if (msg instanceof DefaultHttpResponse) {
// convert to FullHttpResponse ?
}
inChannel.writeAndFlush(msg).addListener((ChannelFutureListener) future -> {
if (future.isSuccess()) {
ctx.channel().read();
} else {
future.channel().close();
}
});
}
...
}
The HttpObjectAggregator in the source initializer pipeline causes no problems, and allows me to record the FullHttpRequest. If I comment pipeLine.addLast(new HttpObjectAggregator(1048576)); in the TargetTransportHandlerInitializer class the proxy works perfect. If I remove that comment the streaming calls begin to fail. Without the aggregator the msg object is only a DefaultHttpResponse for HTTP/1 calls and I can't access the body and headers.
Do I need some kind of conditional pipeline where streaming avoids the aggregator? Or is there some alternative way to construct a FullHttpResponse from a DefaultHttpResponse?
I was really hoping I could just do this FullHttpResponse full = HttpObjectAggregator.aggregate(default) and call it a day.
Netty's HTTP codec works in two distinct ways:
Aggregated: Use the object aggregator as you mention and it produces FullHttpRequest/FullHttpResponse objects.
Non-aggregated (streaming): It produces distinct messages for headers, payload and trailers, which are DefaultHttpRequest/Response, HttpContent and LastHttpContent.
The idiomatic netty way of converting messages is to add a handler in the pipeline, so you would not have APIs like FullHttpResponse full = HttpObjectAggregator.aggregate(default) but instead you will insert a handler after the HttpObjectAggregator and receive the FullHttpResponse object.
The above is an either/or choice, so if you add an aggregator, handlers after the aggregator will only see aggregated messages whereas before the aggregator, they will see the distinct messages as I mention above.
The following example demonstrates of how to handle streaming messages:
https://github.com/netty/netty/blob/4.1/example/src/main/java/io/netty/example/http/snoop/HttpSnoopClientHandler.java#L30
I was able to accomplish this by making a slight modification to the PortUnification example in the Netty project.
I updated the TargetTransportHandlerInitializer with the following:
#Override
protected void initChannel(SocketChannel socketChannel) {
ChannelPipeline pipeLine = socketChannel.pipeline();
pipeLine.addLast("codec", new HttpClientCodec(102400, 102400, 102400));
pipeLine.addLast("switch", new SwitchHandler(inChannel));
}
SwitchHandler is a ChannelInboundHandlerAdapter with the following:
#Override
public void channelRead(final ChannelHandlerContext ctx, Object msg) {
if (recordable(msg))
createRecordingChannel(ctx);
else
createStreamingChannel(ctx);
ctx.fireChannelRead(msg);
}
private void createRecordingChannel(ChannelHandlerContext ctx) {
ChannelPipeline p = ctx.pipeline();
p.addLast("aggregator", new HttpObjectAggregator(1048576));
p.addLast("recordingHandler", new RecordingTargetHandler(inChannel));
p.remove(this);
}
private void createStreamingChannel(ChannelHandlerContext ctx) {
ChannelPipeline p = ctx.pipeline();
p.addLast("simpleHandler", new SimpleTargetHandler(inChannel));
p.remove(this);
}
recordable() contains some business logic to look at the msg and make a determination. In my case was was looking at Headers.
My two new handlers look like the original TargetHandler, with different channelRead methods.
SimpleTargetHandler extends ChannelInboundHandlerAdapter with
#Override
public void channelRead(final ChannelHandlerContext ctx, Object msg) {
inChannel.writeAndFlush(msg).addListener((ChannelFutureListener) future -> {
if (future.isSuccess()) {
ctx.channel().read();
} else {
future.channel().close();
}
});
}
RecordingTargetHandler extends SimpleTargetHandler with:
#Override
public void channelRead(final ChannelHandlerContext ctx, Object msg) {
if (msg instanceof FullHttpResponse ) {
// record response
}
super.channelRead(ctx, msg);
}
Related
The Netty Server can received info from DontNetyy Client
But,DontNetty Client can not received info form Netty Server
They are use protobuf serialize object
c# client
var bootstrap = new Bootstrap();
bootstrap
.Group(group)
.Channel<TcpSocketChannel>()
.Option(ChannelOption.TcpNodelay, true)
.Option(ChannelOption.SoKeepalive,true)
.Handler(new ActionChannelInitializer<ISocketChannel>(channel =>
{
IChannelPipeline pipeline = channel.Pipeline;
pipeline.AddLast(new ProtobufVarint32LengthFieldPrepender());
pipeline.AddLast(new ProtobufVarint32FrameDecoder());
pipeline.AddLast(new EchoClientHandler());
}));
IChannel clientChannel = await bootstrap.ConnectAsync(new IPEndPoint(ClientSettings.Host, ClientSettings.Port));
java server
bootstrap.childHandler(new ChannelInitializer<SocketChannel>() {
#Override
protected void initChannel(SocketChannel socketChannel) throws Exception {
ChannelPipeline p = socketChannel.pipeline();
p.addLast(new ProtobufVarint32FrameDecoder());
p.addLast(MessageCodec.PROTO_DECODER, new ProtobufDecoder(BaseMsgOuterClass.BaseMsg.getDefaultInstance()));
p.addLast(new ProtobufVarint32LengthFieldPrepender());
p.addLast(new ProtobufEncoder());
p.addLast(serverHandler);
}
});
Server send info to client.
but the c# client read0 metho is not called.
public class EchoClientHandler : SimpleChannelInboundHandler<BaseMsgC>
{
protected override void ChannelRead0(IChannelHandlerContext ctx, BaseMsgC msg)
{
Console.Write("ChannelRead0 is not called ");
Console.Write(msg.Msg);
Console.Write(msg.MsgType);
throw new NotImplementedException();
}
public override void ChannelRead(IChannelHandlerContext context, object message)
{
Console.Write("ChannelRead is called");
}
}
}
I haven't worked with DotNetty but my guess is that you add a SimpleChannelInboundHandler that only reads BaseMsgC and you aren't decoding your IByteBuffer into a BaseMsgC. There are two ways to solve that problem.
Change the generic in your SimpleChannelInboundHandler to IByteBuffer or
Change your handler pipeline so it's able to decode the IByteBuffer to a BaseMsgC
PS. Maybe this is helpfull DotNetty.Codecs.Protobuf
I have an hour of experience using RxJava and I am trying to implement it in my project instead of using interfaces and listeners.
I have an async task which calls a google cloud endpoint method in a separate module and receives a List<Profile> when done.
In the onPostExecute() method of the async task, I call onNext so that any subscribers receive this data.
Here is what the AsyncTask looks like:
private BirthpayApi mApi;
private String mUserId;
private ReplaySubject<List<Profile>> notifier = ReplaySubject.create();
public GetFriends(String userId) {
mUserId = userId;
}
public Observable<List<Profile>> asObservable() {
return notifier;
}
#Override
protected List<Profile> doInBackground(Void... params) {
if (mApi == null) {
BirthpayApi.Builder builder = new BirthpayApi.Builder(AndroidHttp.newCompatibleTransport(),
new AndroidJsonFactory(), null)
// options for running against local devappserver
// - 10.0.2.2 is localhost's IP address in Android emulator
// - turn off compression when running against local devappserver
.setRootUrl("http://10.0.2.2:8080/_ah/api/")
.setGoogleClientRequestInitializer(new GoogleClientRequestInitializer() {
#Override
public void initialize(AbstractGoogleClientRequest<?> abstractGoogleClientRequest) throws IOException {
abstractGoogleClientRequest.setDisableGZipContent(true);
}
});
mApi = builder.build();
}
try {
return mApi.getFriends(mUserId).execute().getItems();
} catch (IOException e) {
e.printStackTrace();
return null;
}
}
#Override
protected void onPostExecute(List<Profile> friends) {
super.onPostExecute(friends);
notifier.onNext(friends);
}
In my Fragment I then want to collect this data from the async task calling the onNext() method. I therefore use implements Action1<List<Profile>> when declaring my class which also extends Fragment.
In the onCall() method that comes from the Action1 interface I collect the data sent from the Async task:
#Override
public void call(List<Profile> profiles) {
if (profiles.size() > 0) {
updateAdapter(profiles);
} else
setUpNoFriendsViews();
}
I am following along with treehouse but they use a object to model their data which becomes the observable instead of using an async class and they use an adapter as the observer. Am I doing this wrong, either way how do I get it to work?
It doesn't look like you're subscribing to the Observable you're creating anywhere and it's not clear where you're calling execute on the AsyncTask. Also I don't think you'd want a ReplaySubject but that depends on what you're trying to achieve.
All that aside I'd suggest completely switching over to Rx rather than mixing up Rx and AsyncTasks. In this case in your model class make a method something like:
public Observable<List<Profile>> getProfiles() {
return Observable.defer(new Func0<Observable<List<Profile>>>() {
#Override
public Observable<List<Profile>> call() {
if (mApi == null) {
BirthpayApi.Builder builder = new BirthpayApi.Builder(AndroidHttp.newCompatibleTransport(),
new AndroidJsonFactory(), null)
// options for running against local devappserver
// - 10.0.2.2 is localhost's IP address in Android emulator
// - turn off compression when running against local devappserver
.setRootUrl("http://10.0.2.2:8080/_ah/api/")
.setGoogleClientRequestInitializer(new GoogleClientRequestInitializer() {
#Override
public void initialize(AbstractGoogleClientRequest<?> abstractGoogleClientRequest) throws IOException {
abstractGoogleClientRequest.setDisableGZipContent(true);
}
});
mApi = builder.build();
}
try {
List<Profile> profiles = mApi.getFriends(mUserId).execute().getItems();
return Observable.just(profiles);
} catch (IOException e) {
return Observable.error(e);
}
}
});
}
Then in your Fragment:
modal.getProfiles()
.subscribeOn(Schedulers.io())
.observeOn(AndroidSchedulers.mainThread())
.subscribe(
new Action1<List<Profile>>() {
//...
},
new Action1<Throwable>() {
#Override
public void call(Throwable throwable) {
throwable.printStackTrace();
}
}
);
I'm using netty channel pool for a http client and in the ChannelPoolHandler implementation channelAcquired is not getting called when the channelPool.acquire() invoked. I'm using netty 4.0.32.Final. Here's how I created the chanelpool. I just followed the simple example listed at netty.io. If someone can just explain what I've done wrong or if there is a bug that'll be very helpful. Thanks.
EventLoopGroup group = new NioEventLoopGroup();
final Bootstrap b = new Bootstrap();
b.group(group).channel(NioSocketChannel.class);
AbstractChannelPoolMap<InetSocketAddress, SimpleChannelPool> poolMap = new AbstractChannelPoolMap<InetSocketAddress, SimpleChannelPool>() {
#Override
protected SimpleChannelPool newPool(InetSocketAddress key) {
return new SimpleChannelPool(b.remoteAddress(key), new HttpClientPoolHandler());
}
};
final SimpleChannelPool simpleChannelPool = poolMap.get(new InetSocketAddress(uri.getHost(), uri.getPort()));
final Future<Channel> acquire = simpleChannelPool.acquire();
acquire.addListener(new FutureListener<Channel>() {
public void operationComplete(Future<Channel> f) throws Exception {
if (f.isSuccess()) {
final Channel ch = f.getNow();
// Send the HTTP request.
ChannelFuture channelFuture = ch.writeAndFlush(request);
channelFuture.addListener(new ChannelFutureListener() {
public void operationComplete(ChannelFuture channelFuture) throws Exception {
if (channelFuture.isSuccess()) {
simpleChannelPool.release(ch);
} else {
}
}
});
} else {
System.out.println("ERROR : " + f.cause());
}
}
});
The channelAcquiredmethod will only be called if you "acquire" a previous created channel. In your case there is not channel yet in the pool so it will call channelCreated.
I am using netty for developing my server.
I am also implementing the Idle state handling in netty.
I got it working but an issue I recently found out.
I can't access the channel context attributes inside the userEventTriggered method.
here is my code and can anybody tell me why it is not possible.
I am setting it like
public static final AttributeKey<Agent> CLIENT_MAPPING = AttributeKey.valueOf("clientMapping");
...
ctx.attr(CLIENT_MAPPING).set(agent);
and inside handler, I am getting the value like (this is working perfectly)
Agent agent = ctx.attr(CLIENT_MAPPING).get();
But inside userEventTriggered it is returning null. (I am sure that it is set before this function is being called.)
public class Server
{
...
public void run() throws Exception
{
...
ServerBootstrap b = new ServerBootstrap();
b.group(bossGroup, workerGroup).
channel(NioServerSocketChannel.class).
childHandler(new SslServerInitializer());
...
}
}
class SslServerInitializer extends ChannelInitializer<SocketChannel>
{
#Override
public void initChannel(SocketChannel ch) throws Exception
{
ChannelPipeline pipeline = ch.pipeline();
....
pipeline.addLast("idleStateHandler", new IdleStateHandler(0, 0, Integer.parseInt(Main.configurations.get("netty.idleTimeKeepAlive.ms"))));
pipeline.addLast("idleTimeHandler", new ShelloidIdleTimeHandler());
}
}
class ShelloidIdleTimeHandler extends ChannelDuplexHandler
{
#Override
public void userEventTriggered(ChannelHandlerContext ctx, Object evt) throws Exception
{
if (evt instanceof IdleStateEvent)
{
try
{
// This I am getting null, but I confirmed that I set the attribute from my handler and is accessible inside handler.
Agent agt = ctx.attr(WebSocketSslServerHandler.CLIENT_MAPPING).get();
ctx.channel().writeAndFlush(new TextWebSocketFrame("{\"type\":\"PING\", \"userId\": \"" + agt.getUserId() + "\"}"));
}
catch (Exception ex)
{
ctx.disconnect();
ex.printStackTrace();
}
}
}
}
Are you sure you set and get it in the same ChannelHandler? If you want to set and get it in different ChannelHandler you need to use Channel.attr(...)
I am using Netty 4 RC1. I initialize my pipeline at the client side:
public class NodeClientInitializer extends ChannelInitializer<SocketChannel> {
#Override
protected void initChannel(SocketChannel sc) throws Exception {
// Frame encoding and decoding
sc.pipeline()
.addLast("logger", new LoggingHandler(LogLevel.DEBUG))
// Business logic
.addLast("handler", new NodeClientHandler());
}
}
NodeClientHandler has the following relevant code:
public class NodeClientHandler extends ChannelInboundByteHandlerAdapter {
private void sendInitialInformation(ChannelHandlerContext c) {
c.write(0x05);
}
#Override
public void channelActive(ChannelHandlerContext c) throws Exception {
sendInitialInformation(c);
}
}
I connect to the server using:
public void connect(final InetSocketAddress addr) {
Bootstrap bootstrap = new Bootstrap();
ChannelFuture cf = null;
try {
// set up the pipeline
bootstrap.group(new NioEventLoopGroup())
.channel(NioSocketChannel.class)
.handler(new NodeClientInitializer());
// connect
bootstrap.remoteAddress(addr);
cf = bootstrap.connect();
cf.addListener(new ChannelFutureListener() {
#Override
public void operationComplete(ChannelFuture op) throws Exception {
logger.info("Connect to {}", addr.toString());
}
});
cf.channel().closeFuture().syncUninterruptibly();
} finally {
bootstrap.shutdown();
}
}
So, what I basically want to do is to send some initial information from the client to the server, after the channel is active (i.e. the connect was successful). However, when doing the c.write() I get the following warning and no package is send:
WARNING: Discarded 1 outbound message(s) that reached at the head of the pipeline. Please check your pipeline configuration.
I know there is no outbound handler in my pipeline, but I didn't think I need one (at this point) and I thought Netty would take care to transport the ByteBuffer over to the server. What am I doing wrong here in the pipeline configuration?
Netty only handle messages of type ByteBuf by default if you write to the Channel. So you need to wrap it in a ByteBuf. See also the Unpooled class with its static helpers to create ByteBuf instances.