Develop Reference

Vert.x "Cannot assign requested address" errors caused by high throughput

I"m doing a stress test on vert.x application and send ~10K RPS.
My application send an http async request from a dedicated verticle.
I"m using vert.x http client, and see that around 20 seconds my application sent the http requests successfully.
After 20 seconds i"m starting to get a lot of "Cannot assign requested address" errors.
I tried to deploy more verticles, to set different values to the http client thread pool and nothing helped to solve the issue.
I guess that the issue related to the high throughput in a short time around 1 minute.
Main Class:
public static void main(String[] args) {
final VertxOptions vertxOptions = new VertxOptions()
.setMaxEventLoopExecuteTime(1)
.setMaxEventLoopExecuteTimeUnit(TimeUnit.MILLISECONDS);
final Vertx vertx = Vertx.vertx(vertxOptions);
final Injector injector = Guice.createInjector(new Dependencies(vertx));
CustomCodecRegister.register(vertx.eventBus());
final Stream<Future<String>> deploymentFutures = Stream.of(
deployWorker(vertx, injector, StatsHttpVerticle.class, 10)
).flatMap(stream -> stream);
CompositeFuture.all(deploymentFutures.collect(Collectors.toList()))
.onSuccess(successfulCompositeFuture -> { });}
private static <T> Stream<Future<String>> deployWorker(Vertx vertx, Injector injector, Class<T> workerVerticleClass, int numVerticles) {
final String poolName = workerVerticleClass.getSimpleName()
.toLowerCase()
.replace("verticle", "-worker-pool");
final int numberOfThreads = 50;
final DeploymentOptions options = new DeploymentOptions()
.setWorker(true)
.setWorkerPoolName(poolName)
.setWorkerPoolSize(numberOfThreads);
return IntStream.range(0, numVerticles)
.mapToObj(ignore -> Future.future((Promise<String> promise) ->
vertx.deployVerticle((Verticle) injector.getInstance(workerVerticleClass), options, promise)));
}
EventBusAdapter:
public void send(Map<String, Object> queryParams, HashMap<String, String> headers, boolean followRedirect, Event eventToFire) {
StatsRequest statsRequest = new StatsRequest(queryParams, headers, eventToFire, followRedirect);
eventBus.request(FIRE_GET_METRIC_TO_STATS,statsRequest);
}
WorkerVerticle:
#Override
public void start(Promise<Void> startPromise) throws Exception {
vertx.eventBus().consumer(FIRE_GET_METRIC_TO_STATS, this::fire);
startPromise.complete();
}
private void fire(Message<StatsRequest> message) {
StatsRequest body = message.body();
MultiMap multimapHeader = MultiMap.caseInsensitiveMultiMap();
WebClientOptions webClientOptions = new WebClientOptions();
webClientOptions.setMaxPoolSize(1000);
WebClient httpClient = WebClient.create(vertx, webClientOptions);
httpClient.request(HttpMethod.GET, port, "example.com", "/1x1.gif" + "?" + "queryParamsString")
.followRedirects(false)
.putHeaders(multimapHeader)
.timeout(120000)
.send()
.onSuccess(response -> {
logger.info("All good");
})
.onFailure(err -> {
logger.error("Exception: " + err.getMessage());
});
}
How can i solve this issue?

Does Protocol.registerProtocol for apache HttpClient cause a global static change?

I've come across some code that's using
Protocol.registerProtocol
in an attempt to block some TLS ciphers for a request, and retry the request with it re-enabled at times, depending on other factors.
But is Protocol.registerProtocol causing a global change - ie will other threads be affected by this?
Here's the code in question:
protected static HostConfiguration buildTLSConfig(String uri, HostConfiguration config,
boolean blockTLS1)
throws MalformedURLException
{
scheme = "https";
if (baseHttps == null)
{
baseHttps = Protocol.getProtocol(scheme);
baseFactory = baseHttps.getSocketFactory();
}
URL newUrl = new URL(uri);
defaultPort = baseHttps.getDefaultPort();
if (blockTLS1)
{
ProtocolSocketFactory customFactory =
new CustomHttpsSocketFactory(baseFactory, TLS_PREFERRED_PROTOCOLS);
Protocol applyHttps = new Protocol(scheme, customFactory, defaultPort);
Protocol.registerProtocol(scheme, applyHttps);
config.setHost(newUrl.getHost(), defaultPort, applyHttps);
}
else
{
Protocol.registerProtocol(scheme, baseHttps);
config.setHost(newUrl.getHost(), defaultPort, baseHttps);
}
return config;
}

Yes, all threads will be affected by the change.
If we look at org.apache.commons.httpclient.protocol.Protocol, we see a global protocol Map:
/** The available protocols */
private static final Map PROTOCOLS = Collections.synchronizedMap(new HashMap());
And registerProtocol() simply modifying it:
public static void registerProtocol(String id, Protocol protocol) {
// . . .
PROTOCOLS.put(id, protocol);
}
At least it's synchronized, so there won't be a race during modification.

Netty channelAcquired is not getting called

I'm using netty channel pool for a http client and in the ChannelPoolHandler implementation channelAcquired is not getting called when the channelPool.acquire() invoked. I'm using netty 4.0.32.Final. Here's how I created the chanelpool. I just followed the simple example listed at netty.io. If someone can just explain what I've done wrong or if there is a bug that'll be very helpful. Thanks.
EventLoopGroup group = new NioEventLoopGroup();
final Bootstrap b = new Bootstrap();
b.group(group).channel(NioSocketChannel.class);
AbstractChannelPoolMap<InetSocketAddress, SimpleChannelPool> poolMap = new AbstractChannelPoolMap<InetSocketAddress, SimpleChannelPool>() {
#Override
protected SimpleChannelPool newPool(InetSocketAddress key) {
return new SimpleChannelPool(b.remoteAddress(key), new HttpClientPoolHandler());
}
};
final SimpleChannelPool simpleChannelPool = poolMap.get(new InetSocketAddress(uri.getHost(), uri.getPort()));
final Future<Channel> acquire = simpleChannelPool.acquire();
acquire.addListener(new FutureListener<Channel>() {
public void operationComplete(Future<Channel> f) throws Exception {
if (f.isSuccess()) {
final Channel ch = f.getNow();
// Send the HTTP request.
ChannelFuture channelFuture = ch.writeAndFlush(request);
channelFuture.addListener(new ChannelFutureListener() {
public void operationComplete(ChannelFuture channelFuture) throws Exception {
if (channelFuture.isSuccess()) {
simpleChannelPool.release(ch);
} else {
}
}
});
} else {
System.out.println("ERROR : " + f.cause());
}
}
});

The channelAcquiredmethod will only be called if you "acquire" a previous created channel. In your case there is not channel yet in the pool so it will call channelCreated.

How can I hijack an opened browser session and attach it to the Webdriver than opening a browser by itself

When using Selenium webDriver, I always have to initiate the driver and open a browser, navigate all the way through the flow to validate something, and if it failed throwing an exception, has to do the same spending lot of time.
Always thought of keeping another browser session opened, and let webdriver attached to the opened session and work on it, so the need to go through the flow could be eliminated.
Does any one knows how to do this in Selenium Webdriver, using Java?
Thanks

I've tried this. It works for me.
public class xRemoteWebDriver : RemoteWebDriver
{
public static bool newSession;
public static string capPath = Path.Combine(AppDomain.CurrentDomain.BaseDirectory, "TestFiles", "tmp", "sessionCap");
public static string sessiodIdPath = Path.Combine(AppDomain.CurrentDomain.BaseDirectory, "TestFiles", "tmp", "sessionid");
public xRemoteWebDriver(Uri remoteAddress)
: base(remoteAddress, new DesiredCapabilities())
{
}
protected override Response Execute(DriverCommand driverCommandToExecute, Dictionary<string, object> parameters)
{
if (driverCommandToExecute == DriverCommand.NewSession)
{
if (!newSession)
{
var capText = File.ReadAllText(capPath);
var sidText = File.ReadAllText(sessiodIdPath);
var cap = JsonConvert.DeserializeObject<Dictionary<string, object>>(capText);
return new Response
{
SessionId = sidText,
Value = cap
};
}
else
{
var response = base.Execute(driverCommandToExecute, parameters);
var dictionary = (Dictionary<string, object>) response.Value;
File.WriteAllText(capPath, JsonConvert.SerializeObject(dictionary));
File.WriteAllText(sessiodIdPath, response.SessionId);
return response;
}
}
else
{
var response = base.Execute(driverCommandToExecute, parameters);
return response;
}
}
}
Hope it helps.

HttpClient How to set Connection Timeout in individual GetMethod

In my web application I have a global static HttpClient that is used in many parts of the application. It is created like this:
MultiThreadedHttpConnectionManager connectionManager = new MultiThreadedHttpConnectionManager();
HttpConnectionManagerParams params = new HttpConnectionManagerParams();
params.setConnectionTimeout( 15000 );
params.setSoTimeout( 15000 );
connectionManager.setParams(params);
httpclient = new HttpClient(connectionManager);
HttpClientParams clientParams = new HttpClientParams();
clientParams.setParameter("http.protocol.allow-circular-redirects", true);
clientParams.setParameter("http.protocol.max-redirects", 4);
httpclient.setParams(clientParams);
The timeouts are fine for most use cased, but in a specific call I would like a shorter timeout. So I have:
GetMethod get = new GetMethod(finalUrl);
get.getParams().setParameter("http.socket.timeout", new Integer(1000));
get.getParams().setParameter("http.connection.timeout", new Integer(1000));
HttpClientUtil.getShortTimeoutInstance().executeMethod(get);
It does not work. The connection timeout is still 15000. Can I set a specific connection timeout in the GetMethod without creating a new HttpClient instance (this is because I believe creating a new HttpClient instance would not be a good idea).

you might want to look at
./impl/conn/tsccm/ThreadSafeClientConnManager.java
method
ClientConnectionRequest requestConnection(final HttpRoute route,
final Object state)
Once the connection pool has been initialized for the route, the originally configured value for socket timeOut will continue to be used unless you extend/override it.
public ClientConnectionRequest requestConnection(
final HttpRoute route,
final Object state) {
final PoolEntryRequest poolRequest = pool.requestPoolEntry(
route, state);
return new ClientConnectionRequest() {
public void abortRequest() {
poolRequest.abortRequest();
}
public ManagedClientConnection getConnection(
long timeout, TimeUnit tunit) throws InterruptedException,
ConnectionPoolTimeoutException {
if (route == null) {
throw new IllegalArgumentException("Route may not be null.");
}
if (log.isDebugEnabled()) {
log.debug("Get connection: " + route + ", timeout = " + timeout);
}
BasicPoolEntry entry = poolRequest.getPoolEntry(timeout, tunit);
return new BasicPooledConnAdapter(ThreadSafeClientConnManager.this, entry);
}
};
}