We use akka to stress test one of our systems by sending JMS messages in parallel to our system entry points.
At a very high level, there is a Boss actor and a number of Worker actors that do the job.
When Boss actor is created in its constructor it also creates the Worker actors and puts them in a map:
workers = endPoints.stream().collect(Collectors.toMap(e -> e, e -> newWorker(e, ...)));
Then when the Boss receives the StartTesting message it just iterates to the list of its workers and send a PerformWork message to each one. Upon receiving the PerformWork message each worker goes to the database get the messages to send out, and start sending them to its associated end point. Nothing out of the ordinary here:
logger.info("Number of worker actors: " + workers.values().size());
PerformWork performWork = new PerformWork();
workers.values().forEach(w -> {
logger.info("Sending PerformWork message to " + w);
w.tell(performWork, getSelf());
});
When running we can see in the logs the following:
... Number of worker actors: 43
... Sending a PerformWork message to Actor[akka://.../WORKER-1
... Sending a PerformWork message to Actor[akka://.../WORKER-2
.
... WORKER-1: received message of type PerformWork
... Sending a PerformWork message to Actor[akka://.../WORKER-30
.
The number of Sending a PerformWork log entries are never equals with the number of workers (43 in this case). Usually between 20 and 30 but not necessarily always the same. The number of WORKER-x received message of PerformWork entries are usually smaller than the messages sent. The worker actors that actually received the PerformWork message perform what they are supposed to do without any problem.
However we never seen the rest of the Sending ... or Received ... messages in the logs for the rest of the workers, and obviously those end points associated with them never receive any message.
So my questions would be:
What I am doing wrong? Maybe my approach is too naive. I am not very experienced with akka, never used it beyond building testing tools.
What will cause the sending PerformMessage loop above to never complete.
What will cause the delivery of PerformMessage above to fail.
All this processing is happening inside the same JVM. I am open to any other suggestion that will help me understand what is going on and address the issue.
The testing tool is written in Java. I added the Scala tag thinking that Scala developers would be more familiar with akka, given Actor framework is part of the language.
Thank you in advance.
UPDATE
I worked out what was causing the issue, and I was able to fix it, however I still don't have an explanation about why was happening and it would be good to have an understanding.
So before the fix the Worker code looked like below:
#Override
public void onReceive(final Object message) throws InterruptedException {
if (message instanceof PerformWork) {
// here is the code using jdbcTemplate to get messages
// from database and send them to the system end point
getSender().tell(new WorkDone(), getSelf());
} else {
logger.info("Not prepared to handle message of type " + message.getClass());
unhandled(message);
}
}
And I changed it to start sending messages with a delay of two seconds:
#Override
public void onReceive(final Object message) throws InterruptedException {
if (message instanceof PerformWork) {
new Timer().schedule(
new TimerTask() {
#Override
public void run() {
proceedWithSendingMessages();
}
},
2000
);
} else {
logger.info("Not prepared to handle message of type " + message.getClass());
unhandled(message);
}
}
private void proceedWithSendingMessages() {
// here is the code using jdbcTemplate to get messages
// from database and send them to the system end point
getSender().tell(new WorkDone(), getSelf());
}
After the above change everything started working but the question I have is why the issue existed in the first place. It seems to me that what I experienced is against the asynchronous processing concept.
Again thank you in advance for your inputs.
UPDATE 2
In a normal run each worker runs for 24 hours. Remember it is a stress/load test so no WorkDone message is sent back to the Boss in less than this time. My akka application got hung after less than one second after starting. As I said all the workers that received the PerformWork were running normally and kept sending messages for 24 hours.
When receiving a WorkDone message the Boss verify if all Workers are done and if so it shuts down akka. I am attaching the code here as it was requested but I don't think it has anything to do with my problem.
} else if (message instanceof WorkDone) {
// completed below is a set
completed.add(((WorkDone) message).getSystem())
if (completed.size() == workers.size()) {
shutDownAkka();
}
}
I realized that was the Worker starting the work immediately after receiving a PerformWork message causing my issue by changing the database in such a way that it will be nothing to send so by querying an empty database made everything to work OK but finished straight away.
I suspected something funny would be caused by the spring JdbcTemplate bean which was singleton and being thread safe will lock things when multiple workers will try to use it immediately. I changed it to be prototype but the problem staid the same. I even made it new JdbcTemplate(singletonDatasource) with no luck. On the other hand I thought it must not be JdbcTemplate as 99% of the Java world it is using it.
At this point I added the delay in the worker code as posted in my first UPDATE thinking it was important all workers receive the PerformWork notification. Once I did this all started working and no problems. However I would be interested to know what would have caused this.
Related
Kafka allows for asynchronous message sending through below methods on Producer (KafkaProducer) class:
public java.util.concurrent.Future<RecordMetadata> send(ProducerRecord<K,V> record)
public java.util.concurrent.Future<RecordMetadata> send(ProducerRecord<K,V> record, Callback callback)
Successes can be handled through
1) the Future<RecordMetaData> object or
2) onCompletion method invoked by the callback. Full method signature and usage of onCompletion is as below (taken from kafka docs)
`
ProducerRecord<byte[],byte[]> record = new ProducerRecord<byte[],byte[]>("the-topic", key, value);
producer.send(record,
new Callback() {
public void onCompletion(RecordMetadata metadata, Exception e) {
if(e != null)
e.printStackTrace();
System.out.println("The offset of the record we just sent is: " + metadata.offset());
}
});
While failure needs to be handled through the Exception e passed to the onCompletion method
Fine every thing looks good so far.
But if I am getting it right, any reasonable information that can be obtained from exception or e object is stacktrace and exception message. What I mean to point out here is, e does not contain any information of the actual record sent. Or in other words, it does not contain a reference to the actual record that was sent to kafka broker. So what useful processing or handling can be done by the producer if the record was not sent successfully. Really not much.
Why I say this is - ideally I would like to make a log of the failed message some where and then try to resend it. But with the little information (e) provided by framework, i feel this is not possible.
Can someone point out if I am right or wrong?
You could easily create a callback that receives the producerRecord as a constructor argument. So upon onCompletion with an exception, you can have complete knowledge of the producer record, and even try to send it again.
I dealt with the same issue. Created a callback that gets both producerRecord, and a callback handler that uses an executor service to send the record again. So eventually, I can tolerate any number of failures (e.g. network issues or kafka is down), and recover from it.
I'm playing around with Vert.x and quite new to the servers based on event loop as opposed to the thread/connection model.
public void start(Future<Void> fut) {
vertx
.createHttpServer()
.requestHandler(r -> {
LocalDateTime start = LocalDateTime.now();
System.out.println("Request received - "+start.format(DateTimeFormatter.ISO_DATE_TIME));
final MyModel model = new MyModel();
try {
for(int i=0;i<10000000;i++){
//some simple operation
}
model.data = start.format(DateTimeFormatter.ISO_DATE_TIME) +" - "+LocalDateTime.now().format(DateTimeFormatter.ISO_DATE_TIME);
} catch (Exception e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}
r.response().end(
new Gson().toJson(model)
);
})
.listen(4568, result -> {
if (result.succeeded()) {
fut.complete();
} else {
fut.fail(result.cause());
}
});
System.out.println("Server started ..");
}
I'm just trying to simulate a long running request handler to understand how this model works.
What I've observed is the so called event loop is blocked until my first request completes. Whatever little time it takes, subsequent request is not acted upon until the previous one completes.
Obviously I'm missing a piece here and that's the question that I have here.
Edited based on the answers so far:
Isn't accepting all requests considered to be asynchronous? If a new
connection can only be accepted when the previous one is cleared
off, how is it async?
Assume a typical request takes anywhere between 100 ms to 1 sec (based on the kind and nature of the request). So it means, the
event loop can't accept a new connection until the previous request
finishes(even if its winds up in a second). And If I as a programmer
have to think through all these and push such request handlers to a
worker thread , then how does it differ from a thread/connection
model?
I'm just trying to understand how is this model better from a traditional thread/conn server models? Assume there is no I/O op or
all the I/O op are handled asynchronously? How does it even solve
c10k problem, when it can't start all concurrent requests parallely and have to wait till the previous one terminates?
Even if I decide to push all these operations to a worker thread(pooled), then I'm back to the same problem isn't it? Context switching between threads?
Edits and topping this question for a bounty
Do not completely understand how this model is claimed to asynchronous.
Vert.x has an async JDBC client (Asyncronous is the keyword) which I tried to adapt with RXJava.
Here is a code sample (Relevant portions)
server.requestStream().toObservable().subscribe(req -> {
LocalDateTime start = LocalDateTime.now();
System.out.println("Request for " + req.absoluteURI() +" received - " +start.format(DateTimeFormatter.ISO_DATE_TIME));
jdbc.getConnectionObservable().subscribe(
conn -> {
// Now chain some statements using flatmap composition
Observable<ResultSet> resa = conn.queryObservable("SELECT * FROM CALL_OPTION WHERE UNDERLYING='NIFTY'");
// Subscribe to the final result
resa.subscribe(resultSet -> {
req.response().end(resultSet.getRows().toString());
System.out.println("Request for " + req.absoluteURI() +" Ended - " +LocalDateTime.now().format(DateTimeFormatter.ISO_DATE_TIME));
}, err -> {
System.out.println("Database problem");
err.printStackTrace();
});
},
// Could not connect
err -> {
err.printStackTrace();
}
);
});
server.listen(4568);
The select query there takes 3 seconds approx to return the complete table dump.
When I fire concurrent requests(tried with just 2), I see that the second request completely waits for the first one to complete.
If the JDBC select is asynchronous, Isn't it a fair expectation to have the framework handle the second connection while it waits for the select query to return anything.?
Vert.x event loop is, in fact, a classical event loop existing on many platforms. And of course, most explanations and docs could be found for Node.js, as it's the most popular framework based on this architecture pattern. Take a look at one more or less good explanation of mechanics under Node.js event loop. Vert.x tutorial has fine explanation between "Don’t call us, we’ll call you" and "Verticles" too.
Edit for your updates:
First of all, when you are working with an event loop, the main thread should work very quickly for all requests. You shouldn't do any long job in this loop. And of course, you shouldn't wait for a response to your call to the database.
- Schedule a call asynchronously
- Assign a callback (handler) to result
- Callback will be executed in the worker thread, not event loop thread. This callback, for example, will return a response to the socket.
So, your operations in the event loop should just schedule all asynchronous operations with callbacks and go to the next request without awaiting any results.
Assume a typical request takes anywhere between 100 ms to 1 sec (based on the kind and nature of the request).
In that case, your request has some computation expensive parts or access to IO - your code in the event loop shouldn't wait for the result of these operations.
I'm just trying to understand how is this model better from a traditional thread/conn server models? Assume there is no I/O op or all the I/O op are handled asynchronously?
When you have too many concurrent requests and a traditional programming model, you will make thread per each request. What this thread will do? They will be mostly waiting for IO operations (for example, result from database). It's a waste of resources. In our event loop model, you have one main thread that schedule operations and preallocated amount of worker threads for long tasks. + None of these workers actually wait for the response, they just can execute another code while waiting for IO result (it can be implemented as callbacks or periodical checking status of IO jobs currently in progress). I would recommend you go through Java NIO and Java NIO 2 to understand how this async IO can be actually implemented inside the framework. Green threads is a very related concept too, that would be good to understand. Green threads and coroutines are a type of shadowed event loop, that trying to achieve the same thing - fewer threads because we can reuse system thread while green thread waiting for something.
How does it even solve c10k problem, when it can't start all concurrent requests parallel and have to wait till the previous one terminates?
For sure we don't wait in the main thread for sending the response for the previous request. Get request, schedule long/IO tasks execution, next request.
Even if I decide to push all these operations to a worker thread(pooled), then I'm back to the same problem isn't it? Context switching between threads?
If you make everything right - no. Even more, you will get good data locality and execution flow prediction. One CPU core will execute your short event loop and schedule async work without context switching and nothing more. Other cores make a call to the database and return response and only this. Switching between callbacks or checking different channels for IO status doesn't actually require any system thread's context switching - it's actually working in one worker thread. So, we have one worker thread per core and this one system thread await/checks results availability from multiple connections to database for example. Revisit Java NIO concept to understand how it can work this way. (Classical example for NIO - proxy-server that can accept many parallel connections (thousands), proxy requests to some other remote servers, listen to responses and send responses back to clients and all of this using one or two threads)
About your code, I made a sample project for you to demonstrate that everything works as expected:
public class MyFirstVerticle extends AbstractVerticle {
#Override
public void start(Future<Void> fut) {
JDBCClient client = JDBCClient.createShared(vertx, new JsonObject()
.put("url", "jdbc:hsqldb:mem:test?shutdown=true")
.put("driver_class", "org.hsqldb.jdbcDriver")
.put("max_pool_size", 30));
client.getConnection(conn -> {
if (conn.failed()) {throw new RuntimeException(conn.cause());}
final SQLConnection connection = conn.result();
// create a table
connection.execute("create table test(id int primary key, name varchar(255))", create -> {
if (create.failed()) {throw new RuntimeException(create.cause());}
});
});
vertx
.createHttpServer()
.requestHandler(r -> {
int requestId = new Random().nextInt();
System.out.println("Request " + requestId + " received");
client.getConnection(conn -> {
if (conn.failed()) {throw new RuntimeException(conn.cause());}
final SQLConnection connection = conn.result();
connection.execute("insert into test values ('" + requestId + "', 'World')", insert -> {
// query some data with arguments
connection
.queryWithParams("select * from test where id = ?", new JsonArray().add(requestId), rs -> {
connection.close(done -> {if (done.failed()) {throw new RuntimeException(done.cause());}});
System.out.println("Result " + requestId + " returned");
r.response().end("Hello");
});
});
});
})
.listen(8080, result -> {
if (result.succeeded()) {
fut.complete();
} else {
fut.fail(result.cause());
}
});
}
}
#RunWith(VertxUnitRunner.class)
public class MyFirstVerticleTest {
private Vertx vertx;
#Before
public void setUp(TestContext context) {
vertx = Vertx.vertx();
vertx.deployVerticle(MyFirstVerticle.class.getName(),
context.asyncAssertSuccess());
}
#After
public void tearDown(TestContext context) {
vertx.close(context.asyncAssertSuccess());
}
#Test
public void testMyApplication(TestContext context) {
for (int i = 0; i < 10; i++) {
final Async async = context.async();
vertx.createHttpClient().getNow(8080, "localhost", "/",
response -> response.handler(body -> {
context.assertTrue(body.toString().contains("Hello"));
async.complete();
})
);
}
}
}
Output:
Request 1412761034 received
Request -1781489277 received
Request 1008255692 received
Request -853002509 received
Request -919489429 received
Request 1902219940 received
Request -2141153291 received
Request 1144684415 received
Request -1409053630 received
Request -546435082 received
Result 1412761034 returned
Result -1781489277 returned
Result 1008255692 returned
Result -853002509 returned
Result -919489429 returned
Result 1902219940 returned
Result -2141153291 returned
Result 1144684415 returned
Result -1409053630 returned
Result -546435082 returned
So, we accept a request - schedule a request to the database, go to the next request, we consume all of them and send a response for each request only when everything is done with the database.
About your code sample I see two possible issues - first, it looks like you don't close() connection, which is important to return it to pool. Second, how your pool is configured? If there is only one free connection - these requests will serialize waiting for this connection.
I recommend you to add some printing of a timestamp for both requests to find a place where you serialize. You have something that makes the calls in the event loop to be blocking. Or... check that you send requests in parallel in your test. Not next after getting a response after previous.
How is this asynchronous? The answer is in your question itself
What I've observed is the so called event loop is blocked until my
first request completes. Whatever little time it takes, subsequent
request is not acted upon until the previous one completes
The idea is instead of having a new for serving each HTTP request, same thread is used which you have blocked by your long running task.
The goal of event loop is to save the time involved in context switching from one thread to another thread and utilize the ideal CPU time when a task is using IO/Network activities. If while handling your request it had to other IO/Network operation eg: fetching data from a remote MongoDB instance during that time your thread will not be blocked and instead an another request would be served by the same thread which is the ideal use case of event loop model (Considering that you have concurrent requests coming to your server).
If you have long running tasks which does not involve Network/IO operation, you should consider using thread pool instead, if you block your main event loop thread itself other requests would be delayed. i.e. for long running tasks you are okay to pay the price of context switching for for server to be responsive.
EDIT:
The way a server can handle requests can vary:
1) Spawn a new thread for each incoming request (In this model the context switching would be high and there is additional cost of spawning a new thread every time)
2) Use a thread pool to server the request (Same set of thread would be used to serve requests and extra requests gets queued up)
3) Use a event loop (single thread for all the requests. Negligible context switching. Because there would be some threads running e.g: to queue up the incoming requests)
First of all context switching is not bad, it is required to keep application server responsive, but, too much context switching can be a problem if the number of concurrent requests goes too high (roughly more than 10k). If you want to understand in more detail I recommend you to read C10K article
Assume a typical request takes anywhere between 100 ms to 1 sec (based
on the kind and nature of the request). So it means, the event loop
can't accept a new connection until the previous request finishes(even
if its winds up in a second).
If you need to respond to large number of concurrent requests (more than 10k) I would consider more than 500ms as a longer running operation. Secondly, Like I said there are some threads/context switching involved e.g.: to queue up incoming requests, but, the context switching amongst threads would be greatly reduced as there would be too few threads at a time. Thirdly, if there is a network/IO operation involved in resolving first request second request would get a chance to be resolved before first is resolved, this is where this model plays well.
And If I as a programmer have to think
through all these and push such request handlers to a worker thread ,
then how does it differ from a thread/connection model?
Vertx is trying to give you best of threads and event loop, so, as programmer you can make a call on how to make your application efficient under both the scenario i.e. long running operation with and without network/IO operation.
I'm just trying to understand how is this model better from a
traditional thread/conn server models? Assume there is no I/O op or
all the I/O op are handled asynchronously? How does it even solve c10k
problem, when it can't start all concurrent requests parallely and
have to wait till the previous one terminates?
The above explanation should answer this.
Even if I decide to push all these operations to a worker
thread(pooled), then I'm back to the same problem isn't it? Context
switching between threads?
Like I said, both have pros and cons and vertx gives you both the model and depending on your use case you got to choose what is ideal for your scenario.
In these sort of processing engines, you are supposed to turn long running tasks in to asynchronously executed operations and these is a methodology for doing this, so that the critical thread can complete as quickly as possible and return to perform another task. i.e. any IO operations are passed to the framework to call you back when the IO is done.
The framework is asynchronous in the sense that it supports you producing and running these asynchronous tasks, but it doesn't change your code from being synchronous to asynchronous.
I am using both python and java implementations of a websocket client. However, since onMessage is asynchronous, it will begin executing immediately, even if there is another function being executed. How can I ensure that each onMessage function will finish completely before the next message is handled. Thanks!
EDIT:
I am subscribing to multiple channels, and regardless of which channel sends a message, my onMessage handler will handle the message. I need my onMessage handler to fully process each message it receives before it begins to process the next message, but I cannot lose any messages. I hope this helps to clarify a bit.
It sounds just a concurrent issue. How about this?
private final Object onMessageLock = new Object();
#OnMessage
public void onMessage(String message, Session session)
{
synchronized (onMessageLock)
{
// Handle the message here.
}
}
I tested the solution proposed by Takahiko. It only works based on one client. The messages from different clients will still be processed parallely.
If you want all messages to be processed after the message before has been processed completely (regardless of the client that sent it) you have to make the Lock object static:
private static final Object onMessageLock = new Object();
I have the following code retrieving messages from a SQS queue. I am using a AmazonSQSBufferedAsyncClient to retrieve the message from Queue. A fixed delay SingleThreadedExecutor wakes up every 5 mins calling receiveMessage. Long polling is enabled in the Queue
#Service
public class AmazonQueueService
implements QueueService<String> {
#Autowired
private AmazonSQSBufferedAsyncClient sqsAsyncClient;
#Value("${aws.sqs.queueUrl}")
private String queueUrl;
#Override
public List<Message<String>> receiveMessage() {
ReceiveMessageRequest receiveMessageRequest = new ReceiveMessageRequest(queueUrl);
ReceiveMessageResult result = sqsAsyncClient.receiveMessage(receiveMessageRequest);
LOG.debug("Size=" + result.getMessages().size());
return Lists.transform(result.getMessages(), ......);
}
.....
}
The problem is when I check AWS console, the message is always in-flight, but it is never received in the application (Size is always printed as 0) . Looks like the AmazonSQSBufferedAsyncClient is reading the message from queue but not returning in the receiveMessage call.
Any ideas?
Finally figured this out. The problem manifests with the combination of queue visibility timeout (2 mins) and the scheduledExecutor delay (5 mins).
Increasing the visibility timeout to 15 mins solved the problem.
My theory is -> The AmazonSQSBufferedAsyncClient retrieves message and keeps it in the buffer waiting for receiveMessage call. As the executor delay is 5 mins, the visibility of the message times out befor receiveMessage is called and the message is returned to the queue. It also looks like the message is picked from the queue almost immediately. And now for whatever reason a call to receiveMessage does not receive the message. Increasing the timeout which gave a chance for the receiveMessage call to happen before a timeout event, solved the problem, I guess.
Any other possible explanation?
For me problem was my messages were being read by lambda even before I see them on console. So the result was I was not able to poll for the messages from sqs console.
You have to delete the message from the queue when you're done with it. If you don't, it's going to stay in-flight until it times out and then go right back to the queue. It is designed this way so you will never lose messages. If your program crashes before it finishes handling the message and deleting it, the message will go right back to the queue.
From the basic Java example (SampleDriver.java):
QMessage message = messages.get(0);
System.out.println("\nMessage received");
System.out.println(" message id: " + message.getId());
System.out.println(" receipt handle: " + message.getReceiptHandle());
System.out.println(" message content: " + message.getContent());
testQueue.deleteMessage(message.getReceiptHandle()); // <===== here
In my case, I was using short polling, and sometimes it was receiving more than 1 message but I was processing only one. Thus was losing track of messages. Since the message was never processed its visibility timeout was never reached and thus message appeared in inflight messages.
I'm sending JMS requests to a Weblogic 10.3 server through a named JMS queue, and receive a reply back through a temporary queue.
Client (barebone):
//init
Destination replyQueue = session.createTemporaryQueue();
replyConsumer = session.createConsumer(replyQueue);
...
//loop
TextMessage requestMessage = session.createTextMessage();
requestMessage.setText("Some request")
requestMessage.setJMSReplyTo(replyQueue);
requestProducer.send(requestMessage);
Message msg = replyConsumer.receive(5000);
if (msg instanceof TextMessage) {
...
} else { ... }
//loop end
Server MDB (message driven bean):
public void onMessage(Message msg) {
if (msg instanceof TextMessage) {
...
TextMessage replyMessage = jmsSession.createTextMessage();
replyMessage.setText("Some response");
replyMessage.setJMSCorrelationID(msg.getJMSCorrelationID());
replyProducer.send(replyMessage);
}
}
The problem is that the very first server reply is often lost! That is, the replyConsumer.receive(5000) ends with timeout for every 4th-5th replyConsumer. When the consumer receives the first answer, then it continues to receive all the rest, so the problem is only with the first message send through the temporary queue after the temp queue has been created.
My question: Do I have to set something special for the temporary queue in order it works from the very start after being created? Or any other hint?
Further info:
When testing against my local development machine, the temp queues work without problem. The messages are getting lost only when testing against our clustered Weblogic server. However, I have switched off all cluster members but one instance.
I have verified that the server successfully replies all the requests that the client sends (by counting the sent requests and sent replies). The server replies in the order of milliseconds, even for the lost replies.
When I replace the temporary queue with a regular named queue, the problem disappears! So the problem doesn't seem (to me) to be in my code.
I've also tried to modify expiration, persistency, delay etc. of the reply message, but without success. This way I excluded the scenario that the response arrives earlier than the client begins to read the queue, and then the message immediately expires not giving the client a chance to process it.
Edit: Instead of the synchronous replyConsumer.receive(5000) I've also tried to use the asynchronous replyConsumer.setMessageListener(this). The behaviour hasn't changed, first messages are still getting lost for temp queues.
Edit: It seems that there's something wrong with the Weblogic server (or cluster) I am using. Because when I deployed the server application to another Weblogic cluster we have, everything began to work correctly! Both clusters should be configured identically - so where's a difference? It scares me that the Weblogic signals no error.
Your problem seems to be that sometimes the server is receiving the publish and discarding it before your consumer has started receiving.
The way around it is to use the asynchronous receive (replyConsumer.setMessageListener) calls instead of the blocking call you currently have (replyConsumer.receive(5000)) and to add the call to the code with the rest of your consumer code.
That way, you are already listening for replies before you send out the request.
Hope that helps.
Edit: Just read that you are using a temporary queue, so my first sentence is not correct. However as an experiment try the rest of my response to see if it changes the behaviour you are seeing