I have SocketChannel configured for read only SelectionKey.OP_CONNECT | SelectionKey.OP_READ
Profiler shows runChannel is the most CPU consuming method and actually it is reasonable because it's infinite loop which calls method selector.select() all the time, but on the other hand I have dozens of such connections and it kills CPU.
Is there a possibility to decrease CPU load and in the same time do not miss any incoming message?
public void runChannel() {
while (session.isConnectionAlive()) {
try {
// Wait for an event
int num = selector.select();
// If you don't have any activity, loop around and wait
// again.
if (num == 0) {
continue;
}
} catch (IOException e) {
log.error("Selector error: {}", e.toString());
log.debug("Stacktrace: ", e);
session.closeConnection();
break;
}
handleSelectorkeys(selector.selectedKeys());
}
}
Unsunscribe from OP_CONNECT - select() won't block if you're subscribed to OP_CONNECT and connected.
Related
I have a message stream, where messages comes which I need to process and then store them in database. In Java, I've written polling code which polls stream and consumes messages every 20 seconds.
This is done inside an infinite for-loop, like below:
for (;;) {
try{
//1. Logic for polling.
//2. Logic for processing the message.
//3. Logic for storing the message in database.
Thread.sleep(20000 - <time taken for above 3 steps >);
} catch(Exception E){
//4. Exception handling.
}
}
This logic runs as expected and the stream is polled, but once in a while it hits an exception or something goes wrong and polling stops.
I want to have a mechanism, that as soon as polling stopped, let's say this for loop is not running for 60 seconds, I should receive a mail or ping.
What is the best way to invoke a method if this for loop is not running for 60 seconds?
I am thinking like, each for-loop execution will ping a heartbeat, and when that heartbeat pinging not received from for-loop then a mail sending is invoked.
There are two different reasons why polling stops making progress, and each needs a different approach:
If the logic throws a Throwable other than an Exception, for instance an Error, the catch does not match, and execution will leave the for-loop, and likely reach the thread's UncaughtExceptionHandler, the default implementation of which logs the exception to System.err and terminates the thread. To prevent this, you should catch Throwable rather than Exception.
The second possibility is that some step in your logic doesn't terminate, for instance due to an infinite loop, a deadlock, waiting for I/O operations, or whatever. In this case, you'll want to take a thread dump to see where the thread is stuck. You can automate this as follows:
class Watchdog {
final Duration gracePeriod;
final Thread watchedThread;
volatile Instant lastProgress;
public Watchdog(Duration gracePeriod) {
this.gracePeriod = gracePeriod;
watchedThread = Thread.currentThread();
everythingIsFine();
var t = new Thread(this::keepWatch);
t.setDaemon(true);
t.start();
}
public void everythingIsFine() {
lastProgress = Instant.now();
}
void keepWatch() {
while (true) {
var silence = Duration.between(lastProgress, Instant.now());
if (silence.compareTo(gracePeriod) > 0) {
System.err.println("Watchdog hasn't seen any progress for " + silence.toSeconds() + " seconds. The watched thread is currently at:");
for (var element : watchedThread.getStackTrace()) {
System.err.println("\tat " + element);
}
}
try {
Thread.sleep(gracePeriod);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
}
while you can use as follows:
public class Test {
void step() throws Exception {
System.in.read();
}
void job() {
var snoopy = new Watchdog(Duration.ofSeconds(2));
for (;;) {
try {
step();
snoopy.everythingIsFine();
Thread.sleep(1000);
} catch (Throwable t) {
System.err.println(t);
}
}
}
public static void main(String[] args) throws Exception {
new Test().job();
}
}
once the grace period elapses, the WatchDog will print something like:
Watchdog hasn't seen any progress for 2 seconds. The watched thread is currently at:
at java.base/java.io.FileInputStream.readBytes(Native Method)
at java.base/java.io.FileInputStream.read(FileInputStream.java:293)
at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:255)
at java.base/java.io.BufferedInputStream.implRead(BufferedInputStream.java:289)
at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:276)
at stackoverflow.Test.step(Test.java:48)
at stackoverflow.Test.job(Test.java:55)
at stackoverflow.Test.main(Test.java:65)
I have a custom built API for interacting with their messaging system. But this API doesn't give me any way to confirm that I have established a connection aside from when it is unable to connect an exception will be thrown.
When I receive a exception while connected, I have an exception listener that attempts to reconnect to the server. I'd like this to loop on exception to retry the connection. Doing an infinite loop until I am able to connect, or until the program is closed. I attempted to do this with break labels like so:
reconnect: try{
attemptReconnection();
}catch(Exception e){
log.error(e);
break reconnect;
}
but that was unable to find the reconnect label for me, and is a bit to close to using a GOTO statement than I would be comfortable putting into production.
Proceed this way:
do { // optional loop choice
try{
attemptReconnection();
break; // Connection was successful, break out of the loop
} catch(Exception e){
// Exception thrown, do nothing and move on to the next connection attempt (iteration)
log.error(e);
}
}while(true);
If the execution flow reaches the break; instruction then that means that you successfully connected. Otherwise, it will keep moving on to the next iteration. (Note that the loop choice is optional, you can use pretty much any loop you want)
Can't say I have experience with APIs, but I would think something like this would achieve the result you're after.
boolean success = false;
while (!success){
try{
attemptReconnection();
success = true;
}
catch(Exception e){
log.error(e);
}
}
Once attemptReconnection() executes without errors, success would be set to true and terminate the loop.
Have attemptReconnection return true when connection succeds, false otherwise.
The method attemptReconnection should also catch and log the Exception.
Then :
while(!attemptReconnection()){
log.error("Connection failure");
}
I would suggest controlling the reconnection attempts not with a while loop but with a scheduled event. This you can easily initiate multiple connections and implement a back off mechanism not to over-consume resources while trying to reconnect
private ScheduledExecutorService scheduler;
...
public void connect() {
for (int i = 0; i < numberOfConnections; i++) {
final Runnable r = new Runnable() {
int j = 1;
public void run() {
try {
final Connection connection = createNewConnection();
} catch (IOException e) {
//here we do a back off mechanism every 1,2,4,8,16... 512 seconds
final long sleep = j * 1000L;
if (j < 512) {
j *= 2;
} else {
j = 1;
}
LOGGER.error("Failed connect to host:port: {}:{}. Retrying... in {} millis",
host, port, sleep);
LOGGER.debug("{}", e);
scheduler.schedule(this, sleep, TimeUnit.MILLISECONDS);
} catch (InterruptedException e) {
Thread.currentThread.interrupt();
}
}
};
scheduler.schedule(r, 1, TimeUnit.SECONDS);
}
}
Do not forget to do a scheduler.shutdownNow() if you want to close the application so as to avoid the treadpool being leaking.
You can even implement a reconnect mechanism once you have been connected and you are disconnected by having the listener execute the connect method in case of a status change on the connection.
I have a Runnable that watches for data to send out UDP as well as to send a keep alive every 10 seconds. The process is taking 100% CPU. I tried setting the thread to low priority but didn't seem to make any difference.
private Runnable keepAliveRunnable = new Runnable() {
long nextSend = 0;
byte[] sendData;
#Override
public void run() {
if(DEBUG)
System.out.println("Starting keepAlive.");
while (socket != null) {
synchronized (socketLock) {
try {
sendData = sendQueue.poll();
if (sendData != null) {
socket.send(new DatagramPacket(sendData, sendData.length,
InetAddress.getByName(Main.ipAddress), 10024));
} else if (nextSend < System.currentTimeMillis()) {
if(DEBUG && nextSend < System.currentTimeMillis())
System.out.println("Update keepAlive.");
// Send /xremote
socket.send(new DatagramPacket(("/xremote").getBytes(),
("/xremote").getBytes().length,
InetAddress.getByName(Main.ipAddress), 10024));
nextSend = System.currentTimeMillis() + keepAliveTimeout;
// Send /info
socket.send(new DatagramPacket(("/info").getBytes(),
("/info").getBytes().length,
InetAddress.getByName(Main.ipAddress), 10024));
}
} catch (IOException e) {
e.printStackTrace();
if(!e.getMessage().contains("Socket closed")) {
e.printStackTrace();
}
}
}
}
System.out.println("keepAliveRunnable ended.");
}
};
Make sendQueue a LinkedBlockingQueue, and use poll timeouts.
You are busy waiting, which essentially forces your app to keep running the same logic over and over instead of giving the CPU back to the system.
Don't count on your own implementation of checking the time, that is unreliable and can result in what you're seeing. Instead, use blockingQueue.poll(10, TimeUnit.SECONDS), which automatically handles returning the CPU to the system.
I made a few other changes to your code; I put the duplicated packet construction code in a separate method, and I wrapped the synchronization of the socket only when the socket is actually being used. Notice how much cleaner it is when you let the queue do the work for you.
while(socket != null) {
try {
sendData = sendQueue.poll(10, TimeUnit.SECONDS);
if (sendData != null) {
sendPacket(sendData);
} else {
sendPacket("/xremote".getBytes());
sendPacket("/info".getBytes());
}
} catch (IOException e) {
e.printStackTrace();
if (!e.getMessage().contains("Socket closed")) {
e.printStackTrace();
}
}
}
And here's sendPacket:
private static void sendPacket(byte[] data) throws UnknownHostException, IOException {
// Note, you probably only have to do this once, rather than looking it up every time.
InetAddress address = InetAddress.getByName(Main.ipAddress);
DatagramPacket p = new DatagramPacket(data, data.length, address, 10024);
synchronized(socketLock) {
socket.send(p);
}
}
You should add a Thread.sleep() at the bottom of your while loop, to slow down your loop. As is, you're busy-waiting and churning the CPU while you wait for the nextSend time to be reached. Thread.sleep() will actually pause the thread, allowing other threads and processes to use the CPU while this one sleeps.
Sleeping for a 10th of a second (100 milliseconds) should be a good amount of time to sleep between iterations of your loop, if your goal is to actually do work every 10 seconds.
There are more advanced techniques for dispatching work every so often, like ScheduledExecutorService, which you could also consider using. But for a small application the pattern you're using is fine, just avoid busy waiting.
I think rather than polling your sendqueue, its better to use semaphore signal and wait.
When a packet is inserted in sendqueue, call semaphore signal.
Use semaphore wait instead of call to sendqueue.poll().
I assume you have separate threads for pushing popping data from sendqueue.
This is standard consumer producer problem. https://en.wikipedia.org/wiki/Producer%E2%80%93consumer_problem
After digging through my code, I had realized that over time I had whittled down the number of processes sending data to 1 (duh) so I really didn't need the runnable as I could just send the data directly. I also set up a separate runnable and used ScheduledExecutor. I thought I would just put that here for other to see. Durron597's code is a little prettier but since I'm only sending two packs now I decided to just put the code together.
// In main
pingXAir();
private void pingXAir() {
System.out.println("Start keepAlive");
ScheduledExecutorService executorService = Executors.newScheduledThreadPool(1);
executorService.scheduleAtFixedRate(keepAliveRunnable, 0, 5, TimeUnit.SECONDS);
}
private Runnable keepAliveRunnable = new Runnable() {
#Override
public void run() {
synchronized (socketLock) {
try {
if (DEBUG)
System.out.println("Update keepAlive.");
// Send /xremote
socket.send(new DatagramPacket(("/xremote").getBytes(),
("/xremote").getBytes().length,
InetAddress.getByName(Main.ipAddress), 10024));
// Send /info
socket.send(new DatagramPacket(("/info").getBytes(),
("/info").getBytes().length,
InetAddress.getByName(Main.ipAddress), 10024));
} catch (IOException e) {
e.printStackTrace();
if (!e.getMessage().contains("Socket closed")) {
e.printStackTrace();
}
}
}
}
};
I am using a java NIO based which accept connections from clients(configured non blocking) and only reads data sent by clients. Clients once connected will stick to server for a long time, so i used a single thread for "selector.select" and "Accept", but the clients connected will send messages every 15 sec and number of clients are 5000, each message is of size 150 Bytes.
Instead of creating a new thread for each read from clients i decided to have a thread pool of 100 threads, but server is not able to read data from all clients it simply hangs. When a new thread is created each time it is able to read data from all clients.
Here is my ReadEvent thread
class ReadEvent implements Runnable {
private static Logger logger = Logger.getLogger(ReadEvent.class.getName());
private SelectionKey key;
/**
* Constructor to initialize the thread with the key having read event
* pending.
*
* #param key
* SelectionKey having read event.
**/
public ReadEvent(SelectionKey key) {
this.key = key;
}
/**
* Method to read the data from the key.
**/
#Override
public void run() {
SocketChannel socketChannel = (SocketChannel) key.channel();
synchronized (socketChannel) {
if (socketChannel.isOpen()) {
try {
ByteBuffer readBuffer = ByteBuffer.allocate(150);
int numRead = 0;
try {
/* ".read" is nonblocking */
numRead = socketChannel.read(readBuffer);
/*
* Some other IO error on reading socketChannel.
*/
}
catch (IOException e) {
logger.debug(
"[run] Connection abruptly terminated from client",
e);
key.channel().close();
return;
}
if (numRead == -1) {// socket closed cleanly
key.channel().close();
return;
}
String data = null;
data = new String(readBuffer.array(),
Charset.forName("ASCII"));
/* Send the read data to the DataDispatcher Actor */
Main.getDataDispatcher().tell(data, ActorRef.noSender());
}
catch (IOException e) {
logger.debug("[run] ", e);
return;
}
}
else {// socketChannel is closed
try {
key.channel().close();// Sanitary close operation
return;
}
catch (IOException e) {
}
}
}
}
}
I can't figure out the overload on thread pool, any suggestions on implementation of ReadThread will help me.
UPDATE 1 : java.lang.OutOfMemoryError on fixed thread pool
Snippet of calling read event :
Thread per read:
try {
if (!key.isValid()) {
continue;
}
if (key.isAcceptable()) {
this.accept(key);
}
else if (key.isReadable()) {
new Thread(new ReadEvent(key)).start();
}
}
catch (CancelledKeyException e) {// key has been canceled
}
The above snippet works fine for few thousands of clients.
Using Thread pool
ExecutorService executor = Executors.newFixedThreadPool(100);
try {
if (!key.isValid()) {
continue;
}
if (key.isAcceptable()) {
this.accept(key);
}
else if (key.isReadable()) {
executor.execute(new ReadEvent(key));
}
}
catch (CancelledKeyException e) {// key has been canceled
}
The above snippet doesn't serves all clients and found the heap size is increasing gradually and most(almost 100%) of the CPU is used for GC and finally got java.lang.OutOfMemoryError: GC overhead limit exceeded exception
6.7 years later but hey I had a similar issue. The problem is until the socket is actually read [socketChannel.read(...)] the operation is deemed as valid.
By submitting the SelectionKey to the thread pool, remember that the thread will execute independently from the selecting thread meaning the operation is still ready as per the selecting thread and will keep submitting the operation to your thread pool until one of the submitted threads actually reads [socketChannel.read(...)] the channel and the operation is deemed as invalid and removed from the ready set.
Solution as mentioned by others already is read the bytes within the selecting thread and then pass the channel and the read contents into the thread pool for post processing.
case SelectionKey.OP_READ: {
SocketChannel channel = (SocketChannel) key.channel();
try {
ByteBuffer buffer = ByteBuffer.allocate(1024 * 8);
int read = channel.read(buffer);// read in selecting thread
if (read <= -1) {
key.cancel();//Signals the channel is closed
} else {
threadPool.execute(() -> {
//pass the channel & contents into the thread pool
});
}
} catch (IOException e) {
e.printStackTrace();
key.cancel();
}
}
break;
i've a memory leak problem on java Socket Object communication.
this is my send thread.
// create a new thread to send the packet
#Override
public synchronized void run() {
if(!genericSocket.isConnected()){
if(logger.isEnabled())
logger.logMessage(PFLogging.LEVEL_WARN, "Socket is close");
return;
}
int retry = 0;
boolean packetSent = false;
synchronized (objWriter) {
while ((retry < RETRY) && (!packetSent) && (genericSocket.isConnected())) {
try {
objWriter.writeObject(bean);
objWriter.flush();
// Try until the cache is reset and the memory is free
/*
boolean resetDone = false;
while(!resetDone) {
try {
objWriter.reset();
resetDone = true;
} catch (IOException r) {
Thread.sleep(1);
}
}
*/
// No error and packet sent
continuousError = 0;
packetSent = true;
} catch (Exception e) {
continuousError++;
if(logger.isEnabled())
logger.logMessage(PFLogging.LEVEL_ERROR, "Continuous Error [" + continuousError + "] sending message [" + e.getMessage() + "," + e.getCause() + "]");
// control the number of continuous errors
if(continuousError >= CONTINUOUS_ERROR) {
if(logger.isEnabled())
logger.logMessage(PFLogging.LEVEL_WARN, "I close the socket");
genericSocket.disconnect();
}
// next time is the time!
retry++;
}
}
}
}
the cache, when i sent about i packet per ms grow and grow!
if i add the commented part the cache is clean but when i need to send an async long message (about 3000 char) i see that the other message are lost!
There's another way to clean the cache without reset it??
ObjectOutputStream.reset() is not avoidable as it is the only means of clearing local hash tables, you can refer java source code for ObjectOutputStream for details of what happens in reset(), or else you will get OutOfMemoryError eventually
But you can very well implement a function like
private void writeObject(Object obj, ObjectOutputStream oos) throws IOException
{
synchronized(oos)
{
oos.writeObject(obj);
oos.flush();
oos.reset();
}
}
However you must ensure that all writes to ObjectOutputStream happens through this method.
the only solution i find is, first of starting a sending thread, to check if the thread pool is empty and in that case i reset the output stream.
I run the software all this night to check this.
Thanks all!
Matteo
I would use ObjectOutputStream.reset() periodically to clear the object cache for the stream.
You could even use it after sending every object. ;)
ciao :),
after ObjectOutputStream.flush() you can saftely use ObjectOutputStream.reset()
unless you are using the objWriter somwhere in another thread without using the synchronized (objWriter) statement.
In this case the best way IMHO is to use the objWriter in a thread, it will send object from a syncornized queue (see Queue sub-class http://docs.oracle.com/javase/1.5.0/docs/api/java/util/Queue.html, for example http://docs.oracle.com/javase/1.5.0/docs/api/java/util/concurrent/ConcurrentLinkedQueue.html) that is filled from the other thread (remeber to use object.clone(), because the objcet itself isn't syncornized it can be modified by other thread while you are writing it or is in queue! if you clone it your clone will be a safe copy).
That way you don't need synchronized statment because data-flow between thread and ObjectOutputStream is already synchronized, and you will be less error-prone