My java application has memory leaks - when my resources clearing code is executing task manager shows that memory usage wasn't changed. My code
while (isRunning) {
try
{
selector.select();
long sum=0;
Set keys = selector.selectedKeys();
Iterator it = keys.iterator();
while(it.hasNext())
{
SelectionKey key = (SelectionKey)it.next();
if (key.isReadable())
{
SocketChannel sc = (SocketChannel) key.channel();
ByteBuffer bb;
if(key.attachment()==null)
{
bb = ByteBuffer.allocate(1024*1024);
key.attach(bb);
}
else
{
bb = (ByteBuffer)key.attachment();
bb.clear();
}
int x = sc.read(bb);
System.out.println(x +" bytes were read");
if(x==-1)
{
key.attach(null); //doesn't work
sc.close();
//bb = null; // also doesn't work
}
}
}
keys.clear();
}
catch (Exception ex)
{
ex.printStackTrace(new PrintStream(System.out));
}
finally
{
//stopServer();
}
}
Testing logic - I wrote simple TCP client java programm sending 100 messages to server. I intentionally allocated large buffer - 1MB for each connection. When client finishes his job int x = sc.read(bb); returns -1 and the following code is executed:
if(x==-1)
{
key.attach(null); //doesn't work
sc.close();
//bb = null; // also doesn't work
}
I checked it with debug output, this code was really executed but task manager still showes large memory usage. where is the problem?
Certainly key.attach(null) works. If it didn't, attaching a non-null object wouldn't work either. Same code.
But, in any case, closing the SocketChannel cancels the key, which removes it from all key sets of all Selectors it was registered with, so you will never see the key again anyway, so it becomes eligible for GC, and so does the attachment, regardless of whether you call key.attach(null) or not, which is therefore redundant. Either you have another reference to your attachment somewhere else, or your memory usage problem is elsewhere.
Related
First, I'm not a developer (and I've been coding only for 2 weeks), so feel free to tell me I'm completely misunderstanding the thing (also, I wrote all of this for myself, so I'm sure it's super not cool) :). I want to learn and get it right, so I'm keen to listen to suggestions or complete rewrites.
I want to connect to a socket in non-blocking mode (I'm the client, not the server). I'll mainly need to read from it, but sometimes I'll need to write to it, too. The procedure is as follows:
Connect to socket
Send some initial requests to login to the server
Read from the socket
Sometimes, write some stuff (subscribe to certain information, for example)
My solution is as follows (I'm writing it in Java, because I've read it's a fast and good programming language, but I'm happy to change if required... hopefully not needed though!):
public class SocketClient {
public static void main(String[] args) {
new Feed().init();
}
private boolean isSocketConnected() {
return socket != null && socket.isConnected();
}
public void init() {
try {
if (isSocketConnected()) {
// What here if I'm in non-blocking mode?
// Would be good to know if the "close API" request succeeded
// otherwise next time I won't be able to connect to their socket...
sendCloseRequestToApi();
socket.close();
}
run();
} catch (Exception e) {
if (isSocketConnected()) {
// Same question as above...
sendCloseRequestsToApi();
socket.close();
}
}
}
public void run() throws IOException {
System.out.println("Starting connection in blocking mode...");
SocketChannel channel = SocketChannel.open();
socket = channel.socket();
socket.setReceiveBufferSize(RECEIVE_BUFFER_SIZE);
socket.setSendBufferSize(SEND_BUFFER_SIZE);
channel.connect(new InetSocketAddress("127.0.0.1", 2121));
channel.finishConnect();
System.out.println("Finished connecting in blocking mode");
// Writes to the socket (user and password)
initialiseTheApi();
System.out.println("Sent API requests in blocking mode");
System.out.println("Now we should probably go non-blocking (I guess)");
channel.register(selector, SelectionKey.OP_WRITE | SelectionKey.OP_READ);
selector = Selector.open();
channel.configureBlocking(false);
System.out.println("Selector created and switched to non-blocking mode...");
long timeWithoutData = 0;
boolean needsReconnection = false;
while (!needsReconnection) {
selector.select();
Iterator < SelectionKey > keys = selector.selectedKeys().iterator();
while (keys.hasNext()) {
SelectionKey key = keys.next();
keys.remove();
if (!key.isValid()) {
continue;
}
if (key.isWritable()) {
// Execute write...
// What if I need to know the result to the write operation?
}
if (key.isReadable()) {
int dataRead = readDataFromSocket(buffer);
buffer.flip();
if (buffer.remaining() > 0) {
// I process the data read here,
// but sometimes the data sent is
// "reconnect to API". So I need to close
// the connection and start again.
// How can I do that if I'm in non-blocking mode?
// I mean, I need to make sure when I send that request
// (for reconnection).
// I need to know that the request got to the server and
// was processed OK before moving on and
// reading/writing again...
}
if (dataRead > -1) {
timeWithoutData = 0;
} else {
if (timeWithoutData > 0) {
long diffInMillis = System.currentTimeMillis() - timeWithoutData;
if (diffInMillis > 2000) {
System.out.println("Timeout or something? I need to reconnect I think");
needsReconnection = true;
}
} else {
timeWithoutData = System.currentTimeMillis();
}
}
// Do I even need this? Already did it before, right?
key.interestOps(SelectionKey.OP_READ | SelectionKey.OP_WRITE);
}
}
}
if (needsReconnection) {
// We need full reconnection, go back up and reconnect
init();
}
}
}
I removed imports and other non-useful methods for convenience, and to keep the post short.
As you can see in my questions in the code (plus some added ones):
Reconnection: If I'm in non-blocking mode, how do I know that my request got sent successfully to the server
If I read from the socket and the message is "Reconnect to API", how can I make sure that happens before any other read / write?
Do I need to send the interestedOps over and over again?
I should only connect once to the socket. The fact that I'm non-blocking doesn't change that, right?
I've seen this could all be simplified using Netty or something, but I'm already bloated with so much stuff! :(
I hope my questions are clear. Let me know otherwise, please.
Thanks a lot.
I was trying to do something that just didn't make sense. In my case I can definitely use a blocking connection, which I just didn't know about :/. Internet is a bad source of information sometimes! I kept reading over here not to use a blocking connection :D. But now it makes perfect sense the different scenarios. – Will
I'm receiving a continuous stream of data that I'm saving to a ByteBuffer.
Sometimes I need to write to the channel, however, it's important not to lose any data. Is it possible to use the selector to solve this issue?
If I'm constantly checking the selector for the channel state, it always says that the channel is currently reading and it's like there is no opportunity to perform writing. I can't use multiple connections because the server doesn't support it.
this.socketChannel = SocketChannel.open();
this.socketChannel.configureBlocking(false);
this.socketChannel.connect(new InetSocketAddress(IP, this.port));
try {
this.selector = Selector.open();
int interestSet = SelectionKey.OP_READ | SelectionKey.OP_WRITE;
SelectionKey selectionKey = this.socketChannel.register(selector, interestSet);
while (selector.select() > -1) {
// Wait for an event one of the registered channels
// Iterate over the set of keys for which events are available
Iterator selectedKeys = selector.selectedKeys().iterator();
while (selectedKeys.hasNext()) {
SelectionKey key = (SelectionKey) selectedKeys.next();
selectedKeys.remove();
try {
if (!key.isValid()) {
continue;
} else if (key.isReadable()) {
System.out.println("readable");
} else if (key.isWritable()) {
System.out.println("writable");
}
} catch (Exception e) {
e.printStackTrace();
}
}
}
} catch (IOException e) {
e.printStackTrace();
}
Edit:
Sorry I didn't add more info. This is an important bit of my code. It always prints "readable" to the console and I was hoping that isWritable block also gets executed.
Thanks in advance, Honza
You are using else if operator so if your key is readable checking for if it is writeable will not be performed, but it doesn't mean that the channel is not writeable.
Actually it could be readable and writeable in the same time. But in your program if it is readable you just don't check for writeable.
replace else-if with if and see the result.
i've a memory leak problem on java Socket Object communication.
this is my send thread.
// create a new thread to send the packet
#Override
public synchronized void run() {
if(!genericSocket.isConnected()){
if(logger.isEnabled())
logger.logMessage(PFLogging.LEVEL_WARN, "Socket is close");
return;
}
int retry = 0;
boolean packetSent = false;
synchronized (objWriter) {
while ((retry < RETRY) && (!packetSent) && (genericSocket.isConnected())) {
try {
objWriter.writeObject(bean);
objWriter.flush();
// Try until the cache is reset and the memory is free
/*
boolean resetDone = false;
while(!resetDone) {
try {
objWriter.reset();
resetDone = true;
} catch (IOException r) {
Thread.sleep(1);
}
}
*/
// No error and packet sent
continuousError = 0;
packetSent = true;
} catch (Exception e) {
continuousError++;
if(logger.isEnabled())
logger.logMessage(PFLogging.LEVEL_ERROR, "Continuous Error [" + continuousError + "] sending message [" + e.getMessage() + "," + e.getCause() + "]");
// control the number of continuous errors
if(continuousError >= CONTINUOUS_ERROR) {
if(logger.isEnabled())
logger.logMessage(PFLogging.LEVEL_WARN, "I close the socket");
genericSocket.disconnect();
}
// next time is the time!
retry++;
}
}
}
}
the cache, when i sent about i packet per ms grow and grow!
if i add the commented part the cache is clean but when i need to send an async long message (about 3000 char) i see that the other message are lost!
There's another way to clean the cache without reset it??
ObjectOutputStream.reset() is not avoidable as it is the only means of clearing local hash tables, you can refer java source code for ObjectOutputStream for details of what happens in reset(), or else you will get OutOfMemoryError eventually
But you can very well implement a function like
private void writeObject(Object obj, ObjectOutputStream oos) throws IOException
{
synchronized(oos)
{
oos.writeObject(obj);
oos.flush();
oos.reset();
}
}
However you must ensure that all writes to ObjectOutputStream happens through this method.
the only solution i find is, first of starting a sending thread, to check if the thread pool is empty and in that case i reset the output stream.
I run the software all this night to check this.
Thanks all!
Matteo
I would use ObjectOutputStream.reset() periodically to clear the object cache for the stream.
You could even use it after sending every object. ;)
ciao :),
after ObjectOutputStream.flush() you can saftely use ObjectOutputStream.reset()
unless you are using the objWriter somwhere in another thread without using the synchronized (objWriter) statement.
In this case the best way IMHO is to use the objWriter in a thread, it will send object from a syncornized queue (see Queue sub-class http://docs.oracle.com/javase/1.5.0/docs/api/java/util/Queue.html, for example http://docs.oracle.com/javase/1.5.0/docs/api/java/util/concurrent/ConcurrentLinkedQueue.html) that is filled from the other thread (remeber to use object.clone(), because the objcet itself isn't syncornized it can be modified by other thread while you are writing it or is in queue! if you clone it your clone will be a safe copy).
That way you don't need synchronized statment because data-flow between thread and ObjectOutputStream is already synchronized, and you will be less error-prone
I have a simple non-blocking server with main loop:
try {
while (selector.select() > -1) {
// Wait for an event one of the registered channels
// Iterate over the set of keys for which events are available
Iterator selectedKeys = selector.selectedKeys().iterator();
while (selectedKeys.hasNext()) {
SelectionKey key = (SelectionKey) selectedKeys.next();
selectedKeys.remove();
try {
if (!key.isValid()) {
continue;
}
if (key.isConnectable()) {
connect(key);
}
// Check what event is available and deal with it
if (key.isAcceptable()) {
accept(key);
}
if (key.isReadable()) {
read(key);
}
if (key.isWritable()) {
write(key);
}
} catch (Exception e) {
e.printStackTrace();
close(key);
}
}
}
} catch (Exception e) {
e.printStackTrace();
}
In read/write section I check if there is something to read/write if not - then I try to close channel:
if (channel.read(attachment.buffer) < 1)
close(key);
Close method:
private void close(SelectionKey key) throws IOException {
key.cancel();
key.channel().close();
}
But during processing this code I get exception in main loop (it is catched but I supposed something wrong) I get this stacktrace:
java.nio.channels.CancelledKeyException
at sun.nio.ch.SelectionKeyImpl.ensureValid(Unknown Source)
at sun.nio.ch.SelectionKeyImpl.readyOps(Unknown Source)
at java.nio.channels.SelectionKey.isWritable(Unknown Source)
So it fails on main loop when enter write section, close channel and came back to main loop in 'writable' if section and fails with such exception. Any suggestions?
The error is very simple.
if (!key.isValid()) {
continue;
}
if (key.isConnectable()) {
connect(key);
}
// Check what event is available and deal with it
if (key.isAcceptable()) {
accept(key);
}
if (key.isReadable()) {
read(key);
}
if (key.isWritable()) {
write(key);
}
Your read method is the one which cancels the SelectionKey. However, after returning from read, you again test the key for whether the channel is writable -- potentially after just cancelling that very same key! Your initial check cannot help here.
One solution would be to check for whether the key is valid wherever it might've just been cancelled:
...
if (key.isValid() && key.isWritable()) {
write(key);
}
...
Alternatively, you could also try only registering one interest at a time as you need to on any particular channel, and thus all readiness events are mutually exclusive:
if (!key.isValid()) {
continue;
}
if (key.isConnectable()) {
connect(key);
} else if (key.isAcceptable()) {
accept(key);
} else if (key.isReadable()) {
read(key);
} else if (key.isWritable()) {
write(key);
}
This might be beneficial in situations; as generally a channel will almost always be write-ready, keeping an interest in write-readiness along side read-readiness might keep the Selector loop spinning, which is more than likely not desirable. For the most part, generally register interest in write-readiness only when the underlying socket output buffer is full.
As a side note, know that SocketChannel.read can return a value < 1 without it being an error.
A read operation might not fill the buffer, and in fact it might not read any bytes at all. Whether or not it does so depends upon the nature and state of the channel. A socket channel in non-blocking mode, for example, cannot read any more bytes than are immediately available from the socket's input buffer;
Additionally, Selector.select does not state anything about returning < -1 to indicate it is closed.
Returns: The number of keys, possibly zero, whose ready-operation sets were updated
How can i solve this problem. I got following error:
java.nio.channels.ClosedChannelException
This is coding:
public void run() {
try {
SocketChannel socketChannel = (SocketChannel) key.channel();
ByteBuffer buffer = ByteBuffer.allocate(512);
int i1 = socketChannel.read(buffer);
if (buffer.limit() == 0 || i1 == -1) {
Socket s = null;
try {
s = socketChannel.socket();
s.close();
key.cancel();
} catch (IOException ie) {
if (UnitDataServer.isLog) {
log.error("Error closing socket " + s + ": " + ie);
}
}
} else {
buffer.flip();
if (UnitDataServer.isLog) {
log.info(" Recvd Message from Unit : " + buffer.array());
}
byte byteArray[] = buffer.array();
log.info("Byte Array length :" + byteArray.length);
hexString = new StringBuffer();
for (int i = 0; i < i1 /* byteArray.length */; i++) {
String hex = Integer.toHexString(0xFF & byteArray[i]);
if (hex.length() == 1) {
// could use a for loop, but we're only dealing with a
// single byte
hexString.append('0');
}
hexString.append(hex);
}
hexString.trimToSize();
log.info("Hex String :" + hexString);
Communicator.dataReceive(new DataReceive(
socketChannel, hexString.toString(), dst));
}
} catch (Exception e) {
if (UnitDataServer.isLog) {
// log.error(e);
}
try {
socketChannel.socket().close();
key.cancel();
} catch (IOException ex) {
if (UnitDataServer.isLog) {
log.error(ex);
}
}
}
}
You have closed the channel and are still trying to use it.
There are several issues with your code.
First, your test for EOS is faulty. Remove the limit() == 0 test. That doesn't indicate EOS, it just indicates a zero length read, which can happen in non-blocking mode at any time. It doesn't mean the peer has closed his end of the connection, and it doesn't mean you should close your end.
Second, closing a channel closes the socket as well. You should close the channel only, not the socket.
Third, closing a channel cancels the key. You don't need to follow every close with a cancel.
You may also have failed to check whether a ready key is valid in the select loop before using it, e.g. for reading.
I continue to be amazed, and amused, and bemused, by the claim elsewhere in this thread that 'source code is untrue' under some circumstances.
You need to fix/secure code that is throwing this exception. ClosedChannelException is ...
... thrown when an
attempt is made to invoke or complete
an I/O operation upon channel that is
closed, or at least closed to that
operation. That this exception is
thrown does not necessarily imply that
the channel is completely closed. A
socket channel whose write half has
been shut down, for example, may still
be open for reading
(as described in Java 6 API)
But really, you would need to provide us code snipped and stack trace in order to get more detailed help.