Develop Reference

Memory Consumption by Java Applet

In my applet I have GET call to download file from a remote location. When I am trying to download some large file of around 13MB, then my Applet memory consumption is increasing more than 50MB. I am using the below code to get my memory consumption:
public static long getMemoryUsage()
{
long memory = 0;
// Get the Java runtime
Runtime runtime = Runtime.getRuntime();
memory = runtime.totalMemory() - runtime.freeMemory();
return memory;
}
Code for my get call is
public void getFiles(String filePath, long fileSize)throws MyException
{
InputStream objInputStream = null;
HttpURLConnection conn = null;
BufferedReader br = null;
try
{
URL fileUrl=new URL(filePath);
final String strAPICall=fileUrl.getPath();
final String strHost="some.test.com";
final int iPort=1000;
URL url = null;
url = new java.net.URL
( "https",
strHost, iPort , "/" + strAPICall,
new myHandler() );
conn = (HttpURLConnection)new HttpsURLConn(url);
conn.setRequestMethod("GET");
conn.connect();
if (conn.getResponseCode() != 200) {
objInputStream=conn.getInputStream();
br = new BufferedReader(new InputStreamReader(
(objInputStream)));
String output;
while ((output = br.readLine()) != null) {
System.out.println(output);
}
throw new MyException("Bad response from server",
MyError.BAD_RESPONSE_ERROR);
}
else
{
notifyProgressToObservers(0);
System.out.println("conn.getResponseCode()"+conn.getResponseCode());
System.out.println("conn.getResponseMessage()"+conn.getResponseMessage());
objInputStream = conn.getInputStream();
int count=objInputStream.available();
System.out.println("Stream size: "+count);
System.out.println("fileSize size: "+fileSize);
byte []downloadedData = getBytesFromInputStream
(objInputStream, count,fileSize);
notifyChunkToObservers(downloadedData);
notifyIndivisualFileEndToObservers(true, null);
}
}
catch (MyException pm)
{
throw new MyException
(pm, MyError.CONNECTION_TIMEOUT);
}
catch (IOException pm)
{
throw new MyException
(pm, MyError.CONNECTION_TIMEOUT);
}
catch (Exception e)
{
notifyIndivisualFileEndToObservers(false,new MyException(e.toString()));
}
finally
{
System.out.println("Closing all the streams after getting file");
if(conn !=null)
{
try
{
conn.disconnect();
}
catch(Exception e)
{
}
}
if(objInputStream != null)
{
try {
objInputStream.close();
} catch (IOException e) {
}
}
if (br != null)
{
try {
br.close();
} catch (IOException e) {
}
}
}
}
In the above method, I tried putting the log for memory consumption after each line and found that after conn.connect();, the memory consumption of applet increases by atleast 50MB even though the file I am trying to download is only 13MB.
Is there any memory leak anywhere?
EDIT: Added Implementation for getBytesFromInputStream()
public byte[] getBytesFromInputStream(InputStream is, int len, long fileSize)
throws IOException
{
byte[] readBytes= new byte[8192];
ByteArrayOutputStream getBytes= new ByteArrayOutputStream();
int numRead = 0;
while ((numRead = is.read(readBytes)) != -1) {
getBytes.write(readBytes, 0, numRead);
}
return getBytes.toByteArray();
}

it's because of this line:
byte []downloadedData = getBytesFromInputStream(objInputStream, count,fileSize);
here you are reading the complete amount of bytes of file into the heap. After that you need to track down what happens with this array. Maybe you are copying it somewhere and the GC needs some time to kick in even if you do not use the reference to the object anymore.
Large files should never be read completly to memory, but rather streamed directly to some processor of the data.

The only way to optimize getBytesFromInputStream() is if you know beforehand exactly how many by bytes there are to read. Then you allocate a byte[] of the required size, and read from the input directly into the byte[]. For example:
byte[] buffer = new byte[len];
int pos = 0;
while (pos < len) {
int nosRead = is.read(buffer, pos, len - pos);
if (nosRead == -1) {
throw new IOException("incomplete response");
}
pos += nosRead;
}
return buffer;
(For more information, read the javadoc.)
Unfortunately, your (apparent) attempt at getting the size is incorrect.
int count = objInputStream.available();
This doesn't return the total number of bytes that can be read from the stream. It returns the number of bytes that can be read right now without the possibility of blocking.
If the server is setting the Content-Length header in the response, then you could use that; call getContentLength() (or getContentLengthLong() in other use-cases) once you have the response. But be prepared for the case where that gives you -1.

Is try-catching actually this clunky in Java or am I just not aware of the elegant approach?

public void lock() {
if (this.isLocked()) return;
try {
this.dataOut.flush();
this.dataOut.close();
} catch (IOException e) {
throw new RuntimeException(e);
}
DataInputStream inputStream =
new DataInputStream(
new BufferedInputStream(
new ByteArrayInputStream(
this.byteOut.toByteArray())));
IntStream.Builder intStreamBuilder = IntStream.builder();
try {
try {
while (true) {
intStreamBuilder.accept(inputStream.readInt());
}
} catch (EOFException e) {
// logic to be executed after stream has been fully read
int[] pool = intStreamBuilder.build().toArray();
super.lock(pool);
} finally {
inputStream.close();
}
} catch (IOException e) {
throw new RuntimeException(e);
}
}
What I do here is take an DataOutputStream containing Integers, flush its remaining contents into a ByteArrayOutputStream named this.byteOut and then build an IntStream from it.
I'm from the C# domain and in the process of learning Java, so the code here does not have any actual purpose.
Is there any way to do what I do here more elegantly in Java?
My two main concerns are:
The way I determine that the DataInputStream has been fully read is by catching an EOFException and putting the logic to be executed after reading inside a catch block. I don't like that, since I suppose throwing and catching exceptions is somewhat expensive? Is there a better way to determine that the stream doesn't contain any more Integers?
The fact that I have to wrap a try-catch block around a try-catch block just to be able to call inputStream.close() in the inner finally block. Is there a solution that is not so clunky?

It's mostly you.
If you don't like the try with resources construct,
you can still combine all of your try statments and stack the catch blocks.
public void lock()
{
DataInputStream inputStream = null;
IntStream.Builder intStreamBuilder;
if (isLocked())
{
return;
}
try
{
inputStream = new DataInputStream(
new BufferedInputStream(
new ByteArrayInputStream(
byteOut.toByteArray())));
intStreamBuilder = IntStream.builder();
dataOut.flush();
dataOut.close();
while (true)
{
intStreamBuilder.accept(
inputStream.readInt());
}
}
catch (IOException exception)
{
throw new RuntimeException(exception);
}
catch (EOFException ignoredException)
{
// logic to be executed after stream has been fully read
int[] pool = intStreamBuilder.build().toArray();
super.lock(pool);
}
finally
{
if (inputSream != null)
{
try
{
inputStream.close();
}
catch (IOException exception)
{
throw new RuntimeException(exception);
}
}
}
}
The try inside the finally is required.
I prefer the try-with-resources construct.

If I were you I will change my code to be as the following
//try-with-resources-close
try (DataInputStream inputStream =
new DataInputStream(
new BufferedInputStream(
new ByteArrayInputStream(
this.byteOut.toByteArray())))) {
IntStream.Builder intStreamBuilder = IntStream.builder();
byte[] byts = new byte[4];
while (inputStream.read(byts) > -1) {// read 4 bytes and convert them to int
int result = ByteBuffer.wrap(byts).getInt();
intStreamBuilder.accept(result);
}
// logic to be executed after stream has been fully read
int[] pool = intStreamBuilder.build().toArray();
super.lock(pool);
} catch (IOException e) {
throw new RuntimeException(e);
}
try-with-resources this new way added in Java 7 to close any object that implements AutoCloseable automatically when try block is executed
readInt method it works as the following read 4 bteys then convert them to int
int ch1 = in.read();
int ch2 = in.read();
int ch3 = in.read();
int ch4 = in.read();
if ((ch1 | ch2 | ch3 | ch4) < 0)
throw new EOFException();
return ((ch1 << 24) + (ch2 << 16) + (ch3 << 8) + (ch4 << 0));
//So you also can do the same thing, but instead of throws EOFException
you can add return false or break the loop
for read bytes method if no bytes existed then it reture -1

Well the better solution I found thanks to #Oliver Charlesworth is the following:
try (DataInputStream inputStream =
new DataInputStream(
new BufferedInputStream(
new ByteArrayInputStream(this.byteOut.toByteArray())))) {
while (true)
intStreamBuilder.accept(inputStream.readInt());
} catch (EOFException e) {
int[] pool = intStreamBuilder.build().toArray();
super.lock(pool);
} catch (IOException e) {
throw new RuntimeException(e);
}
This still has logic inside the catch block the code look definitely cleaner.
However, I cannot come up with a better approach that determines that the InputDataStream has been fully read.
What bugs me about this is that reaching the end of the stream is expected and it would actually be exceptional if no exception was thrown, what IMO defeats the purpose of exceptions in the first place.
I asked a separate question of the possible use of Java NIO's IntBuffer class here. My above snippet could be changed to:
IntBuffer intBuffer = ByteBuffer.wrap(this.byteOut.toByteArray()).asIntBuffer();
while (intBuffer.hasRemaining()){
intStreamBuilder.accept(intBuffer.get());
}
int[] pool = intStreamBuilder.build().toArray();
super.lock(pool);

End of File Exception on ObjectInputStream.readObject

My application streams twitter data and writes them to files.
while(true){
Status status = queue.poll();
if (status == null) {
Thread.sleep(100);
}
if(status!=null){
list.add(status);
}
if(list.size()==10){
FileOutputStream fos = null;
ObjectOutputStream out = null;
try {
String uuid = UUID.randomUUID().toString();
String filename = "C:/path/"+topic+"-"+uuid+".ser";
fos = new FileOutputStream(filename);
out = new ObjectOutputStream(fos);
out.writeObject(list);
tweetsDownloaded += list.size();
if(tweetsDownloaded % 100==0)
System.out.println(tweetsDownloaded+" tweets downloaded");
// System.out.println("File: "+filename+" written.");
out.close();
} catch (IOException e) {
e.printStackTrace();
}
list.clear();
}
I have this code which gets data from files.
while(true){
File[] files = folder.listFiles();
if(files != null){
Arrays.sort(//sorting...);
//Here we manage each single file, from data-load until the deletion
for(int i = 0; i<files.length; i++){
loadTweets(files[i].getAbsolutePath());
//TODO manageStatuses
files[i].delete();
statusList.clear();
}
}
}
The method loadTweets() does the following operations:
private static void loadTweets(String filename) {
FileInputStream fis = null;
ObjectInputStream in = null;
try{
fis = new FileInputStream(filename);
in = new ObjectInputStream(fis);
statusList = (List<Status>) in.readObject();
in.close();
}
catch(IOException | ClassNotFoundException ex){
ex.printStackTrace();
}
}
Unfortunately, I don't know why sometimes it throws a
EOFException
when running this line
statusList = (List<Status>) in.readObject();
Anybody knows how I can solve this? Thank you.

I've seen that you're passing the file correctly with the getAbsolutePath() based on a previous question of yours
From what I've read that can be a couple of things, one of them the file being null.
Explaining this idea, you might have written the file but something caused the file to have nothing inside, this might cause an EOFException. The file in fact exists it's just empty
EDIT
Try to enclose the code in while(in.available() > 0)
It would look like this
private static void loadTweets(String filename) {
FileInputStream fis = null;
ObjectInputStream in = null;
try{
fis = new FileInputStream(filename);
in = new ObjectInputStream(fis);
while(in.available() > 0) {
statusList = (List<Status>) in.readObject();
}
in.close();
}
catch(IOException | ClassNotFoundException ex){
ex.printStackTrace();
}
}

Found out what was necessary to solve this. Thanks to #VGR's comment, I thought to pause the executing thread for 0.2 seconds if the file has been created less than a second ago.
if(System.currentTimeMillis()-files[i].lastModified()<1000){
Thread.sleep(200);
This prevents the exception and the application works now fine.

EOFexception caused by empty file

I created a new file roomChecker which is empty. Now when I read it, it throws me an EOFException which is undesirable. Instead I want it to see that, if file is empty then it would run other two functions that are in if(roomFeed.size() == 0) condition. I could write this statement in EOFException catch clause; but that's not what I want to do because then every time when the file will be read and reaches end of file it will execute those functions. Instead when the file has some data it should do what is specified in else.
File fileChecker = new File("roomChecker.ser");
if(!fileChecker.exists()) {
try {
fileChecker.createNewFile();
} catch (IOException e) {
e.printStackTrace();
System.out.println("Unable to create new File");
}
}
try(FileInputStream fis = new FileInputStream("roomChecker.ser"); ObjectInputStream ois = new ObjectInputStream(fis)) {
roomFeed = (List<roomChecker>) ois.readObject();
System.out.println("End of read");
if(roomFeed.size() == 0) {
System.out.println("your in null if statement");
defaultRoomList();
uploadAvailableRooms();
} else {
for(int i=0; i<roomNumber.size(); i++) {
for(int j=0; j<roomFeed.size(); i++) {
if((roomNumber.get(i)).equals(roomFeed.get(i).getRoomNumSearch())){
System.out.println("Reach Dead End for now");
} else {
defaultRoomList();
uploadAvailableRooms();
}
}
}
}
} catch (IOException ioe) {
ioe.printStackTrace();
} catch (ClassNotFoundException e) {
e.printStackTrace();
}

All this:
if(!fileChecker.exists()) {
try {
fileChecker.createNewFile();
} catch (IOException e) {
e.printStackTrace();
System.out.println("Unable to create new File");
}
}
is a complete waste of time, and it is one of two possible causes for your empty file problem. Creating a file just so you can open it and then get a different problem instead of coping correctly with the original problem of the file not being there isn't a rational strategy. Instead, you should do this:
if (fileChecker.isFile() && fileChecker.length() == 0) {
// file is zero length: bail out
}
and, in the following code, this:
try(FileInputStream fis = new FileInputStream(fileChecker); ObjectInputStream ois = new ObjectInputStream(fis)) {
// ...
}
catch (FileNotFoundException exc) {
// no such file ...
}
// other catch blocks as before.
Of course you can still get EOFException if you read the file to its end, or if the file is incomplete, and you still need to handle that.

The gzip compressor is not going with the while loop

I am trying to query from database with Java JDBC and compress data in one column to gzip file in specific directory. I have tested my JDBC query and it working fine, but the Gzip code not going with the while loop, it's run with the loop firt row and stuck there. Why it's stuck? help me please!
These folders already existed: D:\Directory\My\year\id1\id2
//Some query JDBC code here, it's work well. I query all rows Data, year, id1,id2,id3
while (myRs1.next()){
String str = Data;
File myGzipFile = new File("D:\\Directory\\My\\"+year+"\\"+id1+"\\"+id2+"\\"+id3+".gzip");
GZIPOutputStream gos = null;
InputStream is = new ByteArrayInputStream(str.getBytes());
gos = new GZIPOutputStream(new FileOutputStream(myGzipFile));
byte[] buffer = new byte[1024];
int len;
while ((len = is.read(buffer)) != -1) {
gos.write(buffer, 0, len);
System.out.print("done for:"+id3);
}
try { gos.close(); } catch (IOException e) { }
}

Try formatting the source like this to catch exceptions.
public class InputStreamDemo {
public static void main(String[] args) throws Exception {
InputStream is = null;
int i;
char c;
try{
is = new FileInputStream("C://test.txt");
System.out.println("Characters printed:");
// reads till the end of the stream
while((i=is.read())!=-1)
{
// converts integer to character
c=(char)i;
// prints character
System.out.print(c);
}
}catch(Exception e){
// if any I/O error occurs
e.printStackTrace();
}finally{
// releases system resources associated with this stream
if(is!=null)
is.close();
}
}
}