Develop Reference

sevenzipjbinding: How do I properly close file handles as they become unneeded?

We have an application that uses sevenzipjbinding to bundle and transport content. 99% of the time, everything goes smoothly, however there are some edge cases where these zip files contain upwards of a hundred thousand files. When this happens, the number of file handles that the process has a hold on gets too high and then things start to fail.
I haven't been able to find a recommended way to handle this issue. Currently, we create a RandomAccessFileInStream upon request using an instance of IOutCreateCallback, but these never get closed until the very end. There is no apparent method in the interface designed to signify when a stream should be closed.
Currently, my only idea is to keep track of the "last" file that was requested and then every time a new one is requested to close the last file. If I do this though, I'm worried that I risk that 7zip wants to read the file a second time.
Is this a bug in my code, or is it in sevenzipjbinding?
Current implementation:
public ISequentialInStream getStream(int i) throws SevenZipException {
ISequentialInStream is = null;
try {
AbstractItem item = getItem(i);
if (item instanceof FileItem) {
is = new RandomAccessFileInStream(((FileItem) item).getRandomAccessFile());
} else if (item instanceof StreamItem) {
is = new InputStreamSequentialInStream(((StreamItem) item).getInputStream());
}
} catch (IOException e) {
throw new SevenZipException("IO Error!", e);
} finally {
closeables.add(is);
}
return is;
}

Java: Multithreaded complex Getter

I have a get-method that Returns a non-primitive type, and should be thread-safe.
Functional description
I have an Operation working on a separate thread. This Operation Needs the IP-Address and gets it from an other class using my own getter-method (because I have to do some IP-Address calculation). The getter Returns an InetAddress.
Code example
// This is called in a separate thread.
// How can I make Utility.getMyIp() thread-safe?
InetAddress myIp = Utility.getMyIp();
Question
What do I have to note about the IP-Address getter method? I need to use the method from an other thread. Do I have to make it synchronized?
If you need more informations, please ask!
PS: There are other threads asking similar questions, but the ones I found talk only about primitive datatypes, on which should be enough to use the volatile Keyword. Here I need return a type InetAddress.
EDIT: Added Code
This is work in Progress (it may not work in every case and is not finished-code).
/**
* Gets the BroadcastAddress for IPv4.
* #return
* #throws SocketException
*/
public static InetAddress getBroadcastAddress4() {
// Todo: Check other possibility
System.setProperty("java.net.preferIPv4Stack" , "true");
// Init
InetAddress broadcastAddress = null;
try {
Enumeration<NetworkInterface> interfaces = NetworkInterface.getNetworkInterfaces();
while(interfaces != null && interfaces.hasMoreElements()){
NetworkInterface currentInterface = interfaces.nextElement();
if(!currentInterface.isLoopback()){
for(InterfaceAddress address : currentInterface.getInterfaceAddresses()){
InetAddress broadcast = address.getBroadcast();
if(broadcast != null){
broadcastAddress = broadcast;
break;
}
}
}
}
} catch (SocketException e) {
e.printStackTrace();
}
return broadcastAddress;
}

The code is thread safe without any further modifications.
You could also store the result of this method to a variable and do the actual computation once - as the results are unlikely to change during the runtime of your program. Initialize the variable lazily or use a static {} code block to initialize it on classload. The code could look like this:
InetAddress broadcastAddress;
public static InetAddress getBroadcastAddress4() {
return broadcastAddress;
}
static {
System.setProperty("java.net.preferIPv4Stack" , "true");
try {
Enumeration<NetworkInterface> interfaces = NetworkInterface.getNetworkInterfaces();
while(interfaces != null && interfaces.hasMoreElements()){
NetworkInterface currentInterface = interfaces.nextElement();
if(!currentInterface.isLoopback()){
for(InterfaceAddress address : currentInterface.getInterfaceAddresses()){
InetAddress broadcast = address.getBroadcast();
if(broadcast != null){
broadcastAddress = broadcast;
break;
}
}
}
}
} catch (SocketException e) {
// log the damn exception!
Logger.getLogger(YourClass.class).error("exception when getting broadcast address", e);
}
}

If the object which implements your get method exists in a single instance, a synchronized keyword before the getter method declaration will be enough.
You may also be able to use a synchronized collection (like ConcurrentHashMap) to store and access your data.
It is possible, however, that there exists a more efficient solution. As with all multithreaded synchronizations, you have to identify and isolate a critical section. Usually the shorter the critical section, the better the speed of the code.

As far as I can see, your code is Thread-safe so no additional work is needed here. You are not using shared mutable resources that can be shared along threads so should be no problem here.

Java writes writes line multiple times?

static void goOut(String in) {
//instance variables
String fileCopy = currentLine + in;
try {
FileWriter writer = new FileWriter(output,true);
writer.write(line1 + System.getProperty("line.separator", "\r\n"));
writer.write(fileCopy + System.getProperty("line.separator", "\r\n"));
} catch(IOException ex) {
ex.printStackTrace();
}
}
Edited code to the correct standard as pointed out by other users.

of course because thats what you r telling it to do. every time is called it writes both x and the number. a quick fix: you can keep a flag if it is the first run set it flag = true. and check within ur method, sth like this:
public class YourClass{
private boolean didRun = false;
static void goOut(String in) {
...... init ur file and writer
if(!didRun)
writer.write(Y);
writer.write(in);
writer.close();
didRun = true;
}
}
I dont know the rest of the code but i think thats what u need

I believe you want to separate the jobs the "goOut" is responsible for.
You should make "goOut" only write the numbers (in your example).
The writing of the y's (in your example) should not be apart of the method and called once, at the start of writing to the file.
Also, #Jon Skeet is right about the multiple FileWriters. Use one, since its the same file.

Agree, sounds like a disaster.
When you use multiple writers to access the file, I would expect to get unpredictable results.
I dont think there is any guarantee that FileWriter1 would complete the task before FileWriter2.
In addition, the method is not synchronized.

Java ConcurrentHashMap corrupt values

I have a ConcurrentHashMap that exhibits strange behavior on occasion.
When my app first starts up, I read a directory from the file system and load contents of each file into the ConcurrentHashMap using the filename as the key. Some files may be empty, in which case I set the value to "empty".
Once all files have been loaded, a pool of worker threads will wait for external requests. When a request comes in, I call the getData() function where I check if the ConcurrentHashMap contains the key. If the key exists I get the value and check if the value is "empty". If value.contains("empty"), I return "file not found". Otherwise, the contents of the file is returned. When the key does not exist, I try to load the file from the file system.
private String getData(String name) {
String reply = null;
if (map.containsKey(name)) {
reply = map.get(name);
} else {
reply = getDataFromFileSystem(name);
}
if (reply != null && !reply.contains("empty")) {
return reply;
}
return "file not found";
}
On occasion, the ConcurrentHashMap will return the contents of a non-empty file (i.e. value.contains("empty") == false), however the line:
if (reply != null && !reply.contains("empty"))
returns FALSE. I broke down the IF statement into two parts: if (reply != null) and if (!reply.contains("empty")). The first part of the IF statement returns TRUE. The second part returns FALSE. So I decided to print out the variable "reply" in order to determine if the contents of the string does in fact contain "empty". This was NOT the case i.e. the contents did not contain the string "empty". Furthermore, I added the line
int indexOf = reply.indexOf("empty");
Since the variable reply did not contain the string "empty" when I printed it out, I was expecting indexOf to return -1. But the function returned a value approx the length of the string i.e. if reply.length == 15100, then reply.indexOf("empty") was returning 15099.
I experience this issue on a weekly basis, approx 2-3 times a week. This process is restarted on a daily basis therefore the ConcurrentHashMap is re-generated regularly.
Has anyone seen such behavior when using Java's ConcurrentHashMap?
EDIT
private String getDataFromFileSystem(String name) {
String contents = "empty";
try {
File folder = new File(dir);
File[] fileList = folder.listFiles();
for (int i = 0; i < fileList.length; i++) {
if (fileList[i].isFile() && fileList[i].getName().contains(name)) {
String fileName = fileList[i].getAbsolutePath();
FileReader fr = null;
BufferedReader br = null;
try {
fr = new FileReader(fileName);
br = new BufferedReader(fr);
String sCurrentLine;
while ((sCurrentLine = br.readLine()) != null) {
contents += sCurrentLine.trim();
}
if (contents.equals("")) {
contents = "empty";
}
return contents;
} catch (Exception e) {
e.printStackTrace();
if (contents.equals("")) {
contents = "empty";
}
return contents;
} finally {
if (fr != null) {
try {
fr.close();
} catch (Exception e) {
e.printStackTrace();
}
}
if (br != null) {
try {
br.close();
} catch (Exception e) {
e.printStackTrace();
}
}
if (map.containsKey(name)) {
map.remove(name);
}
map.put(name, contents);
}
}
}
} catch (Exception e) {
e.printStackTrace();
if (contents.equals("")) {
contents = "empty";
}
return contents;
}
return contents;
}

I think your problem is that some of your operations should be atomic and they aren't.
For example, one possible thread interleaving scenario is the following:
Thread 1 reads this line in the getData method:
if (map.containsKey(name)) // (1)
the result is false and Thread 1 goes to
reply = getDataFromFileSystem(name); // (2)
in getDataFromFileSystem, you have the following code:
if (map.containsKey(name)) { // (3)
map.remove(name); // (4)
}
map.put(name, contents); // (5)
imagine that another thread (Thread 2) arrives at (1) while Thread 1 is between (4) and (5): name is not in the map, so thread 2 goes to (2) again
Now that does not explain the specific issue you are observing but it illustrates the fact that when you let many threads run concurrently in a section of code without synchronization, weird things can and do happen.
As it stands, I can't find an explanation for the scenario you describe, unless you call reply = map.get(name) more than once in your tests, in which case it is very possible that the 2 calls don't return the same result.

First off, don't even think that there is a bug in ConcurrentHashMap. JDK faults are very rare and even entertaining the idea will pull you away from properly debugging your code.
I think your bug is as follows. Since you are using contains("empty") what happens if the line from the file has the word "empty" in it? Isn't that going to screw things up?
Instead of using contains("empty") I would use ==. Make the "empty" a private static final String then you can use equality on it.
private final static String EMPTY_STRING_REFERENCE = "empty";
...
if (reply != null && reply != EMPTY_STRING_REFERENCE) {
return reply;
}
...
String contents = EMPTY_STRING_REFERENCE;
...
// really this should be if (contents.isEmpty())
if (contents.equals("")) {
contents = EMPTY_STRING_REFERENCE;
}
This is, btw, the only time you should be using == to compare strings. In this case you want to test it by reference and not by contents since lines from your files could actually contain the magic string.
Here are some other points:
In general, whenever you are using the same String in multiple places in your program, it should be pulled up to a static final field. Java will probably do this for you anyway but it makes the code a lot cleaner as well.
#assylias is spot on about race conditions when you make 2 calls to ConcurrentHashMap. For example, instead of doing:
if (map.containsKey(name)) {
reply = map.get(name);
} else {
You should do the following so you do only one.
reply = map.get(name);
if (reply == null) {
In your code you do this:
if (map.containsKey(name)) {
map.remove(name);
}
map.put(name, contents);
That should be rewritten as the following. There is no need to remove before the put which introduces race conditions as #assylias mentioned.
map.put(name, contents);
You said:
if reply.length == 15100, then reply.indexOf("empty") was returning 15099.
This is not possible with the same reply String. I suspect that you were looking at different threads or in some other way misinterpreting the output. Again, don't be fooled into thinking that there are bugs in java.lang.String.

First, using ConcurrentHashMap does not protect you if you call its methods from multiple threads in sequence. If you call containsKey and get afterwards and another thread calls remove in between you will have a null result. Be sure to call only get and check for null instead of containsKey/get. It's also better regarding performance, because both methods nearly have the same cost.
Second, the weird indexOf call result is either due to a programming error, or points to memory corruption. Is there any native code involved in your application? What are you doing in getDataFromFileSystem? I observed memory corruption when using FileChannel objects from multiple threads.

Getting an InputStream to read more than once, regardless of markSupported()

I need to be able to re-use a java.io.InputStream multiple times, and I figured the following code would work, but it only works the first time.
Code
public class Clazz
{
private java.io.InputStream dbInputStream, firstDBInputStream;
private ArrayTable db;
public Clazz(java.io.InputStream defDB)
{
this.firstDBInputStream = defDB;
this.dbInputStream = defDB;
if (db == null)
throw new java.io.FileNotFoundException("Could not find the database at " + db);
if (dbInputStream.markSupported())
dbInputStream.mark(Integer.MAX_VALUE);
loadDatabaseToArrayTable();
}
public final void loadDatabaseToArrayTable() throws java.io.IOException
{
this.dbInputStream = firstDBInputStream;
if (dbInputStream.markSupported())
dbInputStream.reset();
java.util.Scanner fileScanner = new java.util.Scanner(dbInputStream);
String CSV = "";
for (int i = 0; fileScanner.hasNextLine(); i++)
CSV += fileScanner.nextLine() + "\n";
db = ArrayTable.createArrayTableFromCSV(CSV);
}
public void reloadDatabase()//A method called by the UI
{
try
{
loadDatabaseToArrayTable();
}
catch (Throwable t)
{
//Alert the user that an error has occurred
}
}
}
Note that ArrayTable is a class of mine, which uses arrays to give an interface for working with tables.
Question
In this program, the database is shown directly to the user immediately after the reloadDatabase() method is called, and so any solution involving saving the initial read to an object in memory is useless, as that will NOT refresh the data (think of it like a browser; when you press "Refresh", you want it to fetch the information again, not just display the information it fetched the first time). How can I read a java.io.InputStream more than once?

You can't necessarily read an InputStream more than once. Some implementations support it, some don't. What you are doing is checking the markSupported method, which is indeed an indicator if you can read the same stream twice, but then you are ignoring the result. You have to call that method to see if you can read the stream twice, and if you can't, make other arrangements.
Edit (in response to comment): When I wrote my answer, my "other arrangements" was to get a fresh InputStream. However, when I read in your comments to your question about what you want to do, I'm not sure it is possible. For the basics of the operation, you probably want RandomAccessFile (at least that would be my first guess, and if it worked, that would be the easiest) - however you will have file access issues. You have an application actively writing to a file, and another reading that file, you will have problems - exactly which problems will depend on the OS, so whatever solution would require more testing. I suggest a separate question on SO that hits on that point, and someone who has tried that out can perhaps give you more insight.

you never mark the stream to be reset
public Clazz(java.io.InputStream defDB)
{
firstDBInputStream = defDB.markSupported()?defDB:new BufferedInputStream(defDB);
//BufferedInputStream supports marking
firstDBInputStream.mark(500000);//avoid IOException on first reset
}
public final void loadDatabaseToArrayTable() throws java.io.IOException
{
this.dbInputStream = firstDBInputStream;
dbInputStream.reset();
dbInputStream.mark(500000);//or however long the data is
java.util.Scanner fileScanner = new java.util.Scanner(dbInputStream);
StringBuilder CSV = "";//StringBuilder is more efficient in a loop
while(fileScanner.hasNextLine())
CSV.append(fileScanner.nextLine()).append("\n");
db = ArrayTable.createArrayTableFromCSV(CSV.toString());
}
however you could instead keep a copy of the original ArrayTable and copy that when you need to (or even the created string to rebuild it)
this code creates the string and caches it so you can safely discard the inputstreams and just use readCSV to build the ArrayTable
private String readCSV=null;
public final void loadDatabaseToArrayTable() throws java.io.IOException
{
if(readCSV==null){
this.dbInputStream = firstDBInputStream;
java.util.Scanner fileScanner = new java.util.Scanner(dbInputStream);
StringBuilder CSV = "";//StringBuilder is more efficient in a loop
while(fileScanner.hasNextLine())
CSV.append(fileScanner.nextLine()).append("\n");
readCSV=CSV.toString();
fileScanner.close();
}
db = ArrayTable.createArrayTableFromCSV(readCSV);
}
however if you want new information you'll need to create a new stream to read from again