Develop Reference

Quickly find if Java was launched from Windows cmd or Cygwin terminal

I have a Java application that will be used both from the Windows Command Prompt and the Cygwin terminal. The program uses and manipulates file paths. It we be very useful to have a sep variable that would be / when the program is launched from Cygwin but \\ when the program is launched from Windows.
Looking here, I'm not sure it will be possible, but I want to ask.
I will post a small, compilable app that shows the issue in a few minutes. For now, I'll just say that I want a set of functions that something like:
// in main
...
String sep = getSeparatorToUse();
...
// member functions
...
private boolean wasLaunchedFromWinCmd()
{
if (<something-here-that-knows-it-was-cmd-not-cygwin>)
return true;
return false;
}//endof: private boolean wasLaunchedFromWinCmd()
private String getSeparatorToUse()
{
if (wasLaunchedFromWinCmd)
return "\\"
return "/"
}//endof: private String getSeparatorToUse()
Thanks #Raphael_Moita. Those are very useful, and I will likely use them in the Linux version of the app that I will be using. #Luke_Lee, I feel dumb not having realized it. I think you two might have solved my problem while I was getting the compilable code ready. There's still one issue when the program run from a batch script - when it is fed a filename from a find command. I hope what I show will illustrate that.
Examples
All examples are as run from Cygwin.
Works: the way most volunteers use the code, just the filename that's in the same directory as the java code.
$ java FileSeparatorExample pic_4.jpg
Here, something will be done with the file,
C:\Users\bballdave025\Desktop\pic_4.jpg
Works: with relative filepaths and spaces in filenames/file paths
$ java FileSeparatorExample pretty\ pictures/pic\ 1.jpg
Here, something will be done with the file,
C:\Users\me\Desktop\pretty pictures/pic 1.jpg
$ java FileSeparatorExample ../pic_5.jpg
Here, something will be done with the file,
C:\Users\me\Desktop\../pic_5.jpg
DOESN'T WORK. Sometimes, the output of a find command will come with the complete filepath in Cygwin/UNIX format:
$ java FileSeparatorExample /cygdrive/c/David/example/pic.jpg
The file:
C:\Users\bballdave025\Desktop\/cygdrive/c/David/example/pic.jpg
doesn't exist
Compilable Code
I'm just cutting down from my original code, so I'm sorry if it seems too big.
/**********************************
* #file FileSeparatorExample.java
**********************************/
// Import statements
import java.io.File;
import java.io.IOException;
public class FileSeparatorExample
{
// Member variables
private static String sep;
public static void main(String[] args)
{
////****** DOESN'T WORK AS DESIRED ******////
sep = java.io.File.separator;
////** I want **////
// sep = getFileSeparator();
String imageToLoad = null;
boolean argumentExists = ( args != null && args.length != 0 );
if (argumentExists)
{
boolean thereIsExactlyOneArgument = ( args.length == 1 );
if (thereIsExactlyOneArgument)
{
imageToLoad = args[0];
}//endof: if (thereIsExactlyOneArgument)
else
{
// do some other stuff
}
}//endof: if (argumentExists)
String filenamePath = getFilenamePath(imageToLoad);
String filenameFile = getFilenameFile(imageToLoad);
imageToLoad = filenamePath + sep + filenameFile;
File f = new File(imageToLoad);
if (! f.exists())
{
System.err.println("The file:");
System.err.println(imageToLoad);
System.err.println("doesn\'t exist");
System.exit(1);
}//endof: if (! f.exists())
System.out.println("Here, something will be done with the file,");
System.out.println(imageToLoad);
}//endof: main
// member methods
/**
* Separates the filename arg into: full path to directory; bare filename
*/
private static String[] splitFilename(String imageToLoad)
{
String[] fileParts = new String[2];
int indexOfLastSep = imageToLoad.lastIndexOf(sep);
boolean fullFilenameHasSeparator = ( indexOfLastSep != -1 );
if (fullFilenameHasSeparator)
{
fileParts[0] = imageToLoad.substring(0, indexOfLastSep);
fileParts[1] = imageToLoad.substring(indexOfLastSep + 1);
}//endof: if (fullFilenameHasSeparator)
else
{
// Use the user's directory as the path
fileParts[0] = System.getProperty("user.dir");
fileParts[1] = imageToLoad;
}//endof: if/else (fullFilenameHasSeparator)
return fileParts;
}//endof: private static String[] splitFilename(String imageToLoad)
/**
* Gives the full path to the file's directory (from the filename arg)
* but not the bare filename
*/
private static String getFilenamePath(String imageToLoad)
{
String[] fileParts = splitFilename(imageToLoad);
return fileParts[0];
}//endof: private static String getFilenamePath(String imageToLoad)
/**
* Gives the bare filename (no path information)
*/
private static String getFilenameFile(String imageToLoad)
{
String[] fileParts = splitFilename(imageToLoad);
return fileParts[1];
}//endof: private static String getFilenamePath(String imageToLoad)
}//endof: public class FileSeparatorExample

You don't need to know which SO is under your Java. If your goal is to find the correct file separator to use, call this:
java.io.File.separator;
Anyway ... to find out which SO java is running over (not sure how cygwin is detected by this), try:
boolean isWindows = System.getProperty("os.name").startsWith("win");

Here is an answer I've come up with that almost answers my original question. It tries to determine the launcher of the Java code based on the filename argument. A big thanks to #Raphael_Moita and #Luke_Lee, who actually pretty much solved my problem. Their solutions didn't answer the original question, but that's partly because I didn't post the original question completely. As I said, this answer doesn't answer the original question completely. If someone knows the complete solution, please let me know.
My solution was a few methods. As they stand, they only work for my case - Cygwin on Windows. (What they do is tell you if the filename argument for the Java application was consistent with being launched from Windows cmd or not.) I plan on posting a more portable group of methods, i.e. other Operating Systems.
I'm sure there are issues. Please point them out to me.
// in main
...
sep = java.io.File.separator; // Thanks #Luke_Lee
if (args != null && args.length != 0)
sep = getSeparatorToUse(args[0]);
...
// member functions
...
private boolean wasLaunchedFromWinCmd(String firstArg)
{
boolean isWindows = System.getProperty("os.name").startsWith("win");
if (! isWindows) return false; // Thanks #Raphael_Moita
else
{
String launchDir = System.getProperty("user.dir");
String rootOfLaunchDir = getRoot(launchDir);
// This will come back with something like "C:\" or "P:\"
String rootOfArgument = getRoot(firstArg);
if (rootOfArgument.equals("/"))
{
String cygwinBase = "/cygdrive/";
char letterOfRoot = rootOfLaunchDir.charAt(0);
// For, e.g., "/Users/me/Desktop/pic_314.jpg"
if (firstArg.startsWith(cygwinBase))
{
int charsToCut = cygwinBase.length();
letterOfRoot = firstArg.substring(charsToCut,
charsToCut + 1);
}//endof: if (firstArg.startsWith(cygwinBase))
System.out.println("The root directory of your argument will be:");
System.out.println(Character.toUpperCase(letterOfRoot) + ":\\");
System.out.println("In Cygwin, that will be:");
System.out.println(cygwinBase +
Character.toLowerCase(letterOfRoot) + "/");
return false;
// Not always correct, e.g. if someone in Cygwin uses
// $ java FileSeparatorExample "C:\pic_137.jpg"
}//endof: if (rootOfArgument.equals("/"))
return true;
}//endof: if/else (! isWindows)
}//endof: private boolean wasLaunchedFromCmd()
private String getRoot(String fileOrDir)
{
File file = new File(fileOrDir).getAbsoluteFile();
File root = file.getParentFile();
while (root.getParentFile() != null)
root = root.getParentFile();
return root.toString();
}//endof: private String getRoot();
private String getSeparatorToUse(String firstArg)
{
if (wasLaunchedFromWinCmd(firstArg))
return "\\"
return "/"
}//endof: private String getSeparatorToUse(String firstArg)
Parts of this solution are due to #Raphael_Moita and #Luke_Lee, but I also need to reference this SO post. This last one helped with my specific situation, where the files are not all hosted on the C:\ drive.
Note
I won't be accepting mine as the correct solution, because it doesn't answer my original question. I hope it might help someone with answering the original question.

Get a list of disks to read free space in Java, using Sigar

I need to get the free available disk space for all disks in system, or all partitions, I don't mind that. (I dont have to use Sigar, but I am using it already on the project for some other processes, so I can use it for this as well)
I am using Sigar API and got this
public double getFreeHdd() throws SigarException{
FileSystemUsage f= sigar.getFileSystemUsage("/");
return ( f.getAvail());
}
But this only gives me the system partition (root), how can i get a list of all partition and loop them to get their free space?
I tried this
FileSystemView fsv = FileSystemView.getFileSystemView();
File[] roots = fsv.getRoots();
for (int i = 0; i < roots.length; i++) {
System.out.println("Root: " + roots[i]);
}
But it only returns the root dir
Root: /
Thanks
Edit
it seems that I could use
FileSystem[] fslist = sigar.getFileSystemList();
But the results i am getting do not match the ones i get from the terminal. On the other hand on this system I am working on, i have 3 disks with a total 12 partitions, so i might be loosing something there. Will try it on some other system in case i can make something useful out of the results.

We use SIGAR extensively for cross-platform monitoring. This is the code we use to get the file system list:
/**
* #return a list of directory path names of file systems that are local or network - not removable media
*/
public static Set<String> getLocalOrNetworkFileSystemDirectoryNames() {
Set<String> ret = new HashSet<String>();
try {
FileSystem[] fileSystemList = getSigarProxy().getFileSystemList();
for (FileSystem fs : fileSystemList) {
if ((fs.getType() == FileSystem.TYPE_LOCAL_DISK) || (fs.getType() == FileSystem.TYPE_NETWORK)) {
ret.add(fs.getDirName());
}
}
}
catch (SigarException e) {
// log or rethrow as appropriate
}
return ret;
}
You can then use that as the input to other SIGAR methods:
FileSystemUsage usageStats = getSigarProxy().getFileSystemUsage(fileSystemDirectoryPath);
The getSigarProxy() is just a convenience base method:
// The Humidor handles thread safety for a single instance of a Sigar object
static final private SigarProxy sigarProxy = Humidor.getInstance().getSigar();
static final protected SigarProxy getSigarProxy() {
return sigarProxy;
}

You can use java.nio.file.FileSystems to get a list of java.nio.file.FileStorages and then see the usable/available space. Per instance (assuming that you are using Java 7+):
import java.io.IOException;
import java.nio.file.FileStore;
import java.nio.file.FileSystem;
import java.nio.file.FileSystems;
import java.util.function.Consumer;
public static void main(String[] args) {
FileSystem fs = FileSystems.getDefault();
fs.getFileStores().forEach(new Consumer<FileStore>() {
#Override
public void accept(FileStore store) {
try {
System.out.println(store.getTotalSpace());
System.out.println(store.getUsableSpace());
} catch (IOException e) {
e.printStackTrace();
}
}
});
}
Also, keep in mind that FileStore.getUsableSpace() returns the size in bytes. See the docs for more information.

directory not created in ubuntu using java [duplicate]

In Java, I'm dynamically creating a set of files and I'd like to change the file permissions on these files on a linux/unix file system. I'd like to be able to execute the Java equivalent of chmod. Is that possible Java 5? If so, how?
I know in Java 6 the File object has setReadable()/setWritable() methods. I also know I could make a system call to do this, but I'd like to avoid that if possible.

Full control over file attributes is available in Java 7, as part of the "new" New IO facility (NIO.2). For example, POSIX permissions can be set on an existing file with setPosixFilePermissions(), or atomically at file creation with methods like createFile() or newByteChannel().
You can create a set of permissions using EnumSet.of(), but the helper method PosixFilePermissions.fromString() will uses a conventional format that will be more readable to many developers. For APIs that accept a FileAttribute, you can wrap the set of permissions with with PosixFilePermissions.asFileAttribute().
Set<PosixFilePermission> ownerWritable = PosixFilePermissions.fromString("rw-r--r--");
FileAttribute<?> permissions = PosixFilePermissions.asFileAttribute(ownerWritable);
Files.createFile(path, permissions);
In earlier versions of Java, using native code of your own, or exec-ing command-line utilities are common approaches.

Prior to Java 6, there is no support of file permission update at Java level. You have to implement your own native method or call Runtime.exec() to execute OS level command such as chmod.
Starting from Java 6, you can useFile.setReadable()/File.setWritable()/File.setExecutable() to set file permissions. But it doesn't simulate the POSIX file system which allows to set permission for different users. File.setXXX() only allows to set permission for owner and everyone else.
Starting from Java 7, POSIX file permission is introduced. You can set file permissions like what you have done on *nix systems. The syntax is :
File file = new File("file4.txt");
file.createNewFile();
Set<PosixFilePermission> perms = new HashSet<>();
perms.add(PosixFilePermission.OWNER_READ);
perms.add(PosixFilePermission.OWNER_WRITE);
Files.setPosixFilePermissions(file.toPath(), perms);
This method can only be used on POSIX file system, this means you cannot call it on Windows system.
For details on file permission management, recommend you to read this post.

In addition to erickson's suggestions, there's also jna, which allows you to call native libraries without using jni. It's shockingly easy to use, and I've used it on a couple of projects with great success.
The only caveat is that it's slower than jni, so if you're doing this to a very large number of files that might be an issue for you.
(Editing to add example)
Here's a complete jna chmod example:
import com.sun.jna.Library;
import com.sun.jna.Native;
public class Main {
private static CLibrary libc = (CLibrary) Native.loadLibrary("c", CLibrary.class);
public static void main(String[] args) {
libc.chmod("/path/to/file", 0755);
}
}
interface CLibrary extends Library {
public int chmod(String path, int mode);
}

For Windows 7 with NIO 2:
public static void main(String[] args) throws IOException {
Path file = Paths.get("c:/touch.txt");
AclFileAttributeView aclAttr = Files.getFileAttributeView(file, AclFileAttributeView.class);
System.out.println(aclAttr.getOwner());
for (AclEntry aclEntry : aclAttr.getAcl()) {
System.out.println(aclEntry);
}
System.out.println();
UserPrincipalLookupService upls = file.getFileSystem().getUserPrincipalLookupService();
UserPrincipal user = upls.lookupPrincipalByName(System.getProperty("user.name"));
AclEntry.Builder builder = AclEntry.newBuilder();
builder.setPermissions( EnumSet.of(AclEntryPermission.READ_DATA, AclEntryPermission.EXECUTE,
AclEntryPermission.READ_ACL, AclEntryPermission.READ_ATTRIBUTES, AclEntryPermission.READ_NAMED_ATTRS,
AclEntryPermission.WRITE_ACL, AclEntryPermission.DELETE
));
builder.setPrincipal(user);
builder.setType(AclEntryType.ALLOW);
aclAttr.setAcl(Collections.singletonList(builder.build()));
}

Just to update this answer unless anyone comes across this later, since JDK 6 you can use
File file = new File('/directory/to/file');
file.setWritable(boolean);
file.setReadable(boolean);
file.setExecutable(boolean);
you can find the documentation on Oracle File(Java Platform SE 7). Bear in mind that these commands only work if the current working user has ownership or write access to that file. I am aware that OP wanted chmod type access for more intricate user configuration. these will set the option across the board for all users.

If you want to set 777 permission to your created file than you can use the following method:
public void setPermission(File file) throws IOException{
Set<PosixFilePermission> perms = new HashSet<>();
perms.add(PosixFilePermission.OWNER_READ);
perms.add(PosixFilePermission.OWNER_WRITE);
perms.add(PosixFilePermission.OWNER_EXECUTE);
perms.add(PosixFilePermission.OTHERS_READ);
perms.add(PosixFilePermission.OTHERS_WRITE);
perms.add(PosixFilePermission.OTHERS_EXECUTE);
perms.add(PosixFilePermission.GROUP_READ);
perms.add(PosixFilePermission.GROUP_WRITE);
perms.add(PosixFilePermission.GROUP_EXECUTE);
Files.setPosixFilePermissions(file.toPath(), perms);
}

You can use the methods of the File class:
http://docs.oracle.com/javase/7/docs/api/java/io/File.html

Apache ant chmod (not very elegant, adding it for completeness) credit shared with #msorsky
Chmod chmod = new Chmod();
chmod.setProject(new Project());
FileSet mySet = new FileSet();
mySet.setDir(new File("/my/path"));
mySet.setIncludes("**");
chmod.addFileset(mySet);
chmod.setPerm("+w");
chmod.setType(new FileDirBoth());
chmod.execute();

for Oralce Java 6:
private static int chmod(String filename, int mode) {
try {
Class<?> fspClass = Class.forName("java.util.prefs.FileSystemPreferences");
Method chmodMethod = fspClass.getDeclaredMethod("chmod", String.class, Integer.TYPE);
chmodMethod.setAccessible(true);
return (Integer)chmodMethod.invoke(null, filename, mode);
} catch (Throwable ex) {
return -1;
}
}
works under solaris/linux.

There is an example class on Oracle Docs which works very much similar to the UNIX chmod. It works with java se 7+ though.

import java.io.File;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.nio.file.attribute.FileAttribute;
import java.nio.file.attribute.PosixFileAttributes;
import java.nio.file.attribute.PosixFilePermission;
import java.nio.file.attribute.PosixFilePermissions;
import java.util.Set;
public class FileAndDirectory1 {
public static void main(String[] args) {
File file = new File("fileTest1.txt");
System.out.println(file.getAbsoluteFile());
try {
//file.createNewFile();
if(!file.exists())
{
//PosixFilePermission is an enum class, PosixFilePermissions is a final class
//create file permissions from string
Set<PosixFilePermission> filePermissions = PosixFilePermissions.fromString("---------"/* "rwxrwxrwx" */);
FileAttribute<?> permissions = PosixFilePermissions.asFileAttribute(filePermissions);
Files.createFile(file.toPath(), permissions);
// printing the permissions associated with the file
System.out.println("Executable: " + file.canExecute());
System.out.println("Readable: " + file.canRead());
System.out.println("Writable: "+ file.canWrite());
file.setExecutable(true);
file.setReadable(true);
file.setWritable(true);
}
else
{
//modify permissions
//get the permission using file attributes
Set<PosixFilePermission> perms = Files.readAttributes(file.toPath(), PosixFileAttributes.class).permissions();
perms.remove(PosixFilePermission.OWNER_WRITE);
perms.add(PosixFilePermission.OWNER_READ);
perms.add(PosixFilePermission.OWNER_EXECUTE);
perms.add(PosixFilePermission.GROUP_WRITE);
perms.add(PosixFilePermission.GROUP_READ);
perms.add(PosixFilePermission.GROUP_EXECUTE);
perms.add(PosixFilePermission.OTHERS_WRITE);
perms.add(PosixFilePermission.OTHERS_READ);
perms.add(PosixFilePermission.OTHERS_EXECUTE);
Files.setPosixFilePermissions(file.toPath(), perms);
System.out.println("Executable: " + file.canExecute());
System.out.println("Readable: " + file.canRead());
System.out.println("Writable: "+ file.canWrite());
file.delete();
}
} catch (IOException e) {
e.printStackTrace();
}
Path path = Paths.get(String.valueOf(file));
System.out.println(path);
}
}

Permission 777 is the same as rwxrwxrwx which you can set as follows:
Files.setPosixFilePermissions(path, PosixFilePermissions.fromString("rwxrwxrwx"))

'Un'-externalize strings from Eclipse or Intellij

I have a bunch of strings in a properties file which i want to 'un-externalize', ie inline into my code.
I see that both Eclipse and Intellij have great support to 'externalize' strings from within code, however do any of them support inlining strings from a properties file back into code?
For example if I have code like -
My.java
System.out.println(myResourceBundle.getString("key"));
My.properties
key=a whole bunch of text
I want my java code to be replaced as -
My.java
System.out.println("a whole bunch of text");

I wrote a simple java program that you can use to do this.
Dexternalize.java
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileWriter;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.Map.Entry;
import java.util.Properties;
import java.util.Set;
import java.util.Stack;
import java.util.logging.Level;
import java.util.logging.Logger;
public class Deexternalize {
public static final Logger logger = Logger.getLogger(Deexternalize.class.toString());
public static void main(String[] args) throws IOException {
if(args.length != 2) {
System.out.println("Deexternalize props_file java_file_to_create");
return;
}
Properties defaultProps = new Properties();
FileInputStream in = new FileInputStream(args[0]);
defaultProps.load(in);
in.close();
File javaFile = new File(args[1]);
List<String> data = process(defaultProps,javaFile);
buildFile(javaFile,data);
}
public static List<String> process(Properties propsFile, File javaFile) {
List<String> data = new ArrayList<String>();
Set<Entry<Object,Object>> setOfProps = propsFile.entrySet();
int indexOf = javaFile.getName().indexOf(".");
String javaClassName = javaFile.getName().substring(0,indexOf);
data.add("public class " + javaClassName + " {\n");
StringBuilder sb = null;
// for some reason it's adding them in reverse order so putting htem on a stack
Stack<String> aStack = new Stack<String>();
for(Entry<Object,Object> anEntry : setOfProps) {
sb = new StringBuilder("\tpublic static final String ");
sb.append(anEntry.getKey().toString());
sb.append(" = \"");
sb.append(anEntry.getValue().toString());
sb.append("\";\n");
aStack.push(sb.toString());
}
while(!aStack.empty()) {
data.add(aStack.pop());
}
if(sb != null) {
data.add("}");
}
return data;
}
public static final void buildFile(File fileToBuild, List<String> lines) {
BufferedWriter theWriter = null;
try {
// Check to make sure if the file exists already.
if(!fileToBuild.exists()) {
fileToBuild.createNewFile();
}
theWriter = new BufferedWriter(new FileWriter(fileToBuild));
// Write the lines to the file.
for(String theLine : lines) {
// DO NOT ADD windows carriage return.
if(theLine.endsWith("\r\n")){
theWriter.write(theLine.substring(0, theLine.length()-2));
theWriter.write("\n");
} else if(theLine.endsWith("\n")) {
// This case is UNIX format already since we checked for
// the carriage return already.
theWriter.write(theLine);
} else {
theWriter.write(theLine);
theWriter.write("\n");
}
}
} catch(IOException ex) {
logger.log(Level.SEVERE, null, ex);
} finally {
try {
theWriter.close();
} catch(IOException ex) {
logger.log(Level.SEVERE, null, ex);
}
}
}
}
Basically, all you need to do is call this java program with the location of the property file and the name of the java file you want to create that will contain the properties.
For instance this property file:
test.properties
TEST_1=test test test
TEST_2=test 2456
TEST_3=123456
will become:
java_test.java
public class java_test {
public static final String TEST_1 = "test test test";
public static final String TEST_2 = "test 2456";
public static final String TEST_3 = "123456";
}
Hope this is what you need!
EDIT:
I understand what you requested now. You can use my code to do what you want if you sprinkle a bit of regex magic. Lets say you have the java_test file from above. Copy the inlined properties into the file you want to replace the myResourceBundle code with.
For example,
TestFile.java
public class TestFile {
public static final String TEST_1 = "test test test";
public static final String TEST_2 = "test 2456";
public static final String TEST_3 = "123456";
public static void regexTest() {
System.out.println(myResourceBundle.getString("TEST_1"));
System.out.println(myResourceBundle.getString("TEST_1"));
System.out.println(myResourceBundle.getString("TEST_3"));
}
}
Ok, now if you are using eclipse (any modern IDE should be able to do this) go to the Edit Menu -> Find/Replace. In the window, you should see a "Regular Expressions" checkbox, check that. Now input the following text into the Find text area:
myResourceBundle\.getString\(\"(.+)\"\)
And the back reference
\1
into the replace.
Now click "Replace all" and voila! The code should have been inlined to your needs.
Now TestFile.java will become:
TestFile.java
public class TestFile {
public static final String TEST_1 = "test test test";
public static final String TEST_2 = "test 2456";
public static final String TEST_3 = "123456";
public static void regexTest() {
System.out.println(TEST_1);
System.out.println(TEST_1);
System.out.println(TEST_3);
}
}

You may use Eclipse "Externalize Strings" widget. It can also be used for un-externalization. Select required string(s) and press "Internalize" button. If the string was externalized before, it'll be put back and removed from messages.properties file.

May be if you can explain on how you need to do this, then you could get the correct answer.
The Short answer to your question is no, especially in Intellij (I do not know enough about eclipse). Of course the slightly longer but still not very useful answer is to write a plugin. ( That will take a list of property files and read the key and values in a map and then does a regular expression replace of ResourceBundle.getValue("Key") with the value from Map (for the key). I will write this plugin myself, if you can convince me that, there are more people like you, who have this requirement.)
The more elaborate answer is this.
1_ First I will re-factor all the code that performs property file reading to a single class (or module called PropertyFileReader).
2_ I will create a property file reader module, that iterates across all the keys in property file(s) and then stores those information in a map.
4_ I can either create a static map objects with the populated values or create a constant class out of it. Then I will replace the logic in the property file reader module to use a get on the map or static class rather than the property file reading.
5_ Once I am sure that the application performs ok.(By checking if all my Unit Testing passes), then I will remove my property files.
Note: If you are using spring, then there is a easy way to split out all property key-value pairs from a list of property files. Let me know if you use spring.

I would recommend something else: split externalized strings into localizable and non-localizable properties files. It would be probably easier to move some strings to another file than moving it back to source code (which will hurt maintainability by the way).
Of course you can write simple (to some extent) Perl (or whatever) script which will search for calls to resource bundles and introduce constant in this place...
In other words, I haven't heard about de-externalizing mechanism, you need to do it by hand (or write some automated script yourself).

An awesome oneliner from #potong sed 's|^\([^=]*\)=\(.*\)|s#Messages.getString("\1")#"\2"#g|;s/\\/\\\\/g' messages.properties |
sed -i -f - *.java run this inside your src dir, and see the magic.

Is there a workaround for Java's poor performance on walking huge directories?

I am trying to process files one at a time that are stored over a network. Reading the files is fast due to buffering is not the issue. The problem I have is just listing the directories in a folder. I have at least 10k files per folder over many folders.
Performance is super slow since File.list() returns an array instead of an iterable. Java goes off and collects all the names in a folder and packs it into an array before returning.
The bug entry for this is http://bugs.sun.com/view_bug.do;jsessionid=db7fcf25bcce13541c4289edeb4?bug_id=4285834 and doesn't have a work around. They just say this has been fixed for JDK7.
A few questions:
Does anybody have a workaround to this performance bottleneck?
Am I trying to achieve the impossible? Is performance still going to be poor even if it just iterates over the directories?
Could I use the beta JDK7 builds that have this functionality without having to build my entire project on it?

Although it's not pretty, I solved this kind of problem once by piping the output of dir/ls to a file before starting my app, and passing in the filename.
If you needed to do it within the app, you could just use system.exec(), but it would create some nastiness.
You asked. The first form is going to be blazingly fast, the second should be pretty fast as well.
Be sure to do the one item per line (bare, no decoration, no graphics), full path and recurse options of your selected command.
EDIT:
30 minutes just to get a directory listing, wow.
It just struck me that if you use exec(), you can get it's stdout redirected into a pipe instead of writing it to a file.
If you did that, you should start getting the files immediately and be able to begin processing before the command has completed.
The interaction may actually slow things down, but maybe not--you might give it a try.
Wow, I just went to find the syntax of the .exec command for you and came across this, possibly exactly what you want (it lists a directory using exec and "ls" and pipes the result into your program for processing): good link in wayback (Jörg provided in a comment to replace this one from sun that Oracle broke)
Anyway, the idea is straightforward but getting the code right is annoying. I'll go steal some codes from the internets and hack them up--brb
/**
* Note: Only use this as a last resort! It's specific to windows and even
* at that it's not a good solution, but it should be fast.
*
* to use it, extend FileProcessor and call processFiles("...") with a list
* of options if you want them like /s... I highly recommend /b
*
* override processFile and it will be called once for each line of output.
*/
import java.io.*;
public abstract class FileProcessor
{
public void processFiles(String dirOptions)
{
Process theProcess = null;
BufferedReader inStream = null;
// call the Hello class
try
{
theProcess = Runtime.getRuntime().exec("cmd /c dir " + dirOptions);
}
catch(IOException e)
{
System.err.println("Error on exec() method");
e.printStackTrace();
}
// read from the called program's standard output stream
try
{
inStream = new BufferedReader(
new InputStreamReader( theProcess.getInputStream() ));
processFile(inStream.readLine());
}
catch(IOException e)
{
System.err.println("Error on inStream.readLine()");
e.printStackTrace();
}
} // end method
/** Override this method--it will be called once for each file */
public abstract void processFile(String filename);
} // end class
And thank you code donor at IBM

How about using File.list(FilenameFilter filter) method and implementing FilenameFilter.accept(File dir, String name) to process each file and return false.
I ran this on Linux vm for directory with 10K+ files and it took <10 seconds.
import java.io.File;
import java.io.FilenameFilter;
public class Temp {
private static void processFile(File dir, String name) {
File file = new File(dir, name);
System.out.println("processing file " + file.getName());
}
private static void forEachFile(File dir) {
String [] ignore = dir.list(new FilenameFilter() {
public boolean accept(File dir, String name) {
processFile(dir, name);
return false;
}
});
}
public static void main(String[] args) {
long before, after;
File dot = new File(".");
before = System.currentTimeMillis();
forEachFile(dot);
after = System.currentTimeMillis();
System.out.println("after call, delta is " + (after - before));
}
}

An alternative is to have the files served over a different protocol. As I understand you're using SMB for that and java is just trying to list them as a regular file.
The problem here might not be java alone ( how does it behaves when you open that directory with Microsoft Explorer x:\shared ) In my experience it also take a considerably amount of time.
You can change the protocol to something like HTTP, only to fetch the file names. This way you can retrieve the list of files over http ( 10k lines should't be too much ) and let the server deal with file listing. This would be very fast, since it will run with local resources ( those in the server )
Then when you have the list, you can process them one by exactly the way you're doing right now.
The keypoint is to have an aid mechanism in the other side of the node.
Is this feasible?
Today:
File [] content = new File("X:\\remote\\dir").listFiles();
for ( File f : content ) {
process( f );
}
Proposed:
String [] content = fetchViaHttpTheListNameOf("x:\\remote\\dir");
for ( String fileName : content ) {
process( new File( fileName ) );
}
The http server could be a very small small and simple file.
If this is the way you have it right now, what you're doing is to fetch all the 10k files information to your client machine ( I don't know how much of that info ) when you only need the file name for later processing.
If the processing is very fast right now it may be slowed down a bit. This is because the information prefetched is no longer available.
Give it a try.

A non-portable solution would be to make native calls to the operating system and stream the results.
For Linux
You can look at something like readdir. You can walk the directory structure like a linked list and return results in batches or individually.
For Windows
In windows the behavior would be fairly similar using FindFirstFile and FindNextFile apis.

I doubt the problem is relate to the bug report you referenced.
The issue there is "only" memory usage, but not necessarily speed.
If you have enough memory the bug is not relevant for your problem.
You should measure whether your problem is memory related or not. Turn on your Garbage Collector log and use for example gcviewer to analyze your memory usage.
I suspect that it has to do with the SMB protocol causing the problem.
You can try to write a test in another language and see if it's faster, or you can try to get the list of filenames through some other method, such as described here in another post.

If you need to eventually process all files, then having Iterable over String[] won't give you any advantage, as you'll still have to go and fetch the whole list of files.

If you're on Java 1.5 or 1.6, shelling out "dir" commands and parsing the standard output stream on Windows is a perfectly acceptable approach. I've used this approach in the past for processing network drives and it has generally been a lot faster than waiting for the native java.io.File listFiles() method to return.
Of course, a JNI call should be faster and potentially safer than shelling out "dir" commands. The following JNI code can be used to retrieve a list of files/directories using the Windows API. This function can be easily refactored into a new class so the caller can retrieve file paths incrementally (i.e. get one path at a time). For example, you can refactor the code so that FindFirstFileW is called in a constructor and have a seperate method to call FindNextFileW.
JNIEXPORT jstring JNICALL Java_javaxt_io_File_GetFiles(JNIEnv *env, jclass, jstring directory)
{
HANDLE hFind;
try {
//Convert jstring to wstring
const jchar *_directory = env->GetStringChars(directory, 0);
jsize x = env->GetStringLength(directory);
wstring path; //L"C:\\temp\\*";
path.assign(_directory, _directory + x);
env->ReleaseStringChars(directory, _directory);
if (x<2){
jclass exceptionClass = env->FindClass("java/lang/Exception");
env->ThrowNew(exceptionClass, "Invalid path, less than 2 characters long.");
}
wstringstream ss;
BOOL bContinue = TRUE;
WIN32_FIND_DATAW data;
hFind = FindFirstFileW(path.c_str(), &data);
if (INVALID_HANDLE_VALUE == hFind){
jclass exceptionClass = env->FindClass("java/lang/Exception");
env->ThrowNew(exceptionClass, "FindFirstFileW returned invalid handle.");
}
//HANDLE hStdOut = GetStdHandle(STD_OUTPUT_HANDLE);
//DWORD dwBytesWritten;
// If we have no error, loop thru the files in this dir
while (hFind && bContinue){
/*
//Debug Print Statment. DO NOT DELETE! cout and wcout do not print unicode correctly.
WriteConsole(hStdOut, data.cFileName, (DWORD)_tcslen(data.cFileName), &dwBytesWritten, NULL);
WriteConsole(hStdOut, L"\n", 1, &dwBytesWritten, NULL);
*/
//Check if this entry is a directory
if (data.dwFileAttributes & FILE_ATTRIBUTE_DIRECTORY){
// Make sure this dir is not . or ..
if (wstring(data.cFileName) != L"." &&
wstring(data.cFileName) != L"..")
{
ss << wstring(data.cFileName) << L"\\" << L"\n";
}
}
else{
ss << wstring(data.cFileName) << L"\n";
}
bContinue = FindNextFileW(hFind, &data);
}
FindClose(hFind); // Free the dir structure
wstring cstr = ss.str();
int len = cstr.size();
//WriteConsole(hStdOut, cstr.c_str(), len, &dwBytesWritten, NULL);
//WriteConsole(hStdOut, L"\n", 1, &dwBytesWritten, NULL);
jchar* raw = new jchar[len];
memcpy(raw, cstr.c_str(), len*sizeof(wchar_t));
jstring result = env->NewString(raw, len);
delete[] raw;
return result;
}
catch(...){
FindClose(hFind);
jclass exceptionClass = env->FindClass("java/lang/Exception");
env->ThrowNew(exceptionClass, "Exception occured.");
}
return NULL;
}
Credit:
https://sites.google.com/site/jozsefbekes/Home/windows-programming/miscellaneous-functions
Even with this approach, there are still efficiencies to be gained. If you serialize the path to a java.io.File, there is a huge performance hit - especially if the path represents a file on a network drive. I have no idea what Sun/Oracle is doing under the hood but if you need additional file attributes other than the file path (e.g. size, mod date, etc), I have found that the following JNI function is much faster than instantiating a java.io.File object on a network the path.
JNIEXPORT jlongArray JNICALL Java_javaxt_io_File_GetFileAttributesEx(JNIEnv *env, jclass, jstring filename)
{
//Convert jstring to wstring
const jchar *_filename = env->GetStringChars(filename, 0);
jsize len = env->GetStringLength(filename);
wstring path;
path.assign(_filename, _filename + len);
env->ReleaseStringChars(filename, _filename);
//Get attributes
WIN32_FILE_ATTRIBUTE_DATA fileAttrs;
BOOL result = GetFileAttributesExW(path.c_str(), GetFileExInfoStandard, &fileAttrs);
if (!result) {
jclass exceptionClass = env->FindClass("java/lang/Exception");
env->ThrowNew(exceptionClass, "Exception Occurred");
}
//Create an array to store the WIN32_FILE_ATTRIBUTE_DATA
jlong buffer[6];
buffer[0] = fileAttrs.dwFileAttributes;
buffer[1] = date2int(fileAttrs.ftCreationTime);
buffer[2] = date2int(fileAttrs.ftLastAccessTime);
buffer[3] = date2int(fileAttrs.ftLastWriteTime);
buffer[4] = fileAttrs.nFileSizeHigh;
buffer[5] = fileAttrs.nFileSizeLow;
jlongArray jLongArray = env->NewLongArray(6);
env->SetLongArrayRegion(jLongArray, 0, 6, buffer);
return jLongArray;
}
You can find a full working example of this JNI-based approach in the javaxt-core library. In my tests using Java 1.6.0_38 with a Windows host hitting a Windows share, I have found this JNI approach approximately 10x faster then calling java.io.File listFiles() or shelling out "dir" commands.

I wonder why there are 10k files in a directory. Some file systems do not work well with so many files. There are specifics limitations for file systems like max amount of files per directory and max amount of levels of subdirectory.
I solve a similar problem with an iterator solution.
I needed to walk across huge directorys and several levels of directory tree recursively.
I try FileUtils.iterateFiles() of Apache commons io. But it implement the iterator by adding all the files in a List and then returning List.iterator(). It's very bad for memory.
So I prefer to write something like this:
private static class SequentialIterator implements Iterator<File> {
private DirectoryStack dir = null;
private File current = null;
private long limit;
private FileFilter filter = null;
public SequentialIterator(String path, long limit, FileFilter ff) {
current = new File(path);
this.limit = limit;
filter = ff;
dir = DirectoryStack.getNewStack(current);
}
public boolean hasNext() {
while(walkOver());
return isMore && (limit > count || limit < 0) && dir.getCurrent() != null;
}
private long count = 0;
public File next() {
File aux = dir.getCurrent();
dir.advancePostition();
count++;
return aux;
}
private boolean walkOver() {
if (dir.isOutOfDirListRange()) {
if (dir.isCantGoParent()) {
isMore = false;
return false;
} else {
dir.goToParent();
dir.advancePostition();
return true;
}
} else {
if (dir.isCurrentDirectory()) {
if (dir.isDirectoryEmpty()) {
dir.advancePostition();
} else {
dir.goIntoDir();
}
return true;
} else {
if (filter.accept(dir.getCurrent())) {
return false;
} else {
dir.advancePostition();
return true;
}
}
}
}
private boolean isMore = true;
public void remove() {
throw new UnsupportedOperationException();
}
}
Note that the iterator stop by an amount of files iterateds and it has a FileFilter also.
And DirectoryStack is:
public class DirectoryStack {
private class Element{
private File files[] = null;
private int currentPointer;
public Element(File current) {
currentPointer = 0;
if (current.exists()) {
if(current.isDirectory()){
files = current.listFiles();
Set<File> set = new TreeSet<File>();
for (int i = 0; i < files.length; i++) {
File file = files[i];
set.add(file);
}
set.toArray(files);
}else{
throw new IllegalArgumentException("File current must be directory");
}
} else {
throw new IllegalArgumentException("File current not exist");
}
}
public String toString(){
return "current="+getCurrent().toString();
}
public int getCurrentPointer() {
return currentPointer;
}
public void setCurrentPointer(int currentPointer) {
this.currentPointer = currentPointer;
}
public File[] getFiles() {
return files;
}
public File getCurrent(){
File ret = null;
try{
ret = getFiles()[getCurrentPointer()];
}catch (Exception e){
}
return ret;
}
public boolean isDirectoryEmpty(){
return !(getFiles().length>0);
}
public Element advancePointer(){
setCurrentPointer(getCurrentPointer()+1);
return this;
}
}
private DirectoryStack(File first){
getStack().push(new Element(first));
}
public static DirectoryStack getNewStack(File first){
return new DirectoryStack(first);
}
public String toString(){
String ret = "stack:\n";
int i = 0;
for (Element elem : stack) {
ret += "nivel " + i++ + elem.toString()+"\n";
}
return ret;
}
private Stack<Element> stack=null;
private Stack<Element> getStack(){
if(stack==null){
stack = new Stack<Element>();
}
return stack;
}
public File getCurrent(){
return getStack().peek().getCurrent();
}
public boolean isDirectoryEmpty(){
return getStack().peek().isDirectoryEmpty();
}
public DirectoryStack downLevel(){
getStack().pop();
return this;
}
public DirectoryStack goToParent(){
return downLevel();
}
public DirectoryStack goIntoDir(){
return upLevel();
}
public DirectoryStack upLevel(){
if(isCurrentNotNull())
getStack().push(new Element(getCurrent()));
return this;
}
public DirectoryStack advancePostition(){
getStack().peek().advancePointer();
return this;
}
public File[] peekDirectory(){
return getStack().peek().getFiles();
}
public boolean isLastFileOfDirectory(){
return getStack().peek().getFiles().length <= getStack().peek().getCurrentPointer();
}
public boolean gotMoreLevels() {
return getStack().size()>0;
}
public boolean gotMoreInCurrentLevel() {
return getStack().peek().getFiles().length > getStack().peek().getCurrentPointer()+1;
}
public boolean isRoot() {
return !(getStack().size()>1);
}
public boolean isCurrentNotNull() {
if(!getStack().isEmpty()){
int currentPointer = getStack().peek().getCurrentPointer();
int maxFiles = getStack().peek().getFiles().length;
return currentPointer < maxFiles;
}else{
return false;
}
}
public boolean isCurrentDirectory() {
return getStack().peek().getCurrent().isDirectory();
}
public boolean isLastFromDirList() {
return getStack().peek().getCurrentPointer() == (getStack().peek().getFiles().length-1);
}
public boolean isCantGoParent() {
return !(getStack().size()>1);
}
public boolean isOutOfDirListRange() {
return getStack().peek().getFiles().length <= getStack().peek().getCurrentPointer();
}
}

Using an Iterable doesn't imply that the Files will be streamed to you. In fact its usually the opposite. So an array is typically faster than an Iterable.

Are you sure it's due to Java, not just a general problem with having 10k entries in one directory, particularly over the network?
Have you tried writing a proof-of-concept program to do the same thing in C using the win32 findfirst/findnext functions to see whether it's any faster?
I don't know the ins and outs of SMB, but I strongly suspect that it needs a round trip for every file in the list - which is not going to be fast, particularly over a network with moderate latency.
Having 10k strings in an array sounds like something which should not tax the modern Java VM too much either.