I built a classic Hoffman code, with encoder and decoder. I noticed that I had a problem, I use code in "bitset", to compress the input file. But the "bitset" - does not decode all the files I send to, for example when I send a txt file, it works great, but when I send other files like BMP. It doesn't work.
Before I used bitset - the code worked - but without any compression - so I'm afraid the problem is with bitset.
The decoder I built is:
public void Decompress(String[] input_names, String[] output_names) {
HuffmanVerticle tree = new HuffmanVerticle();
tree = readTreeFile(output_names);
restoreInput(tree, output_names, input_names);
}
public static void restoreInput(HuffmanVerticle tree, String[] binary_names, String[] original_names) {
BitSet huffmanCodeBit;
try {
FileOutputStream to_original = new FileOutputStream(original_names[0]);
FileInputStream binary = new FileInputStream(binary_names[0]);
ObjectInputStream s = new ObjectInputStream(binary);
huffmanCodeBit = (BitSet) s.readObject();
System.out.println(huffmanCodeBit.toString());
int index = 0;
while(huffmanCodeBit.length() > index)
{
HuffmanVerticle tmp = tree;
while (!tmp.isNullTree())
{
boolean bit = huffmanCodeBit.get(index);
index++;
System.out.println(bit);
if (!bit)
tmp = tmp.left;
else
tmp = tmp.right;
}
to_original.write(tmp.character);
}
binary.close();
to_original.close();
} catch (Exception e) {
e.printStackTrace();
}
}
What am I missing here? Why doesn't the code work for certain files? I'm trying to run the code on some files but it doesn't work, the files that come back don't work.
The code does not work for bmp files at all, even after half an hour, for example txt files, it runs very fast.
Thank for your help.
I can't find how to get ListArray variables from an AndroidJavaObject in C#.
I'm trying to make a for function using a Count for the number of items in the ListArray that is stored as an AndroidJavaObject. But I need to know how to get the Count from the AndroidJavaObject and how to use it properly.
This is what I'm using to get the variables, also notice that "packages" is not an AndroidJavaObject[].
AndroidJavaClass jc = new AndroidJavaClass("com.unity3d.player.UnityPlayer");
AndroidJavaObject currentActivity = jc.GetStatic<AndroidJavaObject>("currentActivity");
int flag = new AndroidJavaClass("android.content.pm.PackageManager").GetStatic<int>("GET_META_DATA");
AndroidJavaObject pm = currentActivity.Call<AndroidJavaObject>("getPackageManager");
AndroidJavaObject packages = pm.Call<AndroidJavaObject>("getInstalledApplications", flag);
it's very rudimentary at this point, but it works thanks to some help from this How to get installed apps list with Unity3D?
The progress thus far stalls at getting the icon, everything else works perfect, I need a way to get either a string that links to the icon somehow, or the Texture2D of the icon. Another alternative would be a to use a AndroidJavaObject that contains a drawable as if it's a Texture2D. I have no idea how to accomplish any of this though.Another Idea I had was to convert it to another variable, like byte[] that can be transferred and converted back, but I have yet to find a method of that which works under the circumstance.
int count = packages.Call<int>("size");
AndroidJavaObject[] links = new AndroidJavaObject[count];
string[] names = new string[count];
Texture2D[] icons = new Texture2D[count];
int ii =0;
for(int i=0; ii<count;){
//get the object
AndroidJavaObject currentObject = packages.Call<AndroidJavaObject>("get", ii );
try{
//try to add the variables to the next entry
links[i] = pm.Call<AndroidJavaObject>("getLaunchIntentForPackage", currentObject.Get<AndroidJavaObject>("processName"));
names[i] = pm.Call<string>("getApplicationLabel", currentObject);
icons[i] = pm.Call<Texture2D>("getApplicationIcon", currentObject);
Debug.Log ("(" + ii + ") " + i +" "+ names[i]);
//go to the next app and entry
i++;
ii++;
}
catch
{
//if it fails, just go to the next app and try to add to that same entry.
Debug.Log("skipped "+ ii);
ii++;
}
}
I really hate to ask two questions in a row here, and I apologize for having to do so, but this is a difficult and seemingly awkward circumstance, that has little to no documentation (that I can find).
First of all the docs on the interop between Unity3D and Android are scarce at best.
The most important thing is that the only interface Unity exposes to work with java object is AndroidJavaObject and that interface has only a couple of methods defined. You can only use those ones and only those.
This means that you don't get a Count object when working with an java array and you'll still be working with AndroidJavaObject.
int count = packages.Call<int>("size"); //getting the size of the array
for( int i = 0; i < count; i++ )
{
//getting the object at location i
AndroidJavaObject currentObject = packages.Call("get", i );
//use currentObject using the same methods as before: Call, CallStatic, Get, GetStatic
}
I know this is verbose, and writing code like this is hard to test, you need to make and apk and deploy it to a device to check that it runs.
Probably a faster way of doing this is to make your own java class that does all this on the java side and expose one method that gets called from the Unity side. This way at least you get the benefit of static typing when making a jar that you'll add in the Plugins/Android folder.
I'm trying to do the same thing here, but my launcher must only show the CardBoard apps. What i've decided is to make a library in java and import it to Unity as a plugin:
This is my Java class:
public class Launcher extends UnityPlayerActivity {
private List<ApplicationInfo> cbApps;
#Override
protected void onStart() {
super.onStart();
PackageManager pm= getPackageManager();
List<ApplicationInfo> lista = pm.getInstalledApplications(PackageManager.GET_META_DATA);
cbApps= new LinkedList<ApplicationInfo>();
for(ApplicationInfo ai : lista){
Intent intentToResolve = new Intent(Intent.ACTION_MAIN);
intentToResolve.addCategory("com.google.intent.category.CARDBOARD");
intentToResolve.setPackage(ai.packageName);
if(pm.resolveActivity(intentToResolve, 0)!=null) {
cbApps.add(ai);
}
}
}
public int getListSize(){
return cbApps.size();
}
And here my C# method:
void Start () {
AndroidJavaClass unity = new AndroidJavaClass ("com.unity3d.player.UnityPlayer");
AndroidJavaObject currentActivity = unity.GetStatic<AndroidJavaObject> ("currentActivity");
texto.text=""+currentActivity.Call<int> ("getListSize");
}
This way I can create in Java every method that I need to acces the list. The problem I'm still having is trying to get the Texture2D of the icons. I've tried returning the Drawable from Java and accesing with a Call just as you did, but it doesn't work at all. I've been 2 days working with that, I'll let you know if I find a solution.
EDIT:
Finally I could get the images:
First in Java I created this method:
public byte[] getIcon(int i) {
BitmapDrawable icon= (BitmapDrawable)context.getPackageManager().getApplicationIcon(cbApps.get(i));
Bitmap bmp = icon.getBitmap();
ByteArrayOutputStream stream = new ByteArrayOutputStream();
bmp.compress(Bitmap.CompressFormat.PNG, 100, stream);
byte[] byteArray = stream.toByteArray();
return byteArray;
}
That way I can access the Byte array representing the Drawable from Unity and show it as the main texture of a Plane:
void Start () {
using (AndroidJavaClass unity = new AndroidJavaClass ("com.unity3d.player.UnityPlayer")) {
context = unity.GetStatic<AndroidJavaObject> ("currentActivity");
}
using (AndroidJavaClass pluginClass=new AndroidJavaClass("com.droiders.launcher.Launcher")) {
launcher= pluginClass.CallStatic<AndroidJavaObject>("getInstance", context);
byte[] bytes= launcher.Call<byte[]>("getIcon",0);
Texture2D tex= new Texture2D(2,2);
tex.LoadImage(bytes);
plane.GetComponent<Renderer>().material.mainTexture=tex;
}
}
Hope it helps you. It's been hard but this beast has been tamed :P
Also thanks you all for your ideas.
I am in the middle of creating an app that allows users to apply for job positions and upload their CVs. I`m currently stuck on trying to make a search box for the admin to be able to search for Keywords. The app will than look through all the CVs and if it finds such keywords it will show up a list of Cvs that contain the keyword. I am fairly new to Gui design and app creation so not sure how to go about doing it. I wish to have it done via java and am using the Eclipse Window builder to help me design it. Any help will be greatly appreciated, hints, advice anything. Thank You.
Well, this not right design approach as real time search of words in all files of given folder will be slow and not sustainable in long run. Ideally you should have indexed all CV's for keywords. The search should run on index and then get the associated CV for that index ( think of indexes similar to tags). There are many options for indexing - simples DB indexing or using Apache Lucene or follow these steps to create a index using Maps and refer this index for search.
Create a map Map<String, List<File>> for keeping the association of
keywords to files
iterate through all files, and for each word in
each file, add that file to the list corresponding to that word in
your index map
here is the java code which will work for you but I would still suggest to change your design approach and use indexes.
File dir = new File("Folder for CV's");
if(dir.exists())
{
Pattern p = Pattern.compile("Java");
ArrayList<String> list = new ArrayList<String>(); // list of CV's
for(File f : dir.listFiles())
{
if(!f.isFile()) continue;
try
{
FileInputStream fis = new FileInputStream(f);
byte[] data = new byte[fis.available()];
fis.read(data);
String text = new String(data);
Matcher m = p.matcher(text);
if(m.find())
{
list.add(f.getName()); // add file to found-keyword list.
}
fis.close();
}
catch(Exception e)
{
System.out.print("\n\t Error processing file : "+f.getName());
}
}
System.out.print("\n\t List : "+list); // list of files containing keyword.
} // IF directory exists then only process.
else
{
System.out.print("\n Directory doesn't exist.");
}
Here you get the files list to show now for "Java". As I said use indexes :)
Thanks for taking your time to look into my problem.
I have actually come up with a solution of my own. It is probably very amateur like but it works for me.
JButton btnSearch = new JButton("Search");
btnSearch.addActionListener(new ActionListener()
{
public void actionPerformed(ActionEvent arg0)
{
list.clear();
String s = SearchBox.getText();
int i = 0,present = 0;
int id;
try
{
Class.forName(driver).newInstance();
Connection conn = DriverManager.getConnection(url+dbName,userName,password);
Statement st = conn.createStatement();
ResultSet res = st.executeQuery("SELECT * FROM javaapp.test");
while(res.next())
{
i = 0;
present = 0;
while(i < 9)
{
String out = res.getString(search[i]);
if(out.toLowerCase().contains(s.toLowerCase()))
{
present = 1;
break;
}
i++;
}
if(tglbtnNormalshortlist.isSelected())
{
if(present == 1 && res.getInt("Shortlist") == 1)
{
id = res.getInt("Candidate");
String print = res.getString("Name");
list.addElement(print+" "+id);
}
}
else
{
if(present == 1 && res.getInt("Shortlist") == 0)
{
id = res.getInt("Candidate");
String print = res.getString("Name");
list.addElement(print+" "+id);
}
}
}
}
catch (Exception e)
{
e.printStackTrace();
}
}
});
Stackoverflowers,
I am doing a simple project using Android smartphones to create 3D forms. I am using Android Processing to make a simple App.
My code makes a 3D shape and saves it as an .STL file. It works on my laptop and saves the .STL file, but in the App. version, I need it to save to the External storage/SD Card of my phone (HTC Sensation). It does not, because of the way the “save” function (writeSTL) in the Processing library I am using has been written.
I have posted for help here (my code more complete code is here too):
http://forum.processing.org/two/discussion/4809/exporting-geometry-stl-obj-dfx-modelbuilder-and-android
...and Marius Watz who wrote the library says that the writeSTL() code is pretty much standalone and the only thing missing is (or should be) replacing the code creating the output stream, which needs to be modified to work with Android. Basically, this line:
FileOutputStream out=(FileOutputStream)UIO.getOutputStream(p.sketchPath(filename));
I am not a great programmer in that I can usually get Processing to do what I need to do but no more; this problem has me beaten. I am looking for ideas for the correct code to replace the line:...
FileOutputStream out=(FileOutputStream)UIO.getOutputStream(p.sketchPath(filename));
...with something “Android-friendly”. Calling getExternalStorageDirectory() should work but I am at a loss to find the correct structure.
The code for the writeSTL function is below.
import java.io.*;
import java.nio.ByteBuffer;
import java.nio.ByteOrder;
/**
* Output binary STL file of mesh geometry.
* #param p Reference to PApplet instance
* #param filename Name of file to save to
*/
public void customWriteSTL(UGeometry geo, PApplet p, String filename) {
byte [] header;
ByteBuffer buf;
UFace f;
try {
if (!filename.toLowerCase().endsWith("stl")) filename+=".stl";
FileOutputStream out=(FileOutputStream)UIO.getOutputStream(p.sketchPath(filename));
buf = ByteBuffer.allocate(200);
header=new byte[80];
buf.get(header, 0, 80);
out.write(header);
buf.rewind();
buf.order(ByteOrder.LITTLE_ENDIAN);
buf.putInt(geo.faceNum);
buf.rewind();
buf.get(header, 0, 4);
out.write(header, 0, 4);
buf.rewind();
UUtil.logDivider("Writing STL '"+filename+"' "+geo.faceNum);
buf.clear();
header=new byte[50];
if (geo.bb!=null) UUtil.log(geo.bb.toString());
for (int i=0; i<geo.faceNum; i++) {
f=geo.face[i];
if (f.n==null) f.calcNormal();
buf.rewind();
buf.putFloat(f.n.x);
buf.putFloat(f.n.y);
buf.putFloat(f.n.z);
for (int j=0; j<3; j++) {
buf.putFloat(f.v[j].x);
buf.putFloat(f.v[j].y);
buf.putFloat(f.v[j].z);
}
buf.rewind();
buf.get(header);
out.write(header);
}
out.flush();
out.close();
UUtil.log("Closing '"+filename+"'. "+geo.faceNum+" triangles written.\n");
}
catch (Exception e) {
e.printStackTrace();
}
}
Any suggestions are gratefully received.
Thank you in advance.
There are a few ways of doing this - some that will just work and some that are proper ... as with all things Processing/Java. It's really not that different from regular Java though - the only quirk is the root SD path, and checking if it exists or not (note that some phones have "internal" rather than "external" storage (i.e. not removable/swappable), but Android should interpret these the same AFAIK.
In classic Java fashion, you should really be checking IF the SD Card is present beforehand... I use the following structure, taken from this answer by #kaolick
String state = Environment.getExternalStorageState();
if (state.equals(Environment.MEDIA_MOUNTED)) {
// Storage is available and writeable - ALL GOOD
} else if (state.equals(Environment.MEDIA_MOUNTED_READ_ONLY)) {
// Storage is only readable - RUH ROH
} else {
// Storage is neither readable nor writeable - ABORT
}
Note that he provides a full class for you to use, which is great, and has a few convenience functions.
The second thing you might want to look at is creating a custom directory on the SD Card of the device, probably in setup() - something like this:
try{
String dirName = "//sdcard//MyAppName";
File newFile = new File(dirName);
if(newFile.exists() && newFile.isDirectory()) {
println("Directory Exists... All Good");
}
else {
println("Directory Doesn't Exist... We're Making It");
newFile.mkdirs();
}
}
catch(Exception e) {
e.printStacktrace();
}
Of course, instead of HardCoding the Path name, you should do something like
String dirName = Environment.getExternalStorageDirectory().getAbsolutePath() + "/MyAppName";
instead...
Also, note that the above try/catch should go INSIDE the case statement of "if (state.equals(Environment.MEDIA_MOUNTED))" ... or should be wrapped in a separate function anc called from there.
Then, finally, saving it. If you wanted to use a BufferedWriter, it would look like this:
BufferedWriter writer = new BufferedWriter(new FileWriter(dirName, true));
writer.write(STL_STUFF);
writer.flush();
writer.close();
I've only use a FileOutputStream within a BufferedOutput Stream, and it looked like this:
try {
String fileName = "SOME_UNIQUE_NAME_PER_FILE";
String localFile = dirName + "/" +filename;
OutputStream output = new BufferedOutputStream(newFileOutputStream(localFile));
}
catch(Exception e) {
e.printStackTrace();
}
Finally, give my regards to Marius if you talk to him! ;-)
I am trying to process files one at a time that are stored over a network. Reading the files is fast due to buffering is not the issue. The problem I have is just listing the directories in a folder. I have at least 10k files per folder over many folders.
Performance is super slow since File.list() returns an array instead of an iterable. Java goes off and collects all the names in a folder and packs it into an array before returning.
The bug entry for this is http://bugs.sun.com/view_bug.do;jsessionid=db7fcf25bcce13541c4289edeb4?bug_id=4285834 and doesn't have a work around. They just say this has been fixed for JDK7.
A few questions:
Does anybody have a workaround to this performance bottleneck?
Am I trying to achieve the impossible? Is performance still going to be poor even if it just iterates over the directories?
Could I use the beta JDK7 builds that have this functionality without having to build my entire project on it?
Although it's not pretty, I solved this kind of problem once by piping the output of dir/ls to a file before starting my app, and passing in the filename.
If you needed to do it within the app, you could just use system.exec(), but it would create some nastiness.
You asked. The first form is going to be blazingly fast, the second should be pretty fast as well.
Be sure to do the one item per line (bare, no decoration, no graphics), full path and recurse options of your selected command.
EDIT:
30 minutes just to get a directory listing, wow.
It just struck me that if you use exec(), you can get it's stdout redirected into a pipe instead of writing it to a file.
If you did that, you should start getting the files immediately and be able to begin processing before the command has completed.
The interaction may actually slow things down, but maybe not--you might give it a try.
Wow, I just went to find the syntax of the .exec command for you and came across this, possibly exactly what you want (it lists a directory using exec and "ls" and pipes the result into your program for processing): good link in wayback (Jörg provided in a comment to replace this one from sun that Oracle broke)
Anyway, the idea is straightforward but getting the code right is annoying. I'll go steal some codes from the internets and hack them up--brb
/**
* Note: Only use this as a last resort! It's specific to windows and even
* at that it's not a good solution, but it should be fast.
*
* to use it, extend FileProcessor and call processFiles("...") with a list
* of options if you want them like /s... I highly recommend /b
*
* override processFile and it will be called once for each line of output.
*/
import java.io.*;
public abstract class FileProcessor
{
public void processFiles(String dirOptions)
{
Process theProcess = null;
BufferedReader inStream = null;
// call the Hello class
try
{
theProcess = Runtime.getRuntime().exec("cmd /c dir " + dirOptions);
}
catch(IOException e)
{
System.err.println("Error on exec() method");
e.printStackTrace();
}
// read from the called program's standard output stream
try
{
inStream = new BufferedReader(
new InputStreamReader( theProcess.getInputStream() ));
processFile(inStream.readLine());
}
catch(IOException e)
{
System.err.println("Error on inStream.readLine()");
e.printStackTrace();
}
} // end method
/** Override this method--it will be called once for each file */
public abstract void processFile(String filename);
} // end class
And thank you code donor at IBM
How about using File.list(FilenameFilter filter) method and implementing FilenameFilter.accept(File dir, String name) to process each file and return false.
I ran this on Linux vm for directory with 10K+ files and it took <10 seconds.
import java.io.File;
import java.io.FilenameFilter;
public class Temp {
private static void processFile(File dir, String name) {
File file = new File(dir, name);
System.out.println("processing file " + file.getName());
}
private static void forEachFile(File dir) {
String [] ignore = dir.list(new FilenameFilter() {
public boolean accept(File dir, String name) {
processFile(dir, name);
return false;
}
});
}
public static void main(String[] args) {
long before, after;
File dot = new File(".");
before = System.currentTimeMillis();
forEachFile(dot);
after = System.currentTimeMillis();
System.out.println("after call, delta is " + (after - before));
}
}
An alternative is to have the files served over a different protocol. As I understand you're using SMB for that and java is just trying to list them as a regular file.
The problem here might not be java alone ( how does it behaves when you open that directory with Microsoft Explorer x:\shared ) In my experience it also take a considerably amount of time.
You can change the protocol to something like HTTP, only to fetch the file names. This way you can retrieve the list of files over http ( 10k lines should't be too much ) and let the server deal with file listing. This would be very fast, since it will run with local resources ( those in the server )
Then when you have the list, you can process them one by exactly the way you're doing right now.
The keypoint is to have an aid mechanism in the other side of the node.
Is this feasible?
Today:
File [] content = new File("X:\\remote\\dir").listFiles();
for ( File f : content ) {
process( f );
}
Proposed:
String [] content = fetchViaHttpTheListNameOf("x:\\remote\\dir");
for ( String fileName : content ) {
process( new File( fileName ) );
}
The http server could be a very small small and simple file.
If this is the way you have it right now, what you're doing is to fetch all the 10k files information to your client machine ( I don't know how much of that info ) when you only need the file name for later processing.
If the processing is very fast right now it may be slowed down a bit. This is because the information prefetched is no longer available.
Give it a try.
A non-portable solution would be to make native calls to the operating system and stream the results.
For Linux
You can look at something like readdir. You can walk the directory structure like a linked list and return results in batches or individually.
For Windows
In windows the behavior would be fairly similar using FindFirstFile and FindNextFile apis.
I doubt the problem is relate to the bug report you referenced.
The issue there is "only" memory usage, but not necessarily speed.
If you have enough memory the bug is not relevant for your problem.
You should measure whether your problem is memory related or not. Turn on your Garbage Collector log and use for example gcviewer to analyze your memory usage.
I suspect that it has to do with the SMB protocol causing the problem.
You can try to write a test in another language and see if it's faster, or you can try to get the list of filenames through some other method, such as described here in another post.
If you need to eventually process all files, then having Iterable over String[] won't give you any advantage, as you'll still have to go and fetch the whole list of files.
If you're on Java 1.5 or 1.6, shelling out "dir" commands and parsing the standard output stream on Windows is a perfectly acceptable approach. I've used this approach in the past for processing network drives and it has generally been a lot faster than waiting for the native java.io.File listFiles() method to return.
Of course, a JNI call should be faster and potentially safer than shelling out "dir" commands. The following JNI code can be used to retrieve a list of files/directories using the Windows API. This function can be easily refactored into a new class so the caller can retrieve file paths incrementally (i.e. get one path at a time). For example, you can refactor the code so that FindFirstFileW is called in a constructor and have a seperate method to call FindNextFileW.
JNIEXPORT jstring JNICALL Java_javaxt_io_File_GetFiles(JNIEnv *env, jclass, jstring directory)
{
HANDLE hFind;
try {
//Convert jstring to wstring
const jchar *_directory = env->GetStringChars(directory, 0);
jsize x = env->GetStringLength(directory);
wstring path; //L"C:\\temp\\*";
path.assign(_directory, _directory + x);
env->ReleaseStringChars(directory, _directory);
if (x<2){
jclass exceptionClass = env->FindClass("java/lang/Exception");
env->ThrowNew(exceptionClass, "Invalid path, less than 2 characters long.");
}
wstringstream ss;
BOOL bContinue = TRUE;
WIN32_FIND_DATAW data;
hFind = FindFirstFileW(path.c_str(), &data);
if (INVALID_HANDLE_VALUE == hFind){
jclass exceptionClass = env->FindClass("java/lang/Exception");
env->ThrowNew(exceptionClass, "FindFirstFileW returned invalid handle.");
}
//HANDLE hStdOut = GetStdHandle(STD_OUTPUT_HANDLE);
//DWORD dwBytesWritten;
// If we have no error, loop thru the files in this dir
while (hFind && bContinue){
/*
//Debug Print Statment. DO NOT DELETE! cout and wcout do not print unicode correctly.
WriteConsole(hStdOut, data.cFileName, (DWORD)_tcslen(data.cFileName), &dwBytesWritten, NULL);
WriteConsole(hStdOut, L"\n", 1, &dwBytesWritten, NULL);
*/
//Check if this entry is a directory
if (data.dwFileAttributes & FILE_ATTRIBUTE_DIRECTORY){
// Make sure this dir is not . or ..
if (wstring(data.cFileName) != L"." &&
wstring(data.cFileName) != L"..")
{
ss << wstring(data.cFileName) << L"\\" << L"\n";
}
}
else{
ss << wstring(data.cFileName) << L"\n";
}
bContinue = FindNextFileW(hFind, &data);
}
FindClose(hFind); // Free the dir structure
wstring cstr = ss.str();
int len = cstr.size();
//WriteConsole(hStdOut, cstr.c_str(), len, &dwBytesWritten, NULL);
//WriteConsole(hStdOut, L"\n", 1, &dwBytesWritten, NULL);
jchar* raw = new jchar[len];
memcpy(raw, cstr.c_str(), len*sizeof(wchar_t));
jstring result = env->NewString(raw, len);
delete[] raw;
return result;
}
catch(...){
FindClose(hFind);
jclass exceptionClass = env->FindClass("java/lang/Exception");
env->ThrowNew(exceptionClass, "Exception occured.");
}
return NULL;
}
Credit:
https://sites.google.com/site/jozsefbekes/Home/windows-programming/miscellaneous-functions
Even with this approach, there are still efficiencies to be gained. If you serialize the path to a java.io.File, there is a huge performance hit - especially if the path represents a file on a network drive. I have no idea what Sun/Oracle is doing under the hood but if you need additional file attributes other than the file path (e.g. size, mod date, etc), I have found that the following JNI function is much faster than instantiating a java.io.File object on a network the path.
JNIEXPORT jlongArray JNICALL Java_javaxt_io_File_GetFileAttributesEx(JNIEnv *env, jclass, jstring filename)
{
//Convert jstring to wstring
const jchar *_filename = env->GetStringChars(filename, 0);
jsize len = env->GetStringLength(filename);
wstring path;
path.assign(_filename, _filename + len);
env->ReleaseStringChars(filename, _filename);
//Get attributes
WIN32_FILE_ATTRIBUTE_DATA fileAttrs;
BOOL result = GetFileAttributesExW(path.c_str(), GetFileExInfoStandard, &fileAttrs);
if (!result) {
jclass exceptionClass = env->FindClass("java/lang/Exception");
env->ThrowNew(exceptionClass, "Exception Occurred");
}
//Create an array to store the WIN32_FILE_ATTRIBUTE_DATA
jlong buffer[6];
buffer[0] = fileAttrs.dwFileAttributes;
buffer[1] = date2int(fileAttrs.ftCreationTime);
buffer[2] = date2int(fileAttrs.ftLastAccessTime);
buffer[3] = date2int(fileAttrs.ftLastWriteTime);
buffer[4] = fileAttrs.nFileSizeHigh;
buffer[5] = fileAttrs.nFileSizeLow;
jlongArray jLongArray = env->NewLongArray(6);
env->SetLongArrayRegion(jLongArray, 0, 6, buffer);
return jLongArray;
}
You can find a full working example of this JNI-based approach in the javaxt-core library. In my tests using Java 1.6.0_38 with a Windows host hitting a Windows share, I have found this JNI approach approximately 10x faster then calling java.io.File listFiles() or shelling out "dir" commands.
I wonder why there are 10k files in a directory. Some file systems do not work well with so many files. There are specifics limitations for file systems like max amount of files per directory and max amount of levels of subdirectory.
I solve a similar problem with an iterator solution.
I needed to walk across huge directorys and several levels of directory tree recursively.
I try FileUtils.iterateFiles() of Apache commons io. But it implement the iterator by adding all the files in a List and then returning List.iterator(). It's very bad for memory.
So I prefer to write something like this:
private static class SequentialIterator implements Iterator<File> {
private DirectoryStack dir = null;
private File current = null;
private long limit;
private FileFilter filter = null;
public SequentialIterator(String path, long limit, FileFilter ff) {
current = new File(path);
this.limit = limit;
filter = ff;
dir = DirectoryStack.getNewStack(current);
}
public boolean hasNext() {
while(walkOver());
return isMore && (limit > count || limit < 0) && dir.getCurrent() != null;
}
private long count = 0;
public File next() {
File aux = dir.getCurrent();
dir.advancePostition();
count++;
return aux;
}
private boolean walkOver() {
if (dir.isOutOfDirListRange()) {
if (dir.isCantGoParent()) {
isMore = false;
return false;
} else {
dir.goToParent();
dir.advancePostition();
return true;
}
} else {
if (dir.isCurrentDirectory()) {
if (dir.isDirectoryEmpty()) {
dir.advancePostition();
} else {
dir.goIntoDir();
}
return true;
} else {
if (filter.accept(dir.getCurrent())) {
return false;
} else {
dir.advancePostition();
return true;
}
}
}
}
private boolean isMore = true;
public void remove() {
throw new UnsupportedOperationException();
}
}
Note that the iterator stop by an amount of files iterateds and it has a FileFilter also.
And DirectoryStack is:
public class DirectoryStack {
private class Element{
private File files[] = null;
private int currentPointer;
public Element(File current) {
currentPointer = 0;
if (current.exists()) {
if(current.isDirectory()){
files = current.listFiles();
Set<File> set = new TreeSet<File>();
for (int i = 0; i < files.length; i++) {
File file = files[i];
set.add(file);
}
set.toArray(files);
}else{
throw new IllegalArgumentException("File current must be directory");
}
} else {
throw new IllegalArgumentException("File current not exist");
}
}
public String toString(){
return "current="+getCurrent().toString();
}
public int getCurrentPointer() {
return currentPointer;
}
public void setCurrentPointer(int currentPointer) {
this.currentPointer = currentPointer;
}
public File[] getFiles() {
return files;
}
public File getCurrent(){
File ret = null;
try{
ret = getFiles()[getCurrentPointer()];
}catch (Exception e){
}
return ret;
}
public boolean isDirectoryEmpty(){
return !(getFiles().length>0);
}
public Element advancePointer(){
setCurrentPointer(getCurrentPointer()+1);
return this;
}
}
private DirectoryStack(File first){
getStack().push(new Element(first));
}
public static DirectoryStack getNewStack(File first){
return new DirectoryStack(first);
}
public String toString(){
String ret = "stack:\n";
int i = 0;
for (Element elem : stack) {
ret += "nivel " + i++ + elem.toString()+"\n";
}
return ret;
}
private Stack<Element> stack=null;
private Stack<Element> getStack(){
if(stack==null){
stack = new Stack<Element>();
}
return stack;
}
public File getCurrent(){
return getStack().peek().getCurrent();
}
public boolean isDirectoryEmpty(){
return getStack().peek().isDirectoryEmpty();
}
public DirectoryStack downLevel(){
getStack().pop();
return this;
}
public DirectoryStack goToParent(){
return downLevel();
}
public DirectoryStack goIntoDir(){
return upLevel();
}
public DirectoryStack upLevel(){
if(isCurrentNotNull())
getStack().push(new Element(getCurrent()));
return this;
}
public DirectoryStack advancePostition(){
getStack().peek().advancePointer();
return this;
}
public File[] peekDirectory(){
return getStack().peek().getFiles();
}
public boolean isLastFileOfDirectory(){
return getStack().peek().getFiles().length <= getStack().peek().getCurrentPointer();
}
public boolean gotMoreLevels() {
return getStack().size()>0;
}
public boolean gotMoreInCurrentLevel() {
return getStack().peek().getFiles().length > getStack().peek().getCurrentPointer()+1;
}
public boolean isRoot() {
return !(getStack().size()>1);
}
public boolean isCurrentNotNull() {
if(!getStack().isEmpty()){
int currentPointer = getStack().peek().getCurrentPointer();
int maxFiles = getStack().peek().getFiles().length;
return currentPointer < maxFiles;
}else{
return false;
}
}
public boolean isCurrentDirectory() {
return getStack().peek().getCurrent().isDirectory();
}
public boolean isLastFromDirList() {
return getStack().peek().getCurrentPointer() == (getStack().peek().getFiles().length-1);
}
public boolean isCantGoParent() {
return !(getStack().size()>1);
}
public boolean isOutOfDirListRange() {
return getStack().peek().getFiles().length <= getStack().peek().getCurrentPointer();
}
}
Using an Iterable doesn't imply that the Files will be streamed to you. In fact its usually the opposite. So an array is typically faster than an Iterable.
Are you sure it's due to Java, not just a general problem with having 10k entries in one directory, particularly over the network?
Have you tried writing a proof-of-concept program to do the same thing in C using the win32 findfirst/findnext functions to see whether it's any faster?
I don't know the ins and outs of SMB, but I strongly suspect that it needs a round trip for every file in the list - which is not going to be fast, particularly over a network with moderate latency.
Having 10k strings in an array sounds like something which should not tax the modern Java VM too much either.