ResourceTool : to recover a jpg with ResourceNode [velocity] - java

During my searching, I would like a piece of advice about the following situation :
the guy, on my website, choose a parcel to send, when he validates the choice, some carriers appear as results. Now some carriers have different offers with different logos located in a special directory.
Now the business logic I would like to use is :
If in the directory I find the peculiar logo corresponding to the special offer, I will take the logo to display It in the web page with the special offer.
I choose to do this work with the ResourceTool from Velocity
I have to implement 2 methods getLogo() and getLabel().
The getLogo() will look for the special logo.
I think to use this method to recover the object :
public static ResourceNode getResource(Context context, ResourceType resourceType, String...keys) {
try {
if (null != ResourcesTool.instance) {
ResourceNode resource = ResourcesTool.instance.getResourceSet(context, resourceType);
if (null != resource) {
Deque < String > keyDeque = new ArrayDeque < > ();
for (String key: keys) {
keyDeque.add(key);
}
return (ResourceNode) resource.sub(keyDeque);
}
}
} catch (Exception e) {
BoxtaleLogger.debug("[ResourcesTool.getResource] error: ", e);
}
return null;
}
Now I am searching a example to merely use this method to recover the different .jpg
question 2 : I don't understand what is the meaningful of Context context in this method ?
Then the resourceType is an enum either a String or a picture (the logo in fact)

All right I found It :
public String getLogo(ResourceNode node){
//readable variables
String ope_code = (String)((Instance)get("operateur")).get("ope_code");
String paysDest = ((Instance)db.getEntity("tab_pays").fetch((Integer) get("pz_id"))).get("pz_iso");
String path = node.get(ope_code+"_"+get("srv_code")+"_"+paysDest+".png");
if (path=null){
path = node.get(ope_code+"_"+get("srv_code")+".png");
if (path=null){
path = node.get(ope_code+".png");
}
}
return path ;
}
Now I am testing the method, I will tell you after.

Related

How to get Camera RAW metadata info programmatically Android

I have an app, which sets the hardware parameters of the Camera programmatically.
However, as I've been told, and have come to observe, not all chipsets support all parameters.
For example, the Nexus 4 (Qualcomm) has sharpness, and sharpness-max parameters, the Galaxy Note II 3g doesn't have any.
Hence, when I set sharpness parameter, the Nexus responds well, but the Galaxy force closes:
java.lang.RuntimeException: setParameters failed
at android.hardware.Camera.native_setParameters(Native Method)
at android.hardware.Camera.setParameters(Camera.java:1452)
My question is, how can I get the RAW info programmatically? I need to get the parameters, their values, and whether they exist or not.
I wish to get the RAW-Metadata parameters, as like this: database
Alright, thought this would be a fun bit of practice. So, Android does not give a public API into this information. Why? I have no idea. Looks like you can do a Camera.Parameters#get(String) to check for any particular parameter that you're interested in, but lets say you're greedy and want the whole list to yourself. In that case, we can dive in using Reflection, but be aware that there is a strong possibility that this will not work on all versions of Android or may break in future versions. With that said, here's how you do it:
private static Map<String, String> getFullCameraParameters (Camera cam) {
Map<String, String> result = new HashMap<String, String>(64);
final String TAG = "CameraParametersRetrieval";
try {
Class camClass = cam.getClass();
//Internally, Android goes into native code to retrieve this String of values
Method getNativeParams = camClass.getDeclaredMethod("native_getParameters");
getNativeParams.setAccessible(true);
//Boom. Here's the raw String from the hardware
String rawParamsStr = (String) getNativeParams.invoke(cam);
//But let's do better. Here's what Android uses to parse the
//String into a usable Map -- a simple ';' StringSplitter, followed
//by splitting on '='
//
//Taken from Camera.Parameters unflatten() method
TextUtils.StringSplitter splitter = new TextUtils.SimpleStringSplitter(';');
splitter.setString(rawParamsStr);
for (String kv : splitter) {
int pos = kv.indexOf('=');
if (pos == -1) {
continue;
}
String k = kv.substring(0, pos);
String v = kv.substring(pos + 1);
result.put(k, v);
}
//And voila, you have a map of ALL supported parameters
return result;
} catch (NoSuchMethodException ex) {
Log.e(TAG, ex.toString());
} catch (IllegalAccessException ex) {
Log.e(TAG, ex.toString());
} catch (InvocationTargetException ex) {
Log.e(TAG, ex.toString());
}
//If there was any error, just return an empty Map
Log.e(TAG, "Unable to retrieve parameters from Camera.");
return result;
}

How can I create a node recursively using zookeeper client library on Java?

I know this question has been already asked and answered for a zookeeper using python. The answer was good, however, I want something more related with the code. I've already implemented a method to create a node, but I want to do it recursively. The structure for my nodes will be like this:
ZOOKEEPER
WEB SERVER
SERVER1
SERVER2
MODULE CONNECTED
DATABASE MODULE
COMPUTER1
COMPUTER2
SERVICE MODULE
COMPUTER3
SEARCH MODULE
COMPUTER4
I have something like:
Zookeeper zk = new Zookeeper(...);
public void createNodeRecursively(String type) {
final String node = "/" + type + "/" + info.getIP() + ":" + info.getPort(); // Correct line
if (zk.exists("/" + type, null) == null) {
Object ctx = new Object();
StringCallback cb = new StringCallback() {
public void processResult(int rc, String path,
Object ctx, String name) {
if (name.equals("/" + type))// just in case
try {
zk.create(node, info.getBytes(),
Ids.OPEN_ACL_UNSAFE,CreateMode.EPHEMERAL);
} catch (Exception e) {
e.printStackTrace();
}
}
};
zk.create("/" + type, info.getBytes(), Ids.OPEN_ACL_UNSAFE,
CreateMode.PERSISTENT, cb, ctx);
} else
zk.create(node, info.getBytes(), Ids.OPEN_ACL_UNSAFE,
CreateMode.EPHEMERAL);
}
}
As you can see I am using zk.create many times, so I want to make the method recursive in order to gain performance and have a better code, but I don't know how to start, I'll be very grateful if somebody can help me with this. Thank you very much in advance.
Zookeeper has useful properties:
Total order of (write) requests
Its asynchronous nature.
You can put on use that.
Simply issue whole tree as a bunch of asynchronous requests in correct order and then wait, until all of them successfully execute. Of course, you can ignore 'NodeExists' exceptions (but it is not good, due of the fact, that such errors will be written to logs).
I managed to achieve a better performance:
public void createNode(NodePath nodePath, NodeData nodeData, NodeRights nodeRights, NodeCreationHandler nodeCreationHandler) throws KeeperException, InterruptedException, ZookeeperCreationException {
if (zk == null) {
throw new ZookeeperCreationException("The zookeeper client has not been instanced.");
}
String targetPath = nodePath.getFullNodePath();
targetPath = targetPath.substring(1, targetPath.length());
byte[] serializedData = nodeData.serialize(new Object());
String[] array = targetPath.split(ICoordinationConstants.BASE_ROOT_SPTR);
String acum="";
for (int i = 0; i < array.length-1; i++) {
acum+=(ICoordinationConstants.BASE_ROOT_SPTR+array[i]);
if (zk.exists(acum, null) == null) {
zk.create(acum, serializedData, Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
}
}
zk.create(acum+ICoordinationConstants.BASE_ROOT_SPTR+array[array.length-1], serializedData, Ids.OPEN_ACL_UNSAFE, CreateMode.EPHEMERAL);
}

Android - getting the R.raw.value of a file based on filename?

I have a file in the res/raw folder called "book1tabs.txt", but in general I will not know what it is called. Then I have to do something like follows:
InputStream in = this.mCtx.getResources().openRawResource(R.raw.book1tabs);
But I want to use a string variable, like
String param = "book1tabs";
And be able to open up that same input stream.
Is there any way to do this?
Thanks
You can do something like this
String param = "book1tabs";
InputStream in = this.mCtx.getResources().openRawResource(mCtx.getResources().getIdentifier(param,"raw", mCtx.getPackageName()));
getIdentifier() will return the id from R.java of particular parameter you passed.
Exactly what it will do is as follow
Maps the Package Name
Navigate to typeDef you provided
Finds the resource name you provided in the typeDef
For more information http://developer.android.com/reference/android/content/res/Resources.html
I find this method to be very useful to pull all sorts of resources by their string names...
#SuppressWarnings("rawtypes")
public static int getResourceId(String name, Class resType){
try {
Class res = null;
if(resType == R.drawable.class)
res = R.drawable.class;
if(resType == R.id.class)
res = R.id.class;
if(resType == R.string.class)
res = R.string.class;
if(resType == R.raw.class)
res = R.raw.class;
Field field = res.getField(name);
int retId = field.getInt(null);
return retId;
}
catch (Exception e) {
// Log.d(TAG, "Failure to get drawable id.", e);
}
return 0;
}
This will return the numeric id (assuming such a resource exists). For the Class pass in R.drawable and for the string whatever it's xml based ID name is.
I always have this method hanging around all my projects for easy access.

Relative paths in Flying Saucer XHTML?

I am using Flying Saucer to render some PDF documents from strings to XHTML. My code is something like:
iTextRenderer.setDocument(documentGenerator.generate(xhtmlDocumentAsString));
iTextRenderer.layout();
iTextRenderer.createPDF(outputStream);
What I'm trying to understand is, when using this method, where are relative paths in the XHTML resolved from? For example, for images or stylesheets. I am able to use this method to successfully generate a text-based document, but I need to understand how to reference my images and CSS.
The setDocument() method takes two parameters: document and url.
The url parameter indicates the base url used to prepend to relative paths that appear in the xhtml, such as in img tags.
Suppose you have:
<img src="images/img1.jpg">
Now suppose the folder "images" is located at:
C:/physical/route/to/app/images/
You may use setDocument() as:
renderer.setDocument(xhtmlDoc, "file:///C:/physical/route/to/app/");
Notice the trailing slash, it won't work without it.
This is the way it worked for me. I assume you could use other types of urls such as "http://...".
This week I worked on this, and I give you what worked fine for me.
In real life, your XHTML document points to multiple resources (images, css, etc.) with relative paths.
You also have to explain to Flying Saucer where to find them. They can be in your classpath, or in your file system. (If they are on the network, you can just set the base url, so this won't help)
So you have to extend the ITextUserAgent like this:
private static class ResourceLoaderUserAgent extends ITextUserAgent {
public ResourceLoaderUserAgent(ITextOutputDevice outputDevice) {
super(outputDevice);
}
protected InputStream resolveAndOpenStream(String uri) {
InputStream is = super.resolveAndOpenStream(uri);
String fileName = "";
try {
String[] split = uri.split("/");
fileName = split[split.length - 1];
} catch (Exception e) {
return null;
}
if (is == null) {
// Resource is on the classpath
try{
is = ResourceLoaderUserAgent.class.getResourceAsStream("/etc/images/" + fileName);
} catch (Exception e) {
}
if (is == null) {
// Resource is in the file system
try {
is = new FileInputStream(new File("C:\\images\\" + fileName));
} catch (Exception e) {
}
}
return is;
}
}
And you use it like this:
// Output stream containing the result
ByteArrayOutputStream baos = new ByteArrayOutputStream();
ITextRenderer renderer = new ITextRenderer();
ResourceLoaderUserAgent callback = new ResourceLoaderUserAgent(renderer.getOutputDevice());
callback.setSharedContext(renderer.getSharedContext());
renderer.getSharedContext().setUserAgentCallback(callback);
renderer.setDocumentFromString(htmlSourceAsString);
renderer.layout();
renderer.createPDF(baos);
renderer.finishPDF();
Cheers.
The best solution for me was:
renderer.setDocumentFromString(htmlContent, new ClassPathResource("/META-INF/pdfTemplates/").getURL().toExternalForm());
Then all the provided styles and images in html (like
<img class="logo" src="images/logo.png" />
<link rel="stylesheet" type="text/css" media="all" href="css/style.css"></link>
) were rendered as expected.
AtilaUy's answer is spot-on for the default way things work in Flying Saucer.
The more general answer is that it asks the UserAgentContext. It will call setBaseURL() on the UserAgentContext when the document is set in. Then it will call resolveURL() to resolve relative URLs and ultimately resolveAndOpenStream() when it wants to read the actual resource data.
Well, this answer is probably way too late for you to make use of it anyway, but I needed an answer like this when I set out, and setting a custom user agent context is the solution I ended up using.
You can either have file paths, which should be absolute, or http:// urls. Relative paths can work but aren't portable because it depends on what directory you ran your program from
I think a easier approach would be:
DomNodeList<DomElement> images = result.getElementsByTagName("img");
for (DomElement e : images) {
e.setAttribute("src", result.getFullyQualifiedUrl(e.getAttribute("src")).toString());
}
Another way to resolve paths is to override UserAgentCallback#resolveURI, which offers a more dynamic behavior than a fixed URL (as in AtilaUy's answer, which looks quite valid for most cases).
This is how I make an XHTMLPane use dynamically-generated stylesheets:
public static UserAgentCallback interceptCssResourceLoading(
final UserAgentCallback defaultAgentCallback,
final Map< URI, CSSResource > cssResources
) {
return new UserAgentCallback() {
#Override
public CSSResource getCSSResource( final String uriAsString ) {
final URI uri = uriQuiet( uriAsString ) ; // Just rethrow unchecked exception.
final CSSResource cssResource = cssResources.get( uri ) ;
if( cssResource == null ) {
return defaultAgentCallback.getCSSResource( uriAsString ) ;
} else {
return cssResource ;
}
}
#Override
public String resolveURI( final String uriAsString ) {
final URI uri = uriQuiet( uriAsString ) ;
if( cssResources.containsKey( uri ) ) {
return uriAsString ;
} else {
return defaultAgentCallback.resolveURI( uriAsString ) ;
}
}
// Delegate all other methods to defaultUserAgentCallback.
} ;
}
Then I use it like that:
final UserAgentCallback defaultAgentCallback =
xhtmlPanel.getSharedContext().getUserAgentCallback() ;
xhtmlPanel.getSharedContext().setUserAgentCallback(
interceptCssResourceLoading( defaultAgentCallback, cssResources ) ) ;
xhtmlPanel.setDocumentFromString( xhtml, null, new XhtmlNamespaceHandler() ) ;

Is there a workaround for Java's poor performance on walking huge directories?

I am trying to process files one at a time that are stored over a network. Reading the files is fast due to buffering is not the issue. The problem I have is just listing the directories in a folder. I have at least 10k files per folder over many folders.
Performance is super slow since File.list() returns an array instead of an iterable. Java goes off and collects all the names in a folder and packs it into an array before returning.
The bug entry for this is http://bugs.sun.com/view_bug.do;jsessionid=db7fcf25bcce13541c4289edeb4?bug_id=4285834 and doesn't have a work around. They just say this has been fixed for JDK7.
A few questions:
Does anybody have a workaround to this performance bottleneck?
Am I trying to achieve the impossible? Is performance still going to be poor even if it just iterates over the directories?
Could I use the beta JDK7 builds that have this functionality without having to build my entire project on it?
Although it's not pretty, I solved this kind of problem once by piping the output of dir/ls to a file before starting my app, and passing in the filename.
If you needed to do it within the app, you could just use system.exec(), but it would create some nastiness.
You asked. The first form is going to be blazingly fast, the second should be pretty fast as well.
Be sure to do the one item per line (bare, no decoration, no graphics), full path and recurse options of your selected command.
EDIT:
30 minutes just to get a directory listing, wow.
It just struck me that if you use exec(), you can get it's stdout redirected into a pipe instead of writing it to a file.
If you did that, you should start getting the files immediately and be able to begin processing before the command has completed.
The interaction may actually slow things down, but maybe not--you might give it a try.
Wow, I just went to find the syntax of the .exec command for you and came across this, possibly exactly what you want (it lists a directory using exec and "ls" and pipes the result into your program for processing): good link in wayback (Jörg provided in a comment to replace this one from sun that Oracle broke)
Anyway, the idea is straightforward but getting the code right is annoying. I'll go steal some codes from the internets and hack them up--brb
/**
* Note: Only use this as a last resort! It's specific to windows and even
* at that it's not a good solution, but it should be fast.
*
* to use it, extend FileProcessor and call processFiles("...") with a list
* of options if you want them like /s... I highly recommend /b
*
* override processFile and it will be called once for each line of output.
*/
import java.io.*;
public abstract class FileProcessor
{
public void processFiles(String dirOptions)
{
Process theProcess = null;
BufferedReader inStream = null;
// call the Hello class
try
{
theProcess = Runtime.getRuntime().exec("cmd /c dir " + dirOptions);
}
catch(IOException e)
{
System.err.println("Error on exec() method");
e.printStackTrace();
}
// read from the called program's standard output stream
try
{
inStream = new BufferedReader(
new InputStreamReader( theProcess.getInputStream() ));
processFile(inStream.readLine());
}
catch(IOException e)
{
System.err.println("Error on inStream.readLine()");
e.printStackTrace();
}
} // end method
/** Override this method--it will be called once for each file */
public abstract void processFile(String filename);
} // end class
And thank you code donor at IBM
How about using File.list(FilenameFilter filter) method and implementing FilenameFilter.accept(File dir, String name) to process each file and return false.
I ran this on Linux vm for directory with 10K+ files and it took <10 seconds.
import java.io.File;
import java.io.FilenameFilter;
public class Temp {
private static void processFile(File dir, String name) {
File file = new File(dir, name);
System.out.println("processing file " + file.getName());
}
private static void forEachFile(File dir) {
String [] ignore = dir.list(new FilenameFilter() {
public boolean accept(File dir, String name) {
processFile(dir, name);
return false;
}
});
}
public static void main(String[] args) {
long before, after;
File dot = new File(".");
before = System.currentTimeMillis();
forEachFile(dot);
after = System.currentTimeMillis();
System.out.println("after call, delta is " + (after - before));
}
}
An alternative is to have the files served over a different protocol. As I understand you're using SMB for that and java is just trying to list them as a regular file.
The problem here might not be java alone ( how does it behaves when you open that directory with Microsoft Explorer x:\shared ) In my experience it also take a considerably amount of time.
You can change the protocol to something like HTTP, only to fetch the file names. This way you can retrieve the list of files over http ( 10k lines should't be too much ) and let the server deal with file listing. This would be very fast, since it will run with local resources ( those in the server )
Then when you have the list, you can process them one by exactly the way you're doing right now.
The keypoint is to have an aid mechanism in the other side of the node.
Is this feasible?
Today:
File [] content = new File("X:\\remote\\dir").listFiles();
for ( File f : content ) {
process( f );
}
Proposed:
String [] content = fetchViaHttpTheListNameOf("x:\\remote\\dir");
for ( String fileName : content ) {
process( new File( fileName ) );
}
The http server could be a very small small and simple file.
If this is the way you have it right now, what you're doing is to fetch all the 10k files information to your client machine ( I don't know how much of that info ) when you only need the file name for later processing.
If the processing is very fast right now it may be slowed down a bit. This is because the information prefetched is no longer available.
Give it a try.
A non-portable solution would be to make native calls to the operating system and stream the results.
For Linux
You can look at something like readdir. You can walk the directory structure like a linked list and return results in batches or individually.
For Windows
In windows the behavior would be fairly similar using FindFirstFile and FindNextFile apis.
I doubt the problem is relate to the bug report you referenced.
The issue there is "only" memory usage, but not necessarily speed.
If you have enough memory the bug is not relevant for your problem.
You should measure whether your problem is memory related or not. Turn on your Garbage Collector log and use for example gcviewer to analyze your memory usage.
I suspect that it has to do with the SMB protocol causing the problem.
You can try to write a test in another language and see if it's faster, or you can try to get the list of filenames through some other method, such as described here in another post.
If you need to eventually process all files, then having Iterable over String[] won't give you any advantage, as you'll still have to go and fetch the whole list of files.
If you're on Java 1.5 or 1.6, shelling out "dir" commands and parsing the standard output stream on Windows is a perfectly acceptable approach. I've used this approach in the past for processing network drives and it has generally been a lot faster than waiting for the native java.io.File listFiles() method to return.
Of course, a JNI call should be faster and potentially safer than shelling out "dir" commands. The following JNI code can be used to retrieve a list of files/directories using the Windows API. This function can be easily refactored into a new class so the caller can retrieve file paths incrementally (i.e. get one path at a time). For example, you can refactor the code so that FindFirstFileW is called in a constructor and have a seperate method to call FindNextFileW.
JNIEXPORT jstring JNICALL Java_javaxt_io_File_GetFiles(JNIEnv *env, jclass, jstring directory)
{
HANDLE hFind;
try {
//Convert jstring to wstring
const jchar *_directory = env->GetStringChars(directory, 0);
jsize x = env->GetStringLength(directory);
wstring path; //L"C:\\temp\\*";
path.assign(_directory, _directory + x);
env->ReleaseStringChars(directory, _directory);
if (x<2){
jclass exceptionClass = env->FindClass("java/lang/Exception");
env->ThrowNew(exceptionClass, "Invalid path, less than 2 characters long.");
}
wstringstream ss;
BOOL bContinue = TRUE;
WIN32_FIND_DATAW data;
hFind = FindFirstFileW(path.c_str(), &data);
if (INVALID_HANDLE_VALUE == hFind){
jclass exceptionClass = env->FindClass("java/lang/Exception");
env->ThrowNew(exceptionClass, "FindFirstFileW returned invalid handle.");
}
//HANDLE hStdOut = GetStdHandle(STD_OUTPUT_HANDLE);
//DWORD dwBytesWritten;
// If we have no error, loop thru the files in this dir
while (hFind && bContinue){
/*
//Debug Print Statment. DO NOT DELETE! cout and wcout do not print unicode correctly.
WriteConsole(hStdOut, data.cFileName, (DWORD)_tcslen(data.cFileName), &dwBytesWritten, NULL);
WriteConsole(hStdOut, L"\n", 1, &dwBytesWritten, NULL);
*/
//Check if this entry is a directory
if (data.dwFileAttributes & FILE_ATTRIBUTE_DIRECTORY){
// Make sure this dir is not . or ..
if (wstring(data.cFileName) != L"." &&
wstring(data.cFileName) != L"..")
{
ss << wstring(data.cFileName) << L"\\" << L"\n";
}
}
else{
ss << wstring(data.cFileName) << L"\n";
}
bContinue = FindNextFileW(hFind, &data);
}
FindClose(hFind); // Free the dir structure
wstring cstr = ss.str();
int len = cstr.size();
//WriteConsole(hStdOut, cstr.c_str(), len, &dwBytesWritten, NULL);
//WriteConsole(hStdOut, L"\n", 1, &dwBytesWritten, NULL);
jchar* raw = new jchar[len];
memcpy(raw, cstr.c_str(), len*sizeof(wchar_t));
jstring result = env->NewString(raw, len);
delete[] raw;
return result;
}
catch(...){
FindClose(hFind);
jclass exceptionClass = env->FindClass("java/lang/Exception");
env->ThrowNew(exceptionClass, "Exception occured.");
}
return NULL;
}
Credit:
https://sites.google.com/site/jozsefbekes/Home/windows-programming/miscellaneous-functions
Even with this approach, there are still efficiencies to be gained. If you serialize the path to a java.io.File, there is a huge performance hit - especially if the path represents a file on a network drive. I have no idea what Sun/Oracle is doing under the hood but if you need additional file attributes other than the file path (e.g. size, mod date, etc), I have found that the following JNI function is much faster than instantiating a java.io.File object on a network the path.
JNIEXPORT jlongArray JNICALL Java_javaxt_io_File_GetFileAttributesEx(JNIEnv *env, jclass, jstring filename)
{
//Convert jstring to wstring
const jchar *_filename = env->GetStringChars(filename, 0);
jsize len = env->GetStringLength(filename);
wstring path;
path.assign(_filename, _filename + len);
env->ReleaseStringChars(filename, _filename);
//Get attributes
WIN32_FILE_ATTRIBUTE_DATA fileAttrs;
BOOL result = GetFileAttributesExW(path.c_str(), GetFileExInfoStandard, &fileAttrs);
if (!result) {
jclass exceptionClass = env->FindClass("java/lang/Exception");
env->ThrowNew(exceptionClass, "Exception Occurred");
}
//Create an array to store the WIN32_FILE_ATTRIBUTE_DATA
jlong buffer[6];
buffer[0] = fileAttrs.dwFileAttributes;
buffer[1] = date2int(fileAttrs.ftCreationTime);
buffer[2] = date2int(fileAttrs.ftLastAccessTime);
buffer[3] = date2int(fileAttrs.ftLastWriteTime);
buffer[4] = fileAttrs.nFileSizeHigh;
buffer[5] = fileAttrs.nFileSizeLow;
jlongArray jLongArray = env->NewLongArray(6);
env->SetLongArrayRegion(jLongArray, 0, 6, buffer);
return jLongArray;
}
You can find a full working example of this JNI-based approach in the javaxt-core library. In my tests using Java 1.6.0_38 with a Windows host hitting a Windows share, I have found this JNI approach approximately 10x faster then calling java.io.File listFiles() or shelling out "dir" commands.
I wonder why there are 10k files in a directory. Some file systems do not work well with so many files. There are specifics limitations for file systems like max amount of files per directory and max amount of levels of subdirectory.
I solve a similar problem with an iterator solution.
I needed to walk across huge directorys and several levels of directory tree recursively.
I try FileUtils.iterateFiles() of Apache commons io. But it implement the iterator by adding all the files in a List and then returning List.iterator(). It's very bad for memory.
So I prefer to write something like this:
private static class SequentialIterator implements Iterator<File> {
private DirectoryStack dir = null;
private File current = null;
private long limit;
private FileFilter filter = null;
public SequentialIterator(String path, long limit, FileFilter ff) {
current = new File(path);
this.limit = limit;
filter = ff;
dir = DirectoryStack.getNewStack(current);
}
public boolean hasNext() {
while(walkOver());
return isMore && (limit > count || limit < 0) && dir.getCurrent() != null;
}
private long count = 0;
public File next() {
File aux = dir.getCurrent();
dir.advancePostition();
count++;
return aux;
}
private boolean walkOver() {
if (dir.isOutOfDirListRange()) {
if (dir.isCantGoParent()) {
isMore = false;
return false;
} else {
dir.goToParent();
dir.advancePostition();
return true;
}
} else {
if (dir.isCurrentDirectory()) {
if (dir.isDirectoryEmpty()) {
dir.advancePostition();
} else {
dir.goIntoDir();
}
return true;
} else {
if (filter.accept(dir.getCurrent())) {
return false;
} else {
dir.advancePostition();
return true;
}
}
}
}
private boolean isMore = true;
public void remove() {
throw new UnsupportedOperationException();
}
}
Note that the iterator stop by an amount of files iterateds and it has a FileFilter also.
And DirectoryStack is:
public class DirectoryStack {
private class Element{
private File files[] = null;
private int currentPointer;
public Element(File current) {
currentPointer = 0;
if (current.exists()) {
if(current.isDirectory()){
files = current.listFiles();
Set<File> set = new TreeSet<File>();
for (int i = 0; i < files.length; i++) {
File file = files[i];
set.add(file);
}
set.toArray(files);
}else{
throw new IllegalArgumentException("File current must be directory");
}
} else {
throw new IllegalArgumentException("File current not exist");
}
}
public String toString(){
return "current="+getCurrent().toString();
}
public int getCurrentPointer() {
return currentPointer;
}
public void setCurrentPointer(int currentPointer) {
this.currentPointer = currentPointer;
}
public File[] getFiles() {
return files;
}
public File getCurrent(){
File ret = null;
try{
ret = getFiles()[getCurrentPointer()];
}catch (Exception e){
}
return ret;
}
public boolean isDirectoryEmpty(){
return !(getFiles().length>0);
}
public Element advancePointer(){
setCurrentPointer(getCurrentPointer()+1);
return this;
}
}
private DirectoryStack(File first){
getStack().push(new Element(first));
}
public static DirectoryStack getNewStack(File first){
return new DirectoryStack(first);
}
public String toString(){
String ret = "stack:\n";
int i = 0;
for (Element elem : stack) {
ret += "nivel " + i++ + elem.toString()+"\n";
}
return ret;
}
private Stack<Element> stack=null;
private Stack<Element> getStack(){
if(stack==null){
stack = new Stack<Element>();
}
return stack;
}
public File getCurrent(){
return getStack().peek().getCurrent();
}
public boolean isDirectoryEmpty(){
return getStack().peek().isDirectoryEmpty();
}
public DirectoryStack downLevel(){
getStack().pop();
return this;
}
public DirectoryStack goToParent(){
return downLevel();
}
public DirectoryStack goIntoDir(){
return upLevel();
}
public DirectoryStack upLevel(){
if(isCurrentNotNull())
getStack().push(new Element(getCurrent()));
return this;
}
public DirectoryStack advancePostition(){
getStack().peek().advancePointer();
return this;
}
public File[] peekDirectory(){
return getStack().peek().getFiles();
}
public boolean isLastFileOfDirectory(){
return getStack().peek().getFiles().length <= getStack().peek().getCurrentPointer();
}
public boolean gotMoreLevels() {
return getStack().size()>0;
}
public boolean gotMoreInCurrentLevel() {
return getStack().peek().getFiles().length > getStack().peek().getCurrentPointer()+1;
}
public boolean isRoot() {
return !(getStack().size()>1);
}
public boolean isCurrentNotNull() {
if(!getStack().isEmpty()){
int currentPointer = getStack().peek().getCurrentPointer();
int maxFiles = getStack().peek().getFiles().length;
return currentPointer < maxFiles;
}else{
return false;
}
}
public boolean isCurrentDirectory() {
return getStack().peek().getCurrent().isDirectory();
}
public boolean isLastFromDirList() {
return getStack().peek().getCurrentPointer() == (getStack().peek().getFiles().length-1);
}
public boolean isCantGoParent() {
return !(getStack().size()>1);
}
public boolean isOutOfDirListRange() {
return getStack().peek().getFiles().length <= getStack().peek().getCurrentPointer();
}
}
Using an Iterable doesn't imply that the Files will be streamed to you. In fact its usually the opposite. So an array is typically faster than an Iterable.
Are you sure it's due to Java, not just a general problem with having 10k entries in one directory, particularly over the network?
Have you tried writing a proof-of-concept program to do the same thing in C using the win32 findfirst/findnext functions to see whether it's any faster?
I don't know the ins and outs of SMB, but I strongly suspect that it needs a round trip for every file in the list - which is not going to be fast, particularly over a network with moderate latency.
Having 10k strings in an array sounds like something which should not tax the modern Java VM too much either.

Categories

Resources