I'd like to monitor Informatica ETL Workflows from custom Java program, via informatica Development Platform (LMapi), v9.1
I already have got C program, it works fine, but it would be great port to Java.
We get a lot of C DLLs, with asynchronous functions e.g.: JavaLMApi.dll, INFA_LMMonitorServerA (with detailed event-log possibilities)
In header we can see:
PMLM_API_EXTSPEC INFA_API_STATUS
INFA_LMMonitorServerA
(
INFA_UINT32 connectionId,
struct INFA_LMAPI_MONITOR_SERVER_REQUEST_PARAMS *request,
void *clientContext,
INFA_UINT32 *requestId
);
There is no any documentation for this problem, only with this information should I use for soulution.
The question is: how to call/use INFA_LMMonitorServerA in Java? (Dll-load with JNA/JNI is not problem, only the callback).
static INFA_UINT32 nConnectionId = 0;
/* C-"skeleton": */
void GetSrvDetailcallback(struct INFA_API_REPLY_CONTEXT* GetSrvDetailReplyCtxt)
{
INFA_API_STATUS apiRet;
struct INFA_LMAPI_SERVER_DETAILS *serverDetails = NULL;
char *serverStatus = NULL;
/* Check if the return status is Acknowledgement */
if (GetSrvDetailReplyCtxt->returnStatus == INFA_REQUEST_ACKNOWLEDGED)
{
fprintf(stdout, "\nINFA REQUEST ACKNOWLEDGED \n\n",NULL);
return;
}
apiRet = INFA_LMGetServerDetailsResults(GetSrvDetailReplyCtxt, &serverDetails);
/* Check the return code if if is an error */
if (INFA_SUCCESS != apiRet)
{
fprintf(stderr, "Error: INFA_LMGetServerDetailsResults returns %d\n", apiRet);
return;
}
printResults(serverDetails);
}
static void myServer()
{
struct INFA_LMAPI_CONNECT_SERVER_REQUEST_PARAMS_EX connectParamsex;
INFA_API_STATUS apiRet;
struct INFA_LMAPI_LOGIN_REQUEST_PARAMS loginparams;
apiRet = INFA_LMLogin(nConnectionId, &loginparams, NULL);
if (INFA_SUCCESS != apiRet)
{
fprintf(stderr, "Error: INFA_LMLogin returns %d\n", apiRet);
return;
}
struct INFA_LMAPI_MONITOR_SERVER_REQUEST_PARAMS strMonitorRequestParams;
//Only Running Tasks
strMonitorRequestParams.monitorMode = INFA_LMAPI_MONITOR_RUNNING;
strMonitorRequestParams.continuous = INFA_TRUE;
/* Get Server Details */
INFA_UINT32 GetSrvDetailsrequestId = 0;
/* Register a callback function. */
INFA_LMRegisterCallback(INFA_LMAPI_MONITOR_SERVER, &GetSrvDetailcallback);
apiRet = INFA_LMMonitorServerA(nConnectionId, &strMonitorRequestParams, NULL, &GetSrvDetailsrequestId);
if (INFA_SUCCESS != apiRet && INFA_REQUEST_PENDING != apiRet)
{
fprintf(stderr, "Error: INFA_LMMonitorServerA returns %d\n", apiRet);
return;
}
}
Related
I'm trying to make an application that reads the memory of another (non-Java & 32bit) application using JNA. So far I know how to find process ID and base address of modules. And right before reading memory I need to open process and the OpenProcess function simply returns null. Also, I'm using Windows 10.
// process id (pid) is known
final int PROCESS_VM_READ=0x0010;
final int PROCESS_QUERY_INFORMATION=0x0400;
WinNT.HANDLE processHandle = Kernel32.INSTANCE.OpenProcess(PROCESS_VM_READ | PROCESS_QUERY_INFORMATION, true, pid);
How can I get a process handle?
You need to enable debug privilege for your current process in order to query information for processes owned by anyone other than the current user. The link shows the code in C, but you can port that code to JNA.
This is a one-time method call when your program starts up.
Here's how I do it (hat tip to #RbMm for improvements):
/**
* Enables debug privileges for this process, required for OpenProcess() to get
* processes other than the current user
*
* #return {#code true} if debug privileges were successfully enabled.
*/
private static boolean enableDebugPrivilege() {
HANDLEByReference hToken = new HANDLEByReference();
boolean success = Advapi32.INSTANCE.OpenProcessToken(Kernel32.INSTANCE.GetCurrentProcess(),
WinNT.TOKEN_QUERY | WinNT.TOKEN_ADJUST_PRIVILEGES, hToken);
if (!success) {
LOG.error("OpenProcessToken failed. Error: {}", Native.getLastError());
return false;
}
try {
WinNT.LUID luid = new WinNT.LUID();
success = Advapi32.INSTANCE.LookupPrivilegeValue(null, WinNT.SE_DEBUG_NAME, luid);
if (!success) {
LOG.error("LookupPrivilegeValue failed. Error: {}", Native.getLastError());
return false;
}
WinNT.TOKEN_PRIVILEGES tkp = new WinNT.TOKEN_PRIVILEGES(1);
tkp.Privileges[0] = new WinNT.LUID_AND_ATTRIBUTES(luid, new DWORD(WinNT.SE_PRIVILEGE_ENABLED));
success = Advapi32.INSTANCE.AdjustTokenPrivileges(hToken.getValue(), false, tkp, 0, null, null);
int err = Native.getLastError();
if (!success) {
LOG.error("AdjustTokenPrivileges failed. Error: {}", err);
return false;
} else if (err == WinError.ERROR_NOT_ALL_ASSIGNED) {
LOG.debug("Debug privileges not enabled.");
return false;
}
} finally {
Kernel32.INSTANCE.CloseHandle(hToken.getValue());
}
return true;
}
I need to add a timeout for the reply/request transaction using 0MQ. How is this typically accomplished? I tried using the method :
socket.setReceiveTimeOut();
and
socket.setSendTimeout();
but they seem to cause a null pointer exception.
In essence, I want the application to timeout after 10 seconds if the application receiving the request is not available.
Any help is appreciated.
Thanks!
I think jzmq should throw a ZMQException when recv timeout,
But there is no ZMQException, when err = EAGAIN.
https://github.com/zeromq/jzmq/blob/master/jzmq-jni/src/main/c%2B%2B/Socket.cpp
static
zmq_msg_t *do_read(JNIEnv *env, jobject obj, zmq_msg_t *message, int flags)
{
void *socket = get_socket (env, obj);
int rc = zmq_msg_init (message);
if (rc != 0) {
raise_exception (env, zmq_errno());
return NULL;
}
#if ZMQ_VERSION >= ZMQ_MAKE_VERSION(3,0,0)
rc = zmq_recvmsg (socket, message, flags);
#else
rc = zmq_recv (socket, message, flags);
#endif
int err = zmq_errno();
if (rc < 0 && err == EAGAIN) {
rc = zmq_msg_close (message);
err = zmq_errno();
if (rc != 0) {
raise_exception (env, err);
return NULL;
}
return NULL;
}
if (rc < 0) {
raise_exception (env, err);
rc = zmq_msg_close (message);
err = zmq_errno();
if (rc != 0) {
raise_exception (env, err);
return NULL;
}
return NULL;
}
return message;
}
I wonder if your null pointer is related to how your socket was created. I have set a socket timeout successfully in the past.
The following has worked for me when I used the JeroMQ library (native Java implementation of ZMQ). I used this to help do REQ-REP commands via ZMQ.
ZMQ.Context context = ZMQ.context(1);
ZMQ.Socket sock = context.socket(ZMQ.REQ);
sock.setSendTimeOut(10000); // 10 second send timeout
sock.setReceiveTimeOut(10000); // 10 second receive timeout
if (sock.connect("tcp://127.0.0.1:1234")) {
if (sock.send(/* insert data here */)) {
/* Send was successful and did not time out. */
byte[] replyBytes = null;
replyBytes = sock.recv();
if (null == replyBytes) {
/* Receive timed out. */
} else {
/* Receive was successful. Do something with replyBytes. */
}
}
}
How is this [a timeout for the reply/request transaction] typically accomplished ?
I am sad to confirm, there is nothing like this in the ZeroMQ native API. The principle of doing async delivery means, there is no limit for delivery to take place ( in a best-effort model of scheduling, or not at all ).
If new to the ZeroMQ, you may enjoy a fast read into this 5-second read about the main conceptual elements in [ ZeroMQ hierarchy in less than a five seconds ] Section.
I want ... to timeout after 10 seconds if ... receiving the request is not ...
May design your .recv()-method call usage into either a pre-tested / protected after a .poll( 10000 )-method screener, to first explicitly detect a presence of any message for indeed being delivered to your application-code, before ever issuing ( or not ) a call to the actual .recv()-method only upon a previously POSACK-ed message to be ready to get locally read, or may use a bit more "raw" approach, using a handler with a non-blocking form of the method, by a call to the .recv( ZMQ_NOBLOCK )-flagged not to spend a millisecond "there", in cases when "there" are no messages to read right now from the local-side Context()-engine instance, and handle each of the cases accordingly in your code.
A Bonus Point
Also be warned, that using the REQ/REP-Scalable Formal Communication Archetype pattern will not be any easier, as there is a mandatory two-side-step-dance ( sure, if not intentionally artificially ZMQ_RELAXED ), so the both FSA-back-to-back-connected-FSA-s will still have to wait for the next "expected" remote-event, before becoming able to make a chance for handling the next local-event. If interested in details, one will find many posts on un-avoidable, un-salvagable mutual-deadlock, that is sure to happen for REQ/REP, where we only do not know when it happens, but are sure it will.
I use opengl es with jni to play YUV video, the video is OK.
But when i close GLSurfaceView and return to main Activity.
The characters and images display in main Activity become messed.
All icons becomes black solid rectangle. All the characters becomes white solid rectangle.
The code look likes come from WebRTC and modified by someone else.
I'm a C programmer and not familiar with OpenGl. So it's a really big problem for me to solve this problem.
I create handle in JNI like this:
bool isAttached = false;
JNIEnv* env = NULL;
if (_jvm->GetEnv((void**) &env, JNI_VERSION_1_4) != JNI_OK) {
// try to attach the thread and get the env
// Attach this thread to JVM
jint res = _jvm->AttachCurrentThread(&env, NULL);
// Get the JNI env for this thread
if ((res < 0) || !env) {
LOGE("%s: Could not attach thread to JVM (%d, %p)",
__FUNCTION__, res, env);
return -1;
}
isAttached = true;
}
// get the ViEAndroidGLES20 class
jclass javaRenderClassLocal = reinterpret_cast<jclass> (env->FindClass("com/wg/rgc/library/gl/ViEAndroidGLES20"));
if (!javaRenderClassLocal) {
LOGE("%s: could not find ViEAndroidGLES20", __FUNCTION__);
return -1;
}
_javaRenderClass = reinterpret_cast<jclass> (env->NewGlobalRef(javaRenderClassLocal));
if (!_javaRenderClass) {
LOGE("%s: could not create Java SurfaceHolder class reference",
__FUNCTION__);
return -1;
}
// Delete local class ref, we only use the global ref
env->DeleteLocalRef(javaRenderClassLocal);
jmethodID cidUseOpenGL = env->GetStaticMethodID(_javaRenderClass,
"UseOpenGL2",
"(Ljava/lang/Object;)Z");
if (cidUseOpenGL == NULL) {
LOGE("%s: could not get UseOpenGL ID", __FUNCTION__);
return false;
}
jboolean res = env->CallStaticBooleanMethod(_javaRenderClass,
cidUseOpenGL, (jobject) _ptrWindow);
// create a reference to the object (to tell JNI that we are referencing it
// after this function has returned)
_javaRenderObj = reinterpret_cast<jobject> (env->NewGlobalRef((jobject)_ptrWindow));
if (!_javaRenderObj)
{
LOGE("%s: could not create Java SurfaceRender object reference",
__FUNCTION__);
return -1;
}
// get the method ID for the ReDraw function
_redrawCid = env->GetMethodID(_javaRenderClass, "ReDraw", "()V");
if (_redrawCid == NULL) {
LOGE("%s: could not get ReDraw ID", __FUNCTION__);
return -1;
}
_registerNativeCID = env->GetMethodID(_javaRenderClass,
"RegisterNativeObject", "(J)V");
if (_registerNativeCID == NULL) {
LOGE("%s: could not get RegisterNativeObject ID", __FUNCTION__);
return -1;
}
_deRegisterNativeCID = env->GetMethodID(_javaRenderClass,
"DeRegisterNativeObject", "()V");
if (_deRegisterNativeCID == NULL) {
LOGE("%s: could not get DeRegisterNativeObject ID",
__FUNCTION__);
return -1;
}
JNINativeMethod nativeFunctions[2] = {
{ "DrawNative",
"(J)V",
(void*) &AndroidNativeOpenGl2Channel::DrawNativeStatic, },
{ "CreateOpenGLNative",
"(JII)I",
(void*) &AndroidNativeOpenGl2Channel::CreateOpenGLNativeStatic },
};
if (env->RegisterNatives(_javaRenderClass, nativeFunctions, 2) == 0) {
LOGE("%s: Registered native functions", __FUNCTION__);
}
else {
LOGE("%s: Failed to register native functions", __FUNCTION__);
return -1;
}
env->CallVoidMethod(_javaRenderObj, _registerNativeCID, (jlong) this);
if (isAttached) {
if (_jvm->DetachCurrentThread() < 0) {
LOGE("%s: Could not detach thread from JVM", __FUNCTION__);
}
}
LOGE("%s done", __FUNCTION__);
if (_openGLRenderer.SetCoordinates(0, 1.0, 1.0, 0, 0) != 0) {
return -1;
}
isAlreadyInit = 1; //Init finish
return 0;
And i close this handle in JNI like this:
bool isAttached = false;
JNIEnv* env = NULL;
if (_jvm->GetEnv((void**) &env, JNI_VERSION_1_4) != JNI_OK) {
// try to attach the thread and get the env
// Attach this thread to JVM
jint res = _jvm->AttachCurrentThread(&env, NULL);
// Get the JNI env for this thread
if ((res < 0) || !env) {
LOGE("%s: Could not attach thread to JVM (%d, %p)",
__FUNCTION__, res, env);
env = NULL;
} else {
isAttached = true;
}
}
if (env && _deRegisterNativeCID) {
env->CallVoidMethod(_javaRenderObj, _deRegisterNativeCID);
}
env->DeleteGlobalRef(_javaRenderObj);
env->DeleteGlobalRef(_javaRenderClass);
if (isAttached) {
if (_jvm->DetachCurrentThread() < 0) {
LOGE("%s: Could not detach thread from JVM",
__FUNCTION__);
}
}
The method env->CallVoidMethod(_javaRenderObj, _deRegisterNativeCID); called Java method in Java src:
nativeFunctionLock.lock();
nativeFunctionsRegisted = false;
openGLCreated = false;
this.nativeObject = 0;
nativeFunctionLock.unlock();
The code in Java src looks like extends GLSurfaceView and implements GLSurfaceView.Renderer. In JNI the code registered some method as native method in Java.
The code is complicated for me to understand , but my boss request me fix this problem quickly.
I can't figure out why after playing YUV video , every image and characters become rectangles. How did this happen?
you probably delete the texture after rendering, you can try to keep them, and deallocate them when app terminated
I've build a simple test which creates and deletes a file (name does not change) in an infinite loop. The test does run for a couple of seconds (sometimes over 77,000 iterations!) and then fails with this exception:
Exception in thread "main" java.io.IOException: Access is denied
at java.io.WinNTFileSystem.createFileExclusively(Native Method)
at java.io.File.createNewFile(Unknown Source)
at DeleteTest.main(DeleteTest.java:11)
Here's the test logic:
final File f = new File(pathname);
while (true) {
final boolean create = f.createNewFile();
if (!create) {
System.out.println("crate failed");
} else {
final boolean delete = f.delete();
if (!delete) {
System.out.println("delete failed");
}
}
}
How is this possible? The delete call does not fail. It would tell. So delete always succeeds but createNewFile fails. This is what MSDN says about win32 api function DeleteFile:
The DeleteFile function marks a file for deletion on close. Therefore,
the file deletion does not occur until the last handle to the file is
closed. Subsequent calls to CreateFile to open the file fail with
ERROR_ACCESS_DENIED.
So createNewFile does not close the file? The openjdk source tells us that the file is closed:
JNIEXPORT jboolean JNICALL
Java_java_io_Win32FileSystem_createFileExclusively(JNIEnv *env, jclass cls,
jstring pathname)
{
jboolean rv = JNI_FALSE;
DWORD a;
WITH_PLATFORM_STRING(env, pathname, path) {
int orv;
int error;
JVM_NativePath((char *)path);
orv = JVM_Open(path, JVM_O_RDWR | JVM_O_CREAT | JVM_O_EXCL, 0666);
if (orv < 0) {
if (orv != JVM_EEXIST) {
error = GetLastError();
// If a directory by the named path already exists,
// return false (behavior of solaris and linux) instead of
// throwing an exception
a = GetFileAttributes(path);
if ((a == INVALID_FILE_ATTRIBUTES) ||
!(a & FILE_ATTRIBUTE_DIRECTORY)) {
SetLastError(error);
JNU_ThrowIOExceptionWithLastError(env, path);
}
}
} else {
JVM_Close(orv);
rv = JNI_TRUE;
}
} END_PLATFORM_STRING(env, path);
return rv;
}
Can anyone explain this behaviour?
I've found an explanation while writing the question. I still posted the question because I wanted to share what I learned.
My application is not the only process on the system accessing files. The Windows Search Index Service for example could open this file because it wants to add it to it's index. Or the windows Explorer if it is updating the view.
This issue reminds me a problem I experienced recently with the File.renameTo() method. It is (was?) due to this bug in the jvm :
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6213298
A weird workaround is to call System.gc() and to retry renaming the file again (and it works...).
Not sure it has a link with your issue, but it may be worth exploring...
Try this:
final File f = new File("file");
while (true) {
final boolean create = f.createNewFile();
if (!create) {
System.out.println("crate failed");
} else {
final boolean delete = f.delete();
try {
Thread.sleep(10);
} catch (InterruptedException e) {
System.out.println("...");
}
if (!delete) {
System.out.println("delete failed");
}
}
}
In this way we ensure that the file is released by the delete before invoking createNewFile.
I am trying to process files one at a time that are stored over a network. Reading the files is fast due to buffering is not the issue. The problem I have is just listing the directories in a folder. I have at least 10k files per folder over many folders.
Performance is super slow since File.list() returns an array instead of an iterable. Java goes off and collects all the names in a folder and packs it into an array before returning.
The bug entry for this is http://bugs.sun.com/view_bug.do;jsessionid=db7fcf25bcce13541c4289edeb4?bug_id=4285834 and doesn't have a work around. They just say this has been fixed for JDK7.
A few questions:
Does anybody have a workaround to this performance bottleneck?
Am I trying to achieve the impossible? Is performance still going to be poor even if it just iterates over the directories?
Could I use the beta JDK7 builds that have this functionality without having to build my entire project on it?
Although it's not pretty, I solved this kind of problem once by piping the output of dir/ls to a file before starting my app, and passing in the filename.
If you needed to do it within the app, you could just use system.exec(), but it would create some nastiness.
You asked. The first form is going to be blazingly fast, the second should be pretty fast as well.
Be sure to do the one item per line (bare, no decoration, no graphics), full path and recurse options of your selected command.
EDIT:
30 minutes just to get a directory listing, wow.
It just struck me that if you use exec(), you can get it's stdout redirected into a pipe instead of writing it to a file.
If you did that, you should start getting the files immediately and be able to begin processing before the command has completed.
The interaction may actually slow things down, but maybe not--you might give it a try.
Wow, I just went to find the syntax of the .exec command for you and came across this, possibly exactly what you want (it lists a directory using exec and "ls" and pipes the result into your program for processing): good link in wayback (Jörg provided in a comment to replace this one from sun that Oracle broke)
Anyway, the idea is straightforward but getting the code right is annoying. I'll go steal some codes from the internets and hack them up--brb
/**
* Note: Only use this as a last resort! It's specific to windows and even
* at that it's not a good solution, but it should be fast.
*
* to use it, extend FileProcessor and call processFiles("...") with a list
* of options if you want them like /s... I highly recommend /b
*
* override processFile and it will be called once for each line of output.
*/
import java.io.*;
public abstract class FileProcessor
{
public void processFiles(String dirOptions)
{
Process theProcess = null;
BufferedReader inStream = null;
// call the Hello class
try
{
theProcess = Runtime.getRuntime().exec("cmd /c dir " + dirOptions);
}
catch(IOException e)
{
System.err.println("Error on exec() method");
e.printStackTrace();
}
// read from the called program's standard output stream
try
{
inStream = new BufferedReader(
new InputStreamReader( theProcess.getInputStream() ));
processFile(inStream.readLine());
}
catch(IOException e)
{
System.err.println("Error on inStream.readLine()");
e.printStackTrace();
}
} // end method
/** Override this method--it will be called once for each file */
public abstract void processFile(String filename);
} // end class
And thank you code donor at IBM
How about using File.list(FilenameFilter filter) method and implementing FilenameFilter.accept(File dir, String name) to process each file and return false.
I ran this on Linux vm for directory with 10K+ files and it took <10 seconds.
import java.io.File;
import java.io.FilenameFilter;
public class Temp {
private static void processFile(File dir, String name) {
File file = new File(dir, name);
System.out.println("processing file " + file.getName());
}
private static void forEachFile(File dir) {
String [] ignore = dir.list(new FilenameFilter() {
public boolean accept(File dir, String name) {
processFile(dir, name);
return false;
}
});
}
public static void main(String[] args) {
long before, after;
File dot = new File(".");
before = System.currentTimeMillis();
forEachFile(dot);
after = System.currentTimeMillis();
System.out.println("after call, delta is " + (after - before));
}
}
An alternative is to have the files served over a different protocol. As I understand you're using SMB for that and java is just trying to list them as a regular file.
The problem here might not be java alone ( how does it behaves when you open that directory with Microsoft Explorer x:\shared ) In my experience it also take a considerably amount of time.
You can change the protocol to something like HTTP, only to fetch the file names. This way you can retrieve the list of files over http ( 10k lines should't be too much ) and let the server deal with file listing. This would be very fast, since it will run with local resources ( those in the server )
Then when you have the list, you can process them one by exactly the way you're doing right now.
The keypoint is to have an aid mechanism in the other side of the node.
Is this feasible?
Today:
File [] content = new File("X:\\remote\\dir").listFiles();
for ( File f : content ) {
process( f );
}
Proposed:
String [] content = fetchViaHttpTheListNameOf("x:\\remote\\dir");
for ( String fileName : content ) {
process( new File( fileName ) );
}
The http server could be a very small small and simple file.
If this is the way you have it right now, what you're doing is to fetch all the 10k files information to your client machine ( I don't know how much of that info ) when you only need the file name for later processing.
If the processing is very fast right now it may be slowed down a bit. This is because the information prefetched is no longer available.
Give it a try.
A non-portable solution would be to make native calls to the operating system and stream the results.
For Linux
You can look at something like readdir. You can walk the directory structure like a linked list and return results in batches or individually.
For Windows
In windows the behavior would be fairly similar using FindFirstFile and FindNextFile apis.
I doubt the problem is relate to the bug report you referenced.
The issue there is "only" memory usage, but not necessarily speed.
If you have enough memory the bug is not relevant for your problem.
You should measure whether your problem is memory related or not. Turn on your Garbage Collector log and use for example gcviewer to analyze your memory usage.
I suspect that it has to do with the SMB protocol causing the problem.
You can try to write a test in another language and see if it's faster, or you can try to get the list of filenames through some other method, such as described here in another post.
If you need to eventually process all files, then having Iterable over String[] won't give you any advantage, as you'll still have to go and fetch the whole list of files.
If you're on Java 1.5 or 1.6, shelling out "dir" commands and parsing the standard output stream on Windows is a perfectly acceptable approach. I've used this approach in the past for processing network drives and it has generally been a lot faster than waiting for the native java.io.File listFiles() method to return.
Of course, a JNI call should be faster and potentially safer than shelling out "dir" commands. The following JNI code can be used to retrieve a list of files/directories using the Windows API. This function can be easily refactored into a new class so the caller can retrieve file paths incrementally (i.e. get one path at a time). For example, you can refactor the code so that FindFirstFileW is called in a constructor and have a seperate method to call FindNextFileW.
JNIEXPORT jstring JNICALL Java_javaxt_io_File_GetFiles(JNIEnv *env, jclass, jstring directory)
{
HANDLE hFind;
try {
//Convert jstring to wstring
const jchar *_directory = env->GetStringChars(directory, 0);
jsize x = env->GetStringLength(directory);
wstring path; //L"C:\\temp\\*";
path.assign(_directory, _directory + x);
env->ReleaseStringChars(directory, _directory);
if (x<2){
jclass exceptionClass = env->FindClass("java/lang/Exception");
env->ThrowNew(exceptionClass, "Invalid path, less than 2 characters long.");
}
wstringstream ss;
BOOL bContinue = TRUE;
WIN32_FIND_DATAW data;
hFind = FindFirstFileW(path.c_str(), &data);
if (INVALID_HANDLE_VALUE == hFind){
jclass exceptionClass = env->FindClass("java/lang/Exception");
env->ThrowNew(exceptionClass, "FindFirstFileW returned invalid handle.");
}
//HANDLE hStdOut = GetStdHandle(STD_OUTPUT_HANDLE);
//DWORD dwBytesWritten;
// If we have no error, loop thru the files in this dir
while (hFind && bContinue){
/*
//Debug Print Statment. DO NOT DELETE! cout and wcout do not print unicode correctly.
WriteConsole(hStdOut, data.cFileName, (DWORD)_tcslen(data.cFileName), &dwBytesWritten, NULL);
WriteConsole(hStdOut, L"\n", 1, &dwBytesWritten, NULL);
*/
//Check if this entry is a directory
if (data.dwFileAttributes & FILE_ATTRIBUTE_DIRECTORY){
// Make sure this dir is not . or ..
if (wstring(data.cFileName) != L"." &&
wstring(data.cFileName) != L"..")
{
ss << wstring(data.cFileName) << L"\\" << L"\n";
}
}
else{
ss << wstring(data.cFileName) << L"\n";
}
bContinue = FindNextFileW(hFind, &data);
}
FindClose(hFind); // Free the dir structure
wstring cstr = ss.str();
int len = cstr.size();
//WriteConsole(hStdOut, cstr.c_str(), len, &dwBytesWritten, NULL);
//WriteConsole(hStdOut, L"\n", 1, &dwBytesWritten, NULL);
jchar* raw = new jchar[len];
memcpy(raw, cstr.c_str(), len*sizeof(wchar_t));
jstring result = env->NewString(raw, len);
delete[] raw;
return result;
}
catch(...){
FindClose(hFind);
jclass exceptionClass = env->FindClass("java/lang/Exception");
env->ThrowNew(exceptionClass, "Exception occured.");
}
return NULL;
}
Credit:
https://sites.google.com/site/jozsefbekes/Home/windows-programming/miscellaneous-functions
Even with this approach, there are still efficiencies to be gained. If you serialize the path to a java.io.File, there is a huge performance hit - especially if the path represents a file on a network drive. I have no idea what Sun/Oracle is doing under the hood but if you need additional file attributes other than the file path (e.g. size, mod date, etc), I have found that the following JNI function is much faster than instantiating a java.io.File object on a network the path.
JNIEXPORT jlongArray JNICALL Java_javaxt_io_File_GetFileAttributesEx(JNIEnv *env, jclass, jstring filename)
{
//Convert jstring to wstring
const jchar *_filename = env->GetStringChars(filename, 0);
jsize len = env->GetStringLength(filename);
wstring path;
path.assign(_filename, _filename + len);
env->ReleaseStringChars(filename, _filename);
//Get attributes
WIN32_FILE_ATTRIBUTE_DATA fileAttrs;
BOOL result = GetFileAttributesExW(path.c_str(), GetFileExInfoStandard, &fileAttrs);
if (!result) {
jclass exceptionClass = env->FindClass("java/lang/Exception");
env->ThrowNew(exceptionClass, "Exception Occurred");
}
//Create an array to store the WIN32_FILE_ATTRIBUTE_DATA
jlong buffer[6];
buffer[0] = fileAttrs.dwFileAttributes;
buffer[1] = date2int(fileAttrs.ftCreationTime);
buffer[2] = date2int(fileAttrs.ftLastAccessTime);
buffer[3] = date2int(fileAttrs.ftLastWriteTime);
buffer[4] = fileAttrs.nFileSizeHigh;
buffer[5] = fileAttrs.nFileSizeLow;
jlongArray jLongArray = env->NewLongArray(6);
env->SetLongArrayRegion(jLongArray, 0, 6, buffer);
return jLongArray;
}
You can find a full working example of this JNI-based approach in the javaxt-core library. In my tests using Java 1.6.0_38 with a Windows host hitting a Windows share, I have found this JNI approach approximately 10x faster then calling java.io.File listFiles() or shelling out "dir" commands.
I wonder why there are 10k files in a directory. Some file systems do not work well with so many files. There are specifics limitations for file systems like max amount of files per directory and max amount of levels of subdirectory.
I solve a similar problem with an iterator solution.
I needed to walk across huge directorys and several levels of directory tree recursively.
I try FileUtils.iterateFiles() of Apache commons io. But it implement the iterator by adding all the files in a List and then returning List.iterator(). It's very bad for memory.
So I prefer to write something like this:
private static class SequentialIterator implements Iterator<File> {
private DirectoryStack dir = null;
private File current = null;
private long limit;
private FileFilter filter = null;
public SequentialIterator(String path, long limit, FileFilter ff) {
current = new File(path);
this.limit = limit;
filter = ff;
dir = DirectoryStack.getNewStack(current);
}
public boolean hasNext() {
while(walkOver());
return isMore && (limit > count || limit < 0) && dir.getCurrent() != null;
}
private long count = 0;
public File next() {
File aux = dir.getCurrent();
dir.advancePostition();
count++;
return aux;
}
private boolean walkOver() {
if (dir.isOutOfDirListRange()) {
if (dir.isCantGoParent()) {
isMore = false;
return false;
} else {
dir.goToParent();
dir.advancePostition();
return true;
}
} else {
if (dir.isCurrentDirectory()) {
if (dir.isDirectoryEmpty()) {
dir.advancePostition();
} else {
dir.goIntoDir();
}
return true;
} else {
if (filter.accept(dir.getCurrent())) {
return false;
} else {
dir.advancePostition();
return true;
}
}
}
}
private boolean isMore = true;
public void remove() {
throw new UnsupportedOperationException();
}
}
Note that the iterator stop by an amount of files iterateds and it has a FileFilter also.
And DirectoryStack is:
public class DirectoryStack {
private class Element{
private File files[] = null;
private int currentPointer;
public Element(File current) {
currentPointer = 0;
if (current.exists()) {
if(current.isDirectory()){
files = current.listFiles();
Set<File> set = new TreeSet<File>();
for (int i = 0; i < files.length; i++) {
File file = files[i];
set.add(file);
}
set.toArray(files);
}else{
throw new IllegalArgumentException("File current must be directory");
}
} else {
throw new IllegalArgumentException("File current not exist");
}
}
public String toString(){
return "current="+getCurrent().toString();
}
public int getCurrentPointer() {
return currentPointer;
}
public void setCurrentPointer(int currentPointer) {
this.currentPointer = currentPointer;
}
public File[] getFiles() {
return files;
}
public File getCurrent(){
File ret = null;
try{
ret = getFiles()[getCurrentPointer()];
}catch (Exception e){
}
return ret;
}
public boolean isDirectoryEmpty(){
return !(getFiles().length>0);
}
public Element advancePointer(){
setCurrentPointer(getCurrentPointer()+1);
return this;
}
}
private DirectoryStack(File first){
getStack().push(new Element(first));
}
public static DirectoryStack getNewStack(File first){
return new DirectoryStack(first);
}
public String toString(){
String ret = "stack:\n";
int i = 0;
for (Element elem : stack) {
ret += "nivel " + i++ + elem.toString()+"\n";
}
return ret;
}
private Stack<Element> stack=null;
private Stack<Element> getStack(){
if(stack==null){
stack = new Stack<Element>();
}
return stack;
}
public File getCurrent(){
return getStack().peek().getCurrent();
}
public boolean isDirectoryEmpty(){
return getStack().peek().isDirectoryEmpty();
}
public DirectoryStack downLevel(){
getStack().pop();
return this;
}
public DirectoryStack goToParent(){
return downLevel();
}
public DirectoryStack goIntoDir(){
return upLevel();
}
public DirectoryStack upLevel(){
if(isCurrentNotNull())
getStack().push(new Element(getCurrent()));
return this;
}
public DirectoryStack advancePostition(){
getStack().peek().advancePointer();
return this;
}
public File[] peekDirectory(){
return getStack().peek().getFiles();
}
public boolean isLastFileOfDirectory(){
return getStack().peek().getFiles().length <= getStack().peek().getCurrentPointer();
}
public boolean gotMoreLevels() {
return getStack().size()>0;
}
public boolean gotMoreInCurrentLevel() {
return getStack().peek().getFiles().length > getStack().peek().getCurrentPointer()+1;
}
public boolean isRoot() {
return !(getStack().size()>1);
}
public boolean isCurrentNotNull() {
if(!getStack().isEmpty()){
int currentPointer = getStack().peek().getCurrentPointer();
int maxFiles = getStack().peek().getFiles().length;
return currentPointer < maxFiles;
}else{
return false;
}
}
public boolean isCurrentDirectory() {
return getStack().peek().getCurrent().isDirectory();
}
public boolean isLastFromDirList() {
return getStack().peek().getCurrentPointer() == (getStack().peek().getFiles().length-1);
}
public boolean isCantGoParent() {
return !(getStack().size()>1);
}
public boolean isOutOfDirListRange() {
return getStack().peek().getFiles().length <= getStack().peek().getCurrentPointer();
}
}
Using an Iterable doesn't imply that the Files will be streamed to you. In fact its usually the opposite. So an array is typically faster than an Iterable.
Are you sure it's due to Java, not just a general problem with having 10k entries in one directory, particularly over the network?
Have you tried writing a proof-of-concept program to do the same thing in C using the win32 findfirst/findnext functions to see whether it's any faster?
I don't know the ins and outs of SMB, but I strongly suspect that it needs a round trip for every file in the list - which is not going to be fast, particularly over a network with moderate latency.
Having 10k strings in an array sounds like something which should not tax the modern Java VM too much either.