lsof is a nice tool for Unix, showing all currently open file handles.
Does anyone know a similar tool that would show all open files inside a running JVM (via JVMTI or any similar interface)?
In this particular case, it would be sufficient for me to know which class has a handle open. Method/line or even an entire chain to GC root would be fantastic, but handler owner class is already a good start.
I know I could make a heap dump, open it in a profiler and find this out, but this is a tedious task, especially for the big heaps.
The JVMTI option sounds like it wouldn't be a bad choice. The big issue would be ensuring you wrap everything that may open a file handle: you would basically have to go through the JDK source code and find every native function that did a file open (littered throughout java.io., java.nio., I'd think java.net.* as well if you consider sockets as file handles, and just about everywhere else that a file handle may be opened by a native function) and then wrap them all with the SetNativeMethodPrefix call.
I'm assuming that is what some of the profiling folks do: however if you're not required to do this listing in real time then I'd think it would be WAY easier to use lsof or handle (on Windows platforms) and filter for your JVM's process id.
Related
I am trying to develop a framework that will compile and execute (mostly random) C++ and Java packages.
However, given their random nature, I want to check the source (or the executable -- pre-execution) for any linux system calls before execution. If there is such a system call, I don't want to execute the program.
It is safe to assume that these packages wouldn't need to make any system calls to fulfill their functional purpose (they're not complex packages).
Edit: A bash command/script would be simplest, but any answer is fine.
In short, you cannot detect reliably all malicious syscalls (by static analysis of source code); read about the halting problem and Rice theorem... BTW MELT would be slighty better than grep since it works on GCC gimple representation.
Think of (on Linux)
dlopen(3)-ing the libc (or the main executable) then dlsym-ing "system" to get a pointer to the system function
knowing the libc layout and version,, then computing system's address by adding some known offset to address of malloc
using some JIT libary, e.g. the header only GNU lightning
coding the eqivalent of system with fork and execve ....
etc....
Of course, you might be trusting your user (I won't do that for a web application). If you trust all your users and just want to detect mistakes you might be able to filter some of them.
You need some container, e.g. docker
Look in resource limits (setrlimit if you are on POSIX system) as opposed to trying to find the malicious code.
You can limit number of processes, memory, open files, cputime and others. I would suggest you to limit basically everything. And run in chroot jail (even an empty one if you link statically).
Any way to hook on stdout of JVM currently running process or redirect stdout of JVM process to non file location?
I need verbose:gc output of JVM for my application, currently i can see this output in console but unable to store it in database or process it by any means, it invisible to running java application.
System.err
System.out
Both are wrappers over natives, internal JVM output not passes here.
Both can't see this output (ever if redirected or set to null, gc output shows in console).
Writing verbose:gc to file and then reading file not valid option due performance reasons and filesystem locks, checking hotspot sources not given any way to redirect output to nonfile location via JVM launch flags.
Also i was unable to get Process object of running JVM, there are no methods for this, also i was unable to find native that expose Process object, soo still no valid way to read this data.
Any clues about howto read verbose:gc in runtime?
Writing verbose:gc to file and then reading file not valid option due performance reasons and filesystem locks,
The cost of doing this is trivial compared to the cost of actually GCing. Unless you have a system which should never GC normally, I wouldn't worry about it. If you concern is that you can't read the file due to windows locks, you may have a problem (or you could use an OS like Linux which doesn't do this)
You should note that the GC output to a file is buffered, so it is not real time (but close enough for most use cases)
Reading the output programmatically is very difficult esp as it is written by a multi-threaded GC i.e. you get strange re-ordering of information.
I would consider getting the output of jstat which is designed to be read by a program though doesn't get as much detail.
You can get information GC pause information via JMX. It is not as full as GC logs, but will safe you from parsing. SJK tool is a CLI tool which could track GC events of JVM running processes. You could use it or use its code to build customized solution.
jstat is another option, but it exposes less information compared to JMX and its formatting rather misleading in case of CMS collector.
Problematically using PerfCounter API is third option (jstat is using this internally). See sun.management.counter.perf.PerfInstrumentation class for details.
Ok, so I have a couple of Java programs that I'm running using a chron job on a linux server. These jobs run every ten minutes or so, take literally two minutes to run, and then exit. I need to add a way for the programs to detect, when they start up, if there is already an instance of themselves running, and if so to exit without going any further. I'm really not sure of the best way to handle this though and am hoping someone can offer some advice.
One approach I've considered is to run a command line argument from the java code that does some sort of PS command and looks through those to see if it's running. This seems pretty finicky and complex though for something so small. Plus, I'm not all that knowledgeable with linux and am not even sure the best way to do that. If anyone has some better thoughts, please let me know. Or if that is the best way, if you could provide the linux commands I'd need I'd appreciate it. Thanks.
If you have a writable /tmp directory you can use a lockfile.
When your Java program starts up, check for a file with a name unique to your application (e.g. "my-lock-file.lock") in the /tmp directory. If none exists, create one, and remove it when you're done. If one exists, just exit.
You can check the existence of a file with the .exists() method of the java.io.File class.
If your code needs to be portable, you can use System.getProperty("java.io.tmpdir")); to get an appropriate temporary directory for the platform your code is running on.
You could look at JMX and the Attach API to query for running JVMs.
Or, as Andrew logvinov mentioned, by using a lock file.
If you are using Java WebStart, there's already native support for this.
Many programs solve this by creating a temporary file that points to their PID (often referred to as a "lock" file). The filename should encode all relevant information to distinguish this process from other processes that could legitimately run in parallel.
For example, if the process is bound to a user, it should contain the user name. If the process is bound to a machine, it should (also) contain the hostname (if you put it in machine-bound temp. directory, this is debatable. If you put it in a home directory, think of the case of multiple machines sharing a home via NFS).
The location of these files is typically /tmp. This is a great location, as /tmp is typically wiped during system boot, so no orphan files are left in case of a system crash. Another solution employed by some programs is to put the lock file in the user settings directory, if it is related to the settings. E.g. mozilla thunderbird has a file called /home/<username>/.thunderbird/<profilename>.default/lock.
The file should contain the PID of the process. The idea is simple: If the file contains the PID, it is easy to check whether this process is indeed still running. So if the process crashes, the file gets orphaned. The new process instance will check the PID in the file, see that it is not running any more, and ignore the file (overwrite).
Putting it all together, you could create a file like this:
/tmp/myawesomeservice-username-hostname-lock
With the content:
12345
I have an application where I need to check for a file which may be created dynamically during my execution, I will give up after some MAX time where the file has yet to show up. I wanted to know if there was a more efficient method in Java of checking for the file other than polling for it and then sleeping every X seconds? If not what would be the most efficient manner of doing this?
You currently have to poll the file system as you mentioned. Java 7 is supposed to have file system notifications, so this should get easier at some point.
If the same program is doing the creation of the file as the polling, you could instead have the logic that creates the file notify the part of the program using Object.notify(). A general description of the wait() and notify/notifyAll() mechanism can be found here: http://java.sun.com/docs/books/tutorial/essential/concurrency/guardmeth.html
You could try JPoller to poll for the file changes.
If you are running on Windows, you can get directory change notifications, see Obtaining Directory Change Notifications. Of course, this is not cross-platform, and will require use of JNA or similar native bridge. In fact, JNA offsers such as class, the FileMonitor class (in the download) that uses the underlying platform's file change notification.
If you are watching a handlful of files or fewer, then of course, polling is unlikely to be a performance problem, it's just not a "feel-good" solution - but not so bad to warrant the pain of a non pure java solution. Monitoring directories containing thousands of files on the other hand would benefit from direct noficiation from the OS.
Is there a reliable, cross-platform way to do IPC (between two JVMs running on the same host) in Java (J2SE) that doesn't rely on the network stack?
To be more specific, I have a server application that I'd like to provide a small "monitoring" GUI app for. The monitor app would simply talk to the server process and display simple status information. The server app has a web interface for most of its interaction, but sometimes things go wrong (port conflict, user forgot password) that require a local control app.
In the past I've done this by having the server listen on 127.0.01 on a specific port and the client communicates that way. However, this isn't as reliable as I'd like. Certain things can make this not work (Windows's network stack can be bizarre with VPN adapters, MediaSense, laptops lid closing/power saving modes). You can imagine the user's confusion when the tool they use to diagnose the server doesn't even think the server is running.
Named Pipes seem plausible, but Java doesn't seem to have an API for them unless I'm mistaken. Ideas? Third party libraries that support this? My performance requirements are obviously extremely lax in case that helps.
One of my specialties is really low-tech solutions. Especially if your performance requirements aren't critical:
The low-low tech alternative to named pipes is named FILES. Think yourself up a protocol where one app writes a file and another reads it. If need be, you can do semaphoring between them.
Remember that a rename is pretty much an atomic operation, so you could calmly write a file in some process and then make it magically appear in its entirety by renaming/moving it from somewhere that wasn't previously visible.
You can poll for data by checking for appearance of a file (in a loop with a SLEEP in it), and you can signal completion by deleting the file.
An added benefit is that you can debug your app using the DIR command :)
Depending on how much data you need to pass between the server and the diagnostic tool you could:
go low-tech and have a background thread check a file in the file system; fetch commands from it; write ouput into a second to be picked up by the diagnostic tool.
build a component that manages an input/output queue in shared memory connecting to it via JNI.
Consider JMX. I do not know if any of the Windows JVM's allow JMX over shared memory.
Does Windows even have named pipes? I was going to suggest it. You'd just have to use an exec() to create it.
Map a read_write byte buffer into memory from a FileChannel. Write status information into the byte buffer, then call force() to get it written out. On the monitor side, open up the same file and map it into memory too. Poll it periodically to find out the status.