I want create module which update list of usb devices automatically (not only mass storage). Now it works very rude - it has additional thread, which after 1 second update list of devices. But maybe you know how to handle some plug event, which generate OS (for example windows XP and above) or another way how to simplify this process. Thank you for your time!
This question appears to be half design, half implementation. It seems you're most concerned about finding some way for Java USB services like jUSB to service hotplug events from the system, allowing you to have an event called via inversion of control instead of running a thread that passionately and iteratively polls the system in the background.
The support you seek was implemented back in 2005 as JSR-80, and appears to live on, in the matter of all great projects that overstepped their commercial support, as javax-usb on SourceForge. Note that in their toy example (in javax-usb-example/src/ShowTopology.java), their preferred model is still polling based.
Not all is lost, however! In the properties file for javax.usb, we find this:
# If polling should be used instead of waiting for a hotplug
notification.
# Some kernel versions have a bug, and don't update the device node(s) until
many seconds after waking up threads waiting on the devices file.
If you are not getting hotplug events on a moderately to heavily loaded system,
use polling.
# Note that since the new (buggy?) kernel behavior breaks javax.usb, this defaults
to true!
# If you know your kernel isn't broken you can set this to false.
#com.ibm.jusb.os.linux.LinuxUsbServices.topologyUpdateUsePolling = true
And in javax-usb/src/javax/usb/event/UsbServicesListener.java, we see a model for setting up a USB event listener that you can proceed to use when a device is attached or detached from the system. No documentation is externally given on whether this works, however, and the docs themselves appear to be mostly 8 years old.
Therefore, it's reasonable assume that while the support theoretically exists, it's lagged, stale, and in desperate need of an update by the community. This is the kind of thing that this fine community on SO could help fix, so I'll begin to close by pointing directly to the project itself.
For now, until enhanced hardware or hotplug support exists, polling is still the best solution.
(Additional reference implementation of javax-usb)
Related
I'm having trouble with a Jetty 9 server application that seems to go into some kind of resting state after a longer period of idleness. Normally the memory usage of the Java process is ~500 MB, but after being idle for some time it seems to drop down to less than 50MB. The first request that comes takes up to several seconds to respond whereas requests are normally on the scale of tens of milliseconds. But after one or two requests it seems like the application is back to it's normal responsive state.
I'm running on the 32-bit Oracle Java 8 JVM. My JVM configuration is very basic:
java -server -jar start.jar
I was hoping that this issue might be solvable through JVM configuration. Does anyone know if there's any particular parameter to disable this type of behavior?
edit: Based on the comment from Ivan, I was able to identify the source of the issue. Turns out Windows was swapping parts of the Java process out to disk. See my own answer below for a description of my solution.
Based on the comment from Ivan, I was able to identify the source of the issue. Turns out Windows was swapping parts of the Java process out to disk. This was clearly visible when comparing the private working set to the commit size in the task manager.
My solution to this was two-fold. First, I made a simple scheduled job inside my server app that runs every minute and does a simple test run to make sure that the important services never go inactive for long periods. I'm hoping this should ensure that Windows doesn't regard the related pages as inactive.
Afterwards, I also noticed that the process was executing with "Below normal" priority. So I changed the script that starts the server to ensure that it's running with "High" priority going forward. This seems likely to affect swapping behavior and may very well also have been enough to resolve the issue on it's own, but I only found it after already deploying my first solution so that remains unclear. In any case, everything seems to be working as it should now.
I am looking for a Java API that will allow registering for file system mount events, i.e. when a file system is mounted or dismounted. Specifically I want to know when a file system on removable USB devices is available, and also know exactly what type of USB device it was.
The udev subsystem provides notifications on USB plug and unplug events by default but not specifically when the file system on the device is available. It is possible to create udev rules that can do this in pieces, e.g. create a directory and execute a program when devices are added and removed. But my experience with udev rules is that the syntax is arcane and they are fragile and not simple to debug.
I've installed usbmount per this post:
https://serverfault.com/questions/414120/how-to-get-usb-devices-to-automount-in-ubuntu-12-04-server
though I believe the devices were automouting by default.
As an alternative I constructed a JDK 7 WatcherService on /media which can detect changes in /etc/mtab. This works but I have seen cases where the file systems on some USB devices are still not ready - meaning that attempts to read the directory throw an Exception - even after the entry in /etc/mtab is made. I added a timer to sleep for a configurable number of milliseconds and in most cases a 100ms wait time works but not 100% of the time. What this means is that increasing this wait time is not an absolute guarantee nor deterministic.
Clearly at some low level the mount event is being generated because the Nautilus pop-up window gets displayed. I had a case of one flash drive that would put the Nautilus icon in the launch pad menu but it would not mount until the icon was clicked open.
I've also looked at these options:
tailing /var/log/syslog; this may be the next best option. I see lines like the following:
:Dec 2 08:58:07 fred-Inspiron-530 udisksd[1759]: Mounted /dev/sdk1 at /media/fred/USB DISK1 on behalf of uid 1000
I am going to try a WatcherService here to see if the same timing issue exists, i.e. is the directory readable once this message is written.
jlibudev [ github.com/nigelb/jlibudev ] Much better Java API to udev subsystem than writing rules but it still falls short in that you still have to piece a number of different events together. NB: jlibudev depends on JNA [https://github.com/twall/jna] and purejavacomm [ github.com/nyholku/purejavacomm, sparetimelabs.com/purejavacomm/purejavacomm.php] both of which are pretty useful in their own right.
lsusb provides details on the usb device but nothing about where it is mounted.
Ideally I would like a simple API that would allow registering for file system mount/unmount events using the standard Java event listening pattern. I want to believe that such an API exists or is at least possible given that at a macro-level the net effect is occurring. I am still scouring the JDK 7 and JDK 8 APIs for other options.
Any and all pointers and assistance would be greatly appreciated.
Since there's no OS-agnostic way to deal with mounting filesystems, there's definitely no JDK API for this. I'm guessing this problem is not dealt with much (not a lot of programs deal with mounting filesystems directly), so it's unlikely that there's any prebuilt library out there waiting for you.
Of the approaches you mentioned, they all sound roughly equal in terms of how platform-specific they are (all Linux-only), so that just leaves performance and ease of coding as open questions. Regarding performance, running lsusb more than once a second is (a) a giant hack :-) and (b) fork+exec is slow compared to running something in-process, and tailing the event log will create a lot of (unpredictable) work for your program that is not related to USB mounts as well as making your implementation more fragile (what if the message strings change when you upgrade your OS?). Regarding ease of programming, either using jna or JNI to call into libudev or a WatcherService on /media sound about equal -- using libudev seems like the most portable option across Linux distros / user configurations (I'm guessing that's what Nautilus uses).
However, for simplicity of implementation that will work for 99% of users, it's hard to do better than a WatcherService on /media. To help ensure that the filesystem is available before use, I would just use a loop with some kind of randomized exponential backoff in the amount of time to wait between attempts to read the directory -- that way you never wait way longer than necessary for the filesystem to mount, you aren't burning tons of CPU waking up and trying to read, and you don't have to pick a single timeout number that won't work everywhere. If you care enough to ensure you don't tie down a single thread sleeping forever, I'd use a ScheduledExecutorService to issue Runnables that try to access the filesystem, and if it's not available schedule themselves to run again in a bit, otherwise alert your main thread that a new filesystem is available for use using a queue of some kind.
Edit: I just learned that you could also watch for updates to the /proc/mounts file. Hopefully since the kernel is responsible for updating this file things only show up when they're fully mounted, although I don't know for certain. For more details, How to interpret /proc/mounts? and the Red Hat docs were useful.
I have a Java application, and I need it to be high available.
I was thinking of FastMPJ, like running multiple instances on different PCs. Every minute the app will check if master instance is running, and if not, the other will run instead of it.
I'd like to ask if it is a good solution, or if there is any better.
A more general solution is to use a load-balancing system, that is: you have N instances of the application running with the same privileges (if possible on different hardware), then a redundant load-balancer in front selects one of those based on the actual load for each request/task.
The benefit of this solution is obviously, that hardware is actually used and doesn't sit somewhere idle, waiting on the 0.01% case to jump in. Then the instance is actually tested all the time, and errors will be reported when they happen (like faulty hardware), and you prevent a: "Oh... the backup isn't even working". And on top of that you balance the load between machines adaptively.
In one of my project while implementing a exchange we used Apache Qpid for high availability and my experiense was quite satisfaotory. It scales very well too. I have been running application up to 32 node clusters. Please find further details here and let me know in case u need any further infromation:
http://qpid.apache.org/releases/qpid-0.18/books/AMQP-Messaging-Broker-Java-Book/html/High-Availability.html
Hope it helps:)
One often forgets that there must also be high availability from the application to database as well. It is my experience that the data access layer is where most of the application bottlenecks occur. So make sure you have a good application aware DB load balancer. Oracle has a solid solution but is for Oracle databases only. PostGres has an open source version. Heimdall Data is a commercial solution.
I have a directory that continually fills up with "artefact" files. Many different programs dump their temporary files in this directory and it's unlikely that these programs will become self-cleaning any time soon.
Meanwhile, I would like to write a program that continually deletes files in this directory as they become stale, which I'll define as "older than 30 minutes".
A typical approach would be to have a timed mechanism that lists the files in the directory, filters on the old stuff, and deletes the old stuff. However, this approach is not very performant in my case because this directory could conceivably contain 10s or hundreds of thousands of files that do not yet qualify as stale. Consequently, this approach would continually be looping over the same thousands of files to find the old ones.
What I'd really like to do is implement some kind of directory listener that was notified of any new files added to the directory. This listener would then add those files to a queue to be deleted down the road. However, there doesn't appear to be a way to implement such a solution in the languages I program in (JVM languages like Java and Scala).
So: I'm looking for the most efficient way to keep a directory "as clean as it can be" on Windows, preferably with a JVM language. Also, though I've never programmed with Powershell, I'd consider it if it offered this kind of functionality. Finally, if there are 3rd party tools out there to do such things, I'd like to hear about them.
Thanks.
Why can't you issue a directory system command sorted by oldest first:
c:>dir /OD
Take the results and delete all files older than your threshold or sleep if no files are old enough.
Combine that with a Timer or Executor set to a granularity 1 second - 1 minute which guarantees that the files don't keep piling up faster than you can delete them.
If you don't want to write C++, you can use Python. Install pywin32 and you can then use the win32 API as such:
import win32api, win32con
change_handle = win32api.FindFirstChangeNotification(
path_to_watch,
0,
win32con.FILE_NOTIFY_CHANGE_FILE_NAME
)
Full explanation of what to do with that handle by Tim Golden here: http://timgolden.me.uk/python/win32_how_do_i/watch_directory_for_changes.html.
In Java, you can also use Apache Commons JCI FAM. It's is an opensource java library that you can use for free.
JDK 7 (released in beta currently) includes support for file notifications as well. Check out Java NIO2 tutorial.
Both options should work both on Windows and Linux.
http://www.cyberpro.com.au/Tips_n_Tricks/Windows_Related_Tips/Purge_a_Directory_in_Windows_automatically/
I'd go with C++ for a utility like this - lets you interface with the WIN32 API, which does indeed have directory listening facilities (FindFirstChangeNotification or ReadDirectoryChangesW). Use one thread that listens for change notifications and updates your list of files (iirc FFCN requires you to rescan the folder, whereas RDCW gives you the actual changes).
If you keep this list sorted according to modification time, it becomes easy to Sleep() just long enough for a file to go stale, instead of polling at some random fixed interval. You might want to do a WaitForSingleObject with a timeout instead of Sleep, in order to react to outside changes (ie, the file you're waiting for to become stale has been deleted externally, so you'll want to wake up and determine when the next file will become stale).
Sounds like a fun little tool to write :)
You might want to bite the bullet and code it up in C# (or VB). What you're asking for is pretty well handled by the FileSystemWatcher class. It would work basically the way you are describing. Register files as they are added into the directory. Have a periodic timer that scans the list of files for ones that are stale and deletes them if they are still there. I'd probably code it up as a Windows service running under a service id that has enough rights to read/delete files in the directory.
EDIT: A quick google turned up this FileSystemWatcher for Java. Commercial software. Never used it, so can't comment on how well it works.
Is it possible to dump an image of a running JVM and later restore the previous state by loading the image into the JVM? I'm fairly certain the answer is negative, but would love to be wrong.
With all the dynamic languages available for the JVM comes an increase in interactivity, being able to save a coding session would help save time manually restoring the VM to a previous session.
There was a JSR 323 proposed for this a while back but it was rejected. You can find some links in those articles about the research behind this and what it would take. It was mostly rejected as an idea that was too immature.
I have heard of at least one startup (unfortunately don't recall the name) that was working on a virtualization technology over a hypervisor (probably Xen) that was getting pretty close to being able to move JVMs, including even things like file system refs and socket endpoints. Because they were at the hypervisor level, they had access to all of that stuff. By hooking that and the JVM, they had most of the pieces. I think they might have gone under though.
The closest thing you can get today is Terracotta, which allows you to cluster a portion of your JVM heap, storing it in a server array, which can be made persistent. On JVM startup, you connect to the cluster and can continue using whatever portions of your heap are specified as clustered. The actual objects are faulted in on an as-needed basis.
Not possible at present. In general, pausing and restarting a memory image of a process in a different context is incredibly hard to achieve: what are you going to do with open OS resources? Transfers to machines with different instruction sets? database connections?
Also images of the running JVM are probably quite large - maybe much larger than the subset of the state you are actually interested in. So it's not a good idea from a performance perspective.
A much better strategy is to have code that persists and recreates the application state: this is relatively feasible with most JVM dynamic languages. I do so similar stuff in Clojure, where you have an interactive environment (REPL) and it is quite possible to create and run a sequence of operations that rebuild the application state that you want in another JVM.
This is currently not possible in any of the JVMs I know. It would not be very difficult to implement something like this in the JVM if programs run disconnected from their environments. However, many programs have hooks into their environment (think file handles, database connections) which would make implementing something like this very hairy.
As of early 2023, there's some progress in this space and it seems a lot of things can at least be tried, even if without claims for their production readiness.
One such feature is called CRaC. You can check their docs or even get an OpenJDK build that includes the feature. The project has its own repo under OpenJDK and looks quite promising.
Another vendors/products to check:
Azul ReadyNow!
OpenJ9 InstantOn
What's also really exciting, is AWS Lambda SnapStart. It doesn't give you full snapshoting capabilities, and is intrinsically vendor-specific, but it's what a ton of Java engineering who use AWS Lambda were waiting for so long.