Develop Reference

java api to receive notifications when a file system is mounted

I am looking for a Java API that will allow registering for file system mount events, i.e. when a file system is mounted or dismounted. Specifically I want to know when a file system on removable USB devices is available, and also know exactly what type of USB device it was.
The udev subsystem provides notifications on USB plug and unplug events by default but not specifically when the file system on the device is available. It is possible to create udev rules that can do this in pieces, e.g. create a directory and execute a program when devices are added and removed. But my experience with udev rules is that the syntax is arcane and they are fragile and not simple to debug.
I've installed usbmount per this post:
https://serverfault.com/questions/414120/how-to-get-usb-devices-to-automount-in-ubuntu-12-04-server
though I believe the devices were automouting by default.
As an alternative I constructed a JDK 7 WatcherService on /media which can detect changes in /etc/mtab. This works but I have seen cases where the file systems on some USB devices are still not ready - meaning that attempts to read the directory throw an Exception - even after the entry in /etc/mtab is made. I added a timer to sleep for a configurable number of milliseconds and in most cases a 100ms wait time works but not 100% of the time. What this means is that increasing this wait time is not an absolute guarantee nor deterministic.
Clearly at some low level the mount event is being generated because the Nautilus pop-up window gets displayed. I had a case of one flash drive that would put the Nautilus icon in the launch pad menu but it would not mount until the icon was clicked open.
I've also looked at these options:
tailing /var/log/syslog; this may be the next best option. I see lines like the following:
:Dec 2 08:58:07 fred-Inspiron-530 udisksd[1759]: Mounted /dev/sdk1 at /media/fred/USB DISK1 on behalf of uid 1000
I am going to try a WatcherService here to see if the same timing issue exists, i.e. is the directory readable once this message is written.
jlibudev [ github.com/nigelb/jlibudev ] Much better Java API to udev subsystem than writing rules but it still falls short in that you still have to piece a number of different events together. NB: jlibudev depends on JNA [https://github.com/twall/jna] and purejavacomm [ github.com/nyholku/purejavacomm, sparetimelabs.com/purejavacomm/purejavacomm.php] both of which are pretty useful in their own right.
lsusb provides details on the usb device but nothing about where it is mounted.
Ideally I would like a simple API that would allow registering for file system mount/unmount events using the standard Java event listening pattern. I want to believe that such an API exists or is at least possible given that at a macro-level the net effect is occurring. I am still scouring the JDK 7 and JDK 8 APIs for other options.
Any and all pointers and assistance would be greatly appreciated.

Since there's no OS-agnostic way to deal with mounting filesystems, there's definitely no JDK API for this. I'm guessing this problem is not dealt with much (not a lot of programs deal with mounting filesystems directly), so it's unlikely that there's any prebuilt library out there waiting for you.
Of the approaches you mentioned, they all sound roughly equal in terms of how platform-specific they are (all Linux-only), so that just leaves performance and ease of coding as open questions. Regarding performance, running lsusb more than once a second is (a) a giant hack :-) and (b) fork+exec is slow compared to running something in-process, and tailing the event log will create a lot of (unpredictable) work for your program that is not related to USB mounts as well as making your implementation more fragile (what if the message strings change when you upgrade your OS?). Regarding ease of programming, either using jna or JNI to call into libudev or a WatcherService on /media sound about equal -- using libudev seems like the most portable option across Linux distros / user configurations (I'm guessing that's what Nautilus uses).
However, for simplicity of implementation that will work for 99% of users, it's hard to do better than a WatcherService on /media. To help ensure that the filesystem is available before use, I would just use a loop with some kind of randomized exponential backoff in the amount of time to wait between attempts to read the directory -- that way you never wait way longer than necessary for the filesystem to mount, you aren't burning tons of CPU waking up and trying to read, and you don't have to pick a single timeout number that won't work everywhere. If you care enough to ensure you don't tie down a single thread sleeping forever, I'd use a ScheduledExecutorService to issue Runnables that try to access the filesystem, and if it's not available schedule themselves to run again in a bit, otherwise alert your main thread that a new filesystem is available for use using a queue of some kind.
Edit: I just learned that you could also watch for updates to the /proc/mounts file. Hopefully since the kernel is responsible for updating this file things only show up when they're fully mounted, although I don't know for certain. For more details, How to interpret /proc/mounts? and the Red Hat docs were useful.

How do I implement client-side bandwidth throttling for FTP/HTTP?

I am tasked with writing a client-side data download system (on Linux) that uses FTP or HTTP to download terabyte-sized data from external partners to our local site. Our company's network admin tells me that I cannot exceed a certain bandwidth. What is the best way for me to implement such a system? Do existing libraries exist?
I am open to writing my own FTP and HTTP clients (in either C or Java on Linux) but would prefer to stay out of the kernel. I know that I can limit the rate at which my FTP/HTTP client calls a socket read(), but what happens if the server-side calls write() faster than my limit?

You could build another layer on top of an InputStream: In the read method, you can count the bytes so far. If the number of bytes/second exceed a certain limit, let the download thread sleep for a while. TCP's flow control does the rest.

I know Apache JMeter simulates slow connections. You could maybe take look at the code.

If you know the network path delay you could just set your TCP receive buffer size to the desired bandwidth-delay product. That will throttle the sender all right. But the resulting value may be too small for your platform, so it may adjust it upwards. Check the value after you set it.
Does your netadmin know that TCP automatically shares bandwidth fairly?

Are you open to off the shelf GUI or command line products? Filezillia provides this.
There also is a linux command line client called lftp. A settable parameter is net:limit-total-rate which will limit the rate of transfer. Since this client supports multiple transfers at one time, it also has a parameter net:limit-rate.

To keep it simple, if you are on linux you just could use wget instead of re-inventing the wheel? Take a look at the --limit-rate switch.
But back on topic :) This answer could get you started: How can I implement a download rate limited in Java?

Advice for writing Client-Server based game

I'm thinking about writing a game which is based around a server, and several client programs connect to it. The game (very) basically consists of a list of items which a user can 'accept', which would remove it from the list on all connected computers (this needs to update very quickly).
I'm thinking about using a Java applet for the client since I would like this to be portable and run from a browser (mostly in Windows), as well as updating fast, and either a C++ or Java server running on Linux (currently just a home server, but possibly to go on a VPS).
A previous 'incarnation' of this game ran in a browser, and used PHP+mySQL for the backend, but this swamped the server quite a bit when several people connected (that was with about 8 people, this would eventually need to handle a lot more).
The users would probably all be in the same physical location (with the same public IP address), and the system would get several requests per second, all of which would require sending the list back to the clients.
Some computers may have firewall restrictions on them, so would you recommend using HTTP traffic, a custom port, or perhaps through SSH or some existing protocol?
Could anyone suggest some tips (threading, multiple requests of one item?), tools, databases (mySQL?), or APIs which would help me get started on this project? I would prefer C++ for the backend as it would be faster, but using Java would allow me to reuse code.
Thanks!

I wouldn't use C++ because of speed alone. It is highly unlikely that the difference in performance will make a real difference to your game. (Your network is likely to cloud any performance difference, unless you have 10 GigE between the client and server) I would use C++ or Java because you will get it working first using that language.

For anyone looking for a good networking API for c++ I always suggest Boost.Asio. It has the advantage of being platform independent, so you can compile a server for linux, windows etc. However, if you are not too familiar with c++ templates/boost the code can be a little overwhelming. Have a look, give it a try.
In terms of general advice. Given the description above, you seem to need a relatively simple server. I would suggest keeping it very basic, single threaded polling loop. Read a message from your connected clients (wait on multiple sockets), and respond appropriately. This eliminates any issue around multiple accesses to your list and other synchronization problems.
I might also suggest, before you re-write your initial incarnation. Try improving it, as you have stated:
and the system would get several requests per second, all of which would require sending the list back to the clients.
Given that each request removes an item from this list, why not just inform your uses which item is removed, rather than sending the entire list over the network time and time again? If this list is of any significant size, this minor change will result in a large improvement.

How to monitor exceptions or errors generated by other Java applications?

I want to find or develop an application that can run as a daemon, notify the administrator by email or sms when the Java applications running on a host get any exceptions or errors. I know JVMTI can achieve part of my goal, but it will impact performance of the monitored applications(I don't know how much will it be, it will be acceptable if it's slight), besides it seems to be a troublesom job to develop a JVMTI agent and I'm not sure what would happen if several applications running at the same time using the same agent. Is there any better solutions? Thanks in advance.

One way would be to use a logging system like log4j that publishes all errors occuring on system A to a logging server on system B from which you can monitor the errors occured. This isn't a completely generic solutation however, since only exceptions propagated to log4j (or any other logging system) would be handled - but it may be a good start.

The best solution is to have the Java application send its errors via email/sms. The problem is that programs will generate exceptions and handle correctly in normal operation. You only want particular exception.
Failing this you could write a log reader, which reads the logs of the application. This is tricky to get right, but it can be done.
An application can generate 1000+ exception per days and still be behaving normally because the application knows how to handle these exceptions. e.g. every time a socket connection is closed an exception can be thrown.

IMO, the best approach is to deploy an external monitoring system. This can:
monitor multiple applications
monitor infrastructure services
monitor network availability and machine accessibility,
monitor resources such as processor and file system usage.
Applications can be monitored in a variety of ways, including:
by processing log events,
by watching for application restarts,
by "pinging" the application's web apis to check service liveness, and
by using the application's JMX interfaces.
This information can be filtered and prioritized in an intelligent fashion, and critical events can be reported by whatever means is most appropriate.
You don't want individual applications sending emails, because they don't have sufficient information to do a decent job. Furthermore, putting the reporting logic into individual applications is likely to lead to inconsistent implementation, poor configurability, and so on.

There is a nearby alternative to JVMTI : JPDA. This infrastructure allows you to create a remote "debugger" (yes, that's what you're planning to do) using Java code, and connect it to the VM using either local or remote connection.
There will be, like for JVMTI, an overhead to program execution. However, as the Trace.java example shows, it's quite simple to both implement and connect to target VM.
Finally, notice if you want to instrument code run by application server (JBoss, Glassfish, Tomcat, you name it) there are various other means available.

I follow the pattern where every exception gets logged to a table.
Then an RSS feed selects from that table.
I subscribe to the RSS feed in MS Outlook at work and also on my Android phone with a program called NewsRob. NewsRob let me set my phone to alert me when there is something new.
I blog about how to do this HERE. It is in .net, but you get the idea.
As a related step I found a way to notify myself when something DIDN'T happen. That blog is HERE.

There are loads of applications out there that do what you are looking for in a way that does not impact performance. Have you had a look at Kibana/ElasticSearch, or Splunk or Logscape for enterprise solutions ( they both also have free versions).
I'm going to echo what has already been said and highlight what java already provides and what you can do with an external monitoring system. Java already provides:
log4j - log ERRORS, WARNINGS, FATAL and Exceptions to a file
JMX - Create custom application metrics and you also have access to java.lang/* which will give you heap memory usage , garbage collection, thread counters etc.
JVM gc logging - you can log all your garbage collection events to a file and watch for any long Full GC collections.
An external monitoring system will allow you to set alerts triggered off different operational scenarios. You will also get visualisation of your system performance through charts. I've used Logscape's java app in the past to monitor 30 java processes spread out over3 hosts.

In a Java thread running as root, how can we apply Unix rights specific to a logged-in user?

We have a Java program run as root on Unix, that therefore can read for example the content of the folders /home/user1 and /home/user2. However, if the Unix user "user1" is logged in in our application, he should not be able to access "/home/user2" data.
We would like to use directly the Unix rights and not recreate all the permissions in our application !
So, could we...
try to change the UID of our
program depending on the user logged
in ? Sounds difficult, and each file
access is in different threads so
the UID would be different on each
thread of our program...
use JNI to read permissions of
"/home/user2"...And then determine
if user1 has sufficient permissions
on "/home/user2" ? (how ?).

Use SecurityManager!
Put current unix user id into ThreadLocal
Create your own SecurityManager that checks unix user permissions on checkRead() and checkWrite()
System.setSecurityManager(new MySecurityManager())
Enjoy
Update
There is no, of course, standard library to read unix file permissions. It's not WORA.
But I have tried briefly to find a ready to use library, and found this one:
http://jan.newmarch.name/java/posix/ It uses JNI, but you don't need to write your own JNI code, which is a big relief. :) I'm sure there must also be others.
Class Stat from there gives you all required access information:
http://jan.newmarch.name/java/posix/posix.Stat.html
Update 2
As folks mentioned, this approach fails to check for "non-standard" unix security features, such as ACL or Posix Capabilities (may be; not sure if they apply to files). But if the goal of being totally in sync with host OS security is set, then we even more need to use SecurityManager, because it's a JVM-wide protection mechanism! Yes, we can start a child SUID-process to verify the permissions (and keep it running, talking to it via pipe running while the user is logged in), but we need to do so from SecurityManager!

The simplest and most portable way would be to spawn a child process, have it exec a wrapper written in C which changes the UID, drops all the privileges (be careful, writting a wrapper to do that is tricky - it is as hard as writing a setuid wrapper), and execs another java instance to which you talk via RMI. That java instance would do all the filesystem manipulation on behalf of the user.
For single-threaded Linux programs, you could instead use setfsuid()/setfsgid(), but that is not an option for portable or multithreaded programs.

if you only want the app to be allowed to read files by user1 i strongly suggest the app runs as user1.

If everything else fails, you can run a shellscript from java and parse the result.
Described for example here

For those who were wondering, it's apparently not possible to do this by calling setuid with JNI for each independent thread. setuid affects the whole process, not just the thread that invoked it.
Should you want to call setuid within a single-threaded Java program there's a good example at http://www2.sys-con.com/itsg/virtualcd/Java/archives/0510/Silverman/index.html.

Another option would be to invert the approach: instead of the code running as root most of the time and either changing the user ID or somehow checking the permissions whenever it has to use some restricted resource, run as the user most of the time and talk to a smaller daemon running as root when it needs to do something only root can do. This also has the added benefit of reducing the attack surface.
Of course, you then have to authenticate the connection from the process running as the user to the process running as root.

I am also having the exact problem as Mikael, and got to this page looking for answers.
None of the answers are 100% satisfactionary for me. So I am thinking of 4 alternatives:
Use a Linux group that has access to all the users. Run a single Java app under that group. This Java app can communicate to the 'root' app using whatever means.
Potentially, it can be "hotel"-ed. e.g. 1 "hotel" (app with group permissions) per 100 users (or as appropriate). So if you have 10,000 users you need 100 hotels, which is quite manageable.
Spawn a JVM for each child app under its own user ID. This is like calling a script, but rather than using stdin/stdio/stderr, use any communication protocol. In my case, I'm using XMPP and IO Data (which, since it's already in use by other components, it doesn't matter "where" aka which JVM it runs).
Create a Super-Server 'root' app. This can be part of the original 'root' app or a separate service dedicated to service management.
The Super-Server is responsible for handling incoming requests (i.e. it practically becomes a reverse proxy) for the user-specific sub-apps, and launching the real child apps (if they're not running already), and passing messages back and forth between the client and the child app(s).
Additionally, the child apps can be pooled (or even "passivated", if there's such thing), much in the way Java EE EJB Container does it. So even if there are 10,000 users and (potentially) 10,000 child apps servicing, the maximum number of child apps running are capped. Idle apps are shut down to make room for others.
Same as #3 but rather than creating a proprietary service management mechanism, integrate with Upstart (or the service management framework in the underlying OS). i.e. there is a 'root' service that can control Upstart. Upstart can start, stop, restart, can query the status of the child services, just like it can control mysqld, Apache, etc.
For me, now, the quickest and simplest to implement would be #1. However, my ideal solution would be #4, but it will take time and testing whether it works well. (the concept itself borrows from inetd/xinetd and EJB, so I think it's pretty sound fundamentally)