Develop Reference

A better way to call static methods in user-submitted code?

I have a large data set. I am creating a system which allows users to submit java source files, which will then be applied to the data set. To be more specific, each submitted java source file must contain a static method with a specific name, let's say toBeInvoked(). toBeInvoked will take a row of the data set as an array parameter. I want to call the toBeInvoked method of each submitted source file on each row in the data set. I also need to implement security measures (so toBeInvoked() can't do I/O, can't call exit, etc.).
Currently, my implementation is this: I have a list of the names of the java source files. For each file, I create an instance of the custom secure ClassLoader which I coded, which compiles the source file and returns the compiled class. I use reflection to extract the static method toBeInvoked() (e.g. method = c.getMethod("toBeInvoked", double[].class)). Then, I iterate over the rows of the data set, and invoke the method on each row.
There are at least two problems with my approach:
it appears to be painfully slow (I've heard reflection tends to be slow)
the code is more complicated than I would like
Is there a better way to accomplish what I am trying to do?

There is no significantly better approach given the constraints that you have set yourself.
For what it is worth, what makes this "painfully slow" is compiling the source files to class files and loading them. That is many orders of magnitude slower than the use of reflection to call the methods.
(Use of a common interface rather than static methods is not going to make a measurable difference to speed, and the reduction in complexity is relatively small.)
If you really want to simplify this and speed it up, change your architecture so that the code is provided as a JAR file containing all of the compiled classes.

Assuming your #toBeInvoked() could be defined in an interface rather than being static (it should be!), you could just load the class and cast it to the interface:
Class<? extends YourInterface> c = Class.forName("name", true, classLoader).asSubclass(YourInterface.class);
YourInterface i = c.newInstance();
Afterwards invoke #toBeInvoked() directly.
Also have a look into java.util.ServiceLoader, which could be helpful for finding the right class to load in case you have more than one source file.

Personally, I would use an interface. This will allow you to have multiple instance with their own state (useful for multi-threading) but more importantly you can use an interface, first to define which methods must be implemented but also to call the methods.
Reflection is slow but this is only relative to other options such as a direct method call. If you are scanning a large data set, the fact you have to pulling data from main memory is likely to be much more expensive.

I would suggest following steps for your problem.
To check if the method contains any unwanted code, you need to have a check script which can do these checks at upload time.
Create an Interface having a method toBeInvoked() (not a static method).
All the classes which are uploaded must implement this interface and add the logic inside this method.
you can have your custom class loader scan a particular folder for new classes being added and load them accordingly.
When a file is uploaded and successfully validated, you can compile and copy the class file to the folder which class loader scans.
You processor class can lookup for new files and then call toBeInvoked() method on loaded class when required.
Hope this help. (Note that i have used a similar mechanism to load dynamically workflow step classes in Workflow Engine tool which was developed).

Is there a portable way to find out how many files a JVM has open from inside the VM?

I am writing tests for some Java file handling code and want to make sure all files are closed properly. I don't want to run 'lsof' as that will open more files and make the test suite non-portable. Anyone know a way to do this?

If you're looking for something that's part of the JDK, the answer is no.
You might find something that uses JVMTI, but that wouldn't be portable (it's a native interface). Or something that uses JPDA, but that would require a second JVM. I give you those two acronyms as a start for Googling.
If you want to run in-JVM and be portable, you'll have to introduce a factory for your file references: replace all new FileInputStream(), new FileOutputStream(), new RandomAccessFile(), new FileReader, and new FileWriter calls with methods on that factory object. This factory will return subclasses of these objects, that have the close() method overridden. It will also increment an "open files" counter, that is then decremented by the overridden close().
The factory methods and counter will need to be static and synchronized (unless you want to inject the factory), and should use a system property to decide whether to return a subclassed stream or the JDK version.
Personally, I'd take the advice in the comment, and use FindBugs first.

If a referenced Java class is not found, or blacklisted, when is this detected?

I wanted to use, inside Google appengine, a small library; one of the methods of one of its classes (say MyClass.writeToFile()) uses java.io.FileOutputStream, which is among the blacklisted classes (well, not white-listed).
Does this imply that MyClass will fail at classloading time, or just when (if) I try to invoke the offending method? At what point is that checking ("FileOutputStream not allowed") done? When loading MyClass file and detecting that it "refers to/depends on" FileOutputStream (I dont know if this is declared in the MyClass bytecode)? Or when trying to load FileOutputStream, while invoking MyClass.writeToFile() for the first time?
Further, assuming that the method MyClass.writeToFile() is not essential for my library (but MyClass is), is there some workaround, or should one refactor the library (and build, say two different jars, one full fledged and other sandbox-friendly) ?

Do a test deploy. Deployment should fail if I remember it correctly. Classes that are used by a certain class is part of the byte code, and google is verifying it.
Edit: I'm contradicting myself :) This thread indicates that deployment only fail if the FileOutputStream class is loaded:
GoogleAppEngine : possible to disable FileUpload?
Edit2: And this indicates that it is the class loader that is checking / stopping loading of forbidden classes: The Sandbox

Classes are lazyload upon first reference. If your code never tries to use FileOutputStream then there should be no problem.

Java reflecting subclasses

I have a class in Java, I wish to reflect all subclasses of this class, how would I do this?
In this specific case, all subclasses are in the same package, and only subclasses are in this package, so an equivalent solution is to fetch all classes in a package.

I think you could do this using spring's org.springframework.core.io.support.PathMatchingResourcePatternResolver. At least I know you can use it to find all classes with a certain annotation. Finding subclasses seems to be a very similar problem I'd expect to work as well.
Here's some (untested) code to get you started:
PathMatchingResourcePatternResolver match = new PathMatchingResourcePatternResolver();
MetadataReaderFactory f = new SimpleMetadataReaderFactory();
List<Class<?>> matches = ...;
for (Resource r : match.getResources("classpath*:com/example/**/*.class")) {
AnnotationMetadata meta = f.getMetadataReader(r).getAnnotationMetadata();
if (meta.getAnnotationsTypes().contains(MyAnnotation.class.getName()) {
matches.add(Class.forName(meta.getClassName()));
}
}
return matches;
As you might see, it's basically the idea describe by Stephen C

See this answer how to locate all classes in a package.
Next, you'll need a list of all packages which isn't trivial in the general case. But if you only use the standard classloader (which solely relies on the classpath), you can get the system property java.class.path and analyze that to get all JARs. Open them to list the content and then you'll know the class names.

It is possible to do (messily) in some cases; e.g. if your application's classes are loaded from local directories or JAR files. Basically, you need to map the package name to a pathname and use the File and/or ZipFile APIs to look for files in the relevant "directory" whose name ends with ".class". Then you use Class.forName() load the corresponding classes (trying to avoid initializing them), and use clazz.isAssignableFrom(subclazz) to see which of them are subclasses.
But this won't work in all cases. One problem is that the ClassLoader API does not support iterating over all of the classes / packages that it could load.
I would recommend that you find an alternative approach to the problem you are tying to address.
EDIT - in response to this comment.
It seems this is a massive fault in the java reflection API
I understand that the omission is deliberate. If the classloader API did include a method to give all classes in a package, it would limit the schemes that can be used for classloading. For example, URLClassLoader would not be implementable for 'http:' URLs because the HTTP protocol does not provide a standard way to retrieve the entries in a directory.
Besides, there are few situations where a production application really needs to reflectively find all classes in a package.

Find Java classes implementing an interface [duplicate]

This question already has answers here:
How can I get a list of all the implementations of an interface programmatically in Java?
(11 answers)
Closed 9 years ago.
Some time ago, I came across a piece of code, that used some piece of standard Java functionality to locate the classes that implemented a given interface. I know the functions were hidden in some non-logical place, but they could be used for other classes as the package name implied. Back then I did not need it, so I forgot about it, but now I do, and I can't seem to find the functions again. Where can these functions be found?
Edit: I'm not looking for any IDE functions or anything, but rather something that can be executed within the Java application.

Awhile ago, I put together a package for doing what you want, and more. (I needed it for a utility I was writing). It uses the ASM library. You can use reflection, but ASM turned out to perform better.
I put my package in an open source library I have on my web site. The library is here: http://software.clapper.org/javautil/. You want to start with the with ClassFinder class.
The utility I wrote it for is an RSS reader that I still use every day, so the code does tend to get exercised. I use ClassFinder to support a plug-in API in the RSS reader; on startup, it looks in a couple directory trees for jars and class files containing classes that implement a certain interface. It's a lot faster than you might expect.
The library is BSD-licensed, so you can safely bundle it with your code. Source is available.
If that's useful to you, help yourself.
Update: If you're using Scala, you might find this library to be more Scala-friendly.

Spring can do this for you...
BeanDefinitionRegistry bdr = new SimpleBeanDefinitionRegistry();
ClassPathBeanDefinitionScanner s = new ClassPathBeanDefinitionScanner(bdr);
TypeFilter tf = new AssignableTypeFilter(CLASS_YOU_WANT.class);
s.addIncludeFilter(tf);
s.scan("package.you.want1", "package.you.want2");
String[] beans = bdr.getBeanDefinitionNames();
N.B. The TypeFilter is important if you want the correct results!
You can also use exclusion filters here instead.
The Scanner can be found in the spring-context jar, the registry in spring-beans, the type filter is in spring-core.

I really like the reflections library for doing this.
It provides a lot of different types of scanners (getTypesAnnotatedWith, getSubTypesOf, etc), and it is dead simple to write or extend your own.

The code you are talking about sounds like ServiceLoader, which was introduced in Java 6 to support a feature that has been defined since Java 1.3 or earlier. For performance reasons, this is the recommended approach to find interface implementations at runtime; if you need support for this in an older version of Java, I hope that you'll find my implementation helpful.
There are a couple of implementations of this in earlier versions of Java, but in the Sun packages, not in the core API (I think there are some classes internal to ImageIO that do this). As the code is simple, I'd recommend providing your own implementation rather than relying on non-standard Sun code which is subject to change.

Package Level Annotations
I know this question has already been answered a long time ago but another solution to this problem is to use Package Level Annotations.
While its pretty hard to go find all the classes in the JVM its actually pretty easy to browse the package hierarchy.
Package[] ps = Package.getPackages();
for (Package p : ps) {
MyAno a = p.getAnnotation(MyAno.class)
// Recursively descend
}
Then just make your annotation have an argument of an array of Class.
Then in your package-info.java for a particular package put the MyAno.
I'll add more details (code) if people are interested but most probably get the idea.
MetaInf Service Loader
To add to #erickson answer you can also use the service loader approach. Kohsuke has an awesome way of generating the the required META-INF stuff you need for the service loader approach:
http://weblogs.java.net/blog/kohsuke/archive/2009/03/my_project_of_t.html

You could also use the Extensible Component Scanner (extcos: http://sf.net/projects/extcos) and search all classes implementing an interface like so:
Set<Class<? extends MyInterface>> classes = new HashSet<Class<? extends MyInterface>>();
ComponentScanner scanner = new ComponentScanner();
scanner.getClasses(new ComponentQuery() {
#Override
protected void query() {
select().
from("my.package1", "my.package2").
andStore(thoseImplementing(MyInterface.class).into(classes)).
returning(none());
}
});
This works for classes on the file system, within jars and even for those on the JBoss virtual file system. It's further designed to work within standalone applications as well as within any web or application container.

In full generality, this functionality is impossible. The Java ClassLoader mechanism guarantees only the ability to ask for a class with a specific name (including package), and the ClassLoader can supply a class, or it can state that it does not know that class.
Classes can be (and frequently are) loaded from remote servers, and they can even be constructed on the fly; it is not difficult at all to write a ClassLoader that returns a valid class that implements a given interface for any name you ask from it; a List of the classes that implement that interface would then be infinite in length.
In practice, the most common case is an URLClassLoader that looks for classes in a list of filesystem directories and JAR files. So what you need is to get the URLClassLoader, then iterate through those directories and archives, and for each class file you find in them, request the corresponding Class object and look through the return of its getInterfaces() method.

Obviously, Class.isAssignableFrom() tells you whether an individual class implements the given interface. So then the problem is getting the list of classes to test.
As far as I'm aware, there's no direct way from Java to ask the class loader for "the list of classes that you could potentially load". So you'll have to do this yourself by iterating through the visible jars, calling Class.forName() to load the class, then testing it.
However, it's a little easier if you just want to know classes implementing the given interface from those that have actually been loaded:
via the Java Instrumentation framework, you can call Instrumentation.getAllLoadedClasses()
via reflection, you can query the ClassLoader.classes field of a given ClassLoader.
If you use the instrumentation technique, then (as explained in the link) what happens is that your "agent" class is called essentially when the JVM starts up, and passed an Instrumentation object. At that point, you probably want to "save it for later" in a static field, and then have your main application code call it later on to get the list of loaded classes.

If you were asking from the perspective of working this out with a running program then you need to look to the java.lang.* package. If you get a Class object, you can use the isAssignableFrom method to check if it is an interface of another Class.
There isn't a simple built in way of searching for these, tools like Eclipse build an index of this information.
If you don't have a specific list of Class objects to test you can look to the ClassLoader object, use the getPackages() method and build your own package hierarchy iterator.
Just a warning though that these methods and classes can be quite slow.