Develop Reference

Java agent : transform() not applied on all classes

I have been trying to use a Java agent to apply a bytecode transformation with ASM.
I implemented an Agent with the premain method adding a transformer to the Instrumentation.
I added the "Premain-Class" line in the .jar manifest
Premain-Class: <MyAgentPath>
Then I tried to run the application with the agent.
There I have a problem : my transformer modifies some method calls, so if not all involved classes are modified too, it cannot work. And there are some classes which are not modified, like "org.apache.commons.math3.util.FastMath".
Of course then, I got the error :
java.lang.NoSuchMethodError: org.apache.commons.math3.util.FastMath.floor<new_descriptor>
I checked a lot of posts saying it could be the bootstrap loader which does not know the path to this class so I tried to add it using different ways :
Adding the "Boot-Class-Path" line to the manifest :
Boot-Class-Path: <...>/commons-math3<...>.jar
Using the method "appendToBootstrapClassLoaderSearch(JarFile)"
inst.appendToBootstrapClassLoaderSearch("<...>/commons-math3<...>.jar");
Using the JVM argument "-Xbootclasspath/a:"
-Xbootclasspath/a:<...>/commons-math3<...>.jar
None of this changed anything.
I also used the Instrumentation class method getAllLoadedClasses() to see which ones were loaded, and all classes involved in the application process where loaded, including FastMath.
for(Class<?> clazz : MyAgent.getInstInstance().getAllLoadedClasses()){
buffWrite.write(clazz.getName());
As the class "FastMath" gave an error and as the Bootstrap Loader should have its path, I tried adding some method calls to methods from other classes in the same package. It appears the problem does not show for every class of the package.
For example: MathUtils is transformed and a call to the modified method checkFinite(D)V -> checkFinite<new_descriptor>.
So I guess the problem has nothing to do with the paths given to the bootstrap loader.
If you have some ideas about what is happening, I would be glad to hear about it!

A NoSuchMethodError is most likely not caused by not adding something to the bootstrap class loader. The only chance that this could be a problem would be if there were suddenly two such jars available where one was instrumented and the other was not.
If you call change a method checkFinite(D)V to become another method checkFinite<new_descriptor>, then you need to make sure that any class using FastMath.floor updates the descriptor to this method. This means that you need to walk through every method of every class looking for visitMethodIns calls of ASM. It seems like you are missing some. Since you are changing the layout of the FastMath class, you must instrument it while loading it for the first time and you cannot redefine it.
The Java internal classes do not know of FastMath as this is a third-party dependency. Therefore, it should be possible to instrument any call from your agent. It seems to me like you are loading FastMath prematurely.

A better way to call static methods in user-submitted code?

I have a large data set. I am creating a system which allows users to submit java source files, which will then be applied to the data set. To be more specific, each submitted java source file must contain a static method with a specific name, let's say toBeInvoked(). toBeInvoked will take a row of the data set as an array parameter. I want to call the toBeInvoked method of each submitted source file on each row in the data set. I also need to implement security measures (so toBeInvoked() can't do I/O, can't call exit, etc.).
Currently, my implementation is this: I have a list of the names of the java source files. For each file, I create an instance of the custom secure ClassLoader which I coded, which compiles the source file and returns the compiled class. I use reflection to extract the static method toBeInvoked() (e.g. method = c.getMethod("toBeInvoked", double[].class)). Then, I iterate over the rows of the data set, and invoke the method on each row.
There are at least two problems with my approach:
it appears to be painfully slow (I've heard reflection tends to be slow)
the code is more complicated than I would like
Is there a better way to accomplish what I am trying to do?

There is no significantly better approach given the constraints that you have set yourself.
For what it is worth, what makes this "painfully slow" is compiling the source files to class files and loading them. That is many orders of magnitude slower than the use of reflection to call the methods.
(Use of a common interface rather than static methods is not going to make a measurable difference to speed, and the reduction in complexity is relatively small.)
If you really want to simplify this and speed it up, change your architecture so that the code is provided as a JAR file containing all of the compiled classes.

Assuming your #toBeInvoked() could be defined in an interface rather than being static (it should be!), you could just load the class and cast it to the interface:
Class<? extends YourInterface> c = Class.forName("name", true, classLoader).asSubclass(YourInterface.class);
YourInterface i = c.newInstance();
Afterwards invoke #toBeInvoked() directly.
Also have a look into java.util.ServiceLoader, which could be helpful for finding the right class to load in case you have more than one source file.

Personally, I would use an interface. This will allow you to have multiple instance with their own state (useful for multi-threading) but more importantly you can use an interface, first to define which methods must be implemented but also to call the methods.
Reflection is slow but this is only relative to other options such as a direct method call. If you are scanning a large data set, the fact you have to pulling data from main memory is likely to be much more expensive.

I would suggest following steps for your problem.
To check if the method contains any unwanted code, you need to have a check script which can do these checks at upload time.
Create an Interface having a method toBeInvoked() (not a static method).
All the classes which are uploaded must implement this interface and add the logic inside this method.
you can have your custom class loader scan a particular folder for new classes being added and load them accordingly.
When a file is uploaded and successfully validated, you can compile and copy the class file to the folder which class loader scans.
You processor class can lookup for new files and then call toBeInvoked() method on loaded class when required.
Hope this help. (Note that i have used a similar mechanism to load dynamically workflow step classes in Workflow Engine tool which was developed).

Java variable names out of strings

Have a problem with undocumented libraries, where I am trying to replace some classes in the .jar without any source code provided. One class is implemented from an existing one (no java file for it) so i have all the methods and method signatures but the no way to make any sens out of parameters because they are all named arg0, arg1..., because there are a lot of methods and some contain up to 43 parameters trying to loop through the parameters in order to see what is coming in. Is there any way to use String and dynamically get to that variable?
edit:
---more info---
Everything is compiled in the jar file, which I was able to repackage without one class that I want to change. So, the class that I want to change extended another class which is compiled. So, when extended the compiled class my IDE auto-generated all the methods and their signatures whre the parameters are named sequentially and I would like to place a for-loop in every function to see what is coming in instead of go one-by-one and print it to the console. I think I was a little vague in the original question.
Thank you

No, if those symbols have been removed from the compiled class file, you cannot recover them.

You can't get access to the parameter names unless the class was compiled in debug mode. But if you just want to iterate and print the values of each parameter passed in, mockito might be able to help you there if you mock the method. Or you can just step with a graphical debugger.

JNI: How to get method signature for debugging purposes

Can I enumerate all native methods in java, those that have to be
implemented in c/c++ using JNI?
Can I enumerate native methods by
name (there could be multiple overloads with the same name)?
How can I retrieve method signature to be able to generate the method
signature used by JNI?
Is there a way to check if all native jni methods have been bound properly, instead of trying to call them and get java.lang.UnsatisfiedLinkError exceptions. Sometimes method signature changes on either side without properly updating java or c++ side and I'd like to add some debugging code to detect these issues and handle them (perhaps by generating proper method signature and printing it to the log so I can easily fix the code).
I prefer JNI solution, but if something can be done with help on java side then it's ok also.
If I use registerNatives and register methods that weren't declared in java then it fails and prints it to logcat:
E/dalvikvm( 1445): ERROR: couldn't find native method
E/dalvikvm( 1445): Requested: Lcom/bla/bla/bla/Test;.nativeTestXX:()Z
but I'd like to catch this error and handle it myself. Is it possible to do it?
EDIT:
In my JNI code I have a static nativeInit (as suggested in Android JNI tips) that registers all native methods. In that same function I'd like to verify that all native methods are properly bound. That is, I don't need to wait till some uninitialized method is called and the app exists. The problem that I have: there is a lot of jni code written at different times by different ppl and some methods simply became incorrect, but they are used only in some obscure conditions. The best way for me, I think, is to check that all native methods are bound to some c++ function. The other problem, is that part of JNI code uses binding by exporting all these Long_java_names where method signature changes on either side cannot be detected.

There is no call to check for "unbound" native methods. Using RegisterNatives to perform explicit registration ensures that all methods you register have a matching declaration in the Java sources, but there is no way to check for native-declared methods for which there is no implementation (other than calling it and catching the exception).
At the point where a method with a native implementation is called, if nothing has yet been registered then Dalvik will search through the various shared libraries to find a match. What it sounds like you want is a way to force this search and check the result without actually calling the method. There is no such thing.
There are various ways to generate lists of native-declared methods, either statically or at runtime, but you also need a way to determine if an implementation is available. You're better off in the long run having unit tests that exercise the code.

GetPropertyAction vs System.getProperty in obtaining system variables

I have been using quite a lot of
System.getProperty("property")
in order to obtain environmental information. However, it seems to me that Sun prefers the following :
(String) java.security.AccessController.doPrivileged(
new sun.security.action.GetPropertyAction("property"));
The strange thing is that this code involves a cast and as a result should be slightly slower than the
System.getProperty
implementation, that only uses a security manager and then instantly fetches the property from the instance variable props. My question is why did Sun chose to use the second method to obtain most environmental variables in their code internally, while
System.getProperty
seems like the faster way to go?

Both methods have a different meaning, and thus the right one has to be used depending on what the current code needs to do.
The code System.getProperty("property") says "Give me the value of the property, if the current security context allows me to read it."
The code that uses doPrivileged says "Give me the value of the property, if the current class (where this line of code is in) is allowed to read it."
The difference comes into play, when the protection domain of the current class is different from the currently active security context.
For example, consider a framework which executes the code of a plugin, which is untrusted. So the framework uses a SecurityManager to restrict the actions of the untrusted plugin code. But of course the plugin may call some methods of the framework, and suppose that one of these methods needs to read a property. Now as the method is called from untrusted restricted code, it is itself restricted and thus reading the property would fail. But of course the framework trusts itself and wants itself to be able to read that property, even in the case that somewhere in the call stack is untrusted code. That's when you need to use doPrivileged. It basically says "no matter what is up there in the call stack, I am a piece of framework code, and I am allowed to do whatever the framework code is allowed to do". So reading the property using the second method succeeds.
Of course one needs to be careful when using doPrivileged in order to not let the (untrusted) calling code do to much. If, for example, the framework code offers the following method to the plugin:
public String getProp(String key) {
return (String) java.security.AccessController.doPrivileged(
new sun.security.action.GetPropertyAction(key));
}
this would completely invalidate the policy that the untrusted code is not allowed to read system properties, because it can just use your method.
So use this method only when you know it is safe to do it, and only when you need it (which is, when you want your code to be able to do more than some other code should be able to do directly). Inside a normal application (which usually runs with no SecurityManager or the same security context for all code), there is no difference and the first method should be used.

I would recommend to stick with System.getProperty() since sun.security.action.GetPropertyAction seems to be proprietary to SUN and will not work on all Java VM implementations. Even the compiler warns you about it as:
warning: sun.security.action.GetPropertyAction is Sun proprietary API and may be removed in a future release
To understand what it actually means see this answer.

The reason to use a class like sun.security.action.GetPropertyAction is to avoid loading several, basically identical classes.
If you wrote:
(String) java.security.AccessController.doPrivileged(
new java.security.PrivilegedAction<java.lang.String>() {
String run() {
System.getProperty("property");
}
}
);
Each time you wanted to get a system property, you would load a new class for each getProperty call. Each class takes system resources and lives as long as the containing ClassLoader (forever for the bootclassloader).
Check out the javap output for more details:
javap -c -v -p sun.security.action.GetPropertyAction