I have a large 3rd party code base I can't modify, but I need to make a small but important change in many different places. I was hoping to use a ByteBuddy based agent, but I can't figure out how. The call I need to replace is of the form:
SomeSystemClass.someMethod("foo")
and I need to replace it with
SomeSystemClass.someMethod("bar")
while leaving all other calls to the same method untouched
SomeSystemClass.someMethod("ignore me")
Since SomeSystemClass is a JDK class, I do not want to advise it, but only the classes that contain calls to it. How can this be done?
Note that:
someMethod is static and
the calls (at least some of them) are inside a static initializer block
There are two approaches to this with Byte Buddy:
You transform all classes with the call site in question:
new AgentBuilder.Default()
.type(nameStartsWith("my.lib.pkg."))
.transform((builder, type, loader, module) -> builder.visit(MemberSubstitution.relaxed()
.method(SomeSystemClass.class.getMethod("someMethod", String.class))
.replaceWith(MyAlternativeDispatcher.class.getMethod("substitution", String.class)
.on(any()))
.installOn(...);
In this case, I suggest you to implement a class MyAlternativeDispatcher to your class path (it can also be shipped as part of the agent unless you have a more complex class loader setup such as OSGi where you implement the conditional logic:
public class MyAlternativeDispatcher {
public static void substitution(String argument) {
if ("foo".equals(argument)) {
argument = "bar";
}
SomeSystemClass.someMethod(argument);
}
}
Doing so, you can set break points and implement any complex logic without thinking too much of byte code after setting up the agent. You can, as suggested, even ship the substitution method independently of the agent.
Instrument the system class itself and make it caller sensitive:
new AgentBuilder.Default()
.with(RedefinitionStrategy.RETRANSFORMATION)
.disableClassFormatChanges()
.type(is(SomeSystemClass.class))
.transform((builder, type, loader, module) -> builder.visit(Advice.to(MyAdvice.class).on(named("someMethod").and(takesArguments(String.class)))))
.installOn(...);
In this case, you'd need to reflect on the caller class to make sure you only alter behavior for the classes you want to apply this change for. This is not uncommon within the JDK and since Advice inlines ("copy pastes") the code of your advice class into the system class, you can use the JDK internal APIs without restriction (Java 8 and prior) if you cannot use the stack walker API (Java 9 and later):
class MyAdvice {
#Advice.OnMethodEnter
static void enter(#Advice.Argument(0) String argument) {
Class<?> caller = sun.reflect.Reflection.getCallerClass(1); // or stack walker
if (caller.getName().startsWith("my.lib.pkg.") && "foo".equals(argument)) {
argument = "bar";
}
}
}
Which approach should you choose?
The first approach is probably more reliable but it is rather costly since you have to process all classes in a package or subpackages. If there are many classes in this package you will pay quite a price for processing all these classes to check for relevant call sites and therefore delay application startup. Once all classes are loaded, you have however paid the price and everything is in place without having altered a system class. You do however need to take care of class loaders to make sure that your substitution method is visible to everybody. In the simplest case, you can use the Instrumentation API to append a jar with this class to the boot loader what makes it globally visible.
With the second approach, you only need to (re-)transform a single method. This is very cheap to do but you will add a (minimal) overhead to every call to the method. Therefore, if this method is invoked a lot on a critical execution path, you'd pay a price on every invocation if the JIT does not discover an optimization pattern to avoid it. I'd prefer this approach for most cases, I think, a single transformation is often more reliable and performant.
As a third option, you could also use MemberSubstitution and add your own byte code as a replacement (Byte Buddy exposes ASM in the replaceWith step where you can define custom byte code instead of delegating). This way, you could avoid the requirement of adding a replacement method and just add the substitution code in-place. This does however bear the serious requirement that you:
do not add conditional statements
recompute the stack map frames of the class
The latter is required if you add conditional statements and Byte Buddy (or anybody) cannot optimize it in-method. Stack map frame recomputation is very expensive, fails comparable often and can require class loading locks to dead lock. Byte Buddy optimizes ASM's default recomputation, trying to avoid dead locks by avoiding class loading but there is no guarantee either, so you should keep this in mind.
Related
Is the following code considered to be runtime reflection or is it type introspection?
Class c = java.util.ArrayList.class;
String className = c.getName();
I want to use this in the compilation phase, and do not want to use any resources (including time) in runtime. Does it use any runtime resource?
It is both, as #StephenC already points out correctly.
To your request for a compile-time solution: There is a solution, but you'd need to implement your own Builder for that.
First off, some standard-Java implementations:
Because implementing and including your own Builder might be a lot of work for very little effect, here's a much easier answer: class initialization.
public class CompileTimeInit {
// option 1
static public final String ARRAYLIST_CLASS_NAME = java.util.ArrayList.class.getName();
// option 2
static public final String ARRAYLIST_CLASS_NAME_INIT_BLOCK;
static {
ARRAYLIST_CLASS_NAME_INIT_BLOCK = java.util.ArrayList.class.getName();
}
public static void main(final String[] args) {
System.out.println("1:\t" + ARRAYLIST_CLASS_NAME);
System.out.println("2:\t" + ARRAYLIST_CLASS_NAME_INIT_BLOCK);
// option 3
final Class c = java.util.ArrayList.class;
final String className = c.getName();
System.out.println("3:\t" + className);
}
}
static initialization will take its (very minimal) time in class intialization. This is part of the runtime, but it will only take place once a class is referenced/metnioned in some code that gets executed. This initialization will only be done once (per classloader loaded class) and is considered really fast.
Now, to the real compile-time implementations:
You can have really advanced builders implemented, like Project Lombok does, along with integration into IDEs, compilers, build tools, platforms. Check out their page, top menu, "Install", to see the options available.
You can write your own builder and include it into the build cycle of your build cycle. The best way to do it is via an Annotaion "Pre"-Processor.
The third option is to write a complete standalone application. This means you have to implement:
opening .java source files
scanning for your Interfaces or Annotations
replacing the code with the calculated contents
saving .java source files
let the compilation phase continue
and include that into you build cycle. This is also partially described in the article above
I would have said that it is BOTH runtime reflection AND type introspection.
Note that the [introspection] tag defines introspection as:
"A capability of some object-oriented programming languages to determine the type of an object at runtime."
You could also say that your example uses (runtime) reflection to perform type introspection.
I want to use this in the compilation phase.
It is possible to examine types at compile time in an annotation processor. However, I suspect that the effort involved in doing that would be prohibitive. You need to be able to spot a specific pattern in two statements and then replace the statements with something else.
(Have you benchmarked the class name lookup to see how much time it actually takes? I would have thought it was just a few nanoseconds. Not enough to be significant.)
I understand if I have a class file I can load it at run time and execute it's methods through classLoader. However, what if I only have bytecode or java code for a single method? Is it possible to dynamically create a class at run time and then invoke the method?
There is a planned feature, JEP 8158765: Isolated Methods, also on the bugtracking list, which would allow to load and execute such bytecode, without generating a fully materialized Class. It could look like
MethodHandle loadCode(String name, MethodType type, byte[] instructions, Object[] constants)
in the class MethodHandles.Lookup
However, this feature is in draft state, so it could take significant time before becoming an actual API and it might even happen that it gets dropped in favor of an entirely different feature covering the use cases the authors of the JEP have in mind.
Until then, there is no way around generating the necessary bytes before and after the method’s bytecode, to describe a complete class and loading that class. But of course, you can write your own method accepting a method’s byte code and some metadata, like the expected signature, generating such a class and reuse that method.
Note that there’s an alternative to creating a new ClassLoader, Class<?> defineClass(byte[] bytes) in class MethodHandles.Lookup which allows to add a class to an existing class loading context, since Java 9.
The bytecode for a method refers to entries in the class's constant pool, so it doesn't make sense in isolation.
Similar Questions: Here and Here
I guess the situation is pretty uncommon to begin with, and so I admit it is probably too localized for SO.
The Problem
public class bqf implements azj
{
...
public static float b = 0.0F;
...
public void b(...)
{
...
/* b, in both references below,
* is meant to be a class (in the
* default package)
*
* It is being obscured by field
* b on the right side of the
* expression.
*/
b var13 = b.a(var9, var2, new br());
...
}
}
The error is: cannot invoke a(aji, String, br) on primitive type float.
Compromisable limitations:
Field b cannot be renamed.
Class b cannot be renamed or refactored.
Why
I am modifying an obfuscated program. For irrelevant[?], unknown (to me), and uncompromisable reasons the modification must be done via patching the original jar with .class files. Hence, renaming the public field b or class b would require modifying much of the program. Because all of the classes are in the default package, refactoring class b would require me to modify every class which references b (much of the program). Nevertheless there is a substantial amount of modification I do intend on doing, and it is a pain to do it at the bytecode level; just not enough to warrant renaming/refactoring.
Possible Solutions
The most obvious one is to rename/refactor. There are thousands of classes, and every single one is in the default package. It seems like every java program I want to modify has that sort of obfuscation. : (
Anyways sometimes I do take the time to just go about manually renaming/refactoring the program. But when when there's too many errors (I once did 18,000), this is not a viable option.
The second obvious option is to do it in bytecode (via ASM). Sometimes this is ok, when the modifications are small or simple enough. Unfortunately doing bytecode modifications on only the files which I can't compile through java (which is most of them, but this is what I usually try to do) is painfully slow by comparison.
Sometimes I can extend class b, and use that in my modified class. Obviously this won't always work, for example when b is an enum. Unfortunately this means a lot of extra classes.
It may be possible to create a class with static wrapper methods to avoid obscurity. I just thought of this.
A tool which remaps all of the names (not deobfuscate, just unique names), then unmaps them after you make modifications. That would be sweet. I should make one if it doesn't exist.
The problem would also be solved with a way to force the java compiler to require the keyword "this".
b.a(var9, var2, new br());
can easily be rewritten using reflection:
Class.forName("b").getMethod("a", argTypes...).invoke(null, var9, var2, new br());
The problem would also be solved with a way to force the java compiler to require the keyword "this".
I don't think how this would help you for a static member. Compiler would have to require us to qualify everything—basically, disallow simple names altogether except for locals.
Write a helper method elsewhere that invokes b.a(). You can then call that.
Note: In Java the convention is that the class would be named B and not b(which goes for bqf and aqz too) and if that had been followed the problem would not have shown.
The real, long time cure, is not to put classes in the default package.
I'm writing a library that needs to have some code if a particular library is included. Since this code is scattered all around the project, it would be nice if users didn't have to comment/uncomment everything themselves.
In C, this would be easy enough with a #define in a header, and then code blocks surrounded with #ifdefs. Of course, Java doesn't have the C preprocessor...
To clarify - several external libraries will be distributed with mine. I do not want to have to include them all to minimize my executable size. If a developer does include a library, I need to be able to use it, and if not, then it can just be ignored.
What is the best way to do this in Java?
There's no way to do what you want from within Java. You could preprocess the Java source files, but that's outside the scope of Java.
Can you not abstract the differences and then vary the implementation?
Based on your clarification, it sounds like you might be able to create a factory method that will return either an object from one of the external libraries or a "stub" class whose functions will do what you would have done in the "not-available" conditional code.
As other have said, there is no such thing as #define/#ifdef in Java. But regarding your problem of having optional external libraries, which you would use, if present, and not use if not, using proxy classes might be an option (if the library interfaces aren't too big).
I had to do this once for the Mac OS X specific extensions for AWT/Swing (found in com.apple.eawt.*). The classes are, of course, only on the class-path if the application is running on Mac OS. To be able to use them but still allow the same app to be used on other platforms, I wrote simple proxy classes, which just offered the same methods as the original EAWT classes. Internally, the proxies used some reflection to determine if the real classes were on the class-path and would pass through all method calls. By using the java.lang.reflect.Proxy class, you can even create and pass around objects of a type defined in the external library, without having it available at compile time.
For example, the proxy for com.apple.eawt.ApplicationListener looked like this:
public class ApplicationListener {
private static Class<?> nativeClass;
static Class<?> getNativeClass() {
try {
if (ApplicationListener.nativeClass == null) {
ApplicationListener.nativeClass = Class.forName("com.apple.eawt.ApplicationListener");
}
return ApplicationListener.nativeClass;
} catch (ClassNotFoundException ex) {
throw new RuntimeException("This system does not support the Apple EAWT!", ex);
}
}
private Object nativeObject;
public ApplicationListener() {
Class<?> nativeClass = ApplicationListener.getNativeClass();
this.nativeObject = Proxy.newProxyInstance(nativeClass.getClassLoader(), new Class<?>[] {
nativeClass
}, new InvocationHandler() {
public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
String methodName = method.getName();
ApplicationEvent event = new ApplicationEvent(args[0]);
if (methodName.equals("handleReOpenApplication")) {
ApplicationListener.this.handleReOpenApplication(event);
} else if (methodName.equals("handleQuit")) {
ApplicationListener.this.handleQuit(event);
} else if (methodName.equals("handlePrintFile")) {
ApplicationListener.this.handlePrintFile(event);
} else if (methodName.equals("handlePreferences")) {
ApplicationListener.this.handlePreferences(event);
} else if (methodName.equals("handleOpenFile")) {
ApplicationListener.this.handleOpenFile(event);
} else if (methodName.equals("handleOpenApplication")) {
ApplicationListener.this.handleOpenApplication(event);
} else if (methodName.equals("handleAbout")) {
ApplicationListener.this.handleAbout(event);
}
return null;
}
});
}
Object getNativeObject() {
return this.nativeObject;
}
// followed by abstract definitions of all handle...(ApplicationEvent) methods
}
All this only makes sense, if you need just a few classes from an external library, because you have to do everything via reflection at runtime. For larger libraries, you probably would need some way to automate the generation of the proxies. But then, if you really are that dependent on a large external library, you should just require it at compile time.
Comment by Peter Lawrey: (Sorry to edit, its very hard to put code into a comment)
The follow example is generic by method so you don't need to know all the methods involved. You can also make this generic by class so you only need one InvocationHandler class coded to cover all cases.
public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
String methodName = method.getName();
ApplicationEvent event = new ApplicationEvent(args[0]);
Method method = ApplicationListener.class.getMethod(methodName, ApplicationEvent.class);
return method.invoke(ApplicationListener.this, event);
}
In Java one could use a variety of approaches to achieve the same result:
Dependency Injection
Annotations
Reflection
The Java way is to put behaviour that varies into a set of separate classes abstracted through an interface, then plug the required class at run time. See also:
Factory pattern
Builder pattern
Strategy pattern
Well, Java syntax is close enough to C that you could simply use the C preprocessor, which is usually shipped as a separate executable.
But Java isn't really about doing things at compile time anyway. The way I've handled similar situations before is with reflection. In your case, since your calls to the possibly-non-present library are scattered throughout the code, I would make a wrapper class, replace all the calls to the library with calls to the wrapper class, and then use reflection inside the wrapper class to invoke on the library if it is present.
Use a constant:
This week we create some constants
that have all of the benefits of using
the C preprocessor's facilities to
define compile-time constants and
conditionally compiled code.
Java has gotten rid of the entire
notion of a textual preprocessor (if
you take Java as a "descendent" of
C/C++). We can, however, get the best
benefits of at least some of the C
preprocessor's features in Java:
constants and conditional compilation.
I don't believe that there really is such a thing. Most true Java users will tell you that this is a Good Thing, and that relying on conditional compilation should be avoided at almost all costs.
I'm don't really agree with them...
You CAN use constants that can be defined from the compile line, and that will have some of the effect, but not really all. (For example, you can't have things that don't compile, but you still want, inside #if 0... (and no, comments don't always solve that problem, because nesting comments can be tricky...)).
I think that most people will tell you to use some form of inheritance to do this, but that can be very ugly as well, with lots of repeated code...
That said, you CAN always just set up your IDE to throw your java through the pre-processor before sending it to javac...
"to minimize my executable size"
What do you mean by "executable size"?
If you mean the amount of code loaded at runtime, then you can conditionally load classes through the classloader. So you distribute your alternative code no matter what, but it's only actually loaded if the library that it stands in for is missing. You can use an Adapter (or similar) to encapsulate the API, to make sure that almost all of your code is exactly the same either way, and one of two wrapper classes is loaded according to your case. The Java security SPI might give you some ideas how this can be structured and implemented.
If you mean the size of your .jar file, then you can do the above, but tell your developers how to strip the unnecessary classes out of the jar, in the case where they know they aren't going to be needed.
I have one more best way to say.
What you need is a final variable.
public static final boolean LibraryIncluded= false; //or true - manually set this
Then inside the code say as
if(LibraryIncluded){
//do what you want to do if library is included
}
else
{
//do if you want anything to do if the library is not included
}
This will work as #ifdef. Any one of the blocks will be present in the executable code. Other will be eliminated in the compile time itself
Use properties to do this kind of thing.
Use things like Class.forName to identify the class.
Do not use if-statements when you can trivially translate a property directly to a class.
Depending on what you are doing (not quite enough information) you could do something like this:
interface Foo
{
void foo();
}
class FakeFoo
implements Foo
{
public void foo()
{
// do nothing
}
}
class RealFoo
{
public void foo()
{
// do something
}
}
and then provide a class to abstract the instantiation:
class FooFactory
{
public static Foo makeFoo()
{
final String name;
final FooClass fooClass;
final Foo foo;
name = System.getProperty("foo.class");
fooClass = Class.forName(name);
foo = (Foo)fooClass.newInstance();
return (foo);
}
}
Then run java with -Dfoo.name=RealFoo|FakeFoo
Ignored the exception handling in the makeFoo method and you can do it other ways... but the idea is the same.
That way you compile both versions of the Foo subclasses and let the developer choose at runtime which they wish to use.
I see you specifying two mutually exclusive problems here (or, more likely, you have chosen one and I'm just not understanding which choice you've made).
You have to make a choice: Are you shipping two versions of your source code (one if the library exists, and one if it does not), or are you shipping a single version and expecting it to work with the library if the library exists.
If you want a single version to detect the library's existence and use it if available, then you MUST have all the code to access it in your distributed code--you cannot trim it out. Since you are equating your problem with using a #define, I assumed this was not your goal--you want to ship 2 versions (The only way #define can work)
So, with 2 versions you can define a libraryInterface. This can either be an object that wraps your library and forwards all the calls to the library for you or an interface--in either case this object MUST exist at compile time for both modes.
public LibraryInterface getLibrary()
{
if(LIBRARY_EXISTS) // final boolean
{
// Instantiate your wrapper class or reflectively create an instance
return library;
}
return null;
}
Now, when you want to USE your library (cases where you would have had a #ifdef in C) you have this:
if(LIBRARY_EXISTS)
library.doFunc()
Library is an interface that exists in both cases. Since it's always protected by LIBRARY_EXISTS, it will compile out (should never even load into your class loader--but that's implementation dependent).
If your library is a pre-packaged library provided by a 3rd party, you may have to make Library a wrapper class that forwards it's calls to your library. Since your library wrapper is never instantiated if LIBRARY_EXISTS is false, it shouldn't even be loaded at runtime (Heck, it shouldn't even be compiled in if the JVM is smart enough since it's always protected by a final constant.) but remember that the wrapper MUST be available at compile time in both cases.
If it helps have a look at j2me polish or Using preprocessor directives in BlackBerry JDE plugin for eclipse?
this is for mobiles app but this can be reused no ?
Is there a concept of inline functions in java, or its replaced something else? If there is, how is it used? I've heard that public, static and final methods are the inline functions. Can we create our own inline function?
In Java, the optimizations are usually done at the JVM level. At runtime, the JVM perform some "complicated" analysis to determine which methods to inline. It can be aggressive in inlining, and the Hotspot JVM actually can inline non-final methods.
The java compilers almost never inline any method call (the JVM does all of that at runtime). They do inline compile time constants (e.g. final static primitive values). But not methods.
For more resources:
Article: The Java HotSpot Performance Engine: Method Inlining Example
Wiki: Inlining in OpenJDK, not fully populated but contains links to useful discussions.
No, there is no inline function in java. Yes, you can use a public static method anywhere in the code when placed in a public class. The java compiler may do inline expansion on a static or final method, but that is not guaranteed.
Typically such code optimizations are done by the compiler in combination with the JVM/JIT/HotSpot for code segments used very often. Also other optimization concepts like register declaration of parameters are not known in java.
Optimizations cannot be forced by declaration in java, but done by compiler and JIT. In many other languages these declarations are often only compiler hints (you can declare more register parameters than the processor has, the rest is ignored).
Declaring java methods static, final or private are also hints for the compiler. You should use it, but no garantees. Java performance is dynamic, not static. First call to a system is always slow because of class loading. Next calls are faster, but depending on memory and runtime the most common calls are optimized withinthe running system, so a server may become faster during runtime!
Java does not provide a way to manually suggest that a method should be inlined. As #notnoop says in the comments, the inlining is typically done by the JVM at execution time.
What you said above is correct. Sometimes final methods are created as inline, but there is no other way to explicitly create an inline function in java.
Well, there are methods could be called "inline" methods in java, but depending on the jvm. After compiling, if the method's machine code is less than 35 byte, it will be transferred to a inline method right away, if the method's machine code is less than 325 byte, it could be transferred into a inline method, depending on the jvm.
Real life example:
public class Control {
public static final long EXPIRED_ON = 1386082988202l;
public static final boolean isExpired() {
return (System.currentTimeMillis() > EXPIRED_ON);
}
}
Then in other classes, I can exit if the code has expired. If I reference the EXPIRED_ON variable from another class, the constant is inline to the byte code, making it very hard to track down all places in the code that checks the expiry date. However, if the other classes invoke the isExpired() method, the actual method is called, meaning a hacker could replace the isExpired method with another which always returns false.
I agree it would be very nice to force a compiler to inline the static final method to all classes which reference it. In that case, you need not even include the Control class, as it would not be needed at runtime.
From my research, this cannot be done. Perhaps some Obfuscator tools can do this, or, you could modify your build process to edit sources before compile.
As for proving if the method from the control class is placed inline to another class during compile, try running the other class without the Control class in the classpath.
so, it seems there arent, but you can use this workaround using guava or an equivalent Function class implementation, because that class is extremely simple, ex.:
assert false : new com.google.common.base.Function<Void,String>(){
#Override public String apply(Void input) {
//your complex code go here
return "weird message";
}}.apply(null);
yes, this is dead code just to exemplify how to create a complex code block (within {}) to do something so specific that shouldnt bother us on creating any method for it, AKA inline!
Java9 has an "Ahead of time" compiler that does several optimizations at compile-time, rather than runtime, which can be seen as inlining.