Develop Reference

How to determine all runtime (but not static) dependencies of Java source by reading the code?

Let's say I wanted to assert that a given Java source folder contained all of the source code required to run any of the programs contained in the source folder. Simply showing that the whole source folder compiles would be insufficient because perhaps some of those programs use reflection to instantiate objects. As such I could search through all the code and look for invocations of newInstance() to learn what classes are expected to be present at runtime. But what about calls to Class.forName(...) that are not involved in calls to newInstance() ? Better check for those as well. But how many such things do I need to check for?
Is there any sort of exhaustive list I could consult to ensure that I am considering each way in Java that such a runtime dependency could be introduced? Restated, does there exist some list of operations such that if I can show none of the source in a source folder use those operations (and that folder compiles) that all (code) dependencies are present?
If no such list exists, could we start one in this thread and make a best effort to cover all operations that we know of?
Edit
I'd like to narrow the question a little. What I'm most interested in showing is that if a codebase compiles, all dependencies are present. It seems to me to do this I would need to first compile the codebase and then check to see if any of the code ever calls certain methods (e.g. newInstance) that could introduce a runtime dependency. If no such methods are found, I'm reasonably sure that all required code is present in source form and running the program will not generate a ClassNotFoundException.

Answer to the original question
There is no extensive way You can do this as far as I know. I mean consider the following code:
public static Object calleableFromAnywhere(Object o) throws IllegalAccessException, InstantiationException {
Object ret = null;
if(!Objects.isNull(o)){
ret = o.getClass().newInstance();
}
return ret;
}
In this case You do not even know what kind of dependency You will have at runtime.
If you restrict your search to constructor based object creation only you have a lot of choices also:
Constructor<?> constructor = o.getClass().getConstructor();
//there could be a lot of constructors enclosing different paramethers
Object something = o.getClass().newInstance();
//default constructor called
And even those could be obtained by reflection... The rabbit hole is infinitely deep.
The call You mentioned Class.forName(...) could take any parameter String and those could be given from a database or an input field from the user.
I do not think it is possible to predict all the String variables you could encounter at runtime. If you have anything like this You are not likely to succeed.
TL.DR.: there is no exact solution I know of. According to the question You are not only interested in newInstance or instance creations as you could have static methods on classes as dependency as well. The idea is nice, but there is no 100% solution to this problem.
Narrowed/clarified questions answer
The a newInstance call is not introducing a new dependency. It could be called on a loaded class definition only. (Basically you never get to the newInstance if the class could not be loaded.) So if your goal is to answer with high certainty that if a class will not be represented is an another problem.
Altering the previous example shows that you are not likely to have the definit dependent classes:
public static Class getClass(String name) throws ClassNotFoundException {
return Class.forName(name);
}
As name could be anything that you can recieve runtime. Also there is an other ways to load classes as the following example shows.
public Class getClassExample1(String name) throws ClassNotFoundException {
return this.getClass().getClassLoader().loadClass(name);
}
According to the JavaDoc of ClassNotFoundException exception these places are using it:
#see java.lang.Class#forName(java.lang.String)
#see java.lang.ClassLoader#findSystemClass(java.lang.String)
#see java.lang.ClassLoader#loadClass(java.lang.String, boolean)
Those are the places where it is thrown, but you have to check every path to those methods like someInstance.getClass(name).
In addition to runtime dependencies You can encounter NoClassDefFoundError as well. If you are using some containers and set some dependencies as provided for example in maven You can also have some problems runtime, while the compilation was totally fine.
TL.DR.V2: You can only minimize the risk of not having a dependency with a static analysis, but if you want to try it you should check the mentioned methods and all of the call chains (even via reflection) that could lead to these.

Not generally possible
You can, of course, answer this question for trivial cases, but not universally. Its not possible to tell for the general case if the code will ever attempt to load a class not available.
Think of any code that takes input from any external source and loads a class in response (e.g. user input is a JDBC database url, requiring a JDBC driver). Same goes for cases where a class may be resolved in response to internal program state, but the state isn't trivially predictable.
TLDR:
Because its the same as solving the halting problem

Actually something like Annotation Processing would add extra complexity to the problem, you can generate Java code with it at compile time, which can have dependencies you don't see directly in code and would need further understanding of the subject and its limitations on making new dependencies, you can read more about it here: AnnotationProcessing 101

"Find usages" functionality as an IntelliJ plugin

I'm trying to find a way in IntelliJ IDEA to find all the usages of a few library method calls and classes in a particular project.
The goal is to compile a list of classes which make reference to these specific methods or classes.
How can I go about this, I can see there is a MethodReferencesSearch which looks like it could be helpful, however the search method requires an instance of PsiMethod.
How can I create an instance of PSI method that matches the method in a particular lib class (say I wanted to find all the usages of the concat(...) method in Java's String class
Basically I'm trying to build a plugin that will generate a graph of certain method calls from within a project. For example something that would graph a set of routes by looking for certain method calls in a library. I.e. if Class A calls x(T) with type class B and class B calls x(T) with type of Class C, I would have a graph that looks like A -> B -> C etc. Find usages is great, it just doesnt work well for my needs.

You can get the true PsiMethod by JavaPsiFacade.getInstance(...).findClass("java.lang.String", ...allScope(...)).findsMethodByName("concat", false)[0]. This method can then be passed to MethodReferenceSearch.

I am presuming that you can't guarantee that you have a usage of concat easily available (for example, at the user's cursor position in an open document).
A hacky way to do it would be to create a small, correct, self-contained java class in a String, like below:
class Nothing { String s = "a".concat("b"); }
Then, there is a way (if I remember correctly) to use IntelliJ to parse the class contained in this String, thereby giving you a PsiReference to the method you want to find usages on (in this case, concat).
Would this approach be useful to you?
If so, I can dig out a code example on how this can be done.

GWT CodeSplitting [duplicate]

I found that the more than 60% of the javaScript code generated by GWT on my application is for RPC serializers.
Also I found that serializers are not shared between service interfaces, I mean if I have for example AccountDTO type referenced on 2 rpc service interfaces, I will get 2 serializer classes instead of 1 for the same type.
In Order to reduce the size of the compiled code I was thinking that maybe I could use Deferred Binding in order to do a replacement of all the services interfaces I have for one big interface. If that could be possible, maybe then GWTCompiler will produce only one AccountDTO serializer instead of 2.
I'm not sure this is a good idea or if there is a better solution for my problem.
What I was trying to implement was something like this:
// Define new interface that extends all service interfaces
public interface GenericService extends RemoteService,
AccountingService,
FinancialService,..., { }
public interface GenericServiceAsync extends AccountingServiceAsync,
FinancialServiceAsync, ..., { }
// At Application.gwt.xml do:
<module>
...
...
<replace-with class="com.arballon.gwt.core.client.GenericService">
<when-this-is class="com.arballon.gwt.core.client.AccountingService>
</replace-with>
<replace-with class="com.arballon.gwt.core.client.GenericService">
<when-this-is class="com.arballon.gwt.core.client.FinancialService>
</replace-with>
...
...
But at the moment I was receiving the error:
[ERROR] Errors in 'file:/C:/Users/Daniel/EclipseWorkspace/ADK/src/com/arballon/gwt/core/client/FinancialService.java'
[ERROR] Line 31: Rebind result 'com.arballon.gwt.core.client.GenericService' could not be found
Any thoughts about the issue will be appreciated.
Regards
Daniel

GWT's RPC generation code builds several classes to do its work as you've noted: a *_FieldSerializer for each type that goes over the wire, and a *_Proxy class for the RemoteService async type. That proxy type requires a *_TypeSerializer, which is the root of your problem - for some reason, GWT wires up all of the serialization/deserialization methods in a string->js function map, probably to facilitate fast lookups - but this setup code comes at the cost of lines of code that need to be in the final build. A more optimized approach could have each FieldSerializer have a registration method where it adds its methods to the static map owned by the Proxy - this is plagued, however, but GWT's optimization of attempting to not reference instantiate(), deserialize() and serialize() methods if it doesnt appear they will be called.
Your issue stems from having many types that can be serialized, and from your having attempted to build out RemoteService types that each describe specific units of functionality, but re-use many model types. Admirable goal, especially as it will probably make your server-side code look nicer, but apparently GWT bites you for it.
The solution I attempted to offer you on freenode (as niloc132) was to build a single large RemoteService type, which you named GeneralService, and a matching GeneralServiceAsync, each extending all of the existing rpc service types. My first thought was to use a <replace-with> to tell the generator system that when you want each RemoteService type to replace it with GeneralService, but as Tahir points out, this doesn't make sense - GWT doesn't pass rebind results back into itself to keep doing lookups. Instead, I would suggest that when you want a service async type, do the following:
AccountingServiceAsync service = (AccountingServiceAsync) GWT.create(GeneralService.class)
The rebind result from GeneralService will implement GeneralServiceAsync, which is itself assignable to AccountingServiceAsync. If memory serves, you said that you have static methods/fields that provide these services - change those sites to always create a GeneralServiceAsync instance. As long as you do not invoke GWT.create on any RemoteService subtype but GeneralService, you will limit the number of TypeSerializers to one.
As a side note, the RemoteServiceProxy subtypes are stateless, so ensuring that you create only one instance might make it easier to build consistently, but saves no runtime memory or time, as they are almost certainly compiled out to static methods. The *_TypeSerializer classes do have state however, but there is only one instance of each, so combining all of your RemoteServices might save a very small amount of working memory.

Well, after a pair of roundtrips we finally found a solution to our problem I want to share with in case it could help others.
First I have to mention the help of Colin Alworth, without his support this solution wouldn't be possible at all.
Also I have to mention that I'm not really proud of the final solution but it works for us and for the moment is the best we have.
What we finally did was, as Colin remarks on last post was replacing the GWT.create of each of our service interfaces to create instead the GenericBigService interface.
So our first patch goes like this:
1) Create GenericBigService interface which extends all Service interfaces we have (at the moment 52 interfaces), and also create its Async brother. we done this thru a phytom script.
So our GenericBigInterface looks like this:
package com.arballon.gwt.core.client;
import com.google.gwt.user.client.rpc.RemoteService;
public interface GenericBigService extends RemoteService,
AccountingService,
ActionClassifierService,
AFIPWebService,
AnalyticalService,
AuthorizationService,
BudgetService,
BusinessUnitService,
CatalogPartService,
CategoryService,
ClientDepositService,
.....
.....
{ }
2) We have an Util inner static class in each Service interface to instanciate the Async instance, in there we replace the GWT.create to create the GenericBigInterface.
One of our Service interfaces so looks like this:
public interface FinancialPeriodBalanceCategoryService extends RemoteService {
/**
* Utility class for simplifying access to the instance of async service.
*/
public static class Util {
private static FinancialPeriodBalanceCategoryServiceAsync instance;
public static FinancialPeriodBalanceCategoryServiceAsync getInstance() {
if (instance == null) {
instance = GWT.create(GenericBigService.class);
((ServiceDefTarget)instance).setServiceEntryPoint(GWT.getModuleBaseURL()+"FinancialPeriodBalanceCategoryService");
}
return instance;
}
}
we have to do the serServiceEntyPoint call in order to maintain our web.xml unmodified.
When we first compiles this it compiles ok, but it doesn't work because at runtime the server call throws an Exception:
IncompatibleRemoteServiceException Blocked attempt to access interface GenericBigService
, which is not implemented by FinancialPeriodBalanceCategoryService
Well that was absolutelly right we are calling the service with an interface it doesn't implement, and here is when the ugly part cames in.
We couldn't found a the moment a better solution we can code, that the one we decided to implement that is:
We replace RPC.java with our own copy and we replace the code like this:
in the decodeRequest method we did:
if (type != null) {
/*if (!implementsInterface(type, serviceIntfName)) {
// The service does not implement the requested interface
throw new IncompatibleRemoteServiceException(
"Blocked attempt to access interface '" + serviceIntfName
+ "', which is not implemented by '" + printTypeName(type)
+ "'; this is either misconfiguration or a hack attempt");
}*/
if (!implementsInterface(type, serviceIntfName)) {
if(!serviceIntfName.contains("GenericBigService")){
throw new IncompatibleRemoteServiceException(
"Blocked attempt to access interface '" + serviceIntfName
+ "', which is not implemented by '" + printTypeName(type)
+ "'; this is either misconfiguration or a hack attempt");
}
}
The benefit of doing this was :
1) we went to take an 1 hour and 20 minutes to compite to take only 20 minutes for 6 permutarions.
2) In devMode all starts to run more quickly. Startup remains more or less the same but execution once it starts goes really well.
3) Reduction in the size of compilation was other not minor interesting result, we reduce the left over segment from 6Mb to 1.2Mb, we reduce the whole compilation of JS size in aprox. 50% to 60%.
We are really happy with GWT-RPC and we don't want to leave it, but typeSerializers was really a problem basically because of the size of the JS that results.
With this solution, I know is not very elegant but it works, and it works grate.
Thanks again Colin for your help!
Regards
Daniel

For any GWT-RPC Service, GWt will generate one Proxy, one TypeSerializer. And for each object which possibly can be passed via GWT you will have one FieldSerializer class. And there can be only one FieldSerializer per class. So there is no way you can have two FieldSerializers for one AccountDTO.
Deferred binding rule which you trying to use will not work. For example you have something like this:
MyServiceAsync sync = GWT.create(MyService.class);
Deferred binding rules will change it into:
MyServiceAsync sync = new MyServiceAsync_Proxy();
Your rules will actually do something like this:
MyServiceAsync sync = new MyGenericService() ;//not valid since MyGenericService is an interface
So your solution will not work.
Since you are saying that 60% of you application generated code is RPC related stuff, I suspect you have RPC type explosion problem.
Check if GWT doesn't throws any warnings during compilation, or generate stubs for RPC TypeSerializers, most likely you've some very common interface in service.

If you want to have a nicer solution, why not use a command pattern. that way you only need one GWT service that accepts a Command subtype and returns a Result subtype (you can make it typesafe by using generics).
The nice thing is that you only need to declare one method in one gwt servlet and from there on you can dispatch to any other server side service.
The command pattern can give you a lot of added benefit as well since you have a central point of control to do security checks or allows you to transparently batch requests
You exposure to GWT thus becomes much smaller on the server side.

As far as I understand, the GWT code generation is supposed to supply concrete implementations of an interface. This implementation is then transformed into javascript for specific permutations.
Your sample, on the other hand, is replacing one interface with the other. If you see it from GWT compiler's eyes, perhaps you will see the problem with this configuration.
Suppose, you are the GWT compiler, and you see following line in client side code that you are converting into JavaScript
AccountingServiceAsync accountingServiceAsync = (AccountingServiceAsync) GWT.create(AccountingService.class);
accountingServiceAsync.recordTransaction(transaction,callback);
So you need to find out what should happen, at line 2. Specifically, you need to know where to find implementation of accountingServiceAsync.recordTransaction(). So you go looking into all your configuration to find if there is a rule specifying which implementation class should be used for AccountingService (not Async). But sadly you don't find any. But then you notice that AccountingService is also a RemoteService. So you dive into your configuration again. And, aha, there it is, a rule specifying that you can generate RemoteService implementations with ServiceInterfaceProxyGenerator. You happily hand over the task of providing an implementation of AccountingService to ServiceInterfaceProxyGenerator.
But suppose instead of this happy ending, your configuration tells you that AccountingService can be replaced with GenericService, and you say, "hey cool, bring it on". But just then you find out that GenericService is also an interface. Clearly, you'll be turned off, saying "now, what am I going to with another interface, all I needed was an implementation of AccountingService". At this point you'd want to get even with the programmer by throwing a cryptic error at him.
So, far all this explains why your solution (theoretically) won't work . As far as your actual concern of bloated javascript, I am amazed that this problem even exists given the amount of effort that GWT folks put in optimizing the compiled JavaScript. How did you tested your compiled output for duplication?

How to automatically remove methods in java code

I need to remove some methods in a large java project and I was wondering if there are tools that would help me do so. I would basically specify a signature and a source folder in which all the matching method would be removed.
It no such thing exists then I will write a script.

Today, there are a couple of way to achieve this. For example, you can tell the Eclipse compiler to give you the AST (see my blog for an example). You can navigate the AST to find methods and remove nodes to change the source.
This makes it easy to keep the code valid. Using the Eclipse code formatter, you can clean up the formatting afterwards.
Projects like MoDisCo go one step further by analyzing a whole project and giving you ways to search the whole thing. I just found that the documentation for MoDisCo is not very helpful when you're a beginner.

If you use eclipse or other powerful IDE you may have built in support for doing this.
See I can't delete a method using eclipse refactoring? for a way to achieve what you want.

You're talking about a "source transformation", and Google turns up the existence of BeautyJ for Java:
http://beautyj.berlios.de/
"BeautyJ can convert Java source code to XML and back. BeautyJ introduces the XJava format which stores classes or interfaces and their members together with associated source code and Javadoc comments in a single XML file. BeautyJ opens up a wide range of possibilities for automatic structuring of Java source code either by its Sourclet API or by converting Java source code to XML and allowing any external transformations (for example by applying XSLT)."
No clue whether your case is so simple that a script would be better than going through the trouble of learning how to use such a program, though. Although it may be worthwhile to get it in your toolchain for other purposes as well.

Our DMS Software Reengineering Toolkit with its Java Front End could be used for this.
DMS parses languages (using its front ends, in this case Java), builds ASTs and symbol tables, and provide facilities to manipulate the ASTs based on custom analysis. In this case OP wants to provide a method signature (presumably with a context in which it should be interpreted because otherwise the types used in the signature might not be defined), look that signature up in the symbol table, find the point of declaration (this is in the symbol table) as a AST node, and then apply a rewriting rule that replaces the declaration with an empty declaration.
An analysis he may wish to perform on the found method is whether is it used or not. We have that information in the symbol table, too.

You can use something in Java to batch-modify your Java project. The trick is to use a custom annotation processor. With annotation processor you can modify almost everything.
For an example, this library ( for ecilipse and IDEA ) modifies some String field members to it's /**multiline*/ docs.
Studying the library gives me the power to modify the method body as well.
Today, I want to strip out some unwanted methods in the android support library( used as editable local module ). I can manually remove them, but it would be hard to keep udpate after that.
So, instead of using script or mere-hand to modify the code directly, I decide to write another annotation processor which allow you to remove some class members.
For IDEA, the concept-proving code is very simple:
// the custom annotation
#Target({ElementType.TYPE})
#Retention(RetentionPolicy.SOURCE)
public #interface StripMethods {
boolean strip() default true;
String key() default "";
}
// processing the custom annotation
#Override
public boolean process(final Set<? extends TypeElement> annotations, final RoundEnvironment roundEnv) {
Set<? extends Element> fields = roundEnv.getElementsAnnotatedWith(StripMethods.class);
for (Element field : fields) {
StripMethods annotation = field.getAnnotation(StripMethods.class);
ElementKind KIND = field.getKind();
if(KIND == ElementKind.CLASS) {
// the class declaration
JCTree.JCClassDecl laDcl = (JCTree.JCClassDecl) elementUtils.getTree(field);
// the definition tree list
ArrayList<JCTree> defs = new ArrayList<>(laDcl.defs);
// remove the second member which in this case is a string field
defs.remove(2);
// finally modify the class definition
laDcl.defs = List.from(defs);
}
}
}
#StripMethods
public class Test {
// the first member is the default constructor
static {
}
static final int FieldToRemove = 0;
#Test
public void test() {
int variableToRemove = FieldToRemove;
}
}
the result error caused by the member removal:
Test.java:10: error: cannot find symbol
int variableToRemove = FieldToRemove;
^
symbol: variable FieldToRemove
location: class Test
Still a long way to go. I will publish the code when it's finished.
Done. see https://github.com/KnIfER/Metaline
Exmaple usage, removing NightMode from androidx/appcompat:
#StripMethods(key="Night")
public class AppCompatActivity
...
#StripMethods(key="Night")
public abstract class AppCompatDelegate
...
#StripMethods(key="Night")
class AppCompatDelegateImpl
...

I looked for quick solutions and did not find any, I used different tools to get there:
grep to find all the files containing the main methods I wanted
to remove
find/replace in these files to make those methods private
Eclipse clean-up with just one rule: remove unused
private methods
It did the job with the minor side effect of removing other unused private methods. It is not a big deal in the context of that project since pretty much all the files were previously saved with a save action removing unused private methods.
Thanks all for the input, it may be applicable later on.

Can I always use the Reflection API if the code is going to be obfuscated?

I found that there seem to be 2 general solutions:
don't obfuscate what is referred to through the reflection API [Retroguard, Jobfuscate]
replace Strings in reflection API invocations with the obfuscated name.
Those solutions work only for calls within the same project - client code (in another project) may not use the reflection API to access non-public API methods.
In the case of 2 it also only works when the Reflection API is used with Strings known at compile-time (private methods testing?). In those cases dp4j also offers a solution injecting the reflection code after obfuscation.
Reading Proguard FAQ I wondered if 2 otherwise always worked when it says:
ProGuard automatically handles
constructs like
Class.forName("SomeClass") and
SomeClass.class. The referenced
classes are preserved in the shrinking
phase, and the string arguments are
properly replaced in the obfuscation
phase.
With variable string arguments, it's generally not possible to determine
their possible values.
Q: what does the statement in bold mean? Any examples?

With variable string arguments, it's generally not possible to determine their possible values.
public Class loadIt(String clsName) throws ClassNotFoundException {
return Class.forName(clsName);
}
basically if you pass a non-constant string to Class.forName, there's generally no way for proguard or any obfuscation tool to figure out what class you are talking about, and thus can't automatically adjust the code for you.

The Zelix KlassMaster Java obfuscator can automatically handle all Reflection API calls. It has a function called AutoReflection which uses an "encrypted old name" to "obfuscated name" lookup table.
However, it again can only work for calls within the same obfuscated project.
See http://www.zelix.com/klassmaster/docs/tutorials/autoReflectionTutorial.html.

It means that this:
String className;
if (Math.random() <= 0.5) className = "ca.simpatico.Foo";
else className = "ca.simpatico.Bar";
Class cl = Class.forName(className);
Won't work after obfuscation. ProGuard doesn't do a deep enough dataflow analysis to see that the class name which gets loaded came from those two string literals.
Really, your only plausible option is to decide which classes, interfaces, and methods should be accessible through reflection, and then not obfuscate those. You're effectively defining a strange kind of API to clients - one which will only be accessed reflectively.