Develop Reference

How to scan for a particular annotation of java classes loaded at runtime as a bytecode?

If a java class loaded at runtime as a bytecode (e.g. via ASM library or other mechanism), is it on a classpath of java? I don't see it.
How to scan all annotations of classes loaded this way, if it's not on the java classpath?
I use google reflections and a custom classloader of a 3-rd party library.
Scanning for classes works as follows:
Reflections reflections = new Reflections(ClasspathHelper.forPackage("com.mypackage",
MyCustomClassLoader),
new SubTypesScanner(), new TypeAnnotationsScanner());
Set<Class<?>> myClasses = reflections.getTypesAnnotatedWith(MyAnnotation.class);
MyAnnotation - is marked as #Retention(RetentionPolicy.RUNTIME).
The above (in a class) is loaded dynamically at runtime by JVM.
As can be seen, behind the scenes, Reflections tries to get all URLs to be scanned using 2 classloaders by default static and context.
Reflection.scan()
Update: I have found an answer Can you find all classes in a package using reflection? saying that "If there are classes that get generated, or delivered remotely, you will not be able to discover those classes." But there is no proof, however.
Can please anybody give more details on this and confirm?

Dynamically instantiating classes at run time does not change the classpath that your JVM is using. What happens is that some ClassLoader class fetches bytecode from somewhere; and makes it "available" to you. But that does in no way change the "search order" for loading classes (and that is basically what the classpath is about: it only tells the JVM where and in which order to look for classes to load).
Meaning: any "loading" of a class results in some object of class java.lang.Class.
If you want to query the structure of any loaded class; you "just" need to get to the corresponding Class object. Class provides methods like getAnnotation(). It also provides methods to retrieve other objects that represent the methods and fields of the specific Class; and those objects can be queried for annotations in similar ways.
Update, regarding the updates in the question: there are situations where you don't have access to the file system where classes are coming from. You can load classes, when you know their name, but you have no way looking into the "place" where these classes live. And that basically breaks your ability to use reflection as intended.
There is no need to "prove" that, it is simply a consequence of the fact that Java allows you to load classes when you know there name, but "hides" the exact content of "where" those classes are coming from.

Custom classloader trouble with getResources for names ENDING in slash

I am desperate for help but was unable to find anything on the web about this particular subject (many related ones that leave my particular problem unanswered).
Specifically, I need to be able to download code (jars) from a central and external code repository. This is done by the bootstrap code that needs to add this to the classpath of a class loader to be used thereafter. This is when we enter the subject that has been discussed so many times. I don't like hacks, so I tried the following:
Attempt #1: Create an instance of URLClassLoader configured for this purpose, then invoke the "rest" of the code through it.
Failure: There are 1.5 problems here (one may be the cause of another). One is that URLClassLoader, normally, prefers to load stuff from its parent. Some code has to exist in both, possibly different versions. If the code from the parent is used, it continues using the "outer" class loader for the rest of loading, which is not what we want, even when the initial loading is OK. Secondly, some third party libraries seem to access the system class loader directly, either by design or accidentally (may get it from one of the classes loaded by it).
Attempt #2: Create my subclass of the URLClassLoader that prefers self over the parent. Overrode loadClass, getResource, getResources, getPackage, getPackages... later other methods too to make sure of this.
Failure: Didn't help (enough). That third party code still couldn't load some resources.
Attempt #3 Create another custom subclass of the URLClassLoader and set it as the system class loader using -Djava.system.class.loader=...
Failure: This worked better - went further, but still failed trying to get resources. This time it was different resources, though. I added logging to all the overridden methods to log their calls and resource names. Regular resources were MOSTLY found. Some still weren't, even though they are there (confirmed). But something I don't know about even though I tried hard to learn is about many calls with resource names that end with a slash. Some also have slashes where a dollar sign would normally appear (nested/inner class resources). Some examples that were requested but NOT found:
com/acme/foo/bar/ClassName/
com/acme/foo/bar/ClassName/InnerClassName/
When I run the downloaded code with all content on the initial/boot classpath (and do not use my classloader), everything works fine - thus my class loader breaks things, but I need it to work.
My closest guesses are:
Third party code gets hold of the true system class loader somehow, perhaps via some class that was loaded by it, then uses that. I don't see requests to it and they are bound to fail because it does not have the entire class path.
This business with resource names ending in slashes is the cause by being supported by the true system class loader but not by the URLClassLoader I am subclassing. I can only guess that the expected return URL somehow locates the collection of resources with that name as prefix. That would be tough to match, although possible. Furthermore, it appears that some slashes are in positions where a dollar sign separating the inner class name should be, i.e. in the above example (spaces added for clarity):
com/acme/foo/bar/ClassName / InnerClassName/
com/acme/foo/bar/ClassName $ InnerClassName/
Please note that I cannot rely on hacking the actual system classloader by assuming that it is a subclass of the URLClassLoader and using reflection to call its addURL(URL) method.
Is there a way to make this work? Please help!
UPDATE
I just made an additional attempt. I created a dummy wrapper classloader (extending ClassLoader, not URLClassLoader) that only logs requests, then passes them on to the parent (public methods) or superclass (protected methods). I set this to be the system class loader and manually added the entire "inner" class path to the actual outer one, then tried to run the code. That works correctly, just as it does without the custom system class loader. What was logged also identified that even the system class loader return null for these resources ending in slashes for MOST of them, but not all. I did not check whether these also work in the my real code but guessing they may - as they were not the stumbling block. Somehow the custom system classloader is still being bypassed. How?
UPDATE 2
In my custom system class loaders I have let some classes come from the outer/true system class loader, e.g. those in java.lang. I am now suspecting that I should not have and that the inner "world" must be completely isolated. That would make it problematic, though, to communicate with it and all I would have left is reflection... but not sure whether that would even work - i.e. can there be more than one java.lang.Class and/or java.lang.Object?

Given all constraints this does not appear entirely possible in a rock solid fashion as I wanted it:
a) Third party libraries may always "misbehave" and get hold of lassloaders they are not supposed to use one way or another. I looked at OneJar as suggested by fge but they have the same issue - they only detect a possibility of it.
b) There is no way to completely replace/hide the system class loader.
c) Casting the system class loader to a URLClassLoader may stop working at any moment.

It seems, you didn’t understand the class loader structure. Within an ordinary JVM of the current version, there are at least three class loaders:
The bootstrap loader which is responsible for loading the core classes. Since this involves classes like Class and ClassLoader itself, it can’t be represented by a ClassLoader instance. All classes whose getClassLoader() returns null were loaded by the bootstrap loader
The extension loader. It is responsible for loading classes within the ext/ directory of the JRE. Afaik, it may vanish in future versions. Its parent loader is the bootstrap loader
The application loader. This is the one which will be returned by ClassLoader.getSystemClassLoader() and which will be used if no other parent was specified. In the current configurations, it’s parent is the extension loader, but maybe it will have the bootstrap loader as its direct parent in future versions
The conclusion is, if you want to reload your application’s classes without the delegation to the parent loader destroying your effort, you don’t need to manipulate the class loader’s implementation. You just have to specify the right parent. It’s as simple as
URLClassLoader cl=new URLClassLoader(urls, ClassLoader.getSystemClassLoader().getParent());
That way, the new class loader’s parent will be the original application class loader’s parent, thus the already loaded application classes are not in the scope of the new loader while everything else works as usual.
Regarding the resources ending with a slash, they are rather uncommon. They may get resolved when they actually refer to a directory but that depends on the protocol of the URL and the actual handler for that protocol. E.g. it might work for file: URLs but usually doesn’t for jar: URLs unless the jar file contains pseudo-entries for directories. I’ve also seen it working for ftp: URLs.
Another thing to understand is that if one class directly refers to another class, its original defining class loader will be queried, not necessarily the application class loader. E.g. when the class java.lang.String contains a reference to java.lang.Object (it does), this reference will be directly resolved using the bootstrap loader as this is the defining loader of java.lang.String.
This implies that if you manipulate the parent lookup of a loader to not follow the standard parent delegation you are risking to resolve names to different runtime classes as the resolving of the same names when being referenced by classes loaded by the parent loader. You avoid such problems by following the standard procedure as in the solution above. The JRE classes will never contain references to your application classes and the new loader not having the original application loader as its parent will never interfere with the classes loaded by the original application loader.

Force all classes in a certain directory to load using a custom class loader

I'm making a service that watches a certain directory and when it detects changes in code in that directory, it runs all the tests in the directory. Purpose is that it runs the tests on the changes in code in that directory. So if I add a new method in a class and then a new test, it should use the new class and new test and include it in its testrun.
How can I do this dynamically? I can dynamically load all the test classes, by scanning the directory and collecting the tests and using my own custom dynamic loader ( as done by Reload used classes at runtime Java ). But the classes remain the ones before the changes. Before the service started watching, i.e before runtime.
How can I force those classes used in the tests to also be dynamically loaded?

It seems to me that you want to reload a class once it was changed at runtime. You should have a look at this article. The problem with this approach is, that the reloaded class will not be the same anymore, because you can't use the same ClassLoader twice and using another classloader will result in the classes not being the same anymore and ultimately causing ClassCastExceptions.
This can be avoided by programming against interfaces, but you might have to change your class design.
In case you want to reload all referenced classes as well, you probably need to override ClassLoader.findClass(String name). I cannot test this pretty specific scenario now, but I assume that this method is called for all referenced classes. Since the default implementation will just throw a ClassNotFoundException, the parent class loader will be asked to find the class and it will give you the old version.
You could try this:
protected Class<?> findClass(String name) throws ClassNotFoundException {
return loadClass(name);
}

Reloading jar files contents dynamically

I have one jar file in my application's class path. At run time, I add new classes to the jar file and sometimes also modify the fields/methods of the existing classes. Currently I am using URLClassLoader to load the classes dynamically. The new classes added dynamically are loaded correctly and I am able to use them at runtime. But it fails to reload the existing classes that are modified at runtime. I read many articles which states we need to explicitly handle reloading because class once loaded will not be reloaded until all the references to the class are destroyed. Also I tried out sample code that I found but none of them worked.
Can anyone suggest me a proper approach for reloading ? Any sample code for the same will be highly appreciated.

Normally to reload a class you need to unload the entire class loader. i.e. remove all references to all classes loaded for that class loader.
Another option is to use instrumentation to change the byte code of an existing class. This usually comes with limitations and changing fields is something you cannot do. i.e. the objects of that type would have to be translated somehow.
What I normally do is have services which are very quick to start/restart. This way to you easily restart a process which needs updated code ideally by pressing the Run in my IDE. This minimises deployment time as well.

In principle, a class that has already been loaded cannot be reloaded with the same classloader.
For a new load, it is necessary to create a new classloader and thus load the class.
Using URLClassLoader has one problem and that is that the jar file remains open.
If you have multiple classes loaded from one jar file by different instances of URLClassLoader and you change the jar file at runtime, you will usually get this error: java.util.zip.ZipException: ZipFile invalid LOC header (bad signature). The error may be different.
In order for the above errors not to occur, it is necessary to use the close method on all URLClassLoaders using the given jar file. But this is a solution that actually leads to a restart of the entire application.
A better solution is to modify the URLClassLoader so that the contents of the jar file are loaded into the RAM cache. This no longer affects other URLClassloaders that read data from the same jar file. The jar file can then be freely changed while the application is running. For example, you can use this modification of URLClassLoader for this purpose: in-memory URLClassLoader

Can I replace a class file in compiled Java project?

This may, and should be, a question that has been asked and answered many times over, but I just cant find the answer.
If I have a compiled application running on a server, can I compile the application on my local machine, and swap a class that I compiled on my local machine, with the one on the server?
In other words, can I replace a file that has been compiled and is on the server side, with a file almost identical that has been compiled, and is located, on my local machine?
Since the version on the server side has some hard-coded connections in other files, and I dont know all of the places, I would prefer to only swap the one file that I need, instead of doing a recompile for the application as a whole.

The answer to your question is yes, you can replace a class file, but it is somewhat complicated in that you have to be sure that no other dependencies have changed.
For example, if the class you are compiling involved changing method signatures of methods that are used in other classes, you will need to replace those as well. As long as the method signatures of public, protected, or default methods aren't changed, you should be okay.
As a side-note, if this is something you do often, you'll quickly realize why objects are often passed into methods instead of individual parameters.
public MyObject getObject(MyObject2 mySecondObject)
vs
public MyObject getObject(int a, int b, int c)
When you need to add a new property to an object passed into a method, the method signatures don't change, but when you add or remove a parameter on the method signature itself, it creates a chain reaction on all dependencies, requiring that you compile and replace those class files as well.
As a final point to highlight, it's worth noting that changes you make to private methods or private variables, or even the definitions of a method, have no bearing or impact on other class files. The only thing that matters is that you uphold the contract that your methods have with other classes in that the inputs and outputs always take and return the same data types.
This highlights the importance of encapsulation of instance variables and how those dependencies are hidden from other classes.

Yes, you can do it. Moreover due to hot spot the class will be even reloaded at runtime, so you do not have to restart the server. This is very useful during development cycle.
But be careful replacing one separate class. If the class has references to other classes changed in your environment the replacement will fail on server.

I assume you're talking about an 'offline' substitution (I mean you can restart the server).
In this case the following solution can be viable:
Java uses classpath variable. There you define series of jars/folders where your process will look for files.
So you can do the following:
Create your new class (it should be with the same name and reside in the same package) and compile.
Create on server the folder where you're planning to store your updated class.
Update the classpath variable (probably in the startup script of your application) so that this folder will appear before the rest of your classes.
Restart your server. Now your new class will be resolved before the old one and loaded.
Caution:
This works for 'standalone' applications and in general for those that don't fiddle with custom class loaders (for example this is the case in application servers).
More complex solution can be based on define your class loader that will take a look on such a folder (as I've described) and try to load resources from there.
Hope this helps.