I am trying to access a java package loaded into memory and dump it to a file. Here is how the security works: there is an exe packed with Themida that contains the java main class code to be loaded. At runtime the Themida exe loads the clean main class java code into memory. The software is structured with the loader being contained within the exe, but several external libraries can access the packages contained within the exe. So, exe contains com.mysoft.mainloader. But the clean jar library Mylib.jar can call functions within com.mysoft.mainloader. How to I dump com.mysoft.mainloader to a jar file? Can I modify Mylib.jar to dump it as it has access to the package once it is loaded as well?
There is no supported Java SE mechanism to read / retrieve a ".class" that has been loaded by a classloader. So your options would be:
Modify the custom classloader you are using to capture the ".class" before (or after) the classloader calls defineClass.
Burrow into the JVM data structures to try and figure out whether the entire ".class" stream is captured somewhere and then retrieve it.
Modify the JVM ...
Any of these could be feasible. All will be relatively difficult.
It is possible to get loaded classes in runtime using Dynamic Attach and Instrumentation API.
The idea is to inject a Java Agent into the running application.
The agent gets an array of all loaded classes with Instrumentation.getAllLoadedClasses method, then gets their bytecode using Instrumentation.retransformClasses.
The working implementation can be found in the class-file-extractor project.
Usage:
java -jar extractor.jar <pid> mainloader.jar com.mysoft.mainloader
where
<pid> is the process ID of the target JVM application;
mainloader.jar is the output file name;
com.mysoft.mainloader is the name prefix of the classes to extract.
This question goes a little bit further than my previous question:
Obtaining a list of all Java classes used from all JVM's?
Now I need to know the physical location from where those classes are loaded from. I have checked out the jcmd help for other commands but it wasn't useful for me. I also can't find it in jvisualvm, but the information is also not there. Anyone can help me with this?
EDIT:
This is my situation: My company has got different individual java projects (jars) for which we can control whether it starts or stops. We can control this in our own custom build webinterface. Each of such a process gets a process ID (PID) when started, and then runs on the background.
My need: I need a list of all loaded classes by each running PID java process. I already have jcmd <pid> GC.class_histogram, but this only contains a list of which classes are loaded. I also want the information where the classes are actually loaded from (which jar, location on file system).
The classes are loaded from java.lang.ClassLoader's loadClass(String name) method which in turn calls the findClass(String name) method.Usually,the custom ClassLoader overrides the findClass method to retrieve the definition of classes using a specific protocol and location.It may be that classes are loaded from database or from network location,the location of which may be dynamically generated.So you can never know location of all Java classes. The best example is AppletClassLoader which loads classes from network stream or from a remote location.
Bootstrap ClassLoader - core library package such as rt.jar present in JRE lib folder
Extension ClassLoader - jar files present in ext folder or as specified in the environment variable for ext
System ClassLoader - Application's classpath or as specified in environment variable for classpath or through JVM's startup argument parameter for -cp or -classpath
CustomClassLader -accordingly to the classLoader's class loading policy (mostly defined in findClass() method)
We have a java based (jersey+grizzly) REST Server which exposes calls like
foo.com/{game}/rules
foo.com/{game}/players
foo.com/{game}/matches
There can be arbitrary number of games
Each game has different implementations for rules, players, matches
For historical reasons, we would want separate jars for each game implementation
So there is REST Server
as and when there is a call like foo.com/tennis/rules
we want the REST Server to dynamically load 'tennis' jar. The jar does its operation. Then jar should be unloaded
If the next call was for foo.com/football/players
we want the REST Server to dynamically load 'football' jar. The jar does its operation. Then jar should be unloaded
Is there a technique to do this ?
Apparently there is a very old question around [this]: java: is there a framework that allows dynamically loading and unloading of jars (but not osgi)?
I don't know how it works on Java 8, but unloading a class in Java 7 requires unloading not only the class, but its loader, along with all references from other objects that this class might have.
Once all of them were unloaded the System.gc will be called. If other classes are still holding references then the gc won't do its job.
OSGI (as suggested by #Joop Eggen) is a viable option. JRebel, is not.
proxy-object proxy-object library
Load java jar files dynamically in isolated class loader to avoid dependency conflicts and enable modular updates. All jar files in the [main jar folder]/lib/myLib/2.0/*.jar will be loaded.
Code Examples
Create Object from a JAR file located in the myLib/2.0 folder:
File libDir = new File("myLib/2.0");
ProxyCallerInterface caller = ObjectBuilder.builder()
.setClassName("net.proxy.lib.test.LibClass")
.setArtifact(DirArtifact.builder()
.withClazz(ObjectBuilderTest.class)
.withVersionInfo(newVersionInfo(libDir))
.build())
.build();
String version = caller.call("getLibVersion").asString();
I have a a library that is bundled as an executable jar file and added to weblogic / tomcat classpath, how can I execute a main method from the jar file when the server is starting and loading the classes from the jar file.
what I want to is to have some initialization code to be executed first thing when the jar file is loaded and server is starting without any user intervention.
Note: I know I can bundle my jar in a war file, but I have some aspectj code in my library that I want to weave all running applications in the jvm, when I bundle my jar in war file, the aspectj code will only weave into the classes in the war file so I added my library jar file in the classpath.
Thanks in advance.
Add a class inside your JAR with the following code:
public class TomcatStartupListener implements org.apache.catalina.LifecycleListener {
public void lifecycleEvent(org.apache.catalina.LifecycleEvent event) {
if (event.getType().equals("after_start")) {
// call your main method here
}
}
}
Note: In order to compile this, you need to add <tomcat-dir>/lib/catalina.jar to your classpath. Otherwise when compiling it won't be able to find the necessary interfaces (org.apache.catalina.LifecycleListener and org.apache.catalina.LifecycleEvent). Once you're done with the compiling, put the JAR as usual under <tomcat-dir>/lib.
Now open <tomcat-dir>/conf/server.xml and add the following under the <Server> section:
<Listener className="com.yourpackage.TomcatStartupListener" />
Now whenever your Tomcat server starts, this TomcatStartupListener class inside your JAR will be called, and you can invoke your main method. There are a whole lot of other event types too! You can use any of these event types:
before_init
after_init
before_start
configure_start
start
after_start
before_stop
stop
configure_stop
after_stop
before_destroy
after_destroy
This approach is necessary because of the way the classloaders work in Tomcat (or even most JVMs). Here are the important points from that link:
There are three aspects of a class loader behavior
Lazy Loading
Class Caching
Separate Namespaces
The JVM will get very heavy if all classes inside all JARs get loaded indiscriminately. So the classes inside shared JARs are loaded only on-demand. The only way for you to invoke the main method is to add the above lifecycle listener.
Perhaps the simplest thing to do is to deploy a trivial servlet in a .war file that references your .jar file. The servlet can be configured to start up upon deployment/container start, and then it can invoke the class containing your main() method.
As application servers / servlet containers typically have a lot of different classloaders, you'll most likely need a different strategy for weaving aspects into your code than in standalone applications.
I would recommend to add the aspects to every war file deployed at build time. This might be following a common technique - as opposed to a server specific one.
Further, I'm not sure it can actually be done (properly & supported) on a server. Typically a server is built to separate all webapps from each other. You might get it to work, but it might break on the next update of the server.
It might be easier to suggest an alternative technique if you'd state the problem that you want to solve with your proposed approach.
Edit after your comment: Consider the standard web application lifecycle: You can execute some code, e.g. in a servlet, upon it being deployed. If you insist on your code being contained in main, you can call this method from your webapp's initialization code.
You need to register a Java Agent. See this link: java.lang.instrument.
java.lang.instrument provides services that allow Java programming language agents to instrument programs running on the JVM.
This is the right way to do this.
I have read a lot about the Java class loading process lately. Often I came across texts that claimed that it is not possible to add classes to the classpath during runtime and load them without class loader hackery (URLClassLoaders etc.)
As far as I know classes are loaded dynamically. That means their bytecode representation is only loaded and transformed to a java.lang.Class object when needed.
So shouldn't it be possible to add a JAR or *.class file to the classpath after the JVM started and load those classes, provided they haven't been loaded yet? (To be clear: In this case the classpath is simply folder on the filesystem. "Adding a JAR or *.class file" simply means dropping them in this folder.)
And if not, does that mean that the classpath is searched on JVM startup and all fully qualified names of the found classes are cached in an internal "list"?
It would be nice of you if you could point me to some sources in your answers. Preferably the offical SUN documentation: Sun JVM Spec. I have read the spec but could not find anything about the classpath and if it's finalized on JVM startup.
P.s.
This is a theoretical question. I just want to know if it is possible. There is nothing practical I want to achieve. There is just my thirst for knowledge :)
There are two concepts here that are being intermixed: The classpath and the class files in the classpath.
If you point the classpath to a directory, you will generally have no issue adding a file to the directory and having it picked up as part of the classpath. Due to the potential size of all classes in the classpath it isn't really feasible for a modern JVM to load them all at startup. However this is of limited value as it will not include Jar files.
However, changing the classpath itself (which directories, jars, etc. are searched) on a running JVM will depend very much on the implementation. As far as I know, on standard Sun JVMs there is no documented (as in guaranteed to work) method of accomplishing this.
In general, if this is something you need to do (have a dynamic classpath that changes at runtime) then you want to be implementing a ClassLoader, if for no other reason than to be able to throw it away and make a new one that doesn't reference those classes anymore if they need to be unloaded.
However, for a small amount of dynamic loading there are better ways. In Java 1.6 you can specify all the jar files in a directory (*.jar) so you can tell users to put additional libraries in a specified location (although they have to be there at startup).
You also have the option of including a jar file or other location in the classpath even though you don't need it, as a placeholder for someone to put an optional jar or resource file there (such as a log configuration file).
But if you need serious dynamic class loading and especially unloading while the application is running, a Classloader implementation is required.
Since nobody could give my a definite answer nor a link to a corresponding part of the documentation I provide a answer myself. Nevertheless I would like to thank everybody that tried to answer the question.
Short answer:
The classpath is not final upon JVM start.
You actually can put classes in the classpath after the JVM started and they will be loaded.
Long answer:
To answer this question I went after user unknowns suggestion and wrote a little test program.
The basic idea is to have two classes. One is the main class which instantiates the second class. On startup the second class is not on the classpath. After the cli program started it'll prompt you to press enter. Before you press enter you copy the second class on the classpath. After you press enter the second class is instantiated. If the classpath would be final on JVM startup this would throw an Exception. But it doesn't. So I assume the classpath is not final on JVM startup.
Here are the source codes:
JVMTest.java
package jvmtest;
import java.io.Console;
import jvmtest.MyClass;
public class JVMTest {
public static void main(String[] args) {
System.out.println("JVMTest started ...");
Console c = System.console();
String enter = c.readLine("Press Enter to proceed");
MyClass myClass = new MyClass();
System.out.println("Bye Bye");
}
}
MyClass.java
package jvmtest;
public class MyClass {
public MyClass() {
System.out.println("MyClass v2");
}
}
The folder structure looks like this:
jvmtest/
JVMTest.class
MyClass.class
I started the cli program with this command:
> java -cp /tmp/ jvmtest.JVMTest
As you can see I had my jvmtest folder in /tmp/jvmtest. You obviously have to change this according to where you put the classes.
So here are the steps I performed:
Make sure only JVMTest.class is in jvmtest.
Start the program with the command from above.
Just to be sure press enter. You should see an Exception telling you that no class could be found.
Now start the program again.
After the program started and you are prompted to press enter copy the MyClass file into the jvmtest folder.
Press enter. You should see "MyClass v1".
Additional notes:
This also worked when I packed the MyClass class in a jar and run the test above.
I ran this on my Macbook Pro running Mac OS X 10.6.3
> Java -version
results in:
java version "1.6.0_20"
Java(TM) SE Runtime Environment (build 1.6.0_20-b02-279-10M3065)
Java HotSpot(TM) 64-Bit Server VM (build 16.3-b01-279, mixed mode)
#Jen I don't think your experiment can prove your theory, because it is more about object instantiation: your printline happens when an object of this class is instantiated, but not necessarily telling that JVM knows your code, the class, just when it is instantiating.
My opinion is that all Java classes are loaded when JVM is up, and it is possible to plug in more classes into JVM while it is running: this technique is called: Hot deployment.
Bottom line: it is possible to add entries to the system classpath at runtime, and is shown how. This, however, has irreversible side-effects and relies on Sun JVM implementation details.
Class path is final, in the most literal sense:
The system class loader (the one that loads from the main class path) is sun.misc.Launcher$AppClassLoader in rt.jar.
rt.jar:sun/misc/Launcher.class (sources are generated with Java Decompiler):
public class Launcher
{
<...>
static class AppClassLoader
extends URLClassLoader
{
final URLClassPath ucp = SharedSecrets.getJavaNetAccess().getURLClassPath(this);
<...>
rt.jar:sun/misc/URLClassLoader.class:
protected Class<?> findClass(final String paramString)
throws ClassNotFoundException
{
<...>
String str = paramString.replace('.', '/').concat(".class");
Resource localResource = URLClassLoader.this.ucp.getResource(str, false);
<...>
But, even if the field is final, this doesn't mean we can't mutate the object itself if we somehow get access to it. The field is without an access modifier - which means, we can access it if only we make the call from the same package. (the following is IPython with JPype; the commands are readable enough to easily derive their Java counterparts)
#jpype doesn't automatically add top-level packages except `java' and `javax'
In [28]: jpype.sun=jpype._jpackage.JPackage("sun")
In [32]: jpype.sun.misc.Launcher
Out[32]: jpype._jclass.sun.misc.Launcher
In [35]: jpype.sun.misc.Launcher.getLauncher().getClassLoader()
Out[35]: <jpype._jclass.sun.misc.Launcher$AppClassLoader at 0x19e23b0>
In [36]: acl=_
In [37]: acl.ucp
Out[37]: <jpype._jclass.sun.misc.URLClassPath at 0x19e2c90>
In [48]: [u.toString() for u in acl.ucp.getURLs()]
Out[48]: [u'file:/C:/Documents%20and%20Settings/User/']
Now, URLClassPath has a public addURL method. Let's try it out and see what happens:
#normally, this is done with Launcher.getFileURL but we can't call it directly
#public static URLClassPath.pathToURLs also does the same, but it returns an array
In [72]: jpype.sun.net.www.ParseUtil.fileToEncodedURL(
jpype.java.io.File(r"c:\Ivan\downloads\dom4j-2.0.0-RC1.jar")
.getCanonicalFile())
Out[72]: <jpype._jclass.java.net.URL at 0x1a04b50>
In [73]: _.toString()
Out[73]: u'file:/C:/Ivan/downloads/dom4j-2.0.0-RC1.jar'
In [74]: acl.ucp.addURL(_72)
In [75]: [u.toString() for u in acl.ucp.getURLs()]
Out[75]:
[u'file:/C:/Documents%20and%20Settings/User/',
u'file:/C:/Ivan/downloads/dom4j-2.0.0-RC1.jar']
Now, let's try to load some class from the .jar:
In [78]: jpype.org=jpype._jpackage.JPackage("org")
In [79]: jpype.org.dom4j.Entity
Out[79]: jpype._jclass.org.dom4j.Entity
Success!
This will probably fail from a sandbox or such where there are custom class loaders or security settings in the way (AppClassLoader.loadClass does security checks before calling super).
Further code inspection shows that addURL also disables the URLClassPath's lookup cache (implemented in a few native methods), and this is irreversible. Initially, the lookupCacheEnabled flag is set to the value of the sun.cds.enableSharedLookupCache system property.
The interface provides no way to edit the entries. URLs are added to URLClassPath's private ArrayList path and Stack urls. urls is accessible, but it turns out, it only holds entries temporarily, before it's attempted to load from it, at which point the information moves to HashMap lmap and ArrayList loaders. getURLs() returns a copy of path. So, it's theoretically possible to edit it by hacking the accessible fields, but it's nowhere near reliable and won't affect getURLs result.
I can only comment from what I remember of my own experience of hacking a non-sun JVM ten years ago, but it did scan the entire classpath on startup, as an efficiency measure. The names of all classes found were added to an internal hashtable along with their locations (directory or zip/jar file). However, that was ten years ago and I can't help but wonder whether this would still be a reasonable thing to do in most settings considering how disk and memory architectures have evolved.
I believe that the classpath is taken to be static and the result of changing the files is undefined.
If you really want to be able to add and remove classes at runtime, consider doing so in your own classloader. This is what web containers do.
So shouldn't it be possible to add a
JAR or *.class file to the classpath
after the JVM started ...
You add jars and directories to the classpath, not classes. The classes are in either the directory, or the jar.
And if not, does that mean that the
classpath is searched on JVM startup
and all fully qualified names of the
found classes are cached in an
internal "list"?
That could be easily tested: Set the classpath, start your program, move a new class into the CP, call 'Class.forName ("NewClass") from your program. Does it find the new class?
I think you could read documentation of TomCat server. This server implements java classpapth by its own. So, when this server is started, you can deploy new webApp just drag and drop jar in appropriatefolder in hot, without restart server, and it will upload your app.