I have a C binary that calls out to Java via JNI. I set CLASSPATH to somedir/* to pick up all the jars in somedir.
When I run the binary, a required class definition cannot be found. When I run
java that.class's.name
from the same command line, the class is successfully found. If I explicitly add all the jars in somedir/ to the classpath, everything works great, but that leads to a very long classpath which I'd like to avoid.
Does a JVM executed via JNI honour wildcard expansion of the classpath? Can it be made to do so?
I figured out the answer by reading the hotspot source code.
Only paths passed via either CLASSPATH or -cp / -classpath are subject to wildcard expansion. These are then passed as a system property to the running JVM via -Djava.class.path.
You tell a JNI-invoked JVM about a classpath via a JVMOptions structure, which may include -Djava.class.path but -classpath will not necessarily be honoured (and in practice, isn't by the hotspot implementation). Since java.class.path is directly passed to the JVM as a system property, it doesn't get wildcard expanded and therefore wildcards won't work.
No. No, it cannot. Using JNI doesn't help.
The way you would do this is by implementing your own class loader (in Java), but that class loader would have to be in the wildcard-free CLASSPATH.
You could, of course, set the CLASSPATH to its expanded form before invoking the JVM. That would work and could be done via a shell script (no JNI needed).
Related
Our tool (http://plse.cs.washington.edu/daikon) calculates program invariants by inserting instrumentation into the Java byte codes for a program. The user code is instrumented during runtime via the normal ClassFileTransformer::transform method.
It is also necessary to track value flows through JDK methods. Thus, we need to instrument the Java runtime as well. We cannot use transform, because hundreds of runtime methods are loaded prior to the first time we get control at transform.
Prior to Java 9, we handled this in an offline step that reads rt.jar, instruments its methods, and writes out a modified version as dcomp-rt.jar. The user placed dcomp-rt.jar on the bootclasspath to ensure our modified Java runtime methods were loaded instead of the standard ones. The user program invocation would look something like:
java -cp .:.../daikon/daikon.jar \
-Xbootclasspath/p:.../daikon/java/dcomp_rt.jar:.:.../daikon/daikon.jar \
-javaagent:.../daikon/java/dcomp_premain.jar={various dcomp arguments} \
{user program} {user program arguments}
Now to Java 9+. Our first approach was to read in and instrument the class files within the Java runtime jmod files (via the new jrt:/ file system) and create a dcomp_rt.jar as before. The problem we are experiencing is that we cannot get the system to use the contents of this jar instead of jrt:/java.base (for example). We tried various --module-path and -Xbootclasspath (only /a is available now, might be part of problem) options to no avail. Still hoping there might be a way to do this?
If not, I'm guessing we need to make modified versions of each of the interesting runtime jmods and then use a --patch-module argument for each of them. Would this ensure our modified code is loaded instead of the standard runtime?
Any thoughts/suggestions?
Well it looks like --patch-module does the trick. I made the same dcomp_rt.jar but with only classes from java.base.jmod. Then used:
--patch-module java.base={full path}/dcomp_rt.jar
Running java with -verbose:class showed all base classes being loaded from my jar.
Is this the best way to accomplish my goal?
I have to run a java task, with a very large number of classpath (1000, totaling 150k characters if concatenated).
The problem is that java returns an error when I try to execute this class:
/jdk/JAVA8/bin/java: Argument list too long
The error code is 7
I've tried to put the classpaths using "export CLASSPATH=CLASSPATH:....." and so I shouldn't specify them through the -cp java parameter, but it returned the same error.
I'm pretty sure that the problem revolves round a classpath's limit, because if I delete some of the classpath, the error disappears (but then I will have logical errors in the execution, because I need all the classpaths)
You could use classpath wildcards. Especially if many of your jars/class files are in the same directory, this would help a lot.
It could be environment variable size limit or command-line size limit as well rather than javac classpath arg limit.
javac takes arguments from file input as well. You can add all your arguments to this file and pass this file argument to command. Refer this for more.
You didn’t hit a java-specific limitation, but a system dependent limit. This is best illustrated by the fact, that the attempt to set the CLASSPATH variable fails as well, but setting an environment variable via export name=value in the shell isn’t related to Java.
As said by others, you could try to use wildcards for jar files within the same directory, but you have to care that Java does the expansion rather than the shell, as in the latter case, it would again yield a too long command line. So you have to escape the * character to ensure it will not be processed by the shell.
javac supports reading the command line arguments from an external file specified via #filename, but unfortunately, the java launcher doesn’t support this option.
An alternative would be to create symbolic links pointing to the jar files, having shorter paths and specifying these. You could even combine the approaches by creating one directory full of symbolic links and specifying that/directory/* as class path.
But there seems to be a logical error in the requirement. In a comment, you are mentioning “code analysis” and an analyzing tool should not require having the code to analyze in its own application class path. You can access class files via ordinary I/O, if you want to read and parse them. In case you want to load them, e.g. for using the builtin Reflection, you can create new ClassLoader instances pointing to the locations. So the tool doesn’t depend on the application class path and could read the locations from a configuration file, for example.
Using distinct class loaders has the additional advantage that you can close them when you’re done.
JVM does not limit classpath length. However, there is a hard OS limit on command line length and environment variables size.
On Linux check getconf ARG_MAX to see the limit.
On older kernel versions it is only 128KB. On newer kernels it is somewhere around 2MB.
If you want to set really long classpaths, you may need a JAR-Manifest trick. See this question for details.
I know why CLASSPATH has to be provided for java command (and many good answers are given in SO). But I am always confused, why by default it is not deduced from the .class file path given as an argument.
e.g. (It is a silly example...)
Assume I am in /home and there is ./class/SomeClass.class file in it.
/home
|_class
|_SomeClass.class
If I run java -cp /home/class/ SomeClass the program (SomeClass) runs successfully.
But without the -cp option (but with complete path) running
java ./class/SomeClass
throws an exception (java.lang.NoClassDefFoundError) saying Could not find the main class....
And this means, (unless I am in the same directory) the JVM couldn't know where to find SomeClass.class file even if I explicitly stated to look it inside ./class/ directory (by prefixing it before SomeClass).
In such cases I feel the default behavior should be to consider the CLASSPATH the same directory where the .class file exists - by calculating from the path.
Is there any good reason to always use -cp?
I have read a lot about the Java class loading process lately. Often I came across texts that claimed that it is not possible to add classes to the classpath during runtime and load them without class loader hackery (URLClassLoaders etc.)
As far as I know classes are loaded dynamically. That means their bytecode representation is only loaded and transformed to a java.lang.Class object when needed.
So shouldn't it be possible to add a JAR or *.class file to the classpath after the JVM started and load those classes, provided they haven't been loaded yet? (To be clear: In this case the classpath is simply folder on the filesystem. "Adding a JAR or *.class file" simply means dropping them in this folder.)
And if not, does that mean that the classpath is searched on JVM startup and all fully qualified names of the found classes are cached in an internal "list"?
It would be nice of you if you could point me to some sources in your answers. Preferably the offical SUN documentation: Sun JVM Spec. I have read the spec but could not find anything about the classpath and if it's finalized on JVM startup.
P.s.
This is a theoretical question. I just want to know if it is possible. There is nothing practical I want to achieve. There is just my thirst for knowledge :)
There are two concepts here that are being intermixed: The classpath and the class files in the classpath.
If you point the classpath to a directory, you will generally have no issue adding a file to the directory and having it picked up as part of the classpath. Due to the potential size of all classes in the classpath it isn't really feasible for a modern JVM to load them all at startup. However this is of limited value as it will not include Jar files.
However, changing the classpath itself (which directories, jars, etc. are searched) on a running JVM will depend very much on the implementation. As far as I know, on standard Sun JVMs there is no documented (as in guaranteed to work) method of accomplishing this.
In general, if this is something you need to do (have a dynamic classpath that changes at runtime) then you want to be implementing a ClassLoader, if for no other reason than to be able to throw it away and make a new one that doesn't reference those classes anymore if they need to be unloaded.
However, for a small amount of dynamic loading there are better ways. In Java 1.6 you can specify all the jar files in a directory (*.jar) so you can tell users to put additional libraries in a specified location (although they have to be there at startup).
You also have the option of including a jar file or other location in the classpath even though you don't need it, as a placeholder for someone to put an optional jar or resource file there (such as a log configuration file).
But if you need serious dynamic class loading and especially unloading while the application is running, a Classloader implementation is required.
Since nobody could give my a definite answer nor a link to a corresponding part of the documentation I provide a answer myself. Nevertheless I would like to thank everybody that tried to answer the question.
Short answer:
The classpath is not final upon JVM start.
You actually can put classes in the classpath after the JVM started and they will be loaded.
Long answer:
To answer this question I went after user unknowns suggestion and wrote a little test program.
The basic idea is to have two classes. One is the main class which instantiates the second class. On startup the second class is not on the classpath. After the cli program started it'll prompt you to press enter. Before you press enter you copy the second class on the classpath. After you press enter the second class is instantiated. If the classpath would be final on JVM startup this would throw an Exception. But it doesn't. So I assume the classpath is not final on JVM startup.
Here are the source codes:
JVMTest.java
package jvmtest;
import java.io.Console;
import jvmtest.MyClass;
public class JVMTest {
public static void main(String[] args) {
System.out.println("JVMTest started ...");
Console c = System.console();
String enter = c.readLine("Press Enter to proceed");
MyClass myClass = new MyClass();
System.out.println("Bye Bye");
}
}
MyClass.java
package jvmtest;
public class MyClass {
public MyClass() {
System.out.println("MyClass v2");
}
}
The folder structure looks like this:
jvmtest/
JVMTest.class
MyClass.class
I started the cli program with this command:
> java -cp /tmp/ jvmtest.JVMTest
As you can see I had my jvmtest folder in /tmp/jvmtest. You obviously have to change this according to where you put the classes.
So here are the steps I performed:
Make sure only JVMTest.class is in jvmtest.
Start the program with the command from above.
Just to be sure press enter. You should see an Exception telling you that no class could be found.
Now start the program again.
After the program started and you are prompted to press enter copy the MyClass file into the jvmtest folder.
Press enter. You should see "MyClass v1".
Additional notes:
This also worked when I packed the MyClass class in a jar and run the test above.
I ran this on my Macbook Pro running Mac OS X 10.6.3
> Java -version
results in:
java version "1.6.0_20"
Java(TM) SE Runtime Environment (build 1.6.0_20-b02-279-10M3065)
Java HotSpot(TM) 64-Bit Server VM (build 16.3-b01-279, mixed mode)
#Jen I don't think your experiment can prove your theory, because it is more about object instantiation: your printline happens when an object of this class is instantiated, but not necessarily telling that JVM knows your code, the class, just when it is instantiating.
My opinion is that all Java classes are loaded when JVM is up, and it is possible to plug in more classes into JVM while it is running: this technique is called: Hot deployment.
Bottom line: it is possible to add entries to the system classpath at runtime, and is shown how. This, however, has irreversible side-effects and relies on Sun JVM implementation details.
Class path is final, in the most literal sense:
The system class loader (the one that loads from the main class path) is sun.misc.Launcher$AppClassLoader in rt.jar.
rt.jar:sun/misc/Launcher.class (sources are generated with Java Decompiler):
public class Launcher
{
<...>
static class AppClassLoader
extends URLClassLoader
{
final URLClassPath ucp = SharedSecrets.getJavaNetAccess().getURLClassPath(this);
<...>
rt.jar:sun/misc/URLClassLoader.class:
protected Class<?> findClass(final String paramString)
throws ClassNotFoundException
{
<...>
String str = paramString.replace('.', '/').concat(".class");
Resource localResource = URLClassLoader.this.ucp.getResource(str, false);
<...>
But, even if the field is final, this doesn't mean we can't mutate the object itself if we somehow get access to it. The field is without an access modifier - which means, we can access it if only we make the call from the same package. (the following is IPython with JPype; the commands are readable enough to easily derive their Java counterparts)
#jpype doesn't automatically add top-level packages except `java' and `javax'
In [28]: jpype.sun=jpype._jpackage.JPackage("sun")
In [32]: jpype.sun.misc.Launcher
Out[32]: jpype._jclass.sun.misc.Launcher
In [35]: jpype.sun.misc.Launcher.getLauncher().getClassLoader()
Out[35]: <jpype._jclass.sun.misc.Launcher$AppClassLoader at 0x19e23b0>
In [36]: acl=_
In [37]: acl.ucp
Out[37]: <jpype._jclass.sun.misc.URLClassPath at 0x19e2c90>
In [48]: [u.toString() for u in acl.ucp.getURLs()]
Out[48]: [u'file:/C:/Documents%20and%20Settings/User/']
Now, URLClassPath has a public addURL method. Let's try it out and see what happens:
#normally, this is done with Launcher.getFileURL but we can't call it directly
#public static URLClassPath.pathToURLs also does the same, but it returns an array
In [72]: jpype.sun.net.www.ParseUtil.fileToEncodedURL(
jpype.java.io.File(r"c:\Ivan\downloads\dom4j-2.0.0-RC1.jar")
.getCanonicalFile())
Out[72]: <jpype._jclass.java.net.URL at 0x1a04b50>
In [73]: _.toString()
Out[73]: u'file:/C:/Ivan/downloads/dom4j-2.0.0-RC1.jar'
In [74]: acl.ucp.addURL(_72)
In [75]: [u.toString() for u in acl.ucp.getURLs()]
Out[75]:
[u'file:/C:/Documents%20and%20Settings/User/',
u'file:/C:/Ivan/downloads/dom4j-2.0.0-RC1.jar']
Now, let's try to load some class from the .jar:
In [78]: jpype.org=jpype._jpackage.JPackage("org")
In [79]: jpype.org.dom4j.Entity
Out[79]: jpype._jclass.org.dom4j.Entity
Success!
This will probably fail from a sandbox or such where there are custom class loaders or security settings in the way (AppClassLoader.loadClass does security checks before calling super).
Further code inspection shows that addURL also disables the URLClassPath's lookup cache (implemented in a few native methods), and this is irreversible. Initially, the lookupCacheEnabled flag is set to the value of the sun.cds.enableSharedLookupCache system property.
The interface provides no way to edit the entries. URLs are added to URLClassPath's private ArrayList path and Stack urls. urls is accessible, but it turns out, it only holds entries temporarily, before it's attempted to load from it, at which point the information moves to HashMap lmap and ArrayList loaders. getURLs() returns a copy of path. So, it's theoretically possible to edit it by hacking the accessible fields, but it's nowhere near reliable and won't affect getURLs result.
I can only comment from what I remember of my own experience of hacking a non-sun JVM ten years ago, but it did scan the entire classpath on startup, as an efficiency measure. The names of all classes found were added to an internal hashtable along with their locations (directory or zip/jar file). However, that was ten years ago and I can't help but wonder whether this would still be a reasonable thing to do in most settings considering how disk and memory architectures have evolved.
I believe that the classpath is taken to be static and the result of changing the files is undefined.
If you really want to be able to add and remove classes at runtime, consider doing so in your own classloader. This is what web containers do.
So shouldn't it be possible to add a
JAR or *.class file to the classpath
after the JVM started ...
You add jars and directories to the classpath, not classes. The classes are in either the directory, or the jar.
And if not, does that mean that the
classpath is searched on JVM startup
and all fully qualified names of the
found classes are cached in an
internal "list"?
That could be easily tested: Set the classpath, start your program, move a new class into the CP, call 'Class.forName ("NewClass") from your program. Does it find the new class?
I think you could read documentation of TomCat server. This server implements java classpapth by its own. So, when this server is started, you can deploy new webApp just drag and drop jar in appropriatefolder in hot, without restart server, and it will upload your app.
I'm trying to run .class file from command line. It works when I manually move to the directory it's stored in, but when I try something like this:
java C:\Peter\Michael\Lazarus\Main
it says it can't find the main class. Is there any solution to this other than making a .jar file (I know that .jar is the best solution, but at this moment isn't the one I'm looking for)?
The Java application launcher (a.k.a java.exe or simply java) supports up to four different ways to specify what to launch (depending on which Java version you use).
Specifying a class name is the most basic way. Note that the class name is different from the file name.
java -cp path/to/classFiles/ mypackage.Main
Here we start the class mypackage.Main and use the -cp switch to specify the classpath which is used to find the class (the full path to the class mypackage.Main will be path/to/classFiles/mypackage/Main.class.
Starting a jar file.
java -jar myJar.jar
This puts the jar itself and anything specified on its Class-Path entry on the class path and starts the class indicated via the Main-Class entry. Note that in this case you can not specify any additional class path entries (they will be silently ignored).
Java 9 introduced modules and with that it introduce a way to launch a specific module in a way similar to how option #2 works (either by starting that modules dedicated main class or by starting a user-specified class within that module):
java --module my.module
Java 11 introduces support for Single-File Source Code Programs, which makes it very easy to execute Java programs that fit into a single source file. It even does the compile step for you:
java MyMain.java
This option can be useful for experimenting with Java for the first time, but quickly reaches its limits as it will not allow you to access classes that are defined in another source file (unless you compile them separately and put them on the classpath, which defeats the ease of use of this method and means you should probably switch back to option #1 in that case).
This feature was developed as JEP 330 and is still sometimes referred to as such.
For your specific case you'd use option #1 and tell java where to look for that class by using the -classpath option (or its short form -cp):
java -classpath C:\Peter\Michael\Lazarus\ Main
If your Main.java contains the entirety of your source code (and it is in the same directory), then you can use option #4, skip the compile step and directly compile-and-execute it:
java c:\Peter\Michael\Lazarus\Main.java
Assuming that Main.class does not have a package declaration:
java -cp C:\Peter\Michael\Lazarus\ Main
Java looks for classes in a "classpath", which can be set on the command line via the -cp option.
I just had the same issue, I tried running java hello.class, this is wrong.
The command should be java hello.
Do not include the file extension. It is looking for a class file, and will add the name on its own.
So running 'java hello.class' will tell it to go looking for 'hello.class.class' file.
Try this:
java -cp C:\Peter\Michael\Lazarus Main
You need to define the classpath.