I have to run a java task, with a very large number of classpath (1000, totaling 150k characters if concatenated).
The problem is that java returns an error when I try to execute this class:
/jdk/JAVA8/bin/java: Argument list too long
The error code is 7
I've tried to put the classpaths using "export CLASSPATH=CLASSPATH:....." and so I shouldn't specify them through the -cp java parameter, but it returned the same error.
I'm pretty sure that the problem revolves round a classpath's limit, because if I delete some of the classpath, the error disappears (but then I will have logical errors in the execution, because I need all the classpaths)
You could use classpath wildcards. Especially if many of your jars/class files are in the same directory, this would help a lot.
It could be environment variable size limit or command-line size limit as well rather than javac classpath arg limit.
javac takes arguments from file input as well. You can add all your arguments to this file and pass this file argument to command. Refer this for more.
You didn’t hit a java-specific limitation, but a system dependent limit. This is best illustrated by the fact, that the attempt to set the CLASSPATH variable fails as well, but setting an environment variable via export name=value in the shell isn’t related to Java.
As said by others, you could try to use wildcards for jar files within the same directory, but you have to care that Java does the expansion rather than the shell, as in the latter case, it would again yield a too long command line. So you have to escape the * character to ensure it will not be processed by the shell.
javac supports reading the command line arguments from an external file specified via #filename, but unfortunately, the java launcher doesn’t support this option.
An alternative would be to create symbolic links pointing to the jar files, having shorter paths and specifying these. You could even combine the approaches by creating one directory full of symbolic links and specifying that/directory/* as class path.
But there seems to be a logical error in the requirement. In a comment, you are mentioning “code analysis” and an analyzing tool should not require having the code to analyze in its own application class path. You can access class files via ordinary I/O, if you want to read and parse them. In case you want to load them, e.g. for using the builtin Reflection, you can create new ClassLoader instances pointing to the locations. So the tool doesn’t depend on the application class path and could read the locations from a configuration file, for example.
Using distinct class loaders has the additional advantage that you can close them when you’re done.
JVM does not limit classpath length. However, there is a hard OS limit on command line length and environment variables size.
On Linux check getconf ARG_MAX to see the limit.
On older kernel versions it is only 128KB. On newer kernels it is somewhere around 2MB.
If you want to set really long classpaths, you may need a JAR-Manifest trick. See this question for details.
Related
I have a Java Application which generates a temporary JNA folder and a temporary DLL file to the following location in the users' profile:
%OSDRIVE%\Users\ABC-<SOME-USER-ID>\AppData\Local\Temp\jna--881477353\jna7513918229606912988.dll
(the JNA folder and file names contain random numbers as suffix and with prefix "jna--" , "jna" respectively)
The JNA DLL file "Path" needs to be made an exception in Application Control Policies, specifically in AppLocker -> DLL Rules -> [Users] -> Exceptions in order for the Java app to function properly. At the moment, it is blocked by the AppLocker, however, if I add this path to the AppLocker, it will work for the current user.
Now, this is achievable for 1 user, but I have many users with the prefix as "ABC-" and the suffix "SOME-USER-ID" contains a random string with numbers and letters, i.e. many users who have prefix "ABC-" should be able to use this application without the need for manually adding every users' profile path into the AppLocker configuration.
Is there a way I can "Wildcard" this path into the AppLocker configuration?
For example, something like this:
%OSDRIVE%\Users\ABC-<WILDCARD>\AppData\Local\Temp\jna--<WILDCARD>\jna<WILDCARD>.dll
or even:
%OSDRIVE%\Users\ABC-<WILDCARD>\AppData\Local\Temp\jna--<WILDCARD>\*
Is there a realistic way of achieving this via Wildcards?
I do not wish to use "*" inside the path because it will then allow ALL users to be exempted from the AppLocker settings.
Any help is appreciated.
Thanks in advance!
Based on the available documentation I could find, it appears that a wildcard (*) character is only supported in AppLocker at the beginning or end of a path, but not in the middle. So your proposed solution is impossible.
However, rather than using a wildcard in the path for AppLocker, you can pre-extract the JNA native library to a known location for all users. This is a relatively common need for security purposes (exactly your intent), sometimes related to temp directory access permissions or sometimes related to signing binaries.
From the JNA API Overview Loading JNA:
JNA includes a small, platform-specific shared library which enables all native access. When the Native class is first accessed, JNA will first attempt to load this library from the directories specified in jna.boot.library.path. If that fails and jna.nosys=false is set, it will fall back to loading from the system library paths. Finally it will attempt to extract the stub library from from the JNA jar file, and load it.
This gives you two options to avoid the randomly-named temporary file. Copy it to:
a directory of your choice, and either pass -Djna.boot.library.path=C:\your\path on the java command line, or before loading JNA call System.setProperty("jna.boot.library.path", "C:\your\path") in your program.
a system directory on the PATH, e.g. System32, and set jna.nosys=false. (You can also set jna.nounpack=true to prevent the temp file unpacking.)
In addition, the JNA native library will be in a subdirectory of the Java temporary directory specified by the java.io.tmpdir system property. See this SO question: Environment variable to control java.io.tmpdir?
Answers to that question include using the _JAVA_OPTIONS environment variable as a possible means to change the tmpdir even if you're running an executable rather than Java command line.
Our tool (http://plse.cs.washington.edu/daikon) calculates program invariants by inserting instrumentation into the Java byte codes for a program. The user code is instrumented during runtime via the normal ClassFileTransformer::transform method.
It is also necessary to track value flows through JDK methods. Thus, we need to instrument the Java runtime as well. We cannot use transform, because hundreds of runtime methods are loaded prior to the first time we get control at transform.
Prior to Java 9, we handled this in an offline step that reads rt.jar, instruments its methods, and writes out a modified version as dcomp-rt.jar. The user placed dcomp-rt.jar on the bootclasspath to ensure our modified Java runtime methods were loaded instead of the standard ones. The user program invocation would look something like:
java -cp .:.../daikon/daikon.jar \
-Xbootclasspath/p:.../daikon/java/dcomp_rt.jar:.:.../daikon/daikon.jar \
-javaagent:.../daikon/java/dcomp_premain.jar={various dcomp arguments} \
{user program} {user program arguments}
Now to Java 9+. Our first approach was to read in and instrument the class files within the Java runtime jmod files (via the new jrt:/ file system) and create a dcomp_rt.jar as before. The problem we are experiencing is that we cannot get the system to use the contents of this jar instead of jrt:/java.base (for example). We tried various --module-path and -Xbootclasspath (only /a is available now, might be part of problem) options to no avail. Still hoping there might be a way to do this?
If not, I'm guessing we need to make modified versions of each of the interesting runtime jmods and then use a --patch-module argument for each of them. Would this ensure our modified code is loaded instead of the standard runtime?
Any thoughts/suggestions?
Well it looks like --patch-module does the trick. I made the same dcomp_rt.jar but with only classes from java.base.jmod. Then used:
--patch-module java.base={full path}/dcomp_rt.jar
Running java with -verbose:class showed all base classes being loaded from my jar.
Is this the best way to accomplish my goal?
I am developing a project for my Computer Network course.
Actually I ended it, now I need to write a script to compile it, so the teacher will be able to run it
I developed with Netbeans and now I am struggling to compile it by command line.
I have 3 folders (packages)
client: classes of the client process
server: classes of the server process
sharedClasses: classes usefull to both client and server (like User.java)
Also I am using the Gson as a jar file which is needed in the sharedClasses package
for example in sharedClasses there is a class called Message that uses Gson to be transformed in a json string
I tried a lot to create a script that compile it all but every time I get "ClassNotFoundException" or stuff like that: the online guides to understand classpath and so on are pretty bad.
Can someone tell me how to do my script and explain why things are done the way they are? Thanks
Path variables are a concept in all Unix and Windows operating systems. They are not a Java invention, but Java bases its own classpath and module path concepts on them.
A path variable’s value is simply a string which contains a list of file locations, separated by a colon (:) in Unix or a semicolon (;) in Windows.
The most common path variable is simply PATH. (I believe that in Windows, the variable’s canonical name is Path, but environment variables are case-insensitive in Windows, so it can be referred to as PATH in most cases.)
When you try to execute a program on the command line, by specifying a command name with no directory components, the operating system checks each file location in PATH, in order, and for each location which is a directory, the system will look for a match there. The first match is the one the operating system uses.
Java borrows this concept for the classpath. In the very early days of Java, it was exactly the same: If your classpath were /home/giulio:/opt/libraries, and you were looking for a class named com.example.ConnectionFactory, Java would look for a compiled file named com/example/ConnectionFactory.class in /home/giulio and then in /opt/libraries.
It wasn’t long before the classpath was allowed to contain files which are compressed archives of classes, in addition to directories. Your classpath might contain /home/giulio:/opt/libraries/foolib.jar, in which case Java would first check for a requested class in /home/giulio, since that is a directory, and if that failed, it would look for a matching entry in the /opt/libraries/foolib.jar archive file. (Zip files are also acceptable, and in fact a .jar file is really just a zip file with a few special Java-specific entries.)
So, when you want to tell Java to look in certain places for libraries, specify them in the classpath.
For instance, when compiling your client code:
projectroot=`dirname "$0"`
javac -classpath "$projectroot"/sharedClasses/classes \
-d "$projectroot"/client/classes \
"$projectroot"/client/src/*.java
When you run your code:
java -classpath "$projectroot"/sharedClasses/classes:"$projectroot"/client/classes \
edu.acme.giulio.client.Main
So I have a java project with multiple java files.
I know that is almost straight forward to start a java application using batch file. But that is for a pretty simple java program with a single class.
However I am wondering if it is possible to do that with in a scale of a project that you usually create using eclipse. A large project with multiple packages, classes and multiple java files.
My try was to write a script and apply on the main class as following
set path = C:\Program Files\Java\jdk1.7.0_25\bin
javac -classpath twitter/twitter4j-stream-3.0.5.jar;twitter4j-core-3.0.5.jar" sourcepath="lib/twitter4j-core-4.0.1.jar;lib/twitter4j-core-4.0.1.jar;lib/twitter4j-stream-4.0.1.jar;svm_light_lib Program.java
java Program
However when I start the .bat file it automatically closes.
Any Ideas ?
Thanks in advance
First, never overwrite the environment variable path, not even
temporarily. Append your folder instead: set "path=%path%;%mypath%" or set "path=%mypath%;%path%".
(There exists a particular path command but I'm not sure about right syntax: path=%path%;%mypath% with = assignment or path %path%;%mypath% without it).
Use full path to a program if you know it, e.g. "%mypath%\javac".
For better readability, values for -classpath and -sourcepath options are stored to the environment variables mycpth and mysrcp, respectively. Note and use proper " quotation and no spacing around = to avoid any leading and trailing spaces in all set commands.
pause to see all the javac output. Displays the message Press any key to continue . . .
Next code should be (syntax) error-free. However, success depends (among others) on classpath and sourcepath entries visibility as well...
set "mypath=C:\Program Files\Java\jdk1.7.0_25\bin"
set "path=%path%;%mypath%"
set "mycpth=twitter/twitter4j-stream-3.0.5.jar;twitter4j-core-3.0.5.jar"
set "mysrcp=lib/twitter4j-core-4.0.1.jar;lib/twitter4j-core-4.0.1.jar;lib/twitter4j-stream-4.0.1.jar;svm_light_lib"
"%mypath%\javac" -classpath "%mycpth%" -sourcepath "%mysrcp%" Program.java
pause
java Program
However I am wondering if it is possible to do that with in a scale of a project that you usually create using eclipse. A large project with multiple packages, classes and multiple java files.
Of course it is possible!
In this case, I suspect the problem is that you java command doesn't have a "-cp" argument. The java command is probably failing because it can't find twitter classes ... at runtime.
Remember to include "." on the classpath ... or else java won't find the file that you just compiled.
#JB Nizet's suggestion is also very important advice for finding out what is actually happening.
I have a C binary that calls out to Java via JNI. I set CLASSPATH to somedir/* to pick up all the jars in somedir.
When I run the binary, a required class definition cannot be found. When I run
java that.class's.name
from the same command line, the class is successfully found. If I explicitly add all the jars in somedir/ to the classpath, everything works great, but that leads to a very long classpath which I'd like to avoid.
Does a JVM executed via JNI honour wildcard expansion of the classpath? Can it be made to do so?
I figured out the answer by reading the hotspot source code.
Only paths passed via either CLASSPATH or -cp / -classpath are subject to wildcard expansion. These are then passed as a system property to the running JVM via -Djava.class.path.
You tell a JNI-invoked JVM about a classpath via a JVMOptions structure, which may include -Djava.class.path but -classpath will not necessarily be honoured (and in practice, isn't by the hotspot implementation). Since java.class.path is directly passed to the JVM as a system property, it doesn't get wildcard expanded and therefore wildcards won't work.
No. No, it cannot. Using JNI doesn't help.
The way you would do this is by implementing your own class loader (in Java), but that class loader would have to be in the wildcard-free CLASSPATH.
You could, of course, set the CLASSPATH to its expanded form before invoking the JVM. That would work and could be done via a shell script (no JNI needed).