According to the HP Fortify documentation, the Static Code Analyzer first translates the source code into an intermediate format, and then it scans the translated code and generates a vulnerability report.
It says the translation can be done using the following Ant code:
<antcall target="compile">
<param name="build.compiler" value="com.fortify.dev.ant.SCACompiler"/>
</antcall>
This will call your "compile" target but force it to use the SCACompiler instead of the regular javac compiler.
I have run Fortify on our Java code and it produces vulnerability reports. But I do not see the intermediate files anywhere. I ran a diff between the Java class files that the regular javac compiler produced and the Java class files that the SCACompiler produced, and they were exactly the same. Are the intermediate files stored somewhere else, or does Fortify automatically delete them after performing the scan?
The intermediate files are not class or object files. They are NST (Normalized Syntax Tree) files, a proprietary format used by HP Fortify (this is discussed in the book "Secure Programming with Static Analysis". When translating with a build ID, such as:
sourceanalyzer -b test ant
Then it will be stored in the project working directory. In Windows, typically:
%USERPROFILE%\AppData\Local\Fortify\sca<version>\build\test
or on other platforms:
~/.fortify/sca<version>/build/test
this will then contain the canonicalized path to the NST, as was performed during the translation. These can then be used to scan multiple times if needed, but should be "cleaned" if scanning a separate new (or updated) codebase.
For ant integration I think it depends on which version of Ant, and the way you are translating, but this way I think it just calls the sourceanalyzer.jar file (which contains the com.fortify.dev.ant.SCACompiler class) in order to hook into the JVM and follow the build to create the NST files needed for scanning. I don't believe it's actually a separate version of javac, although perhaps there is a separate version under <SCA installation directory>/jre/ which it may use.
Lavamunky is correct about the default path for the working directory. You can change this in the following locations:
1. FortifyInstallRoot\Core\config\fortify.properties: com.fortify.WorkingDirectory
2. FortifyInstallRoot\Core\config\fortify-sca.properties: com.fortify.sca.ProjectRoot
Note that you need to use / as the path delimiter instead of \ inside of the config files. Inside of the folder specified by those paths, the pattern is: sca\build\.
You can also specify these at runtime:
sourceanalyzer -b MyBuild -Dcom.fortify.WorkingDirectory=C:\Fortify\Work -Dcom.fortify.sca.ProjectRoot=C:\Fortify\Work
The path to the working files would then be:
C:\Fortify\Work\sca<version>\build\MyBuild\
Related
Our tool (http://plse.cs.washington.edu/daikon) calculates program invariants by inserting instrumentation into the Java byte codes for a program. The user code is instrumented during runtime via the normal ClassFileTransformer::transform method.
It is also necessary to track value flows through JDK methods. Thus, we need to instrument the Java runtime as well. We cannot use transform, because hundreds of runtime methods are loaded prior to the first time we get control at transform.
Prior to Java 9, we handled this in an offline step that reads rt.jar, instruments its methods, and writes out a modified version as dcomp-rt.jar. The user placed dcomp-rt.jar on the bootclasspath to ensure our modified Java runtime methods were loaded instead of the standard ones. The user program invocation would look something like:
java -cp .:.../daikon/daikon.jar \
-Xbootclasspath/p:.../daikon/java/dcomp_rt.jar:.:.../daikon/daikon.jar \
-javaagent:.../daikon/java/dcomp_premain.jar={various dcomp arguments} \
{user program} {user program arguments}
Now to Java 9+. Our first approach was to read in and instrument the class files within the Java runtime jmod files (via the new jrt:/ file system) and create a dcomp_rt.jar as before. The problem we are experiencing is that we cannot get the system to use the contents of this jar instead of jrt:/java.base (for example). We tried various --module-path and -Xbootclasspath (only /a is available now, might be part of problem) options to no avail. Still hoping there might be a way to do this?
If not, I'm guessing we need to make modified versions of each of the interesting runtime jmods and then use a --patch-module argument for each of them. Would this ensure our modified code is loaded instead of the standard runtime?
Any thoughts/suggestions?
Well it looks like --patch-module does the trick. I made the same dcomp_rt.jar but with only classes from java.base.jmod. Then used:
--patch-module java.base={full path}/dcomp_rt.jar
Running java with -verbose:class showed all base classes being loaded from my jar.
Is this the best way to accomplish my goal?
I am developing a project for my Computer Network course.
Actually I ended it, now I need to write a script to compile it, so the teacher will be able to run it
I developed with Netbeans and now I am struggling to compile it by command line.
I have 3 folders (packages)
client: classes of the client process
server: classes of the server process
sharedClasses: classes usefull to both client and server (like User.java)
Also I am using the Gson as a jar file which is needed in the sharedClasses package
for example in sharedClasses there is a class called Message that uses Gson to be transformed in a json string
I tried a lot to create a script that compile it all but every time I get "ClassNotFoundException" or stuff like that: the online guides to understand classpath and so on are pretty bad.
Can someone tell me how to do my script and explain why things are done the way they are? Thanks
Path variables are a concept in all Unix and Windows operating systems. They are not a Java invention, but Java bases its own classpath and module path concepts on them.
A path variable’s value is simply a string which contains a list of file locations, separated by a colon (:) in Unix or a semicolon (;) in Windows.
The most common path variable is simply PATH. (I believe that in Windows, the variable’s canonical name is Path, but environment variables are case-insensitive in Windows, so it can be referred to as PATH in most cases.)
When you try to execute a program on the command line, by specifying a command name with no directory components, the operating system checks each file location in PATH, in order, and for each location which is a directory, the system will look for a match there. The first match is the one the operating system uses.
Java borrows this concept for the classpath. In the very early days of Java, it was exactly the same: If your classpath were /home/giulio:/opt/libraries, and you were looking for a class named com.example.ConnectionFactory, Java would look for a compiled file named com/example/ConnectionFactory.class in /home/giulio and then in /opt/libraries.
It wasn’t long before the classpath was allowed to contain files which are compressed archives of classes, in addition to directories. Your classpath might contain /home/giulio:/opt/libraries/foolib.jar, in which case Java would first check for a requested class in /home/giulio, since that is a directory, and if that failed, it would look for a matching entry in the /opt/libraries/foolib.jar archive file. (Zip files are also acceptable, and in fact a .jar file is really just a zip file with a few special Java-specific entries.)
So, when you want to tell Java to look in certain places for libraries, specify them in the classpath.
For instance, when compiling your client code:
projectroot=`dirname "$0"`
javac -classpath "$projectroot"/sharedClasses/classes \
-d "$projectroot"/client/classes \
"$projectroot"/client/src/*.java
When you run your code:
java -classpath "$projectroot"/sharedClasses/classes:"$projectroot"/client/classes \
edu.acme.giulio.client.Main
I have to run a java task, with a very large number of classpath (1000, totaling 150k characters if concatenated).
The problem is that java returns an error when I try to execute this class:
/jdk/JAVA8/bin/java: Argument list too long
The error code is 7
I've tried to put the classpaths using "export CLASSPATH=CLASSPATH:....." and so I shouldn't specify them through the -cp java parameter, but it returned the same error.
I'm pretty sure that the problem revolves round a classpath's limit, because if I delete some of the classpath, the error disappears (but then I will have logical errors in the execution, because I need all the classpaths)
You could use classpath wildcards. Especially if many of your jars/class files are in the same directory, this would help a lot.
It could be environment variable size limit or command-line size limit as well rather than javac classpath arg limit.
javac takes arguments from file input as well. You can add all your arguments to this file and pass this file argument to command. Refer this for more.
You didn’t hit a java-specific limitation, but a system dependent limit. This is best illustrated by the fact, that the attempt to set the CLASSPATH variable fails as well, but setting an environment variable via export name=value in the shell isn’t related to Java.
As said by others, you could try to use wildcards for jar files within the same directory, but you have to care that Java does the expansion rather than the shell, as in the latter case, it would again yield a too long command line. So you have to escape the * character to ensure it will not be processed by the shell.
javac supports reading the command line arguments from an external file specified via #filename, but unfortunately, the java launcher doesn’t support this option.
An alternative would be to create symbolic links pointing to the jar files, having shorter paths and specifying these. You could even combine the approaches by creating one directory full of symbolic links and specifying that/directory/* as class path.
But there seems to be a logical error in the requirement. In a comment, you are mentioning “code analysis” and an analyzing tool should not require having the code to analyze in its own application class path. You can access class files via ordinary I/O, if you want to read and parse them. In case you want to load them, e.g. for using the builtin Reflection, you can create new ClassLoader instances pointing to the locations. So the tool doesn’t depend on the application class path and could read the locations from a configuration file, for example.
Using distinct class loaders has the additional advantage that you can close them when you’re done.
JVM does not limit classpath length. However, there is a hard OS limit on command line length and environment variables size.
On Linux check getconf ARG_MAX to see the limit.
On older kernel versions it is only 128KB. On newer kernels it is somewhere around 2MB.
If you want to set really long classpaths, you may need a JAR-Manifest trick. See this question for details.
So I have a java project with multiple java files.
I know that is almost straight forward to start a java application using batch file. But that is for a pretty simple java program with a single class.
However I am wondering if it is possible to do that with in a scale of a project that you usually create using eclipse. A large project with multiple packages, classes and multiple java files.
My try was to write a script and apply on the main class as following
set path = C:\Program Files\Java\jdk1.7.0_25\bin
javac -classpath twitter/twitter4j-stream-3.0.5.jar;twitter4j-core-3.0.5.jar" sourcepath="lib/twitter4j-core-4.0.1.jar;lib/twitter4j-core-4.0.1.jar;lib/twitter4j-stream-4.0.1.jar;svm_light_lib Program.java
java Program
However when I start the .bat file it automatically closes.
Any Ideas ?
Thanks in advance
First, never overwrite the environment variable path, not even
temporarily. Append your folder instead: set "path=%path%;%mypath%" or set "path=%mypath%;%path%".
(There exists a particular path command but I'm not sure about right syntax: path=%path%;%mypath% with = assignment or path %path%;%mypath% without it).
Use full path to a program if you know it, e.g. "%mypath%\javac".
For better readability, values for -classpath and -sourcepath options are stored to the environment variables mycpth and mysrcp, respectively. Note and use proper " quotation and no spacing around = to avoid any leading and trailing spaces in all set commands.
pause to see all the javac output. Displays the message Press any key to continue . . .
Next code should be (syntax) error-free. However, success depends (among others) on classpath and sourcepath entries visibility as well...
set "mypath=C:\Program Files\Java\jdk1.7.0_25\bin"
set "path=%path%;%mypath%"
set "mycpth=twitter/twitter4j-stream-3.0.5.jar;twitter4j-core-3.0.5.jar"
set "mysrcp=lib/twitter4j-core-4.0.1.jar;lib/twitter4j-core-4.0.1.jar;lib/twitter4j-stream-4.0.1.jar;svm_light_lib"
"%mypath%\javac" -classpath "%mycpth%" -sourcepath "%mysrcp%" Program.java
pause
java Program
However I am wondering if it is possible to do that with in a scale of a project that you usually create using eclipse. A large project with multiple packages, classes and multiple java files.
Of course it is possible!
In this case, I suspect the problem is that you java command doesn't have a "-cp" argument. The java command is probably failing because it can't find twitter classes ... at runtime.
Remember to include "." on the classpath ... or else java won't find the file that you just compiled.
#JB Nizet's suggestion is also very important advice for finding out what is actually happening.
Can anyone explain how identical Java sources can end up compiling to binary differing class files?
The question arises from the following situation:
We have a fairly large application (800+ classes) which has been branched, restructured then reintegrated back into the trunk. Prior to reintegration, we merged the trunk into the branch, which is standard procedure.
The end result was a set of directories with the branch sources and a set of directories with the trunk sources. Using Beyond Compare we were able to determine that both sets of sources were identical. However, on compiling (same JDK using maven hosted in IntelliJ v11) we noticed that about a dozen or so of the class files were different.
When we decompiled the source for each pair of apparently different class files we ended up with the same java source, so in terms of the end result, it doesn't seem to matter. But why is it that just a few of the files are different?
Thanks.
Additional thought:
If maven/javac compiles files in a different sequence, might that affect the end result?
Assuming that the JDK versions, build tool versions, and build / compilation options are identical, I can still think of a number of possible sources of differences:
Timestamps - class files may1 contain compilation timestamps. Unless you run the compilations at exactly the same times, different compilations of the same file would result different timestamps.
Source filename paths - each class file includes the pathname of the source file. If you compile two trees with different pathnames the class files will contain different source pathnames.
Values of imported compile-time constants - when a class A uses a compile-time constant defined in another class B (see JLS for the definition of a "compile time constant"), the value of the constant is incorporated into As class file. So if you compile A against different versions of B (with different values for the constants), the code of A is likely to be different.
Differences due to identityHashcode being used in HashMap keys by the compiler could lead to differences in map iteration order in some step. This could affect .class file generation in a way that is not significant, but still shows up as a .class file difference. For example, constant pool entries could end up in a different order.
Differences in signatures of external classes / methods; e.g. if you changed a dependency version in one of your POM files.
Differences in the effective build classpaths might result in differences in the order in which imported classes are found. This might in turn result in non-significant differences in the order of entries in the class file's Constant Pool. This could happen due to things such as:
files appearing in different order in the directories of external JAR files,
files being compiled in different order due to the source files being in different order when your build tool iterates them2, or
parallelism in the build (if you have that enabled).
There is a possible workaround for the problem with file ordering: use the undocumented -XDsortfiles option as described in JDK-7003006. (Kudos to #Holger for knowing about that.)
Note that you don't normally see the actual order of files in file system directories. Commandline tools like ls and dir, and file browsers will typically sort the entries (in name or timestamp order) before displaying them.
1 - This is compiler dependent. Also, it is not guaranteed that javap will show the timestamps ... if they are present.
2 - The OS gives no guarantees that listing a directory (at the syscall level) will return the file system objects in a deterministic order ... or the same order, if you have removed and re-added files.
I should add that the first step to identifying the cause of the differences is to work out exactly what they are. You probably need (needed) to do that the hard way - by manually decoding a pair of class files to identify the places where they actually differences ... and what the differences actually mean.
When you compare using beyond compare, comparision is done based on contents of the files. But in the build process just the timestamp of the source files are checked for change. So it your source file's lastmodified date changes it will be recompiled.
Different JDK produce different binary classes (optimizations, but also class version number). There are compilation options, too (a JDK may compile in an older format, or it can add debug information).
Different versions of Java can add different meta data which is often ignored by a decompiler.
I suggest you try using javap -c -v for more of the details in a file. If this doesn't help you can use the ASMifierClassVisitor which looks at every byte.
same JDK can also have different output depending on how you compile.
you can compile with or without debug info, you can compile to run in an older version, each option will result in other classes.