I wonder how can I quickly take a directory in my computer that contains 1K jars, and let Gradle structure it to the Maven .m2/repository structure. I've read of the Gradle publishing feature, but it seems pretty impossible to use when the amount of jars is that big because they are not taking the information from the META-INF, but want the user to specify the group ID and other details.
I made you wait a little, however I come up with a solution at this point. Have a look at this bash script:
#!/usr/bin/env sh
PWD=$( pwd )
JARS=$( find $PWD -name '*.jar' )
for JARFILE in $JARS
do
echo "found JAR: $JARFILE";
mkdir -p $PWD/extracted/$JARFILE/;
unzip $JARFILE -d $PWD/extracted/ &> /dev/null;
if [ -d "$PWD/extracted/META-INF/maven/" ]; then
POMPROPS=$( find $PWD/extracted/META-INF/maven -name 'pom.properties' );
echo "found POMPROPS: $POMPROPS";
POMXML=$( find $PWD/extracted/META-INF/maven -name 'pom.xml' );
echo "found POMXML: $POMXML";
ARTIFACTID=$( grep 'artifactId' $POMPROPS | cut -d'=' -f2 | tr -d '\r' );
GROUPID=$( grep 'groupId' $POMPROPS | cut -d'=' -f2 | tr -d '\r' );
VERSION=$( grep 'version' $POMPROPS | cut -d'=' -f2 | tr -d '\r' );
if [[ -f $POMPROPS && -f $POMXML ]]; then
GROUPDIR=$( sed 's/\./\//g' <<<$GROUPID);
echo "Group dir: "$GROUPDIR;
mkdir -p ~/.m2test/repository/$GROUPDIR/$ARTIFACTID/$VERSION/
cp $POMXML ~/.m2test/repository/$GROUPDIR/$ARTIFACTID/$VERSION/$ARTIFACTID-$VERSION.pom
cp $JARFILE ~/.m2test/repository/$GROUPDIR/$ARTIFACTID/$VERSION/$ARTIFACTID-$VERSION.jar
else
echo "did not find pom properties and/or xml "
echo "$JARFILE: did not find pom properties and/or xml" >> log.txt
fi
else
echo "$JARFILE: no maven directory found" >> log.txt
fi
rm -rf $PWD/extracted/;
done
First have a look at it and understand it, please. You should not run any script, without understanding it. Of course I don't want to screw you, but always be cautious 😃. You must have sed, grep and unzip installed. However most distributions should have that on board.
Just run that script, when in a terminal inside of the directory. It should organize your JARs inside of ~/.m2test. If some files were not found, it will be logged inside log.txt. Eventually, you can copy-paste or move the files from ~/.m2test to ~/.m2 as soon, as you are optimistic, it worked out.
And of course: Always do make backups (of your ~/.m2 in this case) 😁!
This - of course - only works, if your JARs have that maven directory in META-INF. If it doesn't, I don't have a better guess, how to organize it into the local repository.
I try to make a simple shell script to make a jar file. The jar command combined with -C does not work with wildcards. Therefor I use a wildcard to find the files I want. Write them to a file, and loop over them.
It looks something like this:
the_classes=''
cd "$bin_folder"
tmp_dir=$(mktemp -d -t java_sucks)
find "imui/core/" -type f -name "IMUI_Widget_Agent*.class" >"$tmp_dir/classes.txt"
while IFS="" read -r p || [ -n "$p" ]
do
the_classes="${the_classes} -C '$bin_folder' '$p'"
done < "$tmp_dir/classes.txt"
Using the above I complete the command:
cmd='jar cfm build/IMUI_Widget_Agent.jar'
cmd="${cmd} \"$bin_folder/imui/core/IMUI_Widget_Agent_MANIFEST.MF\" $the_classes"
printf "\n\n\ncmd\n\n\n"
echo $cmd
Now if I copy and paste this command to execute it works!
But I want to avoid the manual labour of doing the copy and paste by hand every time.
Now I have:
eval "$("$cmd")"
But I get an error File name too long. No matter what I try, every fix I do creates a new problem. I have been working 6 hours now to make this script.
What would be a good step forward?
Since you cd "$bin_folder" you don't actually need -C "$bin_folder":
#!/bin/bash
shopt -s globstar
cd "$bin_folder"
jar cfm build/IMUI_Widget_Agent.jar \
imui/core/IMUI_Widget_Agent_MANIFEST.MF \
imui/core/**/IMUI_Widget_Agent*.class
However, if you still want to add them as part of a larger script, you can easily and robustly build your command in an array:
#!/bin/bash
shopt -s globstar
cmd=(jar cfm build/IMUI_Widget_Agent.jar imui/core/IMUI_Widget_Agent_MANIFEST.MF)
cd "$bin_folder"
for file in imui/core/**/IMUI_Widget_Agent*.class
do
cmd+=(-C "$bin_folder" "$file")
done
echo "About to execute: "
printf "%q " "${cmd[#]}"
echo
"${cmd[#]}"
Alternatively, you can simply do eval "$cmd" with your code, which is equivalent to echo and copy-pasting. However, be aware that this is fragile and error prone because it requires careful escaping of the filenames which you're not currently doing.
For some research purpose, I want to download 1000 java classes (".java") files from the given website. I don't want to do this manually.
For example, below has many Java Source files which I want to get using scripting/programming. I've worked with Linux shell scripts, PHP, and Java. So any solution using these is appreciated.
http://www.cs.uic.edu/~sloan/CLASSES/java/
Thanks!
Based on the question
wget -A java -r https://www.cs.uic.edu/~sloan/CLASSES/java/
will download all ".java" files in the same directory structure as on the server.
This will also download the robots.txt file.
For the particular example you gave,
curl -vs https://www.cs.uic.edu/~sloan/CLASSES/java/ 2>&1 | grep -oP '(?<=").*.java(?=")' | sed -e 's|^|https://www.cs.uic.edu/~sloan/CLASSES/java/|' | xargs wget
Explanations
1) Get the page and print to stdout. It will give you entire html.
curl -vs https://www.cs.uic.edu/~sloan/CLASSES/java/ 2>&1
2) Find the word with .java in quotes, but output without quotes "[ANYTHING].java". It will give you something like HelloWorld.java.
grep -oP '(?<=").*.java(?=")'
3) Add prefix to make it full url, so you can download them. It will give you something like https://www.cs.uic.edu/~sloan/CLASSES/java/HelloWorld.java
sed -e 's|^|https://www.cs.uic.edu/~sloan/CLASSES/java/|'
4) Download them to the current directory.
xargs wget
Thank you all !!
I've done using "wget -r -l1 -nd -nc -A.java http://www.cs.uic.edu/~sloan/CLASSES/java/"
This was however my required task. But am just thinking, may be we can improve same "wget" to go on internet and get me 1000 ".java" files. Perhaps, we can invoke google search (from script) for a keyword "java tutorials" and then from the returned URL, scan for ".java" files.
Thank again all
Viki.
How would you find a particular class name inside lots of jar files?
(Looking for the actual class name, not the classes that reference it.)
Unix
On Linux, other Unix variants, Git Bash on Windows, or Cygwin, use the jar (or unzip -v), grep, and find commands.
The following lists all class files that match a given name:
for i in *.jar; do jar -tvf "$i" | grep -Hsi ClassName && echo "$i"; done
If you know the entire list of Java archives you want to search, you could place them all in the same directory using (symbolic) links.
Or use find (case sensitively) to find the JAR file that contains a given class name:
find path/to/libs -name '*.jar' -exec grep -Hls ClassName {} \;
For example, to find the name of the archive containing IdentityHashingStrategy:
$ find . -name '*.jar' -exec grep -Hsli IdentityHashingStrategy {} \;
./trove-3.0.3.jar
If the JAR could be anywhere in the system and the locate command is available:
for i in $(locate "*.jar");
do echo "$i"; jar -tvf "$i" | grep -Hsi ClassName;
done
A syntax variation:
find path/to/libs -name '*.jar' -print | \
while read i; do jar -tvf "$i" | grep -Hsi ClassName && echo "$i"; done
Windows
Open a command prompt, change to the directory (or ancestor directory) containing the JAR files, then:
for /R %G in (*.jar) do #jar -tvf "%G" | find "ClassName" > NUL && echo %G
Here's how it works:
for /R %G in (*.jar) do - loop over all JAR files, recursively traversing directories; store the file name in %G.
#jar -tvf "%G" | - run the Java Archive command to list all file names within the given archive, and write the results to standard output; the # symbol suppresses printing the command's invocation.
find "ClassName" > NUL - search standard input, piped from the output of the jar command, for the given class name; this will set ERRORLEVEL to 1 iff there's a match (otherwise 0).
&& echo %G - iff ERRORLEVEL is non-zero, write the Java archive file name to standard output (the console).
Web
Use a search engine that scans JAR files.
Eclipse can do it, just create a (temporary) project and put your libraries on the projects classpath. Then you can easily find the classes.
Another tool, that comes to my mind, is Java Decompiler. It can open a lot of jars at once and helps to find classes as well.
some time ago, I wrote a program just for that: https://github.com/javalite/jar-explorer
grep -l "classname" *.jar
gives you the name of the jar
find . -name "*.jar" -exec jar -t -f {} \; | grep "classname"
gives you the package of the class
#!/bin/bash
pattern=$1
shift
for jar in $(find $* -type f -name "*.jar")
do
match=`jar -tvf $jar | grep $pattern`
if [ ! -z "$match" ]
then
echo "Found in: $jar"
echo "$match"
fi
done
To locate jars that match a given string:
find . -name \*.jar -exec grep -l YOUR_CLASSNAME {} \;
I didn't know of a utility to do it when I came across this problem, so I wrote the following:
public class Main {
/**
*
*/
private static String CLASS_FILE_TO_FIND =
"class.to.find.Here";
private static List<String> foundIn = new LinkedList<String>();
/**
* #param args the first argument is the path of the file to search in. The second may be the
* class file to find.
*/
public static void main(String[] args) {
if (!CLASS_FILE_TO_FIND.endsWith(".class")) {
CLASS_FILE_TO_FIND = CLASS_FILE_TO_FIND.replace('.', '/') + ".class";
}
File start = new File(args[0]);
if (args.length > 1) {
CLASS_FILE_TO_FIND = args[1];
}
search(start);
System.out.println("------RESULTS------");
for (String s : foundIn) {
System.out.println(s);
}
}
private static void search(File start) {
try {
final FileFilter filter = new FileFilter() {
public boolean accept(File pathname) {
return pathname.getName().endsWith(".jar") || pathname.isDirectory();
}
};
for (File f : start.listFiles(filter)) {
if (f.isDirectory()) {
search(f);
} else {
searchJar(f);
}
}
} catch (Exception e) {
System.err.println("Error at: " + start.getPath() + " " + e.getMessage());
}
}
private static void searchJar(File f) {
try {
System.out.println("Searching: " + f.getPath());
JarFile jar = new JarFile(f);
ZipEntry e = jar.getEntry(CLASS_FILE_TO_FIND);
if (e == null) {
e = jar.getJarEntry(CLASS_FILE_TO_FIND);
if (e != null) {
foundIn.add(f.getPath());
}
} else {
foundIn.add(f.getPath());
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
There are also two different utilities called both "JarScan" that do exactly what you are asking for:
JarScan (inetfeedback.com) and JarScan (java.net)
ClassFinder is a program that's designed to solve this problem. It allows you to search recursively through directories and jar files to find all instances of a class matching a pattern. It is written in Java, not python. It has a nice GUI which makes it easy to use. And it runs fast. This release is precompiled in a runnable jar so you don't have to build it from source.
Download it here: ClassFinder 1.0
user1207523's script works fine for me. Here is a variant that searches for jar files recusively using find instead of simple expansion;
#!/bin/bash
for i in `find . -name '*.jar'`; do jar -tf "$i" | grep $1 | xargs -I{} echo -e "$i : {}" ; done
I've always used this on Windows and its worked exceptionally well.
findstr /s /m /c:"package/classname" *.jar, where
findstr.exe comes standard with Windows and the params:
/s = recursively
/m = print only the filename if there is a match
/c = literal string (in this case your package name + class names
separated by '/')
Hope this helps someone.
A bash script solution using unzip (zipinfo). Tested on Ubuntu 12.
#!/bin/bash
# ./jarwalker.sh "/a/Starting/Path" "aClassName"
IFS=$'\n'
jars=( $( find -P "$1" -type f -name "*.jar" ) )
for jar in ${jars[*]}
do
classes=( $( zipinfo -1 ${jar} | awk -F '/' '{print $NF}' | grep .class | awk -F '.' '{print $1}' ) )
if [ ${#classes[*]} -ge 0 ]; then
for class in ${classes[*]}
do
if [ ${class} == "$2" ]; then
echo "Found in ${jar}"
fi
done
fi
done
To find a class in a folder (and subfolders) bunch of JARs:
https://jarscan.com/
Usage: java -jar jarscan.jar [-help | /?]
[-dir directory name]
[-zip]
[-showProgress]
<-files | -class | -package>
<search string 1> [search string 2]
[search string n]
Help:
-help or /? Displays this message.
-dir The directory to start searching
from default is "."
-zip Also search Zip files
-showProgress Show a running count of files read in
-files or -class Search for a file or Java class
contained in some library.
i.e. HttpServlet
-package Search for a Java package
contained in some library.
i.e. javax.servlet.http
search string The file or package to
search for.
i.e. see examples above
Example:
java -jar jarscan.jar -dir C:\Folder\To\Search -showProgress -class GenericServlet
Just use FindClassInJars util, it's a simple swing program, but useful. You can check source code or download jar file at http://code.google.com/p/find-class-in-jars/
A bit late to the party, but nevertheless...
I've been using JarBrowser to find in which jar a particular class is present. It's got an easy to use GUI which allows you to browse through the contents of all the jars in the selected path.
To search all jar files in a given directory for a particular class, you can do this:
ls *.jar | xargs grep -F MyClass
or, even simpler,
grep -F MyClass *.jar
Output looks like this:
Binary file foo.jar matches
It's very fast because the -F option means search for Fixed string, so it doesn't load the the regex engine for each grep invocation. If you need to, you can always omit the -F option and use regexes.
I found this new way
bash $ ls -1 | xargs -i -t jar -tvf '{}'| grep Abstract
jar -tvf activation-1.1.jar
jar -tvf antisamy-1.4.3.jar
2263 Thu Jan 13 21:38:10 IST 2011 org/owasp/validator/html/scan/AbstractAntiSamyScanner.class
...
So this lists the jar and the class if found, if you want you can give ls -1 *.jar or input to xargs with find command HTH Someone.
To add yet another tool... this is a very simple and useful tool for windows. A simple exe file you click on, give it a directory to search in, a class name and it will find the jar file that contains that class. Yes, it's recursive.
http://sourceforge.net/projects/jarfinder/
Check JBoss Tattletale; although I've never used it personally, this seems to be the tool you need.
Not sure why scripts here have never really worked for me. This works:
#!/bin/bash
for i in *.jar; do jar -tf "$i" | grep $1 | xargs -I{} echo -e "$i : {}" ; done
Script to find jar file: find_jar.sh
IFS=$(echo -en "\n\b") # Set the field separator newline
for f in `find ${1} -iname *.jar`; do
jar -tf ${f}| grep --color $2
if [ $? == 0 ]; then
echo -n "Match found: "
echo -e "${f}\n"
fi
done
unset IFS
Usage: ./find_jar.sh < top-level directory containing jar files > < Class name to find>
This is similar to most answers given here. But it only outputs the file name, if grep finds something. If you want to suppress grep output you may redirect that to /dev/null but I prefer seeing the output of grep as well so that I can use partial class names and figure out the correct one from a list of output shown.
The class name can be both simple class name Like "String" or fully qualified name like "java.lang.String"
Filename: searchForFiles.py
import os, zipfile, glob, sys
def main():
searchFile = sys.argv[1] #class file to search for, sent from batch file below (optional, see second block of code below)
listOfFilesInJar = []
for file in glob.glob("*.jar"):
archive = zipfile.ZipFile(file, 'r')
for x in archive.namelist():
if str(searchFile) in str(x):
listOfFilesInJar.append(file)
for something in listOfFilesInJar:
print("location of "+str(searchFile)+": ",something)
if __name__ == "__main__":
sys.exit(main())
You can easily run this by making a .bat file with the following text (replace "AddWorkflows.class" with the file you are searching for):
(File: CallSearchForFiles.bat)
#echo off
python -B -c "import searchForFiles;x=searchForFiles.main();" AddWorkflows.class
pause
You can double-click CallSearchForFiles.bat to run it, or call it from the command line "CallSearchForFiles.bat SearchFile.class"
Click to See Example Output
You can find a class in a directory full of jars with a bit of shell:
Looking for class "FooBar":
LIB_DIR=/some/dir/full/of/jarfiles
for jarfile in $(find $LIBDIR -name "*.jar"); do
echo "--------$jarfile---------------"
jar -tvf $jarfile | grep FooBar
done
One thing to add to all of the above: if you don't have the jar executable available (it comes with the JDK but not with the JRE), you can use unzip (or WinZip, or whatever) to accomplish the same thing.
shameless self promotion, but you can try a utility I wrote : http://sourceforge.net/projects/zfind
It supports most common archive/compressed files (jar, zip, tar, tar.gz etc) and unlike many other jar/zip finders, supports nested zip files (zip within zip, jar within jar etc) till unlimited depth.
Following script will help you out
for file in *.jar
do
# do something on "$file"
echo "$file"
/usr/local/jdk/bin/jar -tvf "$file" | grep '$CLASSNAME'
done
This one works well in MinGW ( windows bash environment ) ~ gitbash
Put this function into your .bashrc file in your HOME directory:
# this function helps you to find a jar file for the class
function find_jar_of_class() {
OLD_IFS=$IFS
IFS=$'\n'
jars=( $( find -type f -name "*.jar" ) )
for i in ${jars[*]} ; do
if [ ! -z "$(jar -tvf "$i" | grep -Hsi $1)" ] ; then
echo "$i"
fi
done
IFS=$OLD_IFS
}
Grepj is a command line utility to search for classes within jar files. I am the author of the utility.
You can run the utility like grepj package.Class my1.jar my2.war my3.ear
Multiple jar, ear, war files can be provided. For advanced usage use find to provide a list of jars to be searched.
Check this Plugin for eclipse which can do the job you are looking for.
https://marketplace.eclipse.org/content/jarchiveexplorer
Under a Linux environment you could do the following :
$ find <base_dir> -name *.jar -print0 | xargs -0 -l jar tf | grep <name>
Where name is the name of the class file that you are looking inside the jars distributed across the hierarchy of directories rooted at the base_dir.