Develop Reference

How to read needed files from rar archive directly to InputStream (without extracting whole archive)?

Seems quite simple with zip archive using java.util.zip.ZipFile like this:
public static void main(String[] args) throws IOException
{
final ZipFile zipFile = new ZipFile("C:/test.zip");
final Enumeration<? extends ZipEntry> entries = zipFile.entries();
while(entries.hasMoreElements())
{
final ZipEntry entry = entries.nextElement();
if(entry.getName().equals("NEEDED_NAME"))
{
try(InputStream inputStream = zipFile.getInputStream(entry))
{
// Do what's needed with the inputStream.
}
}
}
}
What would be the alternative for rar archives?
I'm aware of Junrar, but didn't found a way to do it without extracting whole archive to some folder.
Edit:
I have added "if sentence for entry.getName()" line just to indicate what I'm interested only in some specific files inside archive and would like to avoid extracting whole archive to some folder and later deleting those files.

I end up using something like this for now (with Junrar):
final Archive archive = new Archive(new File("C:/test.rar"), null);
final LocalFolderExtractor lfe = new LocalFolderExtractor(new File("/path/to/temp/location/"), new FileSystem());
for (final FileHeader fileHeader : archive)
{
if(fileHeader.getFileNameString().equals("NEEDED_NAME"))
{
File file = null;
try
{
file = lfe.extract(archive, fileHeader);
// Create inputStream from file and do what's needed.
}
finally
{
// Fully delete the file + folders if needed.
}
}
}
Maybe there is a better way :)

Getting the list of filenames from resource folder when running in jar

I have some Json files in the folder "resource/json/templates". I want to read these Json files. So far, the snippet of code below is allowing me to do so when running the program in IDE but it fails when I run it in the jar.
JSONParser parser = new JSONParser();
ClassLoader loader = getClass().getClassLoader();
URL url = loader.getResource(templateDirectory);
String path = url.getPath();
File[] files = new File(path).listFiles();
PipelineTemplateRepo pipelineTemplateRepo = new PipelineTemplateRepoImpl();
File templateFile;
JSONObject templateJson;
PipelineTemplateVo templateFromFile;
PipelineTemplateVo templateFromDB;
String templateName;
for (int i = 0; i < files.length; i++) {
if (files[i].isFile()) {
templateFile = files[i];
templateJson = (JSONObject) parser.parse(new FileReader(templateFile));
//Other logic
}
}
}
catch (Exception e) {
e.printStackTrace();
}
Any help would be greatly appreciated.
Thanks a lot.

Assuming that in the class path, in the jar the directory starts with /json (/resource is a root directory), it could be as such:
URL url = getClass().getResource("/json");
Path path = Paths.get(url.toURI());
Files.walk(path, 5).forEach(p -> System.out.printf("- %s%n", p.toString()));
This uses a jar:file://... URL, and opens a virtual file system on it.
Inspect that the jar indeed uses that path.
Reading can be done as desired.
BufferedReader in = Files.newBufferedReader(p, StandardCharsets.UTF_8);

Firstly, remember that Jars are Zip files so you can't get an individual File out of it without unzipping it. Zip files don't exactly have directories, so it's not as simple as getting the children of a directory.
This was a bit of a difficult one but I too was curious, and after researching I have come up with the following.
Firstly, you could try putting the resources into a flat Zip file (resource/json/templates.zip) nested in the Jar, then loading all the resources from that zip file since you know all the zip entries will be the resources you want. This should work even in the IDE.
String path = "resource/json/templates.zip";
ZipInputStream zis = new ZipInputStream(getClass().getResourceAsStream(path));
for (ZipEntry ze = zis.getNextEntry(); ze != null; ze = zis.getNextEntry()) {
// 'zis' is the input stream and will yield an 'EOF' before the next entry
templateJson = (JSONObject) parser.parse(zis);
}
Alternatively, you could get the running Jar, iterate through its entries, and collect the ones that are children of resource/json/templates/ then get the streams from those entries. NOTE: This will only work when running the Jar, add a check to run something else while running in the IDE.
public void runOrSomething() throws IOException, URISyntaxException {
// ... other logic ...
final String path = "resource/json/templates/";
Predicate<JarEntry> pred = (j) -> !j.isDirectory() && j.getName().startsWith(path);
try (JarFile jar = new Test().getThisJar()) {
List<JarEntry> resources = getEntriesUnderPath(jar, pred);
for (JarEntry entry : resources) {
System.out.println(entry.getName());
try (InputStream is = jar.getInputStream(entry)) {
// JarEntry streams are closed when their JarFile is closed,
// so you must use them before closing 'jar'
templateJson = (JSONObject) parser.parse(is);
// ... other logic ...
}
}
}
}
// gets ALL the children, not just direct
// path should usually end in backslash
public static List<JarEntry> getEntriesUnderPath(JarFile jar, Predicate<JarEntry> pred)
{
List<JarEntry> list = new LinkedList<>();
Enumeration<JarEntry> entries = jar.entries();
// has to iterate through all the Jar entries
while (entries.hasMoreElements()) {
JarEntry entry = entries.nextElement();
if (pred.test(entry))
list.add(entry);
}
return list;
}
public JarFile getThisJar() throws IOException, URISyntaxException {
URL url = getClass().getProtectionDomain().getCodeSource().getLocation();
return new JarFile(new File(url.toURI()));
}
I hope this helps.

Getting specific file from ZipInputStream

I can go through ZipInputStream, but before starting the iteration I want to get a specific file that I need during the iteration. How can I do that?
ZipInputStream zin = new ZipInputStream(myInputStream)
while ((entry = zin.getNextEntry()) != null)
{
println entry.getName()
}

If the myInputStream you're working with comes from a real file on disk then you can simply use java.util.zip.ZipFile instead, which is backed by a RandomAccessFile and provides direct access to the zip entries by name. But if all you have is an InputStream (e.g. if you're processing the stream directly on receipt from a network socket or similar) then you'll have to do your own buffering.
You could copy the stream to a temporary file, then open that file using ZipFile, or if you know the maximum size of the data in advance (e.g. for an HTTP request that declares its Content-Length up front) you could use a BufferedInputStream to buffer it in memory until you've found the required entry.
BufferedInputStream bufIn = new BufferedInputStream(myInputStream);
bufIn.mark(contentLength);
ZipInputStream zipIn = new ZipInputStream(bufIn);
boolean foundSpecial = false;
while ((entry = zin.getNextEntry()) != null) {
if("special.txt".equals(entry.getName())) {
// do whatever you need with the special entry
foundSpecial = true;
break;
}
}
if(foundSpecial) {
// rewind
bufIn.reset();
zipIn = new ZipInputStream(bufIn);
// ....
}
(I haven't tested this code myself, you may find it's necessary to use something like the commons-io CloseShieldInputStream in between the bufIn and the first zipIn, to allow the first zip stream to close without closing the underlying bufIn before you've rewound it).

use the getName() method on ZipEntry to get the file you want.
ZipInputStream zin = new ZipInputStream(myInputStream)
String myFile = "foo.txt";
while ((entry = zin.getNextEntry()) != null)
{
if (entry.getName().equals(myFileName)) {
// process your file
// stop looking for your file - you've already found it
break;
}
}
From Java 7 onwards, you are better off using ZipFile instead of ZipStream if you only want one file and you have a file to read from:
ZipFile zfile = new ZipFile(aFile);
String myFile = "foo.txt";
ZipEntry entry = zfile.getEntry(myFile);
if (entry) {
// process your file
}

Look at Finding a file in zip entry
ZipFile file = new ZipFile("file.zip");
ZipInputStream zis = searchImage("foo.png", file);
public searchImage(String name, ZipFile file)
{
for (ZipEntry e : file.entries){
if (e.getName().endsWith(name)){
return file.getInputStream(e);
}
}
return null;
}

I'm late to the party, but all above "answers" does not answer the question and accepted "answer" suggest create temp file which is inefficient.
Lets create sample zip file:
seq 10000 | sed "s/^.*$/a/"> /tmp/a
seq 10000 20000 | sed "s/^.*$/b/"> /tmp/b
seq 20000 30000 | sed "s/^.*$/c/"> /tmp/c
zip /tmp/out.zip /tmp/a /tmp/b /tmp/c
so now we have /tmp/out.zip file, which contains 3 files, each of them full of chars a, b or c.
Now lets read it:
public static void main(String[] args) throws IOException {
ZipInputStream zipStream = new ZipInputStream(new FileInputStream("/tmp/out.zip"));
ZipEntry zipEntry;
while ((zipEntry = zipStream.getNextEntry()) != null) {
String name = zipEntry.getName();
System.out.println("Entry: "+name);
if (name.equals("tmp/c")) {
byte[] bytes = zipStream.readAllBytes();
String s = new String(bytes);
System.out.println(s);
}
}
}
method readAllBytes seems weird, while we're in processing of stream, but it seems to work, I tested it also on some images, where there is higher chance of failure. So it's probably just unintuitive api, but it seems to work.

Scanner and Multithreading issues?

I have following code to read entire file data:
calling method(String zipFile){
ZipInputStream zis =
new ZipInputStream(new FileInputStream(zipFile));
//get the zipped file list entry
ZipEntry ze = zis.getNextEntry();
while (ze != null) {
String fileName = ze.getName();
File newFile =
new File(Constants.OUTPUT_FOLDER + File.separator +
fileName);
if (ze.isDirectory()) {
new File(newFile.getParent()).mkdirs();
} else {
new File(newFile.getParent()).mkdirs();
createBlobDomain(zFile,ze);
}
}
ze = zis.getNextEntry();
}
zis.closeEntry();
zis.close();
}
public String method(ZipFile zf, ZipEntry ze){
scan = new Scanner(zf.getInputStream(ze));
if(scan.hasNext())
fullText = scan.useDelimiter("\\A").next();
return fullText;
}
Please ignore it from compilation perspective as i removed some code not really relevant here. It works fine when run from the webapp as a single instance. But it i run it from two different browsers at the same time then i hit below exception. Please advise what could be going wrong and how to fix it.
java.util.InputMismatchException
at java.util.Scanner.throwFor(Scanner.java:840)
at java.util.Scanner.next(Scanner.java:1347)

I believe the line scan = new Scanner(zf.getInputStream(ze)); is creating the problem. What I understand from you code is scan is an instance variable which you are assigning a new Scanner with every thread. I would suggest to make it as a local variable in your method. Correct me If I misunderstood anything.
Scanner scan = new Scanner(zf.getInputStream(ze))

It looks to me that what you want to do is to copy the contents of a zip into a given folder.
Provided you use Java 7+, it's actually pretty simple to do that; this code uses java7-fs-more to help you do the job:
public static void extractZip(final String zipfile, final String dstdir)
throws IOException
{
final Map<String, ?> env = Collections.singletonMap("readonly", "true);
final Path path = Paths.get(zipfile);
final URI uri = URI.create("jar:" + path.toUri());
try (
final FileSystem zipfs = FileSystems.newFileSystem(uri, env);
) {
MoreFiles.copyRecursive(zipfs.getPath("/"), Paths.get(dstdir),
RecursionMode.FAIL_FAST);
}
}

How to load resources from other JAR file

I have troubles to make loading resources from other jars running. Here is the setup I have
resource.jar # contains resources I want to load
`-res/hwview/file1
engine.jar # my application which need resources
`-res/hwview/file2
Interesting thing is that using the code below I'm able to load file2 (which is in the jar I run) but not the file1.
String dir = "res/hwview";
Enumeration<URL> e = getClass().getClassLoader().getResources(dir);
while(e.hasMoreElements()) {
// prints only file1 from engine.jar
// (actually it's in classes directory because I run it from my IDE)
System.out.println(e.nextElement());
}
[OUTPUT]
/path/to/my/project/SiHwViewUiModel/classes/res/hwview
So I thought maybe the jar was not picked up by the ClassLoader so I printed what was loaded
ClassLoader cl = ClassLoader.getSystemClassLoader();
URL[] urls = ((URLClassLoader)cl).getURLs();
for(URL url: urls){
System.out.println(url.getFile());
}
[OUTPUT]
/path/to/my/project/SiHwViewUiModel/classes/
/path/to/my/project/Resources/deploy/resources.jar
... and other not so important jars
Any ideas?
Thanks for any help!

I found the solution. The problem with getResources() method and similar is that thay cannot be given a directory but only a particular file. This means that if I want to search in the whole classpath for a particular structure I need to create marker file in base directories.
Example: I want to get to my/path directory -> create marker.info (name does not matter) file and then search for it.
resources.jar
`- my/path/
|- my/directories
`- marker.info
resources2.jar
`- my/path/
|- my/other/directories
`- marker.info
# search
Enumeration<URL> urls = getClass().getClassLoader().getResources("my/path/marker.info");
# print
print(urls);
/path/to/resources.jar!/my/path/marker.info
/path/to/resources2.jar!/my/path/marker.info

If the JAR files are on the classpath, you don't need to do anything special. The resources will be found.
If they aren't on the classpath, you need to create a URLClassLoader and use its getResource() method.

In Spring, it can load xml file from all the jar files in the classpath:
ApplicationContext context = new ClassPathXmlApplicationContext(
"classpath*:**/applicationContext*.xml");
You can check the Spring source to see how Spring achieve that.

public final class JarResource
{
private String jarFileName;
private Map<String, Long> hashSizes = new HashMap<String, Long>();
private Map<String, Object> hashJarContents = new HashMap<String, Object>();
public JarResource(String jarFileName) throws Exception
{
this.jarFileName = jarFileName;
ZipFile zipFile = new ZipFile(this.jarFileName);
Enumeration<ZipEntry> e = (Enumeration<ZipEntry>) zipFile.entries();
while (e.hasMoreElements())
{
ZipEntry zipEntry = e.nextElement();
if(!zipEntry.isDirectory())
{
hashSizes.put(getSimpleName(zipEntry.getName()), zipEntry.getSize());
}
}
zipFile.close();
// extract resources and put them into the hashMap.
FileInputStream fis = new FileInputStream(jarFileName);
BufferedInputStream bis = new BufferedInputStream(fis);
ZipInputStream zis = new ZipInputStream(bis);
ZipEntry ze = null;
while ((ze = zis.getNextEntry()) != null)
{
if (ze.isDirectory())
{
continue;
}
else
{
long size = (int) ze.getSize();
// -1 means unknown size.
if (size == -1)
{
size = hashSizes.get(ze.getName());
}
byte[] b = new byte[(int) size];
int rb = 0;
int chunk = 0;
while (((int) size - rb) > 0)
{
chunk = zis.read(b, rb, (int) size - rb);
if (chunk == -1)
{
break;
}
rb += chunk;
}
hashJarContents.put(ze.getName(), b);
}
}
zis.close();
}
public byte[] getResource(String name)
{
return (byte[]) hashJarContents.get(name);
}
private String getSimpleName(String entryName)
{
// Remove ".jar" extension
int index = entryName.indexOf("/");
String fileNameWithoutExt = entryName.substring(index, entryName.length());
return fileNameWithoutExt;
}
}
Then use this class to load your resource:
public static void main(String[] args) throws Exception
{
JarResource jr = new JarResource("/home/mjiang/Downloads/solr-4.8.0/dist/solr-cell-4.8.0-test.jar");
byte[] resource = jr.getResource("/META-INF/NOTICE.txt");
InputStream input = new ByteInputStream(resource, resource.length);
BufferedReader dis = new BufferedReader(new InputStreamReader(input));
String line = "";
while((line = dis.readLine()) != null)
{
System.out.println(line);
}
dis.close();
}